Message Styling

11 Nov, 2020

In 2017 I wrote XEP-0393: Message Styling as a way to standardize the various Markdown-like languages used by many XMPP clients and commercial messengers. Fast forward three years and the spec has recently gone to “draft” status, which is more-or-less XSF lingo for “release candidate” The spec may be tweaked before it reaches “final” (aka “read only”) status, but it’s generally considered ready to be widely implemented and no backwards incompatible changes can be made

This post is a mix of an FAQ, a retrospective, and a look forward to what I would change if I were starting a new styling spec without prior art.

But first:

Overview of Message Styling

Message styling is very small, comprising two block level styles and four inline styles that operate on spans of text. The block styles are:

> Block quote

and

```optional tag
Pre-formatted text block
```

And the inline styles are:

*strong emphasis*, _emphasis_, `pre-formatted span` , and ~~~strike through~~~.

Exactly how each of these are displayed depends on the rendering client. Strong emphasis may mean different things to different people, but it’s the intent that matters, not the exact style. In general strong emph is bold, emph is italicized, pre-formatted spans are monospace, and strike through is exactly what it says on the tin.

Parsing

One of the things I like the least about message styling is that it can’t be easily tokenized. It is not possible to know if any given asterisk is a strong emph styling directive without scanning the rest of the line on which it resides. This was a deliberate choice made on the assumption that it’s more important not to accidentally style the rest of a line because of a false positive “*” existing in the text, and that message styling documents would likely be relatively short. Even in the worst case (one very long line with a styling directive at the beginning) you will run up against the max message length before it becomes a problem.

However, insofar as writing a lexer or defining a formal grammar goes it would be much easier if styling were context free, allowing it to be parsed with pushdown automata, or even regular so you could use a simple regular expression. Doing so would likely increase false positives though, and means we would need a way to disable styling for individual spans, which brings me to my next woe.

Disabling Styling

The current approach to disabling styles takes a “default on” approach and an XML payload that allows you to disable message styling for entire messages. Personally, I don’t think having a way to disable styling for an entire message on the sending side was necessary, but a lot of people in the XSF thought the sending side should have complete control and have a way to turn styling on or off or leave it unspecified. The current “advertise support then be on by default but allow turning it off per message” approach was a compromise that I was willing to live with, and I’m reasonably happy with it.

However, having a way to disable styling on individual spans within a message would be much nicer from a UI perspective. This would allow you to implement an editing experience where you type in some text that should be styled such as “*strong*”, it is automatically styled, and then if you type a backspace instead of removing the final asterisk the text goes back to being unstyled without affecting every other span in the message.

A number of workarounds were tried where zero-width spaces were inserted between the styling directives and the text so that they no longer followed the rules and became normal characters, but none was found that didn’t feel like a hack and the idea was eventually abandoned. I also considered the idea of allowing you to disable styling using a payload outside of the message body, but there is no well-defined way in XMPP (yet) to reference text inside the body, and I decided this could always be added later as a separate spec if necessary.

If I were doing it again, I would design from the ground up with the ability to disable individual spans’ styling while still not requiring any extra XML payloads.

Blockquotes

The blockquote syntax in message styling (lines starting with one or more “>”) is taken more or less directly from markdown, which takes it from the defacto standard used by many email clients. Unfortunately, block quotes are another reason that parsing message styling is somewhat tricky as they are inconsistent with the other styles in that they do not have any sort of “end” marker.

This is outside the scope of Message Styling which was trying to formalize an existing styling language, but if I were implementing a new styling language from scratch I would consider inventing something new at the cost of breaking the users excpectations. Maybe something like this:

>>info
Quote
<<

This has a token that marks the end of the quote and contains an info string like pre-formatted text blocks. This also leads to a natural inline quote version which would be a nice addition that doesn’t exist today. The inline quote would look something like >this<.

Lists

A question I get asked relatively often is “why doesn’t message styling include lists?” While there is no specific styling directive for lists, you can still support them. From a UI perspective if you click a bold button your client might wrap the selected text in asterisks. Message styling is simple, you don’t need to do anything except add a bit of text in the input to add styles. The same can be done for lists without defining any additional styling directives. For example, when you click the lists button your client could insert a “•” (U+2022 BULLET) and a tab at the beginning of the line, then add another the next time you return. For ordered lists you could use simple numbers with periods (ie. “1.”), or the various characters from the Unicode enclosed alphanumerics block such as “⒈” (U+2488 DIGIT ONE FULL STOP) or “⓵” (U+24F5 DOUBLE CIRCLED DIGIT ONE).

• This works pretty well
• I think

⑴ One
⑵ Two
⑶ Three

However, if I were doing it again I probably would make it easy for users to enter a character that exists on their keyboard and have it converted to nicer looking bullets in a way that they can expect to work across clients (clients are still free to do this of course by converting “-” or “*” to “•”, and I encourage it, it’s just not part of the spec).

Strike Out and Underline

I’ve been asked a few times now why we chose strike through as one of the styles instead of, say, underline. This was added for compatibility with existing commercial services that support it in their clients. It’s not hard to add additional span directives once you’ve implemented one already, so we went ahead and added it in case anyone was using it. We did not add underline because I couldn’t think of any particularly interesting use cases for it and it wasn’t widely supported by other messengers (Gajim was a notable exception, and I’m afraid we created some conflicts with their existing styles, for which I’m sorry). I’d probably keep this the same if I were starting over unless someone presented a compelling use case.

Span Directives

The span directives that I picked are also something I probably wouldn’t change because the goal was to maintain compatibility with existing systems as much as possible, but which if given a clean break I might reconsider. If it were entirely up to me I think I would at least change italics to something like /emph/ because the slashes remind me of italicized text and maybe reconsider adding underline just because _underline _ looks so much more like underline than emphasis (this also would have been compatible with Gajim at the time).

Formal Grammar

One complaint I had from a small number of people while developing the spec is that it didn’t come with a formal grammar. While I would personally love to have a formal grammar for Message Styling, I probably wouldn’t want to put one in the spec even if we had one. A lot of IETF specs are full of augmented Backus–Naur form (ABNF) and for the most part I think all it does is intimidate users and make the spec harder to read. Maybe including it as an appendix would be okay.

Never the less, having a formal grammar would be nice, so why didn’t I write one even if it wasn’t going in the XEP? Ignoring for a moment that I just have no desire to learn any of the mess of metalanguages used to specify formal grammars or the tools required to actually test and validate it just for a spec that won’t include it anyways, it may also be impractical due to ambiguities in Message Styling that don’t make it a very good fit for most formal grammar meta languages.

For example: there are no errors or invalid tokens in message styling, so if you specify a formal grammar that includes “*” as a token that matches to the next “*”, you also have to add a rule for normal text that can match “*foo”, “foo*”, and any number of other combinations. Even for a small styling language this makes the grammar bloat significantly. Most meta languages meant for specifying formal grammars are designed for context free grammars, which message styling is not.

The TL;DR is that a formal grammar would be a lot of work that will only ever be looked at by one or two people. Most developers doing implementations will never touch it. The cost-benefit analysis just doesn’t add up.

Conclusion

Though this post is a retrospective of my own work, and therefore somewhat critical, overall I’m very happy with how Message Styling turned out. It accomplishes its original goal: standardize existing formatting used by many messengers both inside and outside the XMPP ecosystem, and has already been adopted by a number of clients.

I’m aware of implementations (some may not be released at the time of this writing or may not support all styles) in:

Styling is also recommended by Modern XMPP for formatting messages and is mentioned in the 2021 XMPP Compliance Suites, so I’m very happy with the rate of adoption.

Finally, if you’d like to play with message styling yourself, you can try my Go library mellium.im/xmpp/styling.

Update 2021-02-08: added a section on having a formal grammar.