SamWhited|blog

Mellium Year in Review

The Mellium project is a collection of libraries and applications in Go relating to XMPP and instant messaging. This post is a copy of a message sent to the xmpp-standards list in response to a call for updates on XMPP related projects from the past year.

But first: Why XMPP?

I recently got asked (and I’m paraphrasing here): “why would you write an XMPP library when trendier technologies are available?” The quick answer is that XMPP is still the most widely used federated instant messaging protocol available on the internet. Despite the demise of XMPP in a few big chat services that users were familiar with (HipChat, Google Talk, Facebook Messenger), it is still widely used in video games (WarFace, Nintendo Switch presence notifications, etc.), in current chat products (Zoom, WhatsApp, Jitsi Meet, etc.), and in industry. XMPP is a core part of internet infrastructure and has a robust ecosystem of both free and commercial products, servers, and clients built around it. We need this infrastructure in Go.

Now, on to the status update.

Last year the main Go module of the Mellium project, mellium.im/xmpp received a huge number of updates, bug fixes, and API overhauls, many of which will be released in the upcoming v0.18.0 release. Some of the highlights include:

The full (much longer) list can be found in the changelog.

Message Styling

The change to mellium.im/xmpp that I’m most proud of in the past year might be the Message Styling implementation which can be found at mellium.im/xmpp/styling.

The actual styling implementation comprises three parts: the Style type, the Scan function, and the Decoder type.

The Scan function is a bufio.SplitFunc that is used to tokenize the byte stream internally. It is exported as a convenience to allow new message styling implementations that behave differently or have other extensions built on top of it.

The Style type is a uint32 that represents a bitmask of styles such as “strong and emph” or “pre-formatted text and block quoted”. It only contains information about the styles, not about metadata related to the styles or the token stream. For example, given a style you can know that the text is in a blockquote, but not the level of nesting (if any) of the blockquote, or you might know that it is in a strong span, but not where the start and endpoints of that span are. The package provides the various styles and a handful of masks for various common combinations as constants for the users convenience.

Finally, the Decoder type pairs the Scan function with the Style type by scanning for tokens and keeping track of the stream state. It allows you to get the style of each token popped, and keeps track of metadata about the styles such as the level of nesting in block quotes or the tag on a fenced code block.

Here is an example of quickly tokenizing an input stream:

	r := strings.NewReader(`The full title is
_Twelfth Night, or What You Will_
but *most* people shorten it.`)
	d := styling.NewDecoder(r)

	var err error
	var tok styling.Token
	for {
		tok, err = d.Token()
		if err != nil {
			break
		}
		fmt.Printf("Token: %q\n", tok.Data)
	}
	if err != nil && err != io.EOF {
		panic(err)
	}
  // Token: "The full title is\n"
  // Token: "_"
  // Token: "Twelfth Night, or What You Will"
  // Token: "_"
  // Token: "\n"
  // Token: "but "
  // Token: "*"
  // Token: "most"
  // Token: "*"
  // Token: " people shorten it."

A more extensive example for converting styling tokens to HTML can be found in the package examples. For more, see my retrospective post, Message Styling.

Stream Initialization and Virtual Hosts

XMPP servers likely want to serve more than one domain, but prior to the upcoming v0.18.0 release this was difficult or impossible. This required changes to the Negotiator type to allow input and output streams to be stored on the session, and to the built-in Negotiator’s StreamConfig type which is used for configuring options on the built in default negotiator to allow the user to select the features in a callback. Unfortunately, these are breaking changes, but it should be one of the last major API changes before we can begin thinking about stabilizing the API in v1.0.0.

The S2S option was also removed from the StreamConfig in favor of new functions for receiving and negotiating a stream which mark the s2s state bit on the session before beginning negotiation. This removes a long standing redundancy, and also a bug where the first stream feature negotiated wouldn’t know that the session was a server-to-server session because it couldn’t be set until the negotiator function returned for the first time. The current list of stream negotiation functions is:

Testing

While it’s not an exciting user-facing feature, one of the most valuable changes this year has been the addition of integration testing against Prosody, Ejabberd, McAbber (Loudmouth), and sendxmpp(1). We’ve already caught several issues both in Mellium and in some of the services we’re testing against thanks to the new tests! This is all facilitated by the internal/integration package which provides a framework for starting an external application, generating certificates, and passing around file handles. It works by creating a temporary directory where any config files can be stored, then launching the process using that temp directory as a working directory. It also provides generic options for writing files to that directory that can be used by the child packages, such as prosody, to write more specific options that write the prosody config file. The integration testing packages also provide functionality for running sub-tests, each one of which will get its own child process with all the configuration automatically. For example, to test sending a ping to Prosody and Ejabberd and receiving a ping on Prosody we might write something like the following:

func TestIntegrationSendPing(t *testing.T) {
	prosodyRun := prosody.Test(context.TODO(), t,
		integration.Log(),
		prosody.ListenC2S(),
	)
	prosodyRun(integrationSendPing)
	prosodyRun(integrationRecvPing)

	ejabberdRun := ejabberd.Test(context.TODO(), t,
		integration.Log(),
		ejabberd.ListenC2S(),
	)
	ejabberdRun(integrationSendPing)
}

func integrationSendPing(ctx context.Context, t *testing.T, cmd *integration.Cmd) {
  …
}

func integrationRecvPing(ctx context.Context, t *testing.T, cmd *integration.Cmd) {
  …
}

More complex options for loading Prosody extensions, generating and using certificates and enabling TLS, changing the default username and server host, etc. are also available.

See Also