Scalable XMPP bots with erlang and exmpp, part II

In part I we focused on learning the basic of the exmpp API.

In this second part we start to think about how one can use the the learned API to write and deploying XMPP components.

The source code for this article can be downloaded from web_status.tar.gz.

Introducing the bot

The example that we will follow through all the articles is a web presence bot. The bot’s only duty is to provide a HTML snippet of the user presence, via HTTP, for each registered user. Users can then embed the URL into an iframe on their web page to show its presence information. The bot acts as a kind of gateway, making the presence information flowing in the XMPP network (push model) available to the HTTP request-response world (pull model).

As any other gateway, our bot must provide two interfaces, one for each of the connecting sides.

At the XMPP network, there are many ways in witch one can plug such a bot:

• As a “normal client”, the bot will behave and communicate with the server as a normal user
• As a server plugin
• As an external component
• As an entirely standalone XMPP server/router

While the first approach seems tempting at first sight, it is well known in the XMPP community (https://metajack.wordpress.com/2008/08/04/thoughts-on-scalable-xmpp-bots/, https://blogs.openaether.org/?p=52, https://metajack.im/2008/09/03/twitters-failures-are-not-xmpps-failures/), that this approach suffers from a couple of serious scalability issues.

As one of our goals is to show how to use exmpp to build highly scalable bots, we will write our component in a way that will allow us to scale it horizontally if needed; we will implement the bot as an external component.

External components aren’t constrained to use a single connection, nor to maintain a roster with all its contacts. Even more, you can deploy a component in a cluster and let ejabberd load-balance the load between all your component’s nodes!

Data Model

Our bot will have two user configurable options:

1. let the user decide if he wants its status message to be shown in the presence webpage.
2. let the user decide if he wants to show his/her full JID on the webpage, or only his presence status.

Internally, we represent registered users of our BOT with a #user{} record. We use the user’s bare JID as identifier (our primary key) and for each user we keep its subscription options. Thus the definition that we are looking for is:

All our data access functions resides in a separate module (I’ve called it web_status_db), whose interface is:

Component startup

The component will have one main process in charge of establish and manage the connection with the server, using Ery Lee’s contributed work on exmpp_component (now incorporated into exmpp SVN, checkout a recent version). We will code this main process as an OTP gen_server.

At least for now, the state that our main process needs to maintain is limited to the socket connection with the server. So for now, the following definition is enough:

At this point we have an authenticated, bidirectional connection with the server. Any process in our bot can send stanzas towards the server as long as it has a reference to the session (the session is thread safe, in the sense that two different process are allowed to send messages on the same session at the same time; the stanzas won’t be mixed)

All stanzas addressed to our component will be received by our main process, in the form of erlang messages. To take advantage of erlang’s concurrency, the main process will spawn a new worker process to handle every incoming stanza that it receives. The worker’s only goal is to interpret the stanza, and send a response if required; after that, the worker process dies.

exmpp_component give us received stanzas as #received_packet{} records (defined in $EXMPP_HOME/include/exmpp_client.hrl). Our component is only interested in the following fields:

Knowing this, we can write the code that receives stanzas and launch the workers. Note that we must give a reference to the session to each spawned worker, to allow them to reply to the user when necessary.

User registration

Our component follows the In-Band Registration XEP to allow users to register. The protocol requires the server entity (our bot) to reply to IQ get requests on the “jabber:iq:register” namespace with a registration form.

We have seen than the main process spawn a worker for each received packet, calling the function process_received_packet/2 on it. Based on the type of stanza the worker is handling, it can dispatch the request further, to a more specialized handler:

Now the real work, its finally time of writing the handler itself!.

If the user is already registered, we want to present a form with the current subscription options, allowing the user to update them; if not we must present and empty form.
The helper function make_form/2 takes a #user_conf{} record and a textual description and builds the proper form from these values. The inverse function is parse_form/1, which parses a XMPP form and constructs a #user_conf{} record based on the form’s values. For brevity, we don’t show these function here, interested readers can look them at the supplied source code.

Note that we are using a base64 encoding of the user’ bare JID as the key in the URL. This make the mapping of JID <-> URL straightforward to implement. If this component is going to be used on the open WEB, we should consider other alternatives, as it isn’t a good idea to expose our users JID to everyone in this way.

It’s the client responsibility to complete the form and sent it back to us. We then handle the submitted form:

We are almost done with the user registration process. The only thing missing, is that our component should advertise its support for In-Band Registration as response to service discovery requests. Easy enough:

Ejabberd setup

We only need to add the corresponding entry to ejabberd.cfg listener definitions, to let ejabberd accept connections from our external component, and register the appropriate route to it.

Supposing your server’s domain is “testdomain”, you will end with something like:

Lets test it!

Here we show how it looks like in PSI, but any client capable of performing Service Discovery will be able to use our bot.

Try to register again, the bot will present you the same form, but this time with the configuration values that you has previously selected.

On the wire

If you look at the XML stanzas exchanged by the previous registration process, you will see the following stanzas being sent and received:

Client ask component’s features:

Component features response:

At this point the client knows that our bot support In-Band registration, and the user is allowed to ask for the registration form:

Bot return (empty) registration form:

User fills and submit the form

Bot confirms succesful registration

Handling HTTP requests

This has nothing to do with exmpp or XMPP, but is actually the reason why we are writing this bot, so I thought it would have been to rude to not include it here :).

We are using mochiweb to handle HTTP requests. So far our bot only knows which users are registered, but has no idea of what is the presence status of those users. With this limitation in mind, the most we can do is return an HTML response if the request is for one of our registered users, or a HTTP error 404 (page not found) if not.

Next steps

In the next article we will make our bot track the presence status of its users, to display the appropriate information on its web page.