We are pleased to announce ejabberd version 19.05. The main focus of this release has been to expand our MQTT stack with Websocket support, to power web based IoT platforms. We also have improved existing features, improved server performance and fixed several issues.

New Features and improvements

MQTT over Websocket

ejabberd supports the Internet of Things in many ways:

• It supports XMPP IoT specifications. It is typically used to provide a global identifier to a device (deviceid@domain.tld), which can be reached in a federated way. It targets devices that are powerful enough to be handling XML parsing. It is ideal on Raspberry devices, for example.
• It supports MQTT 5.0, a specification that is more compact (binary protocol) and easier to parse for smaller devices.

Up to now, MQTT was only supported over raw TCP sockets and TLS.

With ejabberd 19.05, you can now develop MQTT clients that work over Websockets. This major new feature covers several use cases:

• You can use MQTT over Websocket on devices that will need to go through a HTTP proxy (they will need to be Websocket friendly, but this is more and more the case).
• You can use MQTT to write control applications in JavaScript, running in a web browser, connecting directly to ejabberd using MQTT over Websocket.

This makes ejabberd a highly scalable alternative to existing MQTT servers, while still relying on the huge ejabberd ecosystem of extensions that have been developed over the years. Like ejabberd XMPP components, our MQTT service is highly scalable and works in a cluster.

Smoother mobile groupchat

Implementing a Whatsapp-like mobile groupchat experience on top of XMPP is often a challenge. Let’s see how ejabberd helps pushing modern XMPP to provide better mobile support.

The road to a better support for XMPP on mobile: MIX and MucSub

For a historical reason, groupchat on mobile using XMPP has been difficult in the past. XMPP is a session-based, and Multi-User Chat is a presence-based protocol. This means trouble on mobile, as apps cannot typically run all the time. To provide good groupchat support on mobile, you need to provide a way to bridge a presence-based model to a subscription-based model.

The XMPP Standard Foundation is working on a new specification to make implementing groupchat easier. The work is proposed in the experimental XEP-0369: Mediated Information eXchange (MIX). ejabberd implements version 0.13.0 of this specification, which is very close to the state-of-the-art implementation. We are strong supporters of this specification and, more generally, all features that make implementing a mobile XMPP client easier.

That said, the work on MIX is experimental and a work-in-progress. ejabberd needs to provide both a migration path and a working solution, compliant with existing clients. That’s why we developed MucSub, a way to add a thin subscription layer on top of standard groupchat that is compliant with clients that only implement MUC.

We are committed to supporting both approaches until the XMPP community is fully ready to stand behind the MIX specification. The idea is that ejabberd should provide you with a solution for today’s and tomorrow’s needs.

Improving MucSub

After spending a significant effort in ejabberd 19.02 to update our MIX support, we revisited MucSub, based on how our users were leveraging it.

In ejabberd 19.05, we solve two problems with MucSub:

1. Members of a group discussion were expecting to be able to get the list of subscribers to a group chat directly in MucSub. They had to do some awkward workaround to get that info, having to join a MUC room just to get the participant lists.
2. We had designed the implementation of MucSub for small rooms of very active users. That’s why we implemented fan-out on write, duplicating the messages for each subscriber, so that ejabberd could read them fast on login. However, it could cause scalability issue and storage waste if you had MucSub rooms with a lot of subscribers.

So, in ejabberd 19.05, we now:

1. Grant access to subscribers list to all participants, not just the room owner. This will make clients implementation much easier.
2. Offer an option to move the MucSub behaviour to fan-out on read, which is probably the best default behaviour on most servers. It will store the messages just once in the chat room archive, instead of writing it multiple time in all users archive “inbox” (MAM). You can, however, still query your own archive and find your subscriptions here.

Improved error reporting for configuration file mistakes

When using a wrong or invalid entry in ejabberd configuration file, a clear explanation of the error and also some suggestion are dumped in the logs. This feature will really help fix configuration while testing new options and avoid losing time when facing syntax errors.

For example, if you make a typo on a module name (i.e. using certfile instead of certfiles), you used to had the following error in log on previous version:

2019-05-28 16:05:59.841 [error] <0.143.0>@ejabberd_config:validate_opts:1095 Unknown option 'certfile'

Now, ejabberd provide you with suggestions:

16:07:45.673 [error] Unknown option 'certfile', did you mean 'certfiles'?

It even provides suggestions and guidance for module names, which could be custom (i.e. using mod_privates instead of mod_private):

16:08:20.396 [error] Failed to start unknown module mod_privates, did you mean mod_private? Hint: make sure there is no typo and mod_privates.beam exists inside either .../ejabberd/ebin or ~/.ejabberd-modules directory

Technical details

Using MQTT over Websocket

Configuration

Here is an example configuration to enabled MQTT over Websocket.
You first need to add an HTTP request handler, mapping /mqtt path to ejabberd MQTT module.
Then, you need to make sure the MQTT module is started in the module section.

Here are what the listen and modules config fragments look like:

listen:
      ...
      -
        port: 5280
        module: ejabberd_http
        request_handlers:
           ...
           "/mqtt": mod_mqtt

modules:
      ...
      mod_mqtt: {}

From there, you can create a Web MQTT client that you can use to connect to ejabberd.
Here is an example MQTT Web client that you can use to connect to ejabberd over MQTT: mqttjs-demo

The repository also contains small publish and subscribe NodeJS scripts to test MQTT over Websocket from the command-line.

To try the client, you can open the dist/index.html page and open your JavaScript console to see the messages received on the MQTT ‘presence’ topic.

Here is the most basic ejabberd client you can write using MQTTJS:

var mqtt = require('mqtt')
var client  = mqtt.connect('ws://localhost:5280/mqtt', {username: "test@localhost", password: "test"})

client.on('connect', function () {
  client.subscribe('presence', function (err) {
    if (!err) {
      client.publish('presence', 'Hello mqtt')
    }
  })
})

client.on('message', function (topic, message) {
  // message is Buffer
  console.log(topic, message.toString())
  // uncomment if you want to close connection after first message received:
  //client.end()
})

You can get more details reading the mqttjs-demo README.

MucSub IQ to get list of subscribers

As a room subscriber, one can now get list of other subscribers with a simple IQ. This allows clients to use MucSub without any hacks from XEP-0045 in order to provide a good and complete mobile groupchat user experience.

Examples

User asks for subscriptions list:

<iq from='hag66@shakespeare.example'
    to='muc.shakespeare.example'
    type='get'
    id='E6E10350-76CF-40C6-B91B-1EA08C332FC7'>
  <subscriptions xmlns='urn:xmpp:mucsub:0' />
</iq>

Server replies with subscriptions list:

<iq from='muc.shakespeare.example'
    to='hag66@shakespeare.example'
    type='result'
    id='E6E10350-76CF-40C6-B91B-1EA08C332FC7'>
  <subscriptions xmlns='urn:xmpp:mucsub:0'>
    <subscription jid='coven@muc.shakespeare.example'>
      <event node='urn:xmpp:mucsub:nodes:messages'/>
      <event node='urn:xmpp:mucsub:nodes:affiliations'/>
      <event node='urn:xmpp:mucsub:nodes:subject'/>
      <event node='urn:xmpp:mucsub:nodes:config'/>
    </subscription>
    <subscription jid='chat@muc.shakespeare.example'>
      <event node='urn:xmpp:mucsub:nodes:messages'/>
    </subscription>
  </subscriptions>
</iq>

You can read the MucSub documentation for more details on how to use that feature or MucSub in general: Getting Your List of MucSub Subscriptions.

New Option for MucSub + MAM Optimisation

The new option user_mucsub_from_muc_archive for mod_mam is used to enable storage optimization when using MucSub on large chat rooms along with MAM.
It takes boolean argument. Default value is false to keep former behaviour as default.

Enabling this option allows storing mucsub message only once, thus avoiding to store a copy for each individual subscriber. When a user fetches his own account archive, MucSub messages are generated on the fly directly from muc archives instead.

Enabling this option allows to significantly reduce i/o and storage volume on database backend. We encourage users to switch to this mode.

New Option for Offline with MUC / MucSub optimisation

We have also added a new option for mod_offline called bounce_groupchat.
It takes a boolean argument. The default value is false meaning that the optimisation will be enabled as default.

This is an optimisation to avoid bouncing error messages when groupchat message could not be stored as offline. It will reduce chat room load, without any drawback in standard use case. You may change default value only if you have a custom module which uses offline hook after mod_offline.

It can be useful for both standard MUC and MucSub, but the bounce is much more likely to happen in the context of MucSub, so it is even more important to have it on large MucSub services.

Experimental MucSub + Offline option

The new option use_mam_for_storage for mod_offline takes a boolean argument.
The default value is false to keep former behaviour as default.

Enabling this option will make mod_offline not use the former spool table for storing MucSub offline messages, but will use the archive table instead. This use of the archive table is cleaner and it makes it possible for clients to slowly drop the former offline use case and rely on message archive instead. It also further reduce the storage required when you enabled MucSub.

Enabling this option has a known drawback for the moment: most of flexible message retrieval queries don’t work (those that allow retrieval/deletion of messages by id), but this specification is not widely used.

We still consider this option as experimental as it slightly change the behaviour of the offline message delivery. Please, do not enable it on critical production servers yet and do not hesitate to send us your feedback.

The MucSub configuration revisited

Here are an example snippet from ejabberd configuration file for users that want to be all-in on MucSub and MAM support. It enables the new MAM optimisation and makes sure MucSub is enabled as default on all chat rooms (allow_subscription)

modules:
...
  mod_mam:
    user_mucsub_from_muc_archive: true
  mod_muc:
    ...
    max_users: 300
    default_room_options:
      allow_subscription: true

For the developers

While developing internal optimization, we needed to expose three new API hooks.
They are now widely available to all developers when writing new ejabberd plugins and extensions:

• room_destroyed is run when a persistent room is terminated
• muc_subscribed is run when an entity subscribes to a MucSub room
• muc_unsubscribed is run when an entity subscription to a MucSub room is removed

Technical changes

Erlang/OTP requirement

Since this release, Erlang/OTP 19.1 as minimum is required, instead of 19.0.

Database changes

There is no change to perform on the database to move from ejabberd 19.02 to ejabberd 19.05.
Please, make a backup before upgrading, especially if you enable the new MucSub + MAM (Message Archive Management) behaviour.

Note for packagers

The version was tagged a bit too early, before the end of the testing phase. We had to reissue the 19.05 tag on May 28. If you have an earlier copy of the repository, please ensure that your clone is up to date.

Download and install ejabberd 19.05

The source package and binary installers are available at ProcessOne. If you installed a previous version, please read ejabberd upgrade notes.
As usual, the release is tagged in the Git source code repository on Github. If you suspect that you’ve found a bug, please search or fill a bug report in Issues.

Full changelog
===========

Websockets
– Add WebSockets support to mod_mqtt
– Origin header validation on Websockets connection
– Return “Bad request” error when origin in websocket connection doesn’t match

User management & authentication
– Correctly support cache tags in ejabberd_auth
– Don’t process failed EXTERNAL authentication by mod_fail2ban
– Don’t trigger calls to mod_register when it’s not loaded
– Make anonymous auth don’t {de}register user when there are other resources
– Fix cache when using several auth backends

Configuration & server admin
– Improve request_handlers validator
– Provide a suggestion when unknown command, module, option or request handler is detected
– Deprecate some listening options: captcha, register, web_admin, http_bind and xmlrpc
– Add commands to get Mnesia info: mnesia_info and mnesia_table_info
– Fixes in Prosody import: privacy and rooms
– Remove TLS options from the example config
– Fix syntax in example Elixir config file

MUC
– Service admins are allowed to recreate room even if archive is nonempty
– New option user_mucsub_from_muc_archive
– Avoid late arrival of get_disco_item response
– Handle get_subscribed_rooms call from mod_muc_room pid
– Fix room state cleanup from db on change of persistent option change
– Make get_subscribed_rooms work even for non-persistant rooms
– Allow non-moderator subscribers to get list of room subscribers

Offline
– New option bounce_groupchat: make it not bounce mucsub/groupchat messages
– Experimental new option use_mam_for_storage: fetch data from MAM instead of spool table
– When applying limit of max msgs in spool check only spool size
– Do not store MucSub wrapped messages with no-store hint in offline storage
– Always store ActivityMarker messages
– Don’t issue count/message fetch queries for offline from mam when not needed
– Properly handle infinity as max number of message in mam offline storage
– Sort messages by stanza_id when using mam storage in mod_offline
– Return correct value from count_offline_messages with mam storage option
– Make mod_offline put msg ignored by mam in spool when mam storage is on

SQL
– Add missing SQL schemas for MQTT tables
– Report better errors on SQL terms decode failure
– Fix PostgreSQL compatibility in mod_offline_sql:remove_old_messages
– Fix handling of list arguments on pgsql
– Preliminary support for SQL in process_rosteritems command

Other modules
– mod_adhoc: Use xml:lang from stanza when it’s missing in command element
– mod_announce: Add ‘sessionid’ attribute when required
– mod_bosh: Don’t put duplicate polling attribute in bosh payload
– mod_http_api: Improve argument error messages and log messages
– mod_http_upload: Feed whole image to eimp:identify/1
– mod_http_upload: Log nicer warning on unknown host
– mod_http_upload: Case-insensitive host comparison
– mod_mqtt: Support other socket modules
– mod_push: Check for payload in encrypted messages

Developer
– Rename listening callback from start/2 to start/3
– New hook called when room gets destroyed: room_destroyed
– New hooks for tracking mucsub subscriptions changes: muc_subscribed, muc_unsubscribed
– Make static hooks analyzer working again

Tests
– Add tests for user mucsub mam from muc mam
– Add tests for offline with mam storage
– Add tests for offline use_mam_for_storage
– Initial Docker environment to run ejabberd test suite
– Test offline:use_mam_for_storage, mam:user_mucsub_from_muc_archive used together