Home Java Node.js developer tools.The mqtt protocol for working with web sockets

Node.js developer tools.The mqtt protocol for working with web sockets

by admin

Web socket technology allows a web or mobile app to send messages from the server to the client, which can’t be done using REST-API tools. The socket.io library is often used to handle web sockets, or developers work with native browser web socket objects. In this post I will try to show that both ways don’t solve all the problems and it’s much better to use specialized servers such as mqtt-server (formerly called mqtt-broker) for working with web-sockets.
In fairness, and to avoid unnecessary arguments, I will note that in addition to mqtt-server can be used a number of other servers, such as rabbitmq.
Developing applications using web-based sockets seems very easy until we are faced with the reality that connection breakdowns are common. The first challenge we have to deal with is keeping track of broken connections and repairing them. The situation is complicated by the fact that during a connection break and reconnection, the client continues to send new messages, as well as new messages may come to the client which have a high probability of being lost.
We have to track the receipt of messages at the application level, and implement the redelivery of messages. I would like to pay special attention to the phrase "at the application level" (and I wish it was at the protocol level).
As soon as we added the logic to track message delivery – all messages started to arrive, but then it turns out that there were duplicate messages, because the message could have been received, but the confirmation of this fact was lost because the connection was broken. And you still need to double the amount of code to exclude duplicate messages.
As the code becomes more complex, so does its efficiency, which is why the socket.io library is often criticized. It is, of course, less efficient than native web sockets, in particular, because of the connection recovery logic and message delivery confirmation (note immediately that socket.io does not implement redelivery logic).
A more reliable and efficient way would be to take this logic to the protocol level. And such a protocol exists: mqtt. The first version of the mqtt protocol was developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom) in 1999. The MQTT 3.1.1 specification was standardized by the OASIS consortium in 2014.
The mqtt protocol has a "quality of service" (qos) parameter that can take values :
0 – the message is delivered if possible;
1 – the message is delivered guaranteed, but there may be duplicates;
2 – the message is delivered guaranteed and guaranteed once.
So, the mqtt protocol solves the problem with guaranteed message delivery, and this issue is removed from the agenda. But this is not the only issue.
Performance and scaling.
With web sockets, all connected clients leave open connections to the server, even if there is no actual messaging. This load is different in nature from the load in REST-API, which is determined by the flow of requests. The load of open connections over web sockets is difficult to emulate in the testing phase. Therefore, an erroneous assumption is often made about the application’s sufficient performance based on the number of messages sent and received, without considering the load of maintaining a large number of open connections to clients.
If we pass all the work with web sockets to a dedicated mqtt server, our nodejs application only opens one web socket connection (or tcp connection since mqtt supports both) to the mqtt server, and we can scale our application by connecting multiple nodejs instances to the mqtt server.
If one mqtt server runs out of resources, you can cluster the mqtt servers without affecting nodejs applications.
Now let’s move on to an example.
The mqtt server, or broker as it was called in previous specifications, operates on a message/subscription model. Each message is sent to a topic. The recipient subscribes to a message topic. Both sender and receiver have two identifiers: clientId (device ID) and userName.
The Device ID is important because it is associated with the subscription and the messages will be sent to it. The username, unlike the device ID, is not decisive for message delivery and is used to differentiate access to topics.
To work with the mqtt protocol on the client side the library github.com/eclipse/paho.mqtt.javascript There are several server implementations, including some that are free. We will use the emqx server in this example, which is run via docker-compose (see github.com/apapacy/tut-mqtt ).
To test, let’s create a document where we set clientId, userName and message text :

<script src="/paho-mqtt.js"> </script><script src="/messages.js"> </script><form name="sender" onsubmit="return false"><input type="text" name="user"><input type="text" name="client"><input type="text" name="message"><input type="button" onclick="connect()" value="connect"><input type="button" onclick="send()" value="send"></form>

Sending messages is implemented in the message.js file:

var client;var connectOptions = {timeout: 30, reconnect: true, cleanSession: false, mqttVersion: 4, keepAliveInterval: 10, onSuccess: onConnect, onFailure: onFailure}function connect() {try {client = new Paho.Client('localhost', 8083, '/mqtt', document.forms.sender.client.value);connectOptions.userName = document.forms.sender.user.value;client.connect(connectOptions);} catch (ex) {console.log(ex);}}function onConnect() {console.log('on connect');client.onMessageArrived = function(message) {console.log("onMessageArrived: " + message.payloadString);}client.subscribe("test", { qos: 2 });}function onFailure(err) {console.log('on failure', JSON.stringify(err));}function send() {var message = new Paho.Message(document.forms.sender.message.value);message.destinationName = "test";message.qos = 2;client.send(message);}

To check, open the file index.html in your browser, set clientId, userName, message text, and send some messages (you can read them in the console, because the client sends messages to the test thread, and is subscribed to that thread).
Now open another browser or another browser tab, and join with a different (this is important) clientId. Send some more messages from the first browser, and make sure they come to both clients, since they have different clientId and are both subscribed to the test topic.
Now close the second browser (or second browser tab), and send some more messages. After that, reopen the second browser, and join with the same clientId. Make sure in the console logs that all the messages that were sent while the second browser (second tab) was closed came to you. This happened because :

  • When sending a message, the quality level is set to qos=2;
  • You have joined the same topic before with the same clientId by setting qos=2;
  • The connection options set cleanSession: false.

The example code can be downloaded from the repository
September 29, 2019

You may also like