Communicating with Fostrom over MQTT

MQTT is a lightweight messaging protocol that is ideal for IoT applications. It is a publish/subscribe protocol that allows devices to send messages to each other. Fostrom supports MQTT 3.1.1 over TCP/TLS and Secure WebSockets and requires the device to send packets with specific parameters.

The device can send the payload in two serialization formats: JSON and MsgPack.

To use JSON, the CONNECT packet's client_id field must be set to JSON.

To use MsgPack, the CONNECT packet's client_id field must be set to MSGPACK.

Connecting to Fostrom

A TCP/TLS connection can be opened at device.fostrom.dev at port 8883. The MQTT client should then send a specifically crafted CONNECT packet.

The following URL can be used to connect from an MQTT client:

mqtts://device.fostrom.dev:8883

Note that Fostrom does not support any unencrypted connection. You must use TLS.

WebSockets

Fostrom also supports MQTT over Secure WebSockets. The URL for connecting to Fostrom over WebSockets is:

wss://device.fostrom.dev

The WebSockets connection should be opened with the mqtt protocol header:

Sec-WebSocket-Protocol: mqtt

The `CONNECT` Packet

The MQTT CONNECT packet carries the following fields, and they should be set accordingly:

client_id - The client ID usually exists to identify a device, but we do that through the username field. Instead, set the client_id to "MSGPACK" to use MsgPack as the serialization format. Otherwise, set it to "JSON".
username - The username follows the following format: <fleet_id>::<device_id>. A Fleet ID is 8 characters long, and a Device ID is 10 characters long. They are separated by two colons ::, essentially creating a 20-character string.
password - The Device Secret is used as the password. The Device Secret starts with FOS- with 32 characters after it (totaling 36 characters).
keep_alive - The keep alive interval in seconds. If keep_alive is less than 11 seconds, Fostrom assumes the MQTT connection should be terminated after sending a response packet. This is useful for devices that don't need a persistent connection to Fostrom. If you want a persistent connection, set keep_alive to 30 seconds preferably.
clean_session - This flag is ignored by Fostrom.
will - You cannot set a Will in the connect packet at this time. The connection will fail if a will is specified.

Once you send a connect packet, either a CONNACK (0) -> ACCEPTED approving the connection will be returned, or one of two errors will be returned and the connection will be closed:

CONNACK (3) -> SERVER UNAVAILABLE - This is a transient error, the device should retry connecting after a short delay. Devices should use an exponential backoff strategy when trying to reconnect in face of network or server errors.
CONNACK (5) -> UNAUTHORIZED - This is a fatal error, retrying with the same credentials will not help. The device should not attempt to reconnect.

Once the device is connected, it can send subseuqent packets to Fostrom.

Note that devices should use an exponential backoff strategy when trying to reconnect in face of network or server errors.

`CONNECT` Packet:

-> CONNECT
    client_id: "MSGPACK" | "JSON"
    username: "<fleet_id>::<device_id>" # total 20 characters
    password: "<device_secret>"         # starts with `FOS-`, 36 characters
    keep_alive: 0 | 30 | 50
    clean_session: true
    will: nil

<- CONNACK
    code: 0 | 3 | 5
    # 0 = ACCEPTED
    # 3 = SERVER UNAVAILABLE
    # 5 = UNAUTHORIZED

Heartbeats

Fostrom shows whether a device is connected or not, based on heartbeats and open sockets.

MQTT has a PINGREQ packet that acts a heartbeat. The Fostrom server responds with a PINGRESP packet. It is normal for devices to connect to Fostrom and send a PINGREQ packet, and then disconnect. This is useful for devices that don't need a persistent connection to Fostrom but to indicate that they are functioning normally.

-> PINGREQ

<- PINGRESP

There is also another way to send a heartbeat by publishing a packet. This exists because many MQTT clients do not support the user manually triggering a PINGREQ packet. If you need manual control over when the heartbeat should be sent, you can the publish method, as documented below.

Publishing to Fostrom

Fostrom ignores the dup and retain flags in PUBLISH packets. It supports QoS 0 and QoS 1, however we highly recommend using QoS 1 for all packets. There is only one topic a device can publish to: fostrom. The client should send a packet id between 1 and 65535 for QoS 1 packets. Fostrom does not support QoS 2 at this time. The payload should be an object/map, and needs to be serialized with your configured serializer (MsgPack or JSON).

-> PUBLISH
    topic: "fostrom"
    qos: 0 | 1        # we recommend QoS 1
    id: 1-65535
    payload: <msgpacked_payload | json_encoded_payload>

<- PUBACK (for QoS 1 only)
    id: <packet_id>

Heartbeats through Publishing

To send a heartbeat by publishing a packet, the payload should be:

{
  "type": "heartbeat"
}

Send a Datapoint

To send a datapoint, you need to send the following fields in the payload:

{
  "type": "datapoint",
  "name": "<packet-schema-name>",
  "payload": {
    ...fields
  }
}

Send a Message

To send a message, you need to send the following fields in the payload:

{
  "type": "msg",
  "name": "<packet-schema-name>",
  "payload": {
    ...fields
  }
}

Receiving Messages

Step 1: Subscribe to topic `fostrom`

MQTT mandates that a client should subscribe to a topic before a server can send a PUBLISH packet to that topic. The MQTT spec mandates that a client should terminate the connection if it receives a PUBLISH packet on a topic it has not subscribed to. Since many MQTT clients are written to adhere to the spec, Fostrom requires that a client subscribe to the topic fostrom before Fostrom can publish incoming messages to the device.

Once a device sends a SUBSCRIBE packet for topic fostrom, Fostrom will send a SUBACK confirming the subscription.

Step 2: Receive any Messages in the Mailbox

If there are any messages in the device's mailbox currently, Fostrom will also PUBLISH packets on topic fostrom with the next message in the mailbox. If there are no messages, no PUBLISH packets will be sent. The device should send a PUBACK for each PUBLISH packet it receives. PUBACKs are a way of acknowledging the receipt of the PUBLISH packet, not a way of acknowledging the processing of the message, which has to be done separately, see below for details.

Message from Fostrom

The payload for a message sent from Fostrom looks like the following:

{
  "type": "msg",
  "msg_id": "<msg_id>",
  "name": "<packet-schema-name>",
  "payload": {
    ...fields
  },
  "more_msgs_available": true | false
}

Remember you need to deserialize the PUBLISH packet first.

Flow Summary

-> SUBSCRIBE
    topic: "fostrom"
    qos: 1
    id: 1-65535

<- SUBACK
    id: <packet_id>
    code: 0             # 0 = SUCCESS

IF MAILBOX HAS AT LEAST ONE MESSAGE
  <- PUBLISH
      topic: "fostrom"
      qos: 1
      id: 1-65535
      payload: <payload>  # JSON or MsgPack

  -> PUBACK  # device should send a PUBACK for the PUBLISH packet above
      id: <packet_id>

Acknowledging, Rejecting or Requeuing Messages

It is crucial for the device to either acknowledge, reject, or requeue any message that it receives from Fostrom. To acknowledge, reject, or requeue a message, the device should send a PUBLISH packet on the topic fostrom with the payloads described below.

Acknowledging Messages

To acknowledge a message, you need to send the following fields in the payload:

{
  "ack_msg": "<msg_id>"
}

Reject Messages

To reject a message, you need to send the following fields in the payload:

{
  "reject_msg": "<msg_id>"
}

Requeueing Messages

To requeue a message, you need to send the following fields in the payload:

{
  "requeue_msg": "<msg_id>"
}

Notes

The MQTT Specification mandates immediately closing the connection if a malformed packet is received. Hence, Fostrom will close the connection abruptly if any malformed payload is sent, such as incorrectly serialized payloads, or incorrect packet types or incorrect topics for PUBLISH and SUBSCRIBE packets.

Although it is not documented above, the device can send an UNSUBSCRIBE packet for the topic fostrom, at which point Fostrom will send an UNSUBACK packet confirming the unsubscription. Fostrom will stop sending any new messages that arrive in the device's mailbox. The device can subscribe again at any time to receive new messages.

Fostrom implements a robust streaming decoder to handle chunked packets and multiple packets concatenated together. Some MQTT clients such as MQTT.js write bytes in small chunks, which works perfectly with Fostrom.