IoT Data Management

This chapter describes how data is ingested into OS2iot, how it is processed, and how it is sent to data target.

The following sequence diagram illustrates how data is processed when it is received.

image2

Data ingestion

OS2iot supports three types of devices:

  1. LoRaWAN devices (using Chirpstack)

  2. SigFox devices

  3. Generic HTTP devices

OS2iot ingests data for each of these device types in a slightly different way:

LoRaWAN devices

OS2iot connects to LoRaWAN devices using Chirpstack, similarly the data from these are ingested through Chirpstack. In the class: ChirpstackMQTTListenerService in OS2IoT-backend, a MQTT listener is setup to listen to the MQTT broker in chirpstack. It listens to the device data event topic using the pattern: application/+/device/+/event/up. When a message is received, it is parsed into the DTO: ChirpstackMQTTMessageDto, if the device EUI is registered in OS2iot, then the message is further processed, otherwise it is discarded.

SigFox devices

When a SigFox device is created in OS2iot, the device type of it in the SigFox backend is updated to include a bi-directional callback to the backend at the endpoint: /api/v1/sigfox-callback/data/bidir?apiKey={deviceTypeId}. Furthermore the body is set to:

{
    "time": {time},
    "deviceTypeId": "{deviceTypeId}",
    "deviceId": "{device}",
    "data": "{data}",
    "seqNumber": {seqNumber},
    "ack": {ack}
}

When the callback is called, the deviceTypeId is validated to be one that matches what OS2iot knows exists. The deviceId of the body is validated to exist in OS2iot, if it does, then the message is further processed, and the callback is responded with 204 No Content. Note: If the value of ack is true, then the device is ready for a downlink message, then OS2iot will see if it has a message queued for the device, and if it does, remove it from the queue and send it back with status 200 OK.

Generic HTTP devices

OS2iot supports generic HTTP(S) devices, that means that each device of this type gets an API-key at the time of creation, and can send their data using that key, both for identification and authorization. The endpoint /api/v1/receive-data?apiKey=<guid> is used for this, where <guid> is a GUID like: 2704e33f-b678-4d3a-a3b9-b36454cabf22. If the API-key is valid and corresponds to a device in OS2iot, then the response will be 204 No Content, and the message is further processed. If the API-key is unknown, then the resposne will be 403 Bad Request with the message: MESSAGE.DEVICE-DOES-NOT-EXIST

Payload transformation and storage

Once that data is ingested from one of the sources described above, it can be transformed before being sent to its intended target(s). Two examples of payload decoders are shown here as well as an introduction to how they work in OS2iot.

At the same time as payload transformation, the latest message payload is saved, and the metadata (timestamp and signal) is saved for the last 20 messages. This is done for debugging purposes, such that you can test payload decoders on actual and up-to date data.

Sending to data-targets

Once the data is transformed/decoded, it can be sent to the intended data-targets. OS2iot currently supports HTTP push, FIWARE and MQTT as data-targets. It can be extended with other kinds of data-targets, e.g. PostgreSQL, etc.

The data-targets are configured as part of an application, and therefore can only include data from the IoT-Devices in said application. For each IoT-Device, it is up to the user to select an appropriate payload decoder, to translate the raw payload to JSON.

At the time of writing, there is no retry mechanism in OS2iot over HTTP(s). It uses a “fire-and-forget” / “at most once delivery” pattern.

MQTT data-target

OS2iot supports publishing data to a broker when it’s received using MQTT. MQTT is a standard, lightweight messaging protocol based on the publish/subscribe pattern.

When configuring such a data-target, there’s a few terms and keywords to be aware of:

  • QoS: The QoS (Quality of Service) level determines the guarantee of delivery for a specific message. Different network environments may require different QoS levels. Ideally, the level should be set to match the network reliability and application logic. This is the main point of MQTT.

    There are 3 QoS levels:

    • 0 (at most once)

    • 1 (at least once)

    • 2 (exactly once)

    There are a number of well written articles regarding QoS. One such example is this blog entry.

  • Topic: The MQTT data-target must be provided a topic with which it can label the data. This is used by the MQTT broker to filter messages from MQTT clients. Here, OS2iot is a client.

  • Connection authentication: The most common methods of authentication are username and password and/or client certificates. At the time of writing, username and password authentication is supported, but it can be extended to implement other methods.

You can read more on MQTT here