Category: Scalability

  • Kafka vs Z-Mesh

    Both Z-Mesh and Apache Kafka are both distributed event streaming platforms. Both can be used for real-time data processing, data integration and message queuing. But what are the differences?

    TLDR: Z-Mesh uses a request-response pattern and supports publish/subscribe. Kafka uses one-way message queues that supports event-streaming.

    Z-Mesh

    Z-Mesh is a distributed network of interconnected Forwarders, device and systems.

    Message pattern: request-response

    Kafka

    Kafka is a message queue / broker.

    Message pattern: One-way

    Consumers are systems like space-, cleaning or energy-optimization systems, Facility Management- or analytics- or manufacturing intelligence systems.

    Z-Mesh request-response pattern

    In Z-Mesh, the network is tasked with the responsibility to return the requested content (event). There are two scenarios, depending on whether the network has stored (cached) the content or not:

    • If stored, the network returns the content
    • If not stored, the network forwards the request to the producing node, which then responds with the content

    Z-Mesh data-frame

    Content (events) in a Z-Mesh network are pieces of data that are labeled with a Name and a sequence number. IoT-devices and systems (nodes) can request content by using it’s name and sequence number. Content produced by IoT-devices and systems are stored in the Forwarding nodes (Forwarders) for a set period of time (eg. 24 hours) and can be requested at a later time. This enables battery-driven IoT-devices to wake up from sleep and request or publish content.

    Event Streaming or Publish / Subscribe

    Event streaming or publish/subscribe is initiated by the requesting device, by sending a request to the network with a wildcard sequence number and a long timeout. When content is published, it is instantly forwarded to all the requesting nodes.

    Direct data-exchange

    Content between two nodes can be exchanged directly; Two wireless Sub-GHz nodes can exchange directly, without the need for a Forwarder/network being present.

    Kafka one-way message queuing

    In Kafka, producers (devices or systems) sends data/events to a queue (Kafka) which consumers then either receive or retrieve at a later point in time. Content flows one-way and as the request, by the consumer, is served by Kafka, it never reaches the producer.

    Encryption and authorization

    In Z-Mesh, any node (that has access to the network) can request any piece if content at any time, however only the nodes with the encryption key can decode it. The object of encryption is the content (event/data), not the connection like in TCP/IP.

    Kafka is built on top of TCP/IP and so the connection between the sender/receiver and Kafka is encrypted (TLS). It means that data is stored unencrypted in Kafka and that access to data (authorization) is implemented at the connection level.

    Application layer

    The Z-Mesh Device Management protocol defines data-formats for many types of events. This means that when consumers are given access to an event-stream (by providing them with the Content Name and Encryption Key) they are also provided with the data-format, so they know how to interpret the received data. The Z-Mesh network itself has no knowledge of the data-format.

    Like MQTT and Z-Mesh, Kafka is agnostic to the data-format. There is no standard that defines how producers and consumers “speak together”.

    Device Management

    The Z-Mesh Device Management protocol authorizes devices and systems to participate in the network. Each Content Name has it’s own encryption key and data-format and the Z-Mesh Device Management protocol defines how they are managed.

    Kafka, like MQTT, is a transport layer and does not define Device Management.

    Addressing content

    Any device or system in a Z-Mesh network is able to request (address) any piece of content.

    The value of a telecommunications network is proportional to the square (N2) of the number of connected compatible communicating devices
    — Bob Metcalfe

    Kafka is not designed as a network is was designed as a one-way event streaming platform. Addressing content is done by connecting to a message queue (topic). But you could argue that it supports addressing any piece of content by using the Message Queue Name (topic) as the address.

    A note on distributed systems

    Distributed systems has no single-point of failure. Z-Mesh is a distributed network. MQTT is an example of a centralized network.

    Comparison table

    Z-MeshKafkaMQTTLoRa
    PatternReq/ResOne-way
    ArchitectureDistributedDecentralizedCentralizedCentralized
    Central Device ManagementXX
    Application layerX
    Direct commsX
    Group comm.X
    Centrally manage secure data-flowsXX
    Offline supportXX
    Vendor neutral hardwareXXX
    Software update possibleXX(?)
    Uses 20% of battery.
    No vendors support it.
    Royalty freeXXX

  • Why Event Streaming for IoT?

    Event streaming is particularly well-suited for the Internet of Things (IoT) for several reasons, primarily due to the nature of IoT data and the requirements for processing that data in real-time.

    Here are some key factors that make event streaming an ideal fit for IoT applications:

    1. High Volume of Data:

    • Continuous Data Generation: IoT devices generate vast amounts of data continuously, often in real-time. Event streaming platforms can handle high-throughput data streams, making them capable of processing the large volumes of data produced by numerous IoT devices.

    2. Real-Time Processing:

    • Immediate Insights: Many IoT applications require real-time data processing to derive insights, trigger actions, or make decisions. Event streaming allows for the immediate processing of events as they occur, enabling timely responses to changing conditions (e.g., alerts for anomalies, automated adjustments in smart systems).

    3. Decoupling of Data Producers and Consumers:

    • Loose Coupling: Event streaming architectures decouple data producers (IoT devices) from data consumers (applications, analytics engines). This allows for greater flexibility in how data is processed and consumed, enabling multiple applications to subscribe to the same data stream without direct dependencies.

    4. Scalability:

    • Dynamic Scaling: Event streaming platforms can scale horizontally to accommodate increasing numbers of IoT devices and data streams. This scalability is crucial as IoT deployments grow and evolve over time.

    5. Data Integration:

    • Unified Data Pipeline: Event streaming platforms can serve as a central data pipeline that integrates data from various IoT devices and sources. This integration allows for a holistic view of the data and facilitates analytics, machine learning, and other processing tasks.

    6. Event-Driven Architecture:

    • Reactive Systems: IoT applications often benefit from an event-driven architecture, where systems react to events as they occur. Event streaming supports this paradigm, allowing for the development of reactive applications that respond dynamically to incoming data.

    7. Support for Complex Event Processing:

    • Advanced Analytics: Event streaming platforms enables the detection of patterns, trends, and anomalies in real-time. This is particularly valuable for applications like predictive maintenance, fraud detection, and smart city management.

    8. Durability and Reliability:

    • Data Persistence: Many event streaming systems provide durability and reliability features, ensuring that data is not lost even in the event of failures. This is important for IoT applications where data integrity is critical.

    Conclusion:

    In summary, event streaming is well-suited for IoT due to its ability to handle high volumes of real-time data, support for scalable and flexible architectures, and capabilities for immediate processing and integration. These features enable organizations to leverage IoT data effectively, driving insights and actions that enhance operational efficiency and decision-making.

  • IoT solutions must design for “anarchic scalability”

    In semi-open or closed loops (i.e., value chains, whenever a global finality can be settled) the IoT will often be considered and studied as a complex system due to the huge number of different links, interactions between autonomous actors, and its capacity to integrate new actors. At the overall stage (full open loop) it will likely be seen as a chaotic environment.

    Given widespread recognition of the evolving nature of the design and management of the Internet of things, sustainable and secure deployment of IoT solutions must design for “anarchic scalability”

    Source: Wikipedia: Internet of Things

  • Polite Spectrum Access (listen before talk)

    When an IoT device is about to transmit, it should listen if some other IoT device is transmitting before itself transmits. This is just like humans, we wait until the other finishes before talking. This is called Polite Spectrum Access. If you transmit without listening first, it is referred to as Aloha.

    Listen-before-talk (LBT) can increase the spectrum efficiency compared to Aloha by reducing the number of collisions and retransmissions.

    In Aloha, each node transmits its packets without checking if the channel is busy, which can lead to collisions and retransmissions. This can result in a significant waste of bandwidth and reduce the overall spectrum efficiency.

    LBT, on the other hand, requires each node to listen to the channel before transmitting, which can help to reduce the number of collisions and retransmissions. By listening to the channel, a node can determine if the channel is busy and wait until it is clear before transmitting.

    Studies have shown that LBT can increase the spectrum efficiency compared to Aloha by a factor of 2-5, depending on the specific scenario and parameters. For example, a study by the European Telecommunications Standards Institute (ETSI) found that LBT can increase the spectrum efficiency by a factor of 2.5 compared to Aloha in a scenario with a large number of nodes and a high traffic load.

    Here is a rough estimate of the spectrum efficiency gain of LBT compared to Aloha:

    • Aloha: 18% (this is a typical value for Aloha in a scenario with a large number of nodes and a high traffic load)
    • LBT: 45-60% (this is a rough estimate of the spectrum efficiency gain of LBT compared to Aloha, depending on the specific scenario and parameters)

    LoRa is based on a modulation scheme called Chirp Spread Spectrum (CSS). It means that during transmission it increases/decreases the frequency at a rate depending on the speed. This makes it almost impossible for CSS to use Polite Spectrum Access.

    The EU/US regulation states that an IoT device using the sub-GHz band (868MHz/915MHz) must only transmit 1% if the time, unless it is using Polite Spectrum Access (ETSI EN 300 220-2). Being polite allows for a much more efficient use of the frequency spectrum, which is very important as the frequency spectrum is a limited finite natrual resource. Going forward, we will have many more IoT devices using the spectrum, so using it efficiently is very important.

    Almost all commercial off-the-shelf FSK transmitters is able to be polite and listen before talk. Surely we must access the frequency spectrum most efficiently and not use inpolite transmitters.

  • LoRa is not frequency spectrum efficient

    LoRa uses a spread spectrum modulation technique to transmit data. While LoRa has some advantages, such as its ability to transmit data over long distances and its low power consumption, it is not considered to be a spectrum-efficient technology.

    There are several reasons why LoRa is not considered to be spectrum-efficient:

    1. Modulation technique: LoRa uses a modulation technique, which spreads the data signal across a wide frequency band. This can lead to a lower spectral efficiency, as the signal is spread across a wider frequency band than is necessary.
    2. Low data rate: LoRa is designed for low-data-rate applications, such as sensor networks and IoT devices. While this can be beneficial for certain applications, it also means that LoRa is not well-suited for higher-data-rate applications, which can be more spectrum-efficient.
    3. Listen-before-talk (LBT): LoRa is not able to use LBT due to it’s modulation technique. LBT increases the spectrum efficiency reducing the number of collisions and retransmissions.
    4. In-Building environments: LoRa (Long Range) has been developed for use in long range applications (kilometers) but is being used in building environments in which the long range is not needed. Higher data-rate modulation techniques, such as FSK or BPSK, allows for a shorter on-air transmission time therefore being more spectrum efficient.

    LoRa is not as spectrum efficient as other modulation schemes such as BPSK or TurboFSK, which can be implemented in an commercial of-the-shelf radio transceiver. Going forward, The number of IoT devices per square meter will only increase and since the amount of frequency spectrum available is is a natural and finite resource, we should strive to use it as efficiently as possible.

    Fun fact: The first wireless transmissions was done by Marconi using spark-gap transmitters, back in 1896. They basically emitted white noise, which could then be detected at long distances. Whenever a spark-gap transmission was in progress, it was using the entire frequency spectrum and hence only room for one transmission at a time! As time went on, the ability to modulate signals caused the frequency spectrum to be used much more efficiently.