Kafka vs Z-Mesh

Both Z-Mesh and Apache Kafka are both distributed event streaming platforms. Both can be used for real-time data processing, data integration and message queuing. But what are the differences?

TLDR: Z-Mesh uses a request-response pattern and supports publish/subscribe. Kafka uses one-way message queues that supports event-streaming.

Z-Mesh

Z-Mesh is a distributed network of interconnected Forwarders, device and systems.

Message pattern: request-response

Kafka

Kafka is a message queue / broker.

Message pattern: One-way

Consumers are systems like space-, cleaning or energy-optimization systems, Facility Management- or analytics- or manufacturing intelligence systems.

Z-Mesh request-response pattern

In Z-Mesh, the network is tasked with the responsibility to return the requested content (event). There are two scenarios, depending on whether the network has stored (cached) the content or not:

  • If stored, the network returns the content
  • If not stored, the network forwards the request to the producing node, which then responds with the content

Z-Mesh data-frame

Content (events) in a Z-Mesh network are pieces of data that are labeled with a Name and a sequence number. IoT-devices and systems (nodes) can request content by using it’s name and sequence number. Content produced by IoT-devices and systems are stored in the Forwarding nodes (Forwarders) for a set period of time (eg. 24 hours) and can be requested at a later time. This enables battery-driven IoT-devices to wake up from sleep and request or publish content.

Event Streaming or Publish / Subscribe

Event streaming or publish/subscribe is initiated by the requesting device, by sending a request to the network with a wildcard sequence number and a long timeout. When content is published, it is instantly forwarded to all the requesting nodes.

Direct data-exchange

Content between two nodes can be exchanged directly; Two wireless Sub-GHz nodes can exchange directly, without the need for a Forwarder/network being present.

Kafka one-way message queuing

In Kafka, producers (devices or systems) sends data/events to a queue (Kafka) which consumers then either receive or retrieve at a later point in time. Content flows one-way and as the request, by the consumer, is served by Kafka, it never reaches the producer.

Encryption and authorization

In Z-Mesh, any node (that has access to the network) can request any piece if content at any time, however only the nodes with the encryption key can decode it. The object of encryption is the content (event/data), not the connection like in TCP/IP.

Kafka is built on top of TCP/IP and so the connection between the sender/receiver and Kafka is encrypted (TLS). It means that data is stored unencrypted in Kafka and that access to data (authorization) is implemented at the connection level.

Application layer

The Z-Mesh Device Management protocol defines data-formats for many types of events. This means that when consumers are given access to an event-stream (by providing them with the Content Name and Encryption Key) they are also provided with the data-format, so they know how to interpret the received data. The Z-Mesh network itself has no knowledge of the data-format.

Like MQTT and Z-Mesh, Kafka is agnostic to the data-format. There is no standard that defines how producers and consumers “speak together”.

Device Management

The Z-Mesh Device Management protocol authorizes devices and systems to participate in the network. Each Content Name has it’s own encryption key and data-format and the Z-Mesh Device Management protocol defines how they are managed.

Kafka, like MQTT, is a transport layer and does not define Device Management.

Addressing content

Any device or system in a Z-Mesh network is able to request (address) any piece of content.

The value of a telecommunications network is proportional to the square (N2) of the number of connected compatible communicating devices
— Bob Metcalfe

Kafka is not designed as a network is was designed as a one-way event streaming platform. Addressing content is done by connecting to a message queue (topic). But you could argue that it supports addressing any piece of content by using the Message Queue Name (topic) as the address.

A note on distributed systems

Distributed systems has no single-point of failure. Z-Mesh is a distributed network. MQTT is an example of a centralized network.

Comparison table

Z-MeshKafkaMQTTLoRa
PatternReq/ResOne-way
ArchitectureDistributedDecentralizedCentralizedCentralized
Central Device ManagementXX
Application layerX
Direct commsX
Group comm.X
Centrally manage secure data-flowsXX
Offline supportXX
Vendor neutral hardwareXXX
Software update possibleXX(?)
Uses 20% of battery.
No vendors support it.
Royalty freeXXX