Author: adk

  • Kafka vs Z-Mesh

    Both Z-Mesh and Apache Kafka are both distributed event streaming platforms. Both can be used for real-time data processing, data integration and message queuing. But what are the differences?

    TLDR: Z-Mesh uses a request-response pattern and supports publish/subscribe. Kafka uses one-way message queues that supports event-streaming.

    Z-Mesh

    Z-Mesh is a distributed network of interconnected Forwarders, device and systems.

    Message pattern: request-response

    Kafka

    Kafka is a message queue / broker.

    Message pattern: One-way

    Consumers are systems like space-, cleaning or energy-optimization systems, Facility Management- or analytics- or manufacturing intelligence systems.

    Z-Mesh request-response pattern

    In Z-Mesh, the network is tasked with the responsibility to return the requested content (event). There are two scenarios, depending on whether the network has stored (cached) the content or not:

    • If stored, the network returns the content
    • If not stored, the network forwards the request to the producing node, which then responds with the content

    Z-Mesh data-frame

    Content (events) in a Z-Mesh network are pieces of data that are labeled with a Name and a sequence number. IoT-devices and systems (nodes) can request content by using it’s name and sequence number. Content produced by IoT-devices and systems are stored in the Forwarding nodes (Forwarders) for a set period of time (eg. 24 hours) and can be requested at a later time. This enables battery-driven IoT-devices to wake up from sleep and request or publish content.

    Event Streaming or Publish / Subscribe

    Event streaming or publish/subscribe is initiated by the requesting device, by sending a request to the network with a wildcard sequence number and a long timeout. When content is published, it is instantly forwarded to all the requesting nodes.

    Direct data-exchange

    Content between two nodes can be exchanged directly; Two wireless Sub-GHz nodes can exchange directly, without the need for a Forwarder/network being present.

    Kafka one-way message queuing

    In Kafka, producers (devices or systems) sends data/events to a queue (Kafka) which consumers then either receive or retrieve at a later point in time. Content flows one-way and as the request, by the consumer, is served by Kafka, it never reaches the producer.

    Encryption and authorization

    In Z-Mesh, any node (that has access to the network) can request any piece if content at any time, however only the nodes with the encryption key can decode it. The object of encryption is the content (event/data), not the connection like in TCP/IP.

    Kafka is built on top of TCP/IP and so the connection between the sender/receiver and Kafka is encrypted (TLS). It means that data is stored unencrypted in Kafka and that access to data (authorization) is implemented at the connection level.

    Application layer

    The Z-Mesh Device Management protocol defines data-formats for many types of events. This means that when consumers are given access to an event-stream (by providing them with the Content Name and Encryption Key) they are also provided with the data-format, so they know how to interpret the received data. The Z-Mesh network itself has no knowledge of the data-format.

    Like MQTT and Z-Mesh, Kafka is agnostic to the data-format. There is no standard that defines how producers and consumers “speak together”.

    Device Management

    The Z-Mesh Device Management protocol authorizes devices and systems to participate in the network. Each Content Name has it’s own encryption key and data-format and the Z-Mesh Device Management protocol defines how they are managed.

    Kafka, like MQTT, is a transport layer and does not define Device Management.

    Addressing content

    Any device or system in a Z-Mesh network is able to request (address) any piece of content.

    The value of a telecommunications network is proportional to the square (N2) of the number of connected compatible communicating devices
    — Bob Metcalfe

    Kafka is not designed as a network is was designed as a one-way event streaming platform. Addressing content is done by connecting to a message queue (topic). But you could argue that it supports addressing any piece of content by using the Message Queue Name (topic) as the address.

    A note on distributed systems

    Distributed systems has no single-point of failure. Z-Mesh is a distributed network. MQTT is an example of a centralized network.

    Comparison table

    Z-MeshKafkaMQTTLoRa
    PatternReq/ResOne-way
    ArchitectureDistributedDecentralizedCentralizedCentralized
    Central Device ManagementXX
    Application layerX
    Direct commsX
    Group comm.X
    Centrally manage secure data-flowsXX
    Offline supportXX
    Vendor neutral hardwareXXX
    Software update possibleXX(?)
    Uses 20% of battery.
    No vendors support it.
    Royalty freeXXX

  • Why Event Streaming for IoT?

    Event streaming is particularly well-suited for the Internet of Things (IoT) for several reasons, primarily due to the nature of IoT data and the requirements for processing that data in real-time.

    Here are some key factors that make event streaming an ideal fit for IoT applications:

    1. High Volume of Data:

    • Continuous Data Generation: IoT devices generate vast amounts of data continuously, often in real-time. Event streaming platforms can handle high-throughput data streams, making them capable of processing the large volumes of data produced by numerous IoT devices.

    2. Real-Time Processing:

    • Immediate Insights: Many IoT applications require real-time data processing to derive insights, trigger actions, or make decisions. Event streaming allows for the immediate processing of events as they occur, enabling timely responses to changing conditions (e.g., alerts for anomalies, automated adjustments in smart systems).

    3. Decoupling of Data Producers and Consumers:

    • Loose Coupling: Event streaming architectures decouple data producers (IoT devices) from data consumers (applications, analytics engines). This allows for greater flexibility in how data is processed and consumed, enabling multiple applications to subscribe to the same data stream without direct dependencies.

    4. Scalability:

    • Dynamic Scaling: Event streaming platforms can scale horizontally to accommodate increasing numbers of IoT devices and data streams. This scalability is crucial as IoT deployments grow and evolve over time.

    5. Data Integration:

    • Unified Data Pipeline: Event streaming platforms can serve as a central data pipeline that integrates data from various IoT devices and sources. This integration allows for a holistic view of the data and facilitates analytics, machine learning, and other processing tasks.

    6. Event-Driven Architecture:

    • Reactive Systems: IoT applications often benefit from an event-driven architecture, where systems react to events as they occur. Event streaming supports this paradigm, allowing for the development of reactive applications that respond dynamically to incoming data.

    7. Support for Complex Event Processing:

    • Advanced Analytics: Event streaming platforms enables the detection of patterns, trends, and anomalies in real-time. This is particularly valuable for applications like predictive maintenance, fraud detection, and smart city management.

    8. Durability and Reliability:

    • Data Persistence: Many event streaming systems provide durability and reliability features, ensuring that data is not lost even in the event of failures. This is important for IoT applications where data integrity is critical.

    Conclusion:

    In summary, event streaming is well-suited for IoT due to its ability to handle high volumes of real-time data, support for scalable and flexible architectures, and capabilities for immediate processing and integration. These features enable organizations to leverage IoT data effectively, driving insights and actions that enhance operational efficiency and decision-making.

  • EU Landscape report on Interoperability

    Quotes:

    The energy system is therefore progressively evolving from a system centrally operated and optimized into a federated system … As a consequence the associated digital infrastructures need to accelerate their transformation from few central monolithic control room environments … towards new orchestrated platform architectures taking advantage of IOT, edge computing as well as hybrid private and public cloud architectures where real-time data exchanges, interoperability and Open Application Programmable Interfaces have become critical technology building blocks to enable plug and play data interfaces.

    Enterprise service bus (ESB) or other integration solutions were used for the deployment of first Service Oriented architectures to decouple systems. However, the number of connectors, as well as the growing requirements for heterogeneous data consuming interfaces feeding a variety of data interfaces through real-time has led applications, system components and enterprise services to remain closely intertwined, as legacy SCADA platform vendors did not provide sufficient reengineering efforts to establish interoperability across data interfaces although the emergence the IEC CIM standards. This situation has limited the possibility to integrate platform and application from different vendors as well as expand control room platforms with modern technology stacks developed through the opensource community.

    Event streaming platforms:

    This recent transformation is progressively heading towards the migration of original Control Room platforms into new event streaming platforms leveraging events as a core integration principle and orchestrating most of energy management and control business processes around real-time data streams. This new architecture is designed in view of the event data flows while data processing is orchestrated on data while they are in motion. It has the following key benefits:

    • Event-based data flows is a foundation for (near) real-time and batch processing as required in most of flexibility management processes. In previous SOA architectures, applications were built on data stores (data at rest), which was making it impossible to build flexible and agile services to act on data very close to real-time.
    • Scalable architectures for all events shared across infinite source and sink processes. As opposed to centralised monolithic applications, the architecture is built on scalable, distributed infrastructures by design for zero downtime, handling the failure of nodes and networks while being able to roll out upgrades online. Different versions of infrastructure (like Kafka) and applications (business services) can be deployed and managed in an agile, dynamic way. This approach minimises dependencies across application which was particularly complex to manage through historical SOA architectures.
    • Openness to any kind of applications, system components and microservices. Technology does not matter. The streaming environment connects anything: programming languages, APIs like REST or MQTT, open standards, proprietary tools, and legacy applications which reduces the need to redesign existing legacy applications while benefitting from highly scalable cloud containerized environments enabling rapid prototyping of new applications. It also allows to minimize processing speed constraints for real-time control applications.
    • Distributed storage for decoupling applications. The platform data streaming environment allows to store the state of a microservice instead of requiring a separate database.
    • Stateless service and stateful business processes. Business processes typically are stateful processes. They often need to be implemented with events and state changes, for which remote procedure calls and request-response as considered through SOA architecture is not optimal.

    The solid Open Source approach deployed around Apache Kafka makes it the preferred choice…

    Link to the report

  • Digital sovereignty for Europe

    Where is EU is heading in the IoT space?

    Quotes from the PDF

    1. Europe’s ability to act independently in the digital world
    2. Reliable digital infrastructure and services are critical in today’s society
    3. There are calls to build a European cloud and data infrastructure to strengthen Europe’s data sovereignty and address the fact that today, the cloud and data storage market is almost exclusively dominated by non-European suppliers.
    4. Furthermore, investment in frontier technologies, including artificial intelligence, IoT,…
    5. In the long run, building a genuinely sovereign EU digital environment will also require addressing the current lack of coordination between regulators in this field. … Such mechanisms would be critical for instance to ensuring a coherent EU sovereign approach in many areas, such as applications management (e.g. apps or IoT devices in the data privacy field), or platform regulation.

    An open IoT platform approach is absolutely critical and will succeed in the long run.

    What EU is saying

    Rolf Riemenschneider – Head of Sector IoT at European Commission:

    Quotes:

    • 4:38 Aver the air updates
    • 5:00 Avoid Vendor lock in
    • 6:40 Support IoT – Platform approach in the centre – demonstrate orchestation across application sectors (application agnostic)
    • 7:22: So a Platform approach to enforce an open ecosystem
    • 8:00 Open Core – 8M EUR to attract Open Core Open source developers
    • 8:36: Open Calls for a vibrant IoT ecosystem
    • 8:58: Interoperability – most important. Open Interface, Open standards
    • 10:10 Lack of cross sector orchestration
    • 11:20 Data sharing (quite a challenge) – we must go back to the platform strategy/standards

    Max Lemke – Head of Unit for IoT

    Quotes:

    • 1:56 Competitiveness and Standardization are mentioned again and again in the Draghi Report
    • 2:40 Geopolitical tensions … we are either smached in the middle or we find our way … underscoring the need for unified IoT standardization in Europe
    • 3:33 By fostering collaboration and sharing we can better navigate these challenges
    • 7:40 We need to look at Open Platforms … not driven by one actor
    • 8:32 It means a multi-sided market place with different actors … and standards are crutial for scalability
    • 9:00 They enable devices from different manufactures … to ensure interoperability …

  • Sensor to Cloud is only half of the way

    Machine-oriented solutions form the bulk of the interesting new areas that people talk about when discussing M2M and the IoT. Unfortunately, neither of these terms is accurate. These are machines sending readings to a central database or computer that can take action – such as sending an electricity bill or scheduling a garbage truck. This is not really “machine to machine” but rather sensor to database. This may seem to be a pedantic use of semantics but there is an important point here that there is little reason for one “machine” – for example a device such as a smart meter – to talk directly to another. Equally it is not really an Internet of Things. The Internet implies an interconnected network where one computer can access information on another computer. Instead, typically, only the “owner” of the machine, such as the electricity company, will be able to access its readings and communicate with the device. Consumers that wish to read this data will then retrieve it from a cloud-based server, not direct from the machine. It is more like the Intranet of Things” where connectivity is restricted to self-contained groups rather than the “Internet’. Again, this may seem pedantic, but it has important implications for installation, security and network architecture.

    Most applications envisaged are self-contained in that the data generated is used just for that application. So, for example, in smart metering the meters report to electricity companies or with
    parking the car park sensors talk to parking applications.

    While the original version of Metcalfe’s law stated that, the value of a telecommunications network is proportional to the square of the number of compatible communicating devices (n2), it did not
    specify what protocol it should run on.

    Today, many people believe that, for IoT to be a reality, all devices must run the IP protocol so that they can form an interconnected network where all devices can share and access information. Research shows that the IP protocol may not be the best fit for Internet of Things. The question is therefore: How do we make a network for PCs, servers and constrained battery-driven devices that allows for the IoT vision to be a reality.

    The requirements

    A successful IoT network architecture has the following requirements:

    1. Devices in an IoT network must be able to access and share content in near real-time
    2. Devices must be able to send content to groups of devices
    3. Content must be secured so that only authorized parties can access it
    4. Offline devices must be able to pull content from the network when they wake up
    5. Content must be accessible across different types of physical communication mediums, be it
      wired or wireless, using the same naming scheme
    6. Communication standard MUST be open and royalty-free for anyone to implement and use
    7. Transmission rate must be fast enough to support firmware updates, sub-GHz included

    Managing the devices, the data and the data-security is important for organizations deploying IoT
    networks. A successful IoT networking protocol therefore need to implement these features.
    Requirements here are:

    1. Devices and security must be centrally managed
    2. Being able to centrally manage what device sends content and which receives that content

    The rapid growth in Internet of Things (IoT) deployment has posed unprecedented challenges to the underlying network design. Tomorrow’s global-scale IoT systems will focus on service-oriented data sharing and processing rather than point-to-point data collection. Requirements such as global reachability, mobility, communication diversity, and security will also develop naturally along with the change in the communication patterns. However, existing (IP-based)networks focus only on locations and point-to-point channels. The mismatch between the dynamic requirements and the functionalities provided by the network renders IoT communication inefficient and inconvenient.

  • Cellular Non-IP Networking (NIN)

    Telecom IoT-customers has found out that sending a 4-byte floating-point value using MQTT, has an overhead of 702 bytes:

    https://docs.devicewise.com/Content/Products/IoT_Portal_API_Reference_Guide/MQTT_Interface/MQTT-data-usage.htm

    Therefore, the telecoms and standards organisations has come up with Cellular Non-IP Networking. (1NCE has even made their own OS to solve this problem!):

    https://www.icann.org/en/system/files/files/octo-026-12jul21-en.pdf

    Similarly to the Huawei New IP proposal, the starting point of ETSI NIN ISG is the claim that TCP/IP is an old protocol, unsuitable for the new types of applications promised by 5G…

    and

    The initial assumption of this effort is that the TCP/IP protocol suite, now 40 years old, is no longer suitable for modern networking.

  • IoT solutions must design for “anarchic scalability”

    In semi-open or closed loops (i.e., value chains, whenever a global finality can be settled) the IoT will often be considered and studied as a complex system due to the huge number of different links, interactions between autonomous actors, and its capacity to integrate new actors. At the overall stage (full open loop) it will likely be seen as a chaotic environment.

    Given widespread recognition of the evolving nature of the design and management of the Internet of things, sustainable and secure deployment of IoT solutions must design for “anarchic scalability”

    Source: Wikipedia: Internet of Things

  • Interoperability is the devil

    40% of the total economic value of the Industrial IoT will remain locked

    In any interconnected system, all of its component devices must be able to communicate with each other—to speak in the same language, even if these devices work in very different fields. A lack of common software interfaces, standard data formats, and common connectivity protocols are complicating things in IoT-land. For industries, this means that 40% of the total economic value of the Industrial IoT will remain locked because different systems cannot work together.

    Source: Madison.tech

    Standardization creates interoperability

    The lack of standardization creates interoperability issues, which can lead to compatibility problems and limited functionality.

    For example, if two IoT devices cannot communicate with each other because they use different protocols or data formats, they cannot work together to achieve a common goal.

    Source: LinkedIn

  • The problem with TCP/IP for IoT?

    This section will discuss the problems with the IP protocol and why it it not a good fit for the IoT.

    Small MTU

    The maximum transfer unit (MTU) refers to the maximum amount of bytes you can fit in a data-packet. The MTU can be as little as 64 bytes in many wireless IoT systems. This is in clear contrast with today’s IP networks which typically assume a minimum MTU of 1500 bytes or higher.

    Multi-link subnets

    Multi-link subnets is the notion that a subnet may span multiple links connected by routers. RFC 4903, “Multi-Link Subnet Issues” [29], documents the reasons why the IETF decided to abandon the multi-link subnet model in favor of 1:1 mapping between Layer-2 links and IP subnets. An IoT mesh network, on other hand, contains a collection of Layer-2 links joined without any Layer-3 device (i.e., IP routers) in between. This essentially creates a multi-link subnet model that is not anticipated by the original IP addressing architecture.

    Multicast

    Multicast is a group communication model where data-transmission is addressed to a group of destination devices simultaniously. A lot of IP-based protocols make heavy use of IP multicast to achieve one of the two functionalities: notifying all the members in a group and making a query without knowing exactly whom to ask. Using multicast raises a number of concerns:

    • Devices sleeping will not receive the data-transmission
    • Receivers may have different data-transmission rate
    • Broadcasting data is too expensive, so a routing mechanism is necessary
    • Encryption for IP multicast still needs to be invented

    Mesh network routing

    IP based host-routing is a major challenge in constrained IoT devices as each host needs to maintain a routing table. This consumes memory and causes network overhead when the network changes. Also, forwarding traffic may involve decrypting the data from the incoming link and then encrypting it on the outgoing link – an expensive operation for battery-driven devices.

    Transport layer problems

    Due to the energy constraints, devices may frequently go into sleep mode, thus it is infeasible to maintain a long-lived connection in IoT applications. Also, a lot of communication involves only a small amount of data making the overhead of establishing a connection unacceptable.

    Unfortunately, current TCP/IP architecture does not allow to embed application semantics into network packets, thus failing to provide sufficient support for application level framing, which would allow the application more control over the data transmission.

    Application layer problems

    Many IoT applications implement a resource-oriented request-response communication model. ZigBee, CHIP/Matter are such examples. Influenced by the web, many IoT protocols has been working on bringing the same REST architecture into IoT applications. CoAP is an example of such a standard, which is also being used in the Thread protocol. There are a number of problems with this approach:

    • It usually requires resource discovery, such as DNS or Core-RD which in turn uses broadcast.
    • It requires that the client (requester) and the server (resource) is online at the same time
    • It requires a fundamental change to the security model in order to make the in-network caches secure and trustworthy

    Security

    In the IP based host-centric model, TLS/DTLS is used to secure the communication channel between the requester and the resource. However, this model does not fit with the requirements of IoT:

    • TLS/DTLS requires two or more exchanges of data to negotiate the communication channel, a resource and energy extensive task. Also, both ends have to maintain the state of the channel until it is closed, stopping devices from entering sleep-mode
    • Encryption for IP multicast still needs to be invented so security only works in host-based communication

    MQTT and AMQP

    MQTT and AMQP are publish/subscribe protocols that support x-to-many and many-to-x communication models. The underlying protocol is actually connection oriented, meaning that every publisher or subscriber have a connection to the broker (message exchange server). Messages are published to the broker and then the broker sends the message to each of the subscribers in turn, one by one.
    MQTT and AMQP does not have support for offline devices, so using a publish/subscribe protocol, requires the device to be awake 100% of the time. Battery driven sensors sleep and is offline most of the time, therefore not suited for using MQTT and AMQP.
    As these protocols are transport protocols, they do not specify how device management is done. Device management is supposed to be implemented as a layer on top.

    CoAP

    CoAP is a client-server protocol, modeled after the HTTP protocol, but for constrained (battery driven) devices. It allows a device to send a command or retrieve a value directly from another device – still connection oriented. It requires the sensor (server) to be awake (not sleeping) in order for it to respond.

    CoAP has a feature called “Observe”, which as described in RFC 7641, allows clients “subscribe” and be notified, when the condition has occurred. The great thing is that it allows a sensor to sleep until the condition occurs thereby saving battery, the downside of this one-to-many approach is that the sensor (server) must notify each client individually, each time setting up and tearing down a new connection. These connections are secured with DTLS, which is energy- and bandwidth consuming task, so a better way is clearly needed. So why not just use CoAP multicast? Well, you can, but there is no standard for encrypting CoAP multicast traffic. In fact, (DTLS) encryption in IP multicast still needs to be invented, meaning that you can not send encrypted information over the air to other CoAP devices using multicast!

    As for for sending data to an offline sensor, the CoAP standard does not specify how this is done, it is left to the device-creator, to implement this functionallity, making the device non-standard and therefore incompatible with other networks.

    As CoAP is just a transport protocol, it does not specify how device management is done. Device management is supposed to be implemented on top of CoAP.

  • IETF believes in ICN for IoT

    Internet Ingineering Task Force (IETF), the group that governs the Internet Standards (RFCs), believes that Information-Centric Networking (ICN) is much better suited to IoT than the current host based paradigms such as TCP/IP. Here are their findings:

    https://datatracker.ietf.org/doc/html/draft-irtf-icnrg-icniot-03