Your edge gateway reads 200 tags from a PLC every second. The MQTT connection to your cloud broker drops for 3 minutes because someone bumped the cellular antenna. What happens to the 36,000 data points collected during the outage?
If your answer is "they're gone," you have a toy system, not an industrial one.
Reliable telemetry delivery is the hardest unsolved problem in most IIoT architectures. Everyone focuses on the protocol layer — Modbus reads, EtherNet/IP connections, OPC-UA subscriptions — but the real engineering is in what happens between reading a value and confirming it reached the cloud. This article breaks down the buffer architecture that makes zero-data-loss telemetry possible on resource-constrained edge hardware.
The Problem: Three Asynchronous Timelines
In any edge-to-cloud telemetry system, you're managing three independent timelines:
-
PLC read cycle — Tags are read at fixed intervals (1s, 60s, etc.). This never stops. The PLC doesn't care if your cloud connection is down.
-
Batch collection — Raw tag values are grouped into batches by timestamp and device. Batches accumulate until they hit a size limit or a timeout.
-
MQTT delivery — Batches are published to the broker. The broker acknowledges receipt. At QoS 1, the MQTT library handles retransmission, but only if you give it data in the right form.
These three timelines run independently. The PLC read loop runs on a tight 1-second cycle. Batch finalization might happen every 30–60 seconds. MQTT delivery depends on network availability. If any one of these stalls, the others must keep running without data loss.
This is fundamentally a producer-consumer problem with a twist: the consumer (MQTT) can disappear for minutes at a time, and the producer (PLC reads) cannot slow down.
The Batch Layer: Grouping Values for Efficient Transport
Raw tag values are tiny — a temperature reading is 4 bytes, a boolean is 1 byte. Sending each value as an individual MQTT message would be absurdly wasteful. Instead, values are collected into batches — structured payloads that contain multiple timestamped readings from one or more devices.
Batch Structure
A batch is organized as a series of groups, where each group represents one polling cycle (one timestamp, one device):
Batch
├── Group 0: { timestamp: 1709284800, device_type: 5000, serial: 12345 }
│ ├── Value: { id: 2, values: [72.4] } // Delivery Temp
│ ├── Value: { id: 3, values: [68.1] } // Mold Temp
│ └── Value: { id: 5, values: [12.6] } // Flow Value
├── Group 1: { timestamp: 1709284860, device_type: 5000, serial: 12345 }
│ ├── Value: { id: 2, values: [72.8] }
│ ├── Value: { id: 3, values: [68.3] }
│ └── Value: { id: 5, values: [12.4] }
└── ...
Production edge daemons typically support two encoding formats for batches, and the choice has massive implications for bandwidth:
JSON format:
{
"groups": [
{
"ts": 1709284800,
"device_type": 5000,
"serial_number": 12345,
"values": [
{"id": 2, "values": [72.4]},
{"id": 3, "values": [68.1]}
]
}
]
}
Binary format (same data):
Header: F7 (1 byte - magic)
Groups: 00 00 00 01 (4 bytes - group count)
Group 0: 65 E5 A0 00 (4 bytes - timestamp)
13 88 (2 bytes - device type: 5000)
00 00 30 39 (4 bytes - serial number)
00 00 00 02 (4 bytes - value count)
Value 0: 00 02 (2 bytes - tag id)
00 (1 byte - status: OK)
01 (1 byte - values count)
04 (1 byte - element size: 4 bytes)
42 90 CC CD (4 bytes - float 72.4)
Value 1: 00 03
00
01
04
42 88 33 33 (4 bytes - float 68.1)
The JSON version of this payload: ~120 bytes. The binary version: ~38 bytes. That's a 3.2x reduction — and on a metered cellular connection at $0.01/MB, that savings compounds quickly when you're transmitting every 30 seconds 24/7.
The binary format uses a simple TLV-like structure: magic byte, group count (big-endian uint32), then for each group: timestamp (uint32), device type (uint16), serial number (uint32), value count (uint32), then for each value: tag ID (uint16), status byte, value count, element size, and raw value bytes. No field names, no delimiters, no escaping — just packed binary data.
Batch Finalization Triggers
A batch should be finalized (sealed and queued for delivery) when either condition is met:
-
Size limit exceeded — When the accumulated batch size exceeds a configured maximum (e.g., 500KB for JSON, or when the binary buffer is 90%+ full). The 90% threshold for binary avoids the edge case where the next value would overflow the buffer.
-
Collection timeout expired — When elapsed time since the batch started exceeds a configured maximum (e.g., 60 seconds). This ensures data flows even during quiet periods with few value changes.
if (elapsed_seconds > max_collection_time) → finalize
if (batch_size > max_batch_size) → finalize
Both checks happen after every group is closed (after every polling cycle). This means finalization granularity is tied to your polling interval — if you poll every 1 second and your batch timeout is 60 seconds, each batch will contain roughly 60 groups.
The "Do Not Batch" Exception
Some values are too important to wait for batch finalization. Equipment alarms, pump state changes, emergency stops — these need to reach the cloud immediately. These tags are flagged as "do not batch" in the configuration.
When a do-not-batch tag changes value, it bypasses the normal batch pipeline entirely. A mini-batch is created on the spot — containing just that single value — and pushed directly to the outgoing buffer. This ensures sub-second cloud visibility for critical state changes, while bulk telemetry still benefits from batch efficiency.
Tag: "Pump Status" interval: 1s do_not_batch: true
Tag: "Heater Status" interval: 1s do_not_batch: true
Tag: "Delivery Temp" interval: 60s do_not_batch: false ← normal batching
The Buffer Layer: Surviving Disconnections
This is where most IIoT implementations fail. The batch layer produces data. The MQTT layer consumes it. But what sits between them? If it's just an in-memory queue, you'll lose everything on disconnect.
Page-Based Ring Buffer Architecture
The production-grade answer is a page-based ring buffer — a fixed-size memory region divided into equal-sized pages that cycle through three states:
States:
FREE → Available for writing
WORK → Currently being filled with batch data
USED → Filled, waiting for MQTT delivery
Lifecycle:
FREE → WORK (when first data is added)
WORK → USED (when page is full or batch is finalized)
USED → transmit → delivery ACK → FREE (recycled)
Here's how it works:
Memory layout: At startup, a contiguous block of memory is allocated (e.g., 2MB). This block is divided into pages of a configured size (matching the MQTT max packet size, typically matching the batch size). Each page has a small header tracking its state and a data area.
┌──────────────────────────────────────────────┐
│ [Page 0: USED] [Page 1: USED] [Page 2: WORK]│
│ [Page 3: FREE] [Page 4: FREE] [Page 5: FREE]│
│ [Page 6: FREE] ... [Page N: FREE] │
└──────────────────────────────────────────────┘
Writing data: When a batch is finalized, its serialized bytes are written to the current WORK page. Each message gets a small header: a 4-byte message ID slot (filled later by the MQTT library) and a 4-byte size field. If the current page can't fit the next message, it transitions to USED and a fresh FREE page becomes the new WORK page.
Overflow handling: When all FREE pages are exhausted, the buffer reclaims the oldest USED page — the one that's been waiting for delivery the longest. This means you lose old data rather than new data, which is the right trade-off: the most recent readings are the most valuable. An overflow warning is logged so operators know the buffer is under pressure.
Delivery: When the MQTT connection is active, the buffer walks through USED pages and publishes their contents. Each publish gets a packet ID from the MQTT library. When the broker ACKs the packet (via the PUBACK callback for QoS 1), the corresponding page is recycled to FREE.
Disconnection recovery: When the MQTT connection drops:
- The disconnect callback fires
- The buffer marks itself as disconnected
- Data continues accumulating in pages (WORK → USED)
- When reconnected, the buffer immediately starts draining USED pages
No data is lost unless the buffer physically overflows. With 2MB of buffer and 500KB page size, you get 4 pages of headroom — enough to survive several minutes of disconnection at typical telemetry rates.
Thread Safety
The PLC read loop and the MQTT event loop run on different threads. The buffer must be thread-safe. Every buffer operation acquires a mutex:
buffer_add_data() — called from the PLC read thread after batch finalizationbuffer_process_data_delivered() — called from the MQTT callback thread on PUBACKbuffer_process_connect() / buffer_process_disconnect() — called from MQTT lifecycle callbacks
Without proper locking, you'll see corrupted pages, double-free crashes, and mysterious data loss under load. This is non-negotiable.
Sizing the Buffer
Buffer sizing depends on three variables:
- Data rate: How many bytes per second does your polling loop produce?
- Expected outage duration: How long do you need to survive without MQTT?
- Available memory: Edge devices (especially industrial routers) have limited RAM
Example calculation:
- 200 tags, average 6 bytes each (including binary overhead) = 1,200 bytes/group
- Polling every 1 second = 1,200 bytes/second = 72KB/minute
- Target: survive 30-minute outage = 2.16MB buffer
- With 500KB pages = 5 pages minimum (round up for safety)
In practice, 2–4MB covers most scenarios. On a 32MB industrial router, that's well within budget.
The MQTT Layer: QoS, Reconnection, and Watchdogs
QoS 1: At-Least-Once Delivery
For industrial telemetry, QoS 1 is the right choice:
- QoS 0 (fire and forget): No delivery guarantee. Unacceptable for production data.
- QoS 1 (at least once): Broker ACKs every message. Duplicates possible but data loss prevented. Good trade-off.
- QoS 2 (exactly once): Eliminates duplicates but doubles the handshake overhead. Rarely worth it for telemetry.
The page buffer's recycling logic depends on QoS 1: pages are only freed when the PUBACK arrives. If the ACK never comes (connection drops mid-transmission), the page stays in USED state and will be retransmitted after reconnection.
Connection Watchdog
MQTT connections can enter a zombie state — the TCP socket is open, the MQTT loop is running, but no data is actually flowing. This happens when network routing changes, firewalls silently drop the connection, or the broker becomes unresponsive.
The fix: a watchdog timer that monitors delivery acknowledgments. If no PUBACK has been received within a timeout window (e.g., 120 seconds) and data has been queued for transmission, force a reconnect:
if (now - last_delivered_packet_time > 120s) {
if (has_pending_data) {
// Force MQTT reconnection
reset_mqtt_client();
}
}
This catches the edge case where the MQTT library thinks it's connected but the network is actually dead. Without this watchdog, your edge daemon could silently accumulate hours of undelivered data in the buffer, eventually overflowing and losing it all.
Asynchronous Connection
MQTT connection establishment (DNS resolution, TLS handshake, CONNACK) can take several seconds, especially over cellular links. This must not block the PLC read loop. The connection should happen on a separate thread:
- Main thread detects connection is needed
- Connection thread starts
connect_async()
- Main thread continues reading PLCs
- On successful connect, the callback fires and buffer delivery begins
If the connection thread is still working when a new connection attempt is needed, skip it — don't queue multiple connection attempts or you'll thrash the network stack.
TLS for Production
Any MQTT connection leaving your plant network must use TLS. Period. Industrial telemetry data — temperatures, pressures, equipment states, alarm conditions — is operationally sensitive. On the wire without encryption, anyone on the network path can see (and potentially modify) your readings.
For cloud brokers like Azure IoT Hub, TLS is mandatory. The edge daemon should:
- Load the CA certificate from a PEM file
- Use MQTT v3.1.1 protocol (widely supported, well-tested)
- Monitor the SAS token expiration timestamp and alert before it expires
- Automatically reinitialize the MQTT client when the certificate or connection string changes (file modification detected via
stat())
Daemon Status Reporting
A well-designed edge daemon reports its own health back through the same MQTT channel it uses for telemetry. A periodic status message should include:
- System uptime and daemon uptime — detect restarts
- PLC link state — is the PLC connection healthy?
- Buffer state — how full is the outgoing buffer?
- MQTT state — connected/disconnected, last ACK time
- SAS token expiration — days until credentials expire
- Software version — for remote fleet management
An extended status format can include per-tag state: last read time, last delivery time, current value, and error count. This is invaluable for remote troubleshooting — you can see from the cloud exactly which tags are stale and why.
Value Comparison and Change Detection
Not all values need to be sent every polling cycle. A temperature that's been 72.4°F for the last hour doesn't need to be transmitted 3,600 times. Change detection — comparing the current value to the last sent value — can dramatically reduce bandwidth.
The implementation: each tag stores its last transmitted value. After reading, compare:
if (tag.compare_enabled && tag.has_been_read_once) {
if (current_value == tag.last_value) {
skip_this_value(); // Don't add to batch
}
}
Important caveats:
- Not all tags should use comparison. Continuous process variables (temperatures, flows) should always send, even if unchanged — the recipient needs the full time series to calculate trends and detect flatlines (a stuck sensor reads the same value forever, which is itself a fault condition).
- Discrete state tags (booleans, enums) are ideal for comparison — they change rarely and each change is significant.
- Floating-point comparison should use an epsilon threshold, not exact equality, to avoid sending noise from ADC jitter.
Putting It All Together: The Main Loop
The complete edge daemon main loop ties all these layers together:
1. Parse configuration (device addresses, tag lists, MQTT credentials)
2. Allocate memory (PLC config pool + output buffer)
3. Format output buffer into pages
4. Start MQTT connection thread
5. Detect PLC device (probe address, determine type/protocol)
6. Load device-specific tag configuration
MAIN LOOP (runs every 1 second):
a. Check for config file changes → restart if changed
b. Read PLC tags (coalesced Modbus/EtherNet/IP)
c. Add values to batch (with comparison filtering)
d. Check batch finalization triggers (size/timeout)
e. Process incoming commands (config updates, force reads)
f. Check MQTT connection watchdog
g. Sleep 1 second
Every component — polling, batching, buffering, delivery — operates within this single loop iteration, keeping the system deterministic and debuggable.
How machineCDN Implements This
The machineCDN edge runtime implements this full stack natively on resource-constrained industrial routers. The page-based ring buffer runs in pre-allocated memory (no dynamic allocation after startup), the MQTT layer handles Azure IoT Hub and local broker configurations interchangeably, and the batch layer supports both JSON and binary encoding selectable per-device.
On a Teltonika RUT9xx router with 256MB RAM, the daemon typically uses under 4MB total — including 2MB of buffer space that can store 20+ minutes of telemetry during a connectivity outage. Tags are automatically sorted, coalesced, and dispatched with zero configuration beyond listing the tag names and addresses.
The result: edge gateways that have been running continuously for years in production environments, surviving cellular dropouts, network reconfigurations, and even firmware updates without losing a single data point.
Conclusion
Reliable telemetry delivery isn't about the protocol — it's about the pipeline. Modbus reads are the easy part. The hard engineering is in the layers between: batching values efficiently, buffering them through disconnections, and confirming delivery before recycling memory.
The key design principles:
- Never block the read loop — PLC polling is sacred
- Buffer with finite, pre-allocated memory — dynamic allocation on embedded systems is asking for trouble
- Reclaim oldest data first — in overflow, recent values matter more
- Acknowledge before recycling — a page stays USED until the broker confirms receipt
- Watch for zombie connections — a connected socket doesn't mean data is flowing
Get these right, and your edge infrastructure becomes invisible — which is exactly what production IIoT should be.
