One post tagged with "buffer"

Here's the scenario every IIoT engineer dreads: your edge gateway is collecting temperature, pressure, and vibration data from 200 tags across 15 PLCs. The cellular modem on the factory roof drops its connection — maybe for 30 seconds during a handover, maybe for 4 hours because a backhoe hit a fiber line. When connectivity returns, what happens to the data?

If your answer is "it's gone," you have a buffer management problem. And fixing it properly requires understanding paged ring buffers — the unsung hero of reliable industrial telemetry.

Why Naive Buffering Fails​

The simplest approach — queue MQTT messages in memory and retry on reconnect — has three fatal flaws:

1. Memory exhaustion: A gateway reading 200 tags at 1-second intervals generates ~12,000 readings per minute. At ~100 bytes per JSON reading, that's 1.2 MB/minute. A 4-hour outage accumulates ~288 MB. Your 256 MB embedded gateway just died.

2. No delivery confirmation: MQTT QoS 1 guarantees "at least once" delivery, but the Mosquitto client library's in-flight message queue is finite. If you publish 50,000 messages into a disconnected client, most will be silently dropped by the client library's internal buffer long before the broker sees them.

3. Thundering herd on reconnect: When connectivity returns, dumping 288 MB of queued messages simultaneously will choke the cellular uplink (typically 1–5 Mbps), cause broker-side backpressure, and likely trigger another disconnect.

The Paged Ring Buffer Architecture​

The solution is a fixed-size, page-based circular buffer that sits between the data collection layer and the MQTT client. Here's how it works:

Memory Layout​

The buffer is allocated as a single contiguous block — typically 2 MB on an embedded gateway. This block is divided into equal-sized pages, where each page can hold one complete MQTT payload.

┌─────────────────────────────────────────────────┐
│                 2 MB Buffer Memory              │
├────────┬────────┬────────┬────────┬────────┬────┤
│ Page 0 │ Page 1 │ Page 2 │ Page 3 │ Page 4 │ ...│
│ 4 KB   │ 4 KB   │ 4 KB   │ 4 KB   │ 4 KB   │    │
└────────┴────────┴────────┴────────┴────────┴────┘

With a 4 KB page size and 2 MB total buffer, you get approximately 500 pages. Each page holds multiple MQTT messages packed sequentially.

Page States​

Every page exists in exactly one of three states:

• Free: Available for new data. Part of a singly-linked free list.
• Work: Currently being filled with incoming data. Only one work page exists at a time.
• Used: Full of data, waiting to be transmitted. Part of a singly-linked FIFO queue.

Free Pages → [P5] → [P6] → [P7] → null
Work Page  → [P3] (currently filling)
Used Pages → [P0] → [P1] → [P2] → null
              ↑ sending        waiting →

The Write Path​

When a batch of PLC tag values arrives from the data collection layer:

1. Check the work page: If there's no current work page, pop one from the free list. If the free list is empty, steal the oldest used page (overflow — we're losing old data to make room for new data, which is the correct trade-off for operational monitoring).

2. Calculate fit: Each message is packed as: [4-byte message ID] [4-byte message size] [message payload]. Check if the current work page has enough remaining space for this overhead plus the payload.

3. If it fits: Write the message ID (initially zero — will be filled by the MQTT client), the size, and the payload. Advance the write pointer.

4. If it doesn't fit: Move the current work page to the tail of the used queue. Pop a new page from the free list (or steal from used queue). Write into the new page.

Page Internal Layout:
┌──────────┬──────────┬─────────────┬──────────┬──────────┬─────────────┐
│ msg_id_1 │ msg_sz_1 │ payload_1   │ msg_id_2 │ msg_sz_2 │ payload_2   │
│ (4 bytes) │ (4 bytes) │ (N bytes)  │ (4 bytes) │ (4 bytes) │ (M bytes)  │
└──────────┴──────────┴─────────────┴──────────┴──────────┴─────────────┘
                                     ↑ write_p (current position)

The Send Path​

The MQTT send logic runs after every write operation and follows strict rules:

1. Check prerequisites: Connection must be up (connected == 1) AND no packet currently in-flight (packet_sent == 0). If either fails, do nothing — the data is safely buffered.

2. Select the send source: If there are used pages, send from the first one in the FIFO. If no used pages exist but the work page has data, promote the work page to used and send from it.

3. Read the next message from the current page's read pointer: extract the size, get the data pointer, and call mosquitto_publish() with QoS 1.

4. Mark packet as in-flight: Set packet_sent = 1. This is critical — only one message can be in-flight at a time. This prevents the thundering herd problem and ensures ordered delivery.

5. Wait for acknowledgment: The MQTT client library calls the publish callback when the broker confirms receipt (PUBACK for QoS 1). Only then do we advance the read pointer and send the next message.

The Acknowledgment Path​

When the Mosquitto library fires the on_publish callback with a packet ID:

1. Verify the ID matches the in-flight message on the current used page
2. Advance the read pointer past the delivered message (skip message ID + size + payload bytes)
3. Check if page is fully delivered: If read_p >= write_p, move the page back to the free list
4. Clear the in-flight flag: Set packet_sent = 0
5. Immediately attempt to send the next message — this creates a natural flow control where messages are delivered as fast as the broker can acknowledge them

Delivery Flow:
                     publish()
  [Used Page] ──────────────────→ [MQTT Broker]
       ↑                               │
       │              PUBACK            │
       └────────────────────────────────┘
       advance read_p, try next

Thread Safety: The Mutex Dance​

In a real gateway, data collection and MQTT delivery run on different threads. The PLC polling loop writes data every second, while the Mosquitto client library fires callbacks from its own network thread. Every buffer operation — add, send, acknowledge, connect, disconnect — must be wrapped in a mutex:

// Data collection thread:
mutex_lock(buffer)
  add_data(payload)
  try_send_next()  // opportunistic send
mutex_unlock(buffer)

// MQTT callback thread:
mutex_lock(buffer)
  mark_delivered(packet_id)
  try_send_next()  // chain next send
mutex_unlock(buffer)

The key insight is that try_send_next() is called from both threads — after every write (in case we're connected and idle) and after every acknowledgment (to chain the next message). This ensures maximum throughput without busy-waiting.

Handling Disconnects Gracefully​

When the MQTT connection drops, two things happen:

1. The disconnect callback fires: Set connected = 0 and packet_sent = 0. The in-flight message is NOT lost — it's still in the page at the current read pointer. When connectivity returns, it will be re-sent.

2. Data keeps flowing in: The PLC polling loop doesn't stop. New data continues to fill pages. The used queue grows. If it fills all available pages, new pages will steal from the oldest used pages — but this only happens under extreme sustained outages.

When the connection re-establishes:

1. The connect callback fires: Set connected = 1 and trigger try_send_next()
2. Buffered data starts flowing: Messages are delivered in FIFO order, one at a time, with acknowledgment pacing

This means the broker receives data in chronological order, with timestamps embedded in each batch. Analytics systems downstream can seamlessly handle the gap — they see a burst of historical data followed by real-time data, all correctly timestamped.

The Cloud Watchdog: Detecting Silent Failures​

There's a subtle failure mode: the MQTT connection appears healthy (no disconnect callback), but data isn't actually being delivered. This can happen with certain TLS middlebox issues, stale TCP connections that haven't timed out, or Azure IoT Hub token expirations.

The solution is a delivery watchdog:

1. Track the timestamp of the last successful packet delivery
2. On a periodic check (every 120 seconds), compare the current time against the last delivery timestamp
3. If no data has been delivered in 120 seconds AND the connection claims to be up, force a reconnection:
   • Reset the MQTT configuration timestamp (triggers config reload)
   • Clear the watchdog timer
   • The main loop will detect the stale configuration and restart the MQTT client

if (now - last_delivery_time > 120s) AND (connected) {
    log("No data delivered in 120s — forcing MQTT reconnect")
    force_mqtt_restart()
}

This catches the "zombie connection" problem that plagues many IIoT deployments — the gateway thinks it's sending, but nothing is actually arriving at the cloud.

Binary vs. JSON: The Bandwidth Trade-off​

The paged buffer doesn't care about the payload format — it stores raw bytes. But the choice between JSON and binary encoding has massive implications for buffer utilization:

JSON payload for one tag reading:

{"id":42,"values":[23.7],"ts":1709337600}

~45 bytes per reading.

Binary payload for the same reading:

Tag ID:    2 bytes (uint16)
Status:    1 byte
Value Cnt: 1 byte  
Value Sz:  1 byte
Value:     4 bytes (float32)
─────────────────────
Total:     9 bytes per reading

That's a 5x reduction. With batching (multiple readings per batch header), the per-reading overhead drops further because the timestamp and device identity are shared across a group of values.

On a cellular connection billing per megabyte, this isn't academic — it's the difference between $15/month and $75/month per gateway. On satellite connections (Iridium, Starlink maritime), it can be $50 vs. $250.

Binary Batch Wire Format​

A binary batch on the wire follows this structure:

[0xF7]                          — 1 byte, magic/version marker
[num_groups]                    — 4 bytes, big-endian uint32
For each group:
  [timestamp]                   — 4 bytes, big-endian time_t
  [device_type]                 — 2 bytes, big-endian uint16
  [serial_number]               — 4 bytes, big-endian uint32
  [num_values]                  — 4 bytes, big-endian uint32
  For each value:
    [tag_id]                    — 2 bytes, big-endian uint16
    [status]                    — 1 byte (0 = OK, else error code)
    If status == 0:
      [values_count]            — 1 byte
      [value_size]              — 1 byte (1, 2, or 4)
      [values...]               — values_count × value_size bytes

A batch of 50 tag readings fits in ~600 bytes binary versus ~3,000 bytes JSON. Over a 4-hour outage with 200 tags at 60-second intervals, that's the difference between buffering ~4.8 MB (binary) versus ~24 MB (JSON) — within or far exceeding a typical gateway's buffer.

Sizing Your Buffer: The Math​

For a given deployment, calculate your buffer needs:

Tags: 200
Read interval: 60 seconds
Binary payload per reading: ~9 bytes
Readings per minute: 200
Bytes per minute: 200 × 9 = 1,800 bytes
With batch overhead (~15 bytes per group): ~1,815 bytes/min

Buffer size: 2 MB = 2,097,152 bytes
Retention: 2,097,152 / 1,815 = ~1,155 minutes = ~19.2 hours

So a 2 MB buffer can hold approximately 19 hours of data for 200 tags at 60-second intervals using binary encoding. With JSON, that drops to ~3.8 hours. Size your buffer accordingly.

What machineCDN Does Differently​

machineCDN's edge gateway implements this paged ring buffer architecture natively. Every gateway shipped includes:

• Fixed 2 MB paged buffer with configurable page sizes matching the MQTT broker's maximum packet size
• Automatic binary encoding for all telemetry — 5x bandwidth reduction over JSON
• Single-message flow control with QoS 1 acknowledgment tracking — no thundering herd on reconnect
• 120-second delivery watchdog that detects zombie connections and forces reconnect
• Graceful overflow handling — when buffer fills, oldest data is recycled (not newest), preserving the most recent operational state

For plant engineers, this means deploying a gateway on a cellular connection and knowing that a connectivity outage — whether 30 seconds or 12 hours — won't result in lost data. The buffer holds, the watchdog monitors, and data flows in order when the link comes back.

Key Takeaways​

1. Never use unbounded queues for industrial telemetry buffering — use fixed-size paged buffers that degrade gracefully under memory pressure
2. One message in-flight at a time prevents the thundering herd problem and ensures ordered delivery
3. Always track delivery acknowledgments — don't just publish and forget; verify the broker received each packet before advancing
4. Implement a delivery watchdog — silent MQTT failures are harder to detect than disconnects
5. Use binary encoding — 5x bandwidth reduction means 5x longer buffer retention on the same memory
6. Size for your worst outage — calculate how much buffer you need based on tag count, interval, and the longest connectivity gap you expect
7. Thread safety is non-negotiable — data collection and MQTT delivery run concurrently; every buffer operation needs mutex protection

The paged ring buffer isn't exotic computer science — it's a practical engineering pattern that's been battle-tested in thousands of industrial deployments. The difference between a prototype IIoT system and a production one often comes down to exactly this kind of infrastructure.