Backpressure

In high-throughput systems like TBMQ, backpressure handling is essential for preventing out-of-memory errors and maintaining stability under load. Backpressure must be managed in two directions: inbound (publishers to the broker) and outbound (broker to subscribers).

TBMQ handles virtually unlimited publisher load by immediately persisting incoming messages to Kafka before any further processing. This decouples ingestion from delivery and lets the broker scale horizontally to meet demand. Two complementary mechanisms govern inbound flow control: TCP-level backpressure and rate limiting.

TBMQ uses Netty for network I/O, and the socket receive buffer size can be tuned via the following configuration:

listener:
  # Netty socket receive buffer size (in KB). When the buffer limit is reached, TCP triggers backpressure
  # and notifies the sender to slow down.
  # Defaults to 0, which uses the system's default buffer size.
  so_receive_buffer: "${NETTY_SO_RECEIVE_BUFFER:0}"

The value is expressed in kilobytes and applies globally to all MQTT listeners.

When the receive buffer fills up, TCP signals the sender to slow down, preventing the broker from being overwhelmed at the network layer.

Unlike TCP-level backpressure, which is reactive, rate limiting is a proactive control layer. It enforces traffic constraints before the system becomes overloaded, preventing individual publishers from overwhelming the broker and maintaining fairness across clients.

The cluster-wide quota is sized by your TBMQ subscription plan — the Throughput (msg/sec) tier you purchase is the budget. It is always enforced and cannot be enabled, disabled, or enlarged from the configuration; the total section only tunes how each node draws from that shared budget. Per-client publish rate limits are configured separately:

rate-limits:
  total:
    # Tokens a node draws from Redis per round trip. 0 derives max(1, sustainedRate / 10).
    block-size: "${MQTT_TOTAL_RATE_LIMITS_BLOCK_SIZE:0}"
    # Period of the tick returning a node's unused drawn tokens to the shared bucket (in ms).
    lease-return-ms: "${MQTT_TOTAL_RATE_LIMITS_LEASE_RETURN_MS:1000}"
  incoming-publish:
    # Enables or disables publish rate limits per client for incoming messages to the broker.
    enabled: "${MQTT_INCOMING_RATE_LIMITS_ENABLED:false}"
    # Limits the count of publish messages per publisher per time interval (in s).
    client-config: "${MQTT_INCOMING_RATE_LIMITS_CLIENT_CONFIG:10:1,300:60}"

When the quota refuses a message, TBMQ also raises a licensed throughput reached indicator in the license usage notification, so an operator can tell a licensing ceiling apart from an ordinary rate limit. It clears when acknowledged, and otherwise expires five minutes after it was first raised.

Per-client limits are configured as a comma-separated list of limit:seconds pairs, and all pairs must be satisfied. The default 10:1,300:60 means at most 10 messages in any second and at most 300 in any minute — so a client cannot burst at 10 msg/sec for a whole minute.

Per-client rate limits are disabled by default. When an incoming limit is hit, the PUBLISH is dropped, the droppedMsgs counter is incremented, and the broker’s response to the publisher depends on the protocol version:

The cluster-wide quota counts MQTT PUBLISH packets — incoming and outgoing combined — against a single budget shared by every node in the cluster. Packets are charged at admission: when the publish is processed, before it is stored. A publish costs

• 1 packet for the incoming message itself, plus
• 1 packet for every persistent subscriber, shared subscription group, and integration it is routed to.

The bill is therefore 1 + fanout, computed at publish time and independent of whether those subscribers happen to be online. Once a message has been charged and stored, delivering it costs nothing more: a persistent session that reconnects after being offline receives its whole backlog however large, and QoS 1/2 retransmissions are free.

Two kinds of packet fall outside that one charge. Delivery to a non-persistent subscriber is charged as it is sent, because nothing was stored — for those clients, delivery is admission. A retained message sent to a client that subscribes later is charged on SUBSCRIBE, since it is a new outgoing packet that nothing has paid for yet.

When the budget runs out, the outcome depends on which packet is being charged:

Because refusals happen at admission, the quota only ever sheds work the broker has not yet acknowledged; a message that is already durable is never discarded to save budget.

The grace is fixed — long enough to ride out a Redis Cluster or Sentinel failover, and deliberately not configurable: the quota is what enforces your licensed throughput, so a setting able to lengthen the grace could suspend that limit for as long as one chose, simply by taking Redis away.

Refusals past the grace do not raise the licensed throughput reached indicator: the budget is unknown rather than spent. During a Redis outage the only signals are droppedMsgs and throughputQuotaDegraded_total.

Two parameters govern how each node draws from the shared budget, and rarely need changing:

• block-size — how many tokens a node draws from Redis per round trip. Defaults to 0, which derives max(1, sustainedRate / 10) from the sustained rate. Larger blocks mean fewer Redis calls but let an idle node hold more of the cluster budget between lease returns. A publish whose fan-out exceeds the node’s drawn tokens draws the shortfall on demand, so this value never caps a single publish.
• lease-return-ms — how often a node returns unused drawn tokens to the shared bucket, in milliseconds. Defaults to 1000. Setting it to 0 disables the tick, and a node that draws a block and then goes quiet holds those tokens indefinitely — the cluster’s effective budget shrinks by the sum of every idle node’s lease. Leave the tick on unless traffic is even across all nodes.

The outbound channel buffer may become overwhelmed when a subscriber cannot consume messages as fast as the broker delivers them. TBMQ detects non-writable Netty channels and pauses delivery to affected clients, resuming automatically once the channel becomes writable again.

TBMQ monitors channel writability via the channelWritabilityChanged event. Write buffer watermarks define the thresholds:

• High Watermark: When the write buffer exceeds this threshold, the channel is marked non-writable and message delivery is paused.
• Low Watermark: When the write buffer drops below this threshold, the channel is marked writable again and delivery resumes.

listener:
  # Threshold (in KB) at which Netty considers the channel non-writable.
  write_buffer_high_water_mark: "${NETTY_WRITE_BUFFER_HIGH_WATER_MARK:1280}"
  # Threshold (in KB) at which Netty considers the channel writable again.
  write_buffer_low_water_mark: "${NETTY_WRITE_BUFFER_LOW_WATER_MARK:640}"

Both values are in kilobytes and apply globally to all MQTT listeners. They bound bytes queued in the outbound channel, not a message count — with the default 1280 KB high watermark, a subscriber receiving 1 KB payloads goes non-writable after roughly 1,280 unsent messages have accumulated, while one receiving 100 KB payloads goes non-writable after about 13. Size the watermarks against your payload size and how much memory per slow subscriber you are willing to spend.

When a non-persistent client’s channel is non-writable, the broker skips delivery of that message. Dropped messages are not retained or retried; the droppedMsgs counter records them so you can see backpressure losses under load. Dropped retained messages are deliberately not counted, since a client can still recover them from the retained-message store. This approach avoids memory buildup for short-lived or unreliable clients that are not expected to maintain state.

Non-persistent sessions are skipped per message rather than paused: they never transition into the internal CHANNEL_NON_WRITABLE session state, because there is nothing to resume from. They do, however, still count toward the nonWritableClients gauge while their channel is over the high watermark.

For persistent clients, a non-writable channel pauses delivery instead of discarding messages. The pause itself never loses anything: the message is already in durable storage before the broker attempts to write it to the socket, so “skip the delivery” means “leave it where it is” rather than “drop it”. Messages are only lost if storage limits are exceeded while the client stays behind — the bounds described below.

This protection covers only what TBMQ stores, which means QoS 1 and 2 publishes delivered to a subscription of QoS 1 or 2. A message published at QoS 0 is never stored, even for a persistent session, so it takes the non-persistent path above and is skipped and counted while the channel is congested. The same applies to a persistent client that subscribed at QoS 0.

Device clients use Redis as the buffer.

The order of operations is what makes the pause safe. A Kafka consumer reads the tbmq.msg.persisted topic and writes the batch into Redis first, then hands the messages to the client’s device actor for delivery. Redis is therefore the authoritative copy at all times, not a fallback that gets populated once trouble starts.

What happens across a backpressure episode:

1. Channel goes non-writable. The client’s device actor records that its channel is congested and stops passing messages to it. Messages that keep arriving for this client are still persisted to Redis as usual; they are simply not written to the socket, so the outbound buffer is given a chance to drain.
2. Channel becomes writable again. The broker does not immediately resume at full rate. The actor first checks how many messages it has sent but not yet had acknowledged. While that count is above zero it reschedules itself using an exponential backoff — roughly 1, 2, 3, 5, 9 and 10 seconds, each with up to a second of jitter, so about half a minute of grace in total — giving the client time to work through its PUBACK/PUBREC backlog instead of being flooded the instant its buffer clears.
3. Delivery resumes from Redis. Once the in-flight count reaches zero, or the retry budget is exhausted, the actor re-reads the client’s queue from Redis and continues from there. Recovery is driven by durable state, so it is unaffected by whatever was in memory when congestion began — and it works the same way whether the client was merely slow or fully disconnected.
4. Expired messages are discarded on the way out. A message whose expiry has passed while it waited is skipped at delivery time rather than sent late.

The two bounds on how long a client can stay behind:

• Each client has a per-client message queue bounded by MQTT_PERSISTENT_SESSION_DEVICE_PERSISTED_MESSAGES_LIMIT (default: 10,000). When the queue exceeds the limit, TBMQ trims the oldest messages first — so a client that stays congested past the limit keeps the newest messages and silently loses the earliest ones. The limit cannot exceed 65535 — the broker fails to start with a larger value.
• Messages in the queue have a configurable TTL controlled by MQTT_PERSISTENT_SESSION_DEVICE_PERSISTED_MESSAGES_TTL (default: 604800 s — seven days), applied per message.
• For MQTT 5.0 publishers, the message’s Message Expiry Interval property overrides that default TTL.

Application clients use Kafka consumer pause.

Here the mechanism is simpler, because each persistent APPLICATION client already has its own Kafka topic and its own consumer. Instead of buffering anywhere, the broker just stops reading:

• When the channel becomes non-writable, the broker pauses that client’s Kafka consumer. Messages continue to be produced into the client’s topic; they simply accumulate there, unread. Nothing is buffered in broker memory, so a slow APPLICATION client costs disk on the Kafka side rather than heap on the broker.
• When the channel becomes writable again, the broker resumes the consumer, which picks up from its committed offset — so delivery continues exactly where it left off, with no gap and no re-reading of already-acknowledged messages.
• The bound here is Kafka retention rather than a message count, configured via TB_KAFKA_APP_PERSISTED_MSG_TOPIC_PROPERTIES. Defaults: retention.ms=604800000 — seven days, retention.bytes=1048576000 — 1 GB. A client that stays paused longer than retention allows loses the oldest unread messages, the same failure mode as the DEVICE queue trim.

This is why APPLICATION clients absorb backpressure far better than DEVICE clients: the backlog lives in Kafka, sized in gigabytes, instead of a per-client Redis queue capped at 65535 messages.

See shared subscriptions for a full overview of shared subscriptions in TBMQ.

For non-persistent and persistent DEVICE groups, TBMQ picks the target in two steps, and only the first one considers writability: it checks whether the group has any usable member — a connected subscriber that is either on another cluster node or, if local, has a writable channel — and then round-robin picks the next connected member. Because that second step ignores writability, a congested member can still be picked.

If no usable member exists — every subscriber disconnected, or every local one non-writable — the message is dropped. If round-robin lands on a congested member, that copy is dropped and counted in droppedMsgs too, even when another member of the group was writable.

Landing on a congested member costs nothing here: the message goes to that subscriber’s device actor, which keeps it in Redis and delivers it once the channel recovers. If no usable member exists at all, the message is saved to a per-group Redis queue and delivered to whichever member becomes available first. The same configuration parameters apply as for individual persistent DEVICE clients (MQTT_PERSISTENT_SESSION_DEVICE_PERSISTED_MESSAGES_LIMIT and MQTT_PERSISTENT_SESSION_DEVICE_PERSISTED_MESSAGES_TTL).

Here selection is Kafka’s, not TBMQ’s: the group is a real Kafka consumer group on the shared topic. When a member’s channel becomes non-writable, TBMQ pauses that member’s consumer. It stays in the group and keeps its partition assignment, so its share of the messages waits in the topic rather than being handed to the other members. Retention for shared APPLICATION topics is configured via TB_KAFKA_APP_PERSISTED_MSG_SHARED_TOPIC_PROPERTIES.

How much of the group a congested member holds up therefore depends on the Partitions count you set on the Application shared subscription when you create it: Kafka gives each member a subset of the partitions, and only that member’s own subset waits while it is paused. The count cannot be changed afterwards.

Writability monitoring reacts to a channel that is already congested. Two proactive limits cap outbound and persistence load before that happens:

rate-limits:
  outgoing-publish:
    # Enables or disables publish rate limits per client for outgoing messages to subscribers.
    # Applies only to non-persistent subscribers with QoS = 0 ("AT_MOST_ONCE").
    enabled: "${MQTT_OUTGOING_RATE_LIMITS_ENABLED:false}"
    # Limits the count of publish messages per subscriber per time interval (in s).
    client-config: "${MQTT_OUTGOING_RATE_LIMITS_CLIENT_CONFIG:10:1,300:60}"
  device-persisted-messages:
    # Enables or disables Device clients persisted messages rate limits for the broker (per whole cluster).
    enabled: "${MQTT_DEVICE_PERSISTED_MSGS_RATE_LIMITS_ENABLED:false}"
    # Limits the count of Device clients persisted messages per time interval (in s).
    config: "${MQTT_DEVICE_PERSISTED_MSGS_RATE_LIMITS_CONFIG:100:1,1000:60}"

outgoing-publish is a per-subscriber limit that applies only to messages published at QoS 0, the ones TBMQ never stores. Messages over the limit are dropped and counted in droppedMsgs. QoS 1 and 2 publishes are unaffected, since dropping them would violate the delivery guarantee — but a persistent subscriber is not exempt: its QoS 0 traffic goes through the same non-persistent path and is subject to this limit.

device-persisted-messages is a cluster-wide limit on how many messages TBMQ writes into persistent DEVICE storage per interval, sharing one Redis-backed bucket across all nodes. When a publish matches more persistent DEVICE subscribers than the bucket has tokens left, the surplus is dropped and counted. Use it to protect Redis when a burst fans out to many persistent DEVICE subscribers.

Both are disabled by default.

• Monitor the nonWritableClients gauge on /actuator/prometheus to detect backpressure conditions early, and watch droppedMsgs to quantify what backpressure and rate limits are actually costing you.
• Size the watermarks against your payload size (see above) rather than a target message rate, and change them only after load testing — raising them trades broker memory for tolerance of slow subscribers.
• Ensure sufficient Redis capacity for persistent DEVICE client queues: worst case is number of persistent DEVICE clients × MQTT_PERSISTENT_SESSION_DEVICE_PERSISTED_MESSAGES_LIMIT × average message size.
• Ensure sufficient Kafka capacity and appropriate retention settings for APPLICATION client topics — retention, not the session, is what bounds how long a disconnected APPLICATION client can fall behind before losing messages.
• Enable the rate limits that match your risk: incoming-publish against a misbehaving publisher, device-persisted-messages against Redis saturation.
• Use horizontal scaling to distribute load across multiple TBMQ nodes.
• Test your deployment under realistic peak load before going to production.

When a drop counter does move, the Dropped messages page breaks the total down by reason, client, and topic, so you can tell which limit fired and for which client.