TBMQ cluster supports 100M MQTT connections

A core job of an MQTT broker is to receive published messages, filter them by topic, and distribute them to subscribers — work that becomes critical under substantial workloads. This article demonstrates how a TBMQ cluster reliably handles 100 million connected clients while sustaining 6M MQTT PUBLISH packets per second across both directions.

The test environment was deployed on AWS. A TBMQ cluster of 25 nodes ran on an EKS cluster (one broker pod per EC2 instance), connected to RDS and Kafka. RDS was a single instance; Kafka consisted of 9 brokers across 3 Availability Zones. For a detailed overview of the broker’s internal design, see the TBMQ architecture page.

The test emulated smart tracker devices sending 5-field messages of approximately 114 bytes:

{ "lat": 40.761894, "long": -73.970455, "speed": 55.5, "fuel": 92, "batLvl": 81 }

Publishers: 500 groups of 200k publishers each — 100M publishers in total, producing 6k msg/sec per group. Each publishes to its own topic in the format CountryCode/RandomString/GroupId/ClientId, creating 100M unique topics.

Subscribers: 500 APPLICATION subscriber groups, each with a single persistent subscriber using the topic filter CountryCode/RandomString/GroupId/+, receiving 6k msg/sec.

The TBMQ cluster sustained 100,000,500 connections and handled 6M messages per second (3M incoming × 2 for publish + delivery). Over the 1-hour test, 3M incoming msg/sec added up to 10,800M messages published.

Including a warm-up phase (6 iterations × 100M publishers × 1 message each = 600M warm-up messages in ~7 min), the grand total was 11,400M messages — approximately 1TB stored in the tbmq.msg.all Kafka topic.

The test agent orchestrates the provisioning of MQTT clients, allowing flexible configuration of their count. Publisher clients stay connected for the whole run, continuously publishing time series data over MQTT to designated topics. The agent also provisions subscriber clients that subscribe by topic filter to receive the published messages.

Each of the 500 APPLICATION subscribers received 22.8M messages in its dedicated per-client Kafka topic. Because every subscriber was an APPLICATION client, Kafka was the only persistence layer in the message path and PostgreSQL handled just a few operations per second.

Tip: to plan and manage Kafka disk space, adjust the size retention policy and period retention policy. For details on topic-level configuration, see the configuration reference.

Latency:

Where:

• Msg latency avg: average time from publisher transmission to subscriber receipt
• Pub ACK avg: average time from publisher transmission to PUBACK receipt
• 95th: 95th percentile

The tbmq.client.session Kafka topic confirms all 100M+ client sessions were established:

Kafka topic statistics (100% message delivery success):

Publish msg topic — received all 11,400M messages Application topic example 1 — 22.8M messages Application topic example 2 — 22.8M messages

All topics show 100% message delivery success. Topic data is compressed — producers send compressed data and Kafka brokers retain the original compression codec (compression.type property), ensuring efficient storage and transmission.

Grafana monitoring:

AWS instance monitoring (TBMQ ~45% avg CPU, PostgreSQL minimal, Kafka has headroom for more load):

AWS EC2 TBMQ monitoring AWS EC2 Kafka monitoring AWS EBS Kafka monitoring AWS RDS monitoring

JVM monitoring (forward the JMX port then open VisualVM):

kubectl port-forward tb-broker-0 9999:9999

• The TBMQ cluster had capacity to process even higher loads.
• Kafka provides reliable, highly available message processing.
• No direct communication between TBMQ nodes enables horizontal scaling.
• Using QoS 0 would push the message rate higher still, but QoS 1 was chosen to demonstrate a more practical setup. QoS 1 is widely favored as it balances message delivery speed and reliability.
• TBMQ excels at fan-in, P2P, and fan-out scenarios at both small and large scale.

Kafka producer disconnects — caused message loss. Resolved by implementing a dedicated executor service for processing publish callbacks.

Publishing queue overhead — eliminated by using the thread-safe nature of Kafka producers directly, with message ordering achieved another way.

Throughput tuning — additional improvements to message pack processing and UUID generation and sending messages without explicit flushing.

Memory optimization — reduced unnecessary object creation, improving Garbage Collector performance and reducing stop-the-world pauses: ClientSessionInfo reuse, application publish msg copies, Bytebuf in publish msg.

Uneven client distribution — resolved by implementing even distribution of clients among broker nodes.

Approximate monthly cost for this setup on AWS us-east-1 (prices are illustrative — consult AWS for accurate pricing; Savings Plans (up to 72% discount), RDS Reserved Instances (up to 69%), and MSK Tiered Storage (50% or more) can reduce costs significantly):

~$0.0003 per device per month.

Load configuration:

• 100M publish MQTT clients (smart tracker devices)
• 500 persistent subscribe MQTT clients (APPLICATION consumers for analysis)
• 6M msg/sec throughput over MQTT; each message contains five data points, message size is 114 bytes
• PostgreSQL database to store MQTT client credentials and client session states
• Kafka queue to persist messages

At this message rate and size, the entry-point tbmq.msg.all Kafka topic receives approximately ~1 TB of data per hour, and each subscriber topic receives around ~1.6 GB per hour. It is not necessary to store data for an extended period — TBMQ distributes messages among subscribers and optionally stores them temporarily for offline clients. Configure an appropriate storage size and adjust the size and period retention policies for Kafka topics based on your requirements.

The test agent consisted of 2,000 publisher and 500 subscriber Kubernetes pods, with a single pod as the orchestrator.

Refer to the AWS cluster installation guide to deploy TBMQ on AWS.

• TBMQ branch with test scripts
• Performance testing tool
• Publisher configuration
• Subscriber configuration

Publisher group configuration:

{
    "id": 1,
    "publishers": 200000,
    "topicPrefix": "usa/ydwvv/1/",
    "clientIdPrefix": null
}

• id — identifier of the publisher group.
• publishers — number of publisher clients in the group.
• topicPrefix — topic prefix to which messages are published.
• clientIdPrefix — client ID prefix for publishers.

Subscriber group configuration:

{
    "id": 1,
    "subscribers": 1,
    "topicFilter": "usa/ydwvv/1/+",
    "expectedPublisherGroups": [1],
    "persistentSessionInfo": {
        "clientType": "APPLICATION"
    },
    "clientIdPrefix": null
}

• id — identifier of the subscriber group.
• subscribers — number of subscriber clients in the group.
• topicFilter — topic filter to subscribe to.
• expectedPublisherGroups — list of publisher group IDs whose messages these subscribers receive (used for debugging and statistics).
• persistentSessionInfo — persistent session object containing the client type.
• clientIdPrefix — client ID prefix for subscribers.

This test demonstrates the TBMQ cluster’s capability to process 6M messages per second from diverse devices while handling 100M concurrent connections. TBMQ excels in high-throughput scenarios and scales both vertically and horizontally. Follow the project on GitHub for future performance updates.