Kafka integration

TBMQ Kafka Integration enables communication with Apache Kafka, allowing TBMQ to publish messages to external Kafka clusters. This is useful for the following scenarios:

• Streaming IoT Data — forwarding device telemetry, logs, or events to Kafka for processing and storage.
• Event-Driven Architectures — publishing messages to Kafka topics for real-time analytics and monitoring.
• Decoupled System Communication — using Kafka as a buffer between TBMQ and downstream applications.

1. Device (client) publishes an MQTT message to a topic matching the integration’s topic filters.
2. TBMQ broker receives the message and forwards it to the TBMQ Integration Executor.
3. Integration Executor processes the message and sends it to a configured Kafka topic.
4. Kafka consumers process the message in downstream systems.

• A running TBMQ instance, version 2.1 or later, with the TBMQ Integration Executor service running. Without a reachable executor, saving an integration fails with a validation timeout.
• An external Kafka cluster ready to receive messages (e.g., Confluent Cloud).
• A client capable of publishing MQTT messages (e.g., TBMQ WebSocket Client).

1. Go to the Integrations page and click the “+” button.
2. Select Kafka as the integration type and click Next.
3. On the Filters and events step, keep the default Topic filters entry tbmq/#, leave Lifecycle events empty, and click Next.
4. In the Configuration step, enter the Bootstrap servers (Kafka broker addresses). See below for common and Confluent Cloud configurations. Leave Topic at its default tbmq.messages — that is the topic this tutorial reads from at the end.
5. Click Add to save the integration.

Navigate to Integrations page and click the plus icon Select Kafka as the integration type and click Next Click Next to subscribe to default topic tbmq/#

• Common / Docker
• Confluent Cloud

Specify the bootstrap server address of your own Kafka cluster (e.g., localhost:9092 for a broker you run locally — not TBMQ’s internal Kafka, which uses the same address in the default deployments). The screenshot below shows the basic configuration for establishing a connection between TBMQ and a Kafka broker.

Topic filters define MQTT-based subscriptions that trigger the integration. When TBMQ receives a message matching a configured topic filter, the integration processes it and forwards the data to the Kafka cluster.

Topic filters are optional if you select at least one lifecycle event type instead — an integration only has to have one of the two.

For example, with the topic filter tbmq/devices/+/status, any of the following messages will trigger the integration:

tbmq/devices/device-01/status
tbmq/devices/gateway-01/status

With Send only message payload enabled, the record value is the payload alone — as UTF-8 text if it is valid UTF-8, Base64-encoded otherwise.

Besides the messages matched by topic filters, this integration can deliver client lifecycle events — a client connecting, disconnecting, changing its subscriptions, or failing authentication or authorization.

Pick the event types on the Filters and events step while creating the integration, or add them later:

1. Open the integration on the Integrations page and click the Toggle edit mode button (pencil icon).
2. Click the Lifecycle events field to list the available event types, then select the ones you need: Client connected, Client disconnected, Client subscribed, Client unsubscribed, Client authentication failed, Client authorization failed, Client connection failed. Each one is added as a chip; remove it with its x.
3. Click Apply changes.

Events are published to the same Topic as messages, using the configured Key, Kafka headers, and producer settings. Send only message payload applies to message payloads only — an event is always published as its full JSON body.

These are the integration’s own events, recorded inside TBMQ for debugging and troubleshooting. They are not the client lifecycle events described above: those are about MQTT clients and are published to your Kafka topic, while these describe the integration itself and never leave TBMQ.

• Lifecycle Events — logs events such as Started, Created, Updated, Stopped.
• Statistics — insights into integration performance, including processed message counts and error rates.
• Errors — captures failures related to authentication, timeouts, payload formatting, or connectivity issues.

Lifecycle events for the Kafka integration Statistics events showing processed message counts Error events capturing integration failures

1. Navigate to the WebSocket Client page.
2. Select WebSocket Default Connection (or any working connection) and click Connect. Verify the connection status shows Connected.
3. Set the Topic field to tbmq/kafka-integration to match the integration’s topic filter tbmq/#.
4. Leave the pre-filled Payload {"temperature": 25} as is — it is what lands in the Kafka topic below.
5. Click the Send icon to publish the message.

Select a working connection and click Connect Set topic to tbmq/kafka-integration and click Send In Confluent Cloud open Topics and click on the topic tbmq.messages The message from TBMQ Kafka integration is available in the messages table

If successful, the message should be available in your Kafka service under the topic tbmq.messages:

{
    "payload": "eyJ0ZW1wZXJhdHVyZSI6MjV9",
    "topicName": "tbmq/kafka-integration",
    "clientId": "tbmq_df52bNUQ",
    "eventType": "PUBLISH_MSG",
    "qos": 1,
    "retain": false,
    "tbmqIeNode": "tbmq_ie_node",
    "tbmqNode": "tbmq_node",
    "ts": 1742554969254,
    "props": {},
    "metadata": {
        "integrationName": "Kafka integration"
    }
}

Message field descriptions: