Troubleshooting

This guide walks through diagnosing a misbehaving TBMQ deployment: a playbook for each symptom you are likely to hit, followed by the reference those playbooks draw on — where the logs are, how to turn on a component logger, and what the Kafka lag line tells you.

Every metric named here is documented in full on the Prometheus metrics page, and Reading logs below covers where the log files live and how to tail them on Docker Compose and Kubernetes.

A client’s CONNECT either comes back with a failure CONNACK, or the connection ends before a CONNACK is ever sent.

Check

Interpret

connectionRefused_total rising while connectionAccepted_total stays flat means the CONNECT is reaching TBMQ and getting a CONNACK back — just not a successful one. Bad credentials and failed authorization both send this CONNACK. So does a handful of validation failures that TBMQ only runs after authentication has already succeeded — an empty client identifier on a non-clean-start session, an invalid last-will topic, an MQTT 5 receive-maximum of 0. Seeing one of these specifically is a useful signal in its own right: it confirms the client’s credentials were accepted, and the connection failed on a protocol/session detail instead. It’s also how a broker-side quota shows up — TBMQ enforces a configurable limit on total sessions and on APPLICATION clients, and refuses new connections with a QUOTA_EXCEEDED reason once either is hit.

connectionError_total rising instead means the failure sits below the MQTT layer, before a CONNACK is even possible. It counts pre-CONNECT failures — a TLS handshake problem, non-MQTT traffic arriving on the port, or a protocol violation caught before a session exists — and it is deliberately narrow: it only fires while the connection has no client id yet, so an error on an already-established session is counted as a disconnect instead, not a connection error (see Unexpected disconnects).

Act

• Check the credentials and authorization rules against what the client is actually presenting.
• Verify the listener and TLS configuration — the right port, a valid certificate chain, a TLS-capable client talking to a TLS listener.
• Review the Unauthorized clients page for the specific client id and rejection reason.
• See Security for how authentication and authorization are configured end to end.

A client connects successfully and then drops — either on its own or because TBMQ closed the session.

Check

Interpret

The counter alone can’t tell reasons apart, so the logged DisconnectReasonType is what you read for the breakdown. The ones that show up most often in practice:

• ON_KEEP_ALIVE — the client stopped sending anything within its keep-alive window.
• ON_CONFLICTING_SESSIONS / ON_CLUSTER_CONFLICTING_SESSIONS — another connection using the same client id took over the session, on this node or elsewhere in the cluster.
• ON_RATE_LIMITS — an MQTT 3.x client exceeded a configured incoming-message rate limit. An MQTT 5 client hitting the same limit stays connected instead: a QoS 1 or QoS 2 publish gets a PUBACK/PUBREC reporting “Quota exceeded”; a QoS 0 publish is silently dropped with no ack at all.
• ON_QUOTA_EXCEEDED — an internal limit was exceeded, for example too many messages queued while a CONNECT was still being processed.
• ON_CHANNEL_CLOSED — the client’s TCP connection went away.
• ON_ADMINISTRATIVE_ACTION — an operator or API call disconnected the client.
• ON_SERVER_SHUTTING_DOWN — the node is shutting down and is closing every session on it.

The takeover case is worth calling out on its own: two clients sharing one client id will keep disconnecting each other, as long as both reconnect automatically, as each new CONNECT kicks off the previous session and each kicked-off client’s own reconnect logic kicks the other one straight back off. The signature is a clientDisconnects_total rate that stays high while connectedSessions barely moves — sessions are churning, not draining. That reading holds cleanly for a same-node takeover (ON_CONFLICTING_SESSIONS), where the old and new session are on the same node. For the cross-node case (ON_CLUSTER_CONFLICTING_SESSIONS), each node’s own connectedSessions does move — one node loses the old session while another gains the new one — so look at whether the cluster-wide sum stays flat rather than at any single node’s gauge.

Act

• Give every client a unique client identifier — this is the fix for the takeover case above. See Client ID.
• Review the keep-alive interval against the client’s actual network conditions — see Keep alive.
• Check whether a configured rate limit is tighter than the client’s real traffic.

A publisher gets its PUBLISH acknowledged (or sends QoS 0 without expecting one), but the message never reaches a subscriber.

Check

Interpret

Several different things collapse into “a drop,” and telling them apart matters:

• No matching subscription — nobody is subscribed to a topic a client just published to. This still counts toward droppedMsgs_total, but it’s a normal, often expected outcome rather than a problem to chase.
• dropsOverflow — a subscriber’s in-flight window and its delay queue behind it are both full, almost always because that subscriber has fallen behind the publisher for a while, not just momentarily. See the Slow subscribers playbook. This only adds to droppedMsgs_total for a non-persistent, non-retained message — a persistent subscriber’s copy stays recoverable from Kafka or Redis, so it isn’t counted as lost.
• dropsTtl — the message aged out of the delay queue before TBMQ could send it. This isn’t an independent cause: a message only lands in the delay queue in the first place because the in-flight window was already full, so dropsTtl is usually the same slow-subscriber situation as dropsOverflow, just showing up on a longer delay — see Slow subscribers here too.
• failedMsgs on the device processor — persistence failed, not delivery. That points at Redis (the persistent DEVICE message store), not at the messaging path — see the Dependency failures playbook.
• Rate limiting — a client’s incoming messages, or the broker’s total-message budget, hit a configured limit and the PUBLISH is dropped at ingest before it is ever dispatched to a subscriber.

Act

• Confirm a subscription actually exists on the topic and that its filter matches what was published — see Topics and wildcards.
• Check the configured rate limits against the client’s or the broker’s real traffic.
• For a slow subscriber or a failing dependency, follow through to the corresponding playbook below rather than treating the drop counter itself as the fix.
• See Backpressure for how TBMQ handles a subscriber that can’t keep up.

A client subscribes successfully but a publish that should reach it never arrives.

Check

Interpret

Ordinary causes, roughly in order of how often they turn out to be it:

• The topic filter doesn’t match the publish topic — wildcards and exact segments have to line up; test the filter directly rather than assuming.
• A QoS 0 message published to an offline persistent session — QoS 0 is never stored for later delivery, regardless of whether the subscribing session itself is persistent. A persistent client that was offline when a QoS 0 publish went out will never see that message, even after it reconnects.
• Shared-subscription semantics — only one member of a shared-subscription group receives any given message. If you’re testing with more than one group member, look at all of them before concluding nothing was delivered — see Shared subscriptions.
• The subscription hasn’t propagated across the cluster yet — subscription state replicates between nodes over Kafka; a publish arriving on another node right after the SUBSCRIBE can briefly miss it.

Act

• Verify the topic filter by subscribing with the WebSocket client and publishing a test message to the exact topic.
• Confirm the session type and clean-start behavior match what you expect — see Non-persistent and persistent sessions.
• Review shared-subscription group membership if other members are receiving messages the one you’re testing isn’t — see Shared subscriptions.
• See Topics and wildcards for filter syntax.

Publishes succeed and subscribers eventually get them, but end-to-end latency is high or climbing — the broker is falling behind rather than failing outright.

Check

Interpret

Growing lag on msg-all-consumer-group means the broker is ingesting PUBLISH messages faster than the consumer pool behind it can process them — usually the first sign of trouble, ahead of anything that implies loss. Rising msgInQueueTime on either actor family means the actor thread pool doing that work is behind, not just the Kafka consumers feeding it. Rising kafkaProducer_send_seconds points at Kafka itself — broker load, network, or replication — rather than at TBMQ’s own processing. Timeouts on the app processor (tmpTimeoutPublish climbing) mean persistent APPLICATION clients are not acknowledging fast enough to keep the pipeline draining.

Act

• Scale broker nodes horizontally, or increase the consumer count for the pipeline that’s behind (queue.msg-all.consumers-count, queue.device-persisted-msg.consumers-count) — each is capped by its topic’s partition count, so raising it past that point is a silent no-op. Persistent APPLICATION clients have no equivalent setting to raise; each gets its own per-client consumer instead — see Kafka consumer group lag.
• If the slowdown traces to one specific downstream — a subscriber, Kafka, or the database — follow through to that playbook instead of tuning consumer counts here.
• See Architecture for how the consumer pools relate to broker nodes, and MQTT broker configuration for the consumer-count parameters.

A publish reaches the broker fine and most subscribers get it promptly, but one subscriber consistently lags — or starts losing messages under load.

Check

Interpret

nonWritableClients above zero for a sustained period means at least one subscriber’s channel cannot drain as fast as the broker is writing to it. delayedQueueSize climbing is the early warning; dropsOverflow climbing means the buffer already overflowed and messages were lost — see Messages are dropped above for what counts as lost and what doesn’t. A brief spike during a reconnect burst is ordinary; a level that stays elevated is not.

Act

• Fix the slow consumer, or reduce its subscription fan-out.
• Reconsider the client’s type: persistent APPLICATION clients absorb backpressure in Kafka, sized in days/GB of retention, while persistent DEVICE clients are bounded by a much smaller per-client Redis-backed queue. See TBMQ client type.
• See Backpressure for the full mechanism — how TBMQ buffers, pauses, and eventually drops for a subscriber that can’t keep up. This playbook only points at the symptoms; that page is the full treatment.

A node is pegged on CPU or memory, threads pile up, or the process restarts under load — the broker is resource-bound rather than waiting on a downstream dependency.

Check

Interpret

jvm_gc_overhead approaching 1 means the JVM is spending nearly all its CPU time collecting rather than doing broker work — the node is effectively down even though the process is still running. Open file descriptors approaching process_files_max_files cap how many more connections the node can accept, since every MQTT connection holds a socket descriptor open. Growing subscriptionTrieNodes or retainMsgTrieNodes while message traffic stays flat points at subscription or retained-message accumulation, not at load. retainedMessages and those two trie gauges are cluster-wide replicated, while runningActors is per node — read them accordingly rather than summing what’s already cluster-wide; see Aggregating across a cluster on the metrics page for the full breakdown.

Act

• When Prometheus isn’t reachable, top gives a fast read on whether a node is CPU- or memory-bound — a stand-in for a quick check, not a replacement for watching these series over time.
• Sustained high CPU across every node calls for horizontal scaling — add broker nodes. Memory pressure on a single node with a modest connection count usually calls for vertical scaling — raise the heap — instead.
• Raise the file descriptor limit before raising the connection target per node.
• See cluster setup and Scaling P2P messaging to 1M msg/sec for sizing guidance.

The broker itself is up and accepting connections, but something behind it — Kafka, PostgreSQL, or the Redis-backed store — is slow or unreachable, and TBMQ is stalling on it.

Check

Interpret

sqlQueue_queueSize climbing while sqlQueue_total{statsName="successfulMsgs"} stays flat means TBMQ is producing SQL writes faster than PostgreSQL is draining them. hikaricp_connections_pending above zero means threads are waiting for a connection — the pool is undersized for the load, or queries themselves are slow. Rising kafkaProducer_send_seconds or kafkaConsumer_commit_seconds with no matching rise in TBMQ’s own CPU usage points outside the broker, at Kafka. deviceProcessor_total{statsName="failedMsgs"} climbing on its own, with the SQL queues above flat, narrows the problem to the Redis (or Valkey) side specifically.

Act

• Check the dependency’s own health first — Health API reports db, kafka, and redis as separate components once HEALTH_SHOW_DETAILS=always is set, so a DOWN there tells you which one to chase before looking anywhere else.
• Review connection-pool sizing against real concurrency — the HikariCP pool, and the Kafka producer/consumer counts.
• Verify TLS and credentials for Redis or Valkey specifically when the persistent DEVICE path is the one failing.
• See MQTT broker configuration for the pool-size and connection parameters.

TBMQ writes to thingsboard-mqtt-broker.log in /var/log/thingsboard-mqtt-broker, plus stdout — both by default. On Kubernetes, each pod logs into its own subdirectory under that path instead, named after the pod (TB_SERVICE_ID is set to the pod’s own name) — see the Kubernetes tab below.

docker compose logs -f tbmq1 tbmq2
docker compose logs tbmq1 tbmq2 | grep ERROR
docker compose logs -f tbmq1 tbmq2 > tbmq.log
docker ps

kubectl get pods
kubectl logs -f tbmq-0
kubectl logs -f tbmq-0 | grep ERROR
kubectl logs -f tbmq-0 > tbmq-0.log

TBMQ looks for logback.xml at /config/logback.xml first, falling back to the packaged copy at /usr/share/thingsboard-mqtt-broker/conf/logback.xml only if nothing is mounted at /config. The cluster Docker Compose setup and every Kubernetes manifest both mount something at /config, so /config/logback.xml is the file actually in effect there — the packaged path only matters for a single-node Docker install, which doesn’t mount /config by default. Either way, scan="true" and scanPeriod="10 seconds" mean an edit takes effect within ten seconds once it’s in the right place — no restart required. This is different from the environment-variable parameters covered in how to change configuration: logback.xml is its own file, and Docker and Kubernetes each ship their own copy of it.

The shipped configuration:

<configuration scan="true" scanPeriod="10 seconds">

    <appender name="fileLogAppender"
              class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/thingsboard-mqtt-broker/thingsboard-mqtt-broker.log</file>
        <rollingPolicy
                class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>/var/log/thingsboard-mqtt-broker/thingsboard-mqtt-broker.%d{yyyy-MM-dd}.%i.log
            </fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="fileLogAppender"/>
        <appender-ref ref="STDOUT"/>
    </root>

</configuration>

The RollingFileAppender caps each file at 100MB, keeps 30 days of history, and stops growing past a 3GB total cap. STDOUT mirrors the same pattern to the console, and <root level="INFO"> sends everything to both.

To turn on a specific component, add one or more <logger> elements between the appenders and <root>. For example, to see every detail of the CONNECT/CONNACK path while silencing actor lifecycle noise:

<logger name="org.thingsboard.mqtt.broker.actors.client.service.connect" level="TRACE"/>
<logger name="org.thingsboard.mqtt.broker.actors.DefaultTbActorSystem" level="OFF"/>

Loggers worth enabling. Every name below is relative to org.thingsboard.mqtt.broker. — prepend that prefix when you write the <logger name="…"> entry. Rows marked (package) cover every class beneath them.

./scripts/docker-refresh-config.sh

kubectl apply -f tbmq-configmap.yml

Consumer-group lag reporting is enabled by default (TB_KAFKA_CONSUMER_STATS_ENABLED, default true), printed every TB_KAFKA_CONSUMER_STATS_PRINT_INTERVAL_MS (default 60000 ms). There is nothing to turn on.

2022-11-27 02:33:23,625 [kafka-consumer-stats-1-thread-1] INFO  o.t.m.b.q.k.s.TbKafkaConsumerStatsService - [msg-all-consumer-group] Topic partitions with lag: [[topic=[tbmq.msg.all], partition=[2], lag=[5]]].

Generic form:

[<consumer group>] Topic partitions with lag: [[topic=[<topic>], partition=[<partition>], lag=[<count>]], ...].

Each consumer group belongs to one subsystem, so a lag line points you straight at what’s falling behind:

Every group name is prefixed with the Kafka prefix from configuration (TB_KAFKA_PREFIX, empty by default — see Kafka parameters), so on a broker with a prefix configured, look for <prefix>msg-all-consumer-group rather than the bare name above.

Most groups also carry a trailing id, and the placeholders above mean different things depending on the group:

• <serviceId> — just that node’s TB_SERVICE_ID, stable across restarts.
• <id> — that node’s TB_SERVICE_ID plus a suffix generated fresh each time the node starts. The exact group name for these six changes on every restart, so don’t expect to find the same name in an old log after a node has restarted — search for the group’s name prefix instead.
• <clientId> / <shareName>-<sharedTopic> — the two APPLICATION-client rows aren’t per-node at all: each persistent APPLICATION client, and each shared-subscription group, gets its own dedicated consumer group identified by its own client ID or share name and topic.
• <integrationId> — the two ie-…-consumer-group-… rows are per integration rather than per node: each integration gets its own pair of groups, identified by that integration’s ID written as a 32-character UUID with the hyphens stripped (for example, integration 1d2f5d40-1111-2222-3333-444455556666 lags as ie-msg-consumer-group-1d2f5d40111122223333444455556666). Lag on one of these points at a single integration falling behind, not at the integration subsystem as a whole.

If none of the playbooks above resolve the issue, have the logs and the relevant metric series ready before reaching out.

If you have questions or run into issues, the TBMQ team and community are here to help.