What a Retained-Message Audit Actually Finds: Orphaned MQTT Discovery Topics and the Automations Still Pointing at Them

There is no 30-day study behind this article, and I’m not going to pretend there is. What follows is a procedure you can run on a Tuesday afternoon, plus an explanation of what it surfaces. The “30 days” in the title is a schedule choice, not a finding: it’s roughly how long it takes for a decommissioned device to stop being obvious and start being a mystery. The audit itself is the point.

Here is the short version of what a retained-topic audit reveals. It finds discovery payloads on your broker whose publishers no longer exist. Those payloads are replayed to Home Assistant on every restart, which recreates entities that point at state topics nothing will ever publish to again. Automations that reference those entities keep running, keep evaluating conditions against stale or unavailable states, and keep failing in ways that look like something else. The audit is how you find the debt before it bills you.

Why the ghost exists at all

Retained messages are a broker feature, not a Home Assistant feature. The broker stores the last retained message per topic and hands it to any new subscriber that matches. HiveMQ’s protocol explainer puts it plainly: the broker stores only one retained message per topic, and each client subscribing to a matching topic pattern receives it immediately after subscribing. That is the whole mechanism, and it is working as designed.

Home Assistant’s MQTT discovery uses that mechanism deliberately. The discovery topic format is documented as <discovery_prefix>/<component>/[<node_id>/]<object_id>/config, with homeassistant as the default prefix. The integration docs describe retaining the discovery payload as a supported approach: it stores the payload at the broker and offers it to the MQTT integration as soon as it subscribes for discovery. The docs also note the tradeoff — with many entities this can cause high IO loads — but the reliability benefit is real. Your entities survive a Home Assistant restart without the publisher needing to be awake to re-announce itself.

The failure mode is the same mechanism pointed at a device that no longer exists. When you remove a Zigbee or Z-Wave device, re-pair it under a new node ID, or retire an ESPHome node, the publisher stops. Nothing tells the broker to forget the retained discovery payload. On the next Home Assistant start, the integration subscribes, receives the stale payload, and recreates the entity. The Home Assistant docs are explicit that on MQTT startup, existing MQTT devices, entities, tags, and device triggers remain unavailable until a discovery message is received and processed. A retained payload is a discovery message. It gets processed.

A community write-up on the Home Assistant forum describes exactly this loop: you follow the normal deletion procedure, the entity gets re-discovered, because the discovery payload was published as a retained message. A first-person homelab post describes the same thing after a service teardown — the retained discovery message stayed on the broker, and Home Assistant kept reading it on every restart, faithfully recreating an entity pointing at a state topic that would never update again. These are community reports, not vendor documentation, but they match the documented mechanism.

Why deleting the entity in the UI is not enough

The broker is the source of truth for MQTT-discovered entities. The Home Assistant entity registry is a downstream consumer. If you delete the entity in the UI and the retained discovery payload is still on the broker, the next restart re-delivers the payload and the entity comes back. Any deletion procedure that starts and ends in the UI is incomplete by construction.

There is a wrinkle worth stating as uncertainty rather than fact. Older community reports describe the entity lingering in the MQTT integration view after the retained topic is cleared, sometimes until a restart, sometimes requiring a manual “Remove Entity” step afterward. I have not verified the current behavior on the latest release, and I would not promise that clearing the retained topic alone produces a clean registry. Treat the registry check as a separate step, not an assumption.

The audit procedure

MQTT has no standard feature for listing all retained messages or topics on a broker. HiveMQ states this directly. So the audit is a subscription trick: subscribe to the discovery prefix with a wildcard, let the broker replay every retained message that matches, and capture the output. Home Assistant’s own docs show the pattern for reading all messages under the homeassistant topic from a local broker:

mosquitto_sub -h 127.0.0.1 -v -t "homeassistant/#"

That command prints topic and payload for everything the broker replays on subscription. For an audit you want a bounded capture rather than an open stream, so add a timeout and write to a file. The exact flags depend on your client version; check mosquitto_sub --help on your system rather than trusting a flag from a blog post, including this one.

What you are capturing is the retained tree under the discovery prefix. Each line is a candidate: a topic, a JSON payload, and the implied entity it would create. Record three things per topic:

• Topic — the full discovery topic path.
• Payload — the JSON, or at least the unique_id, state_topic, and device fields if present.
• Publisher status — whether anything is currently publishing to the state topic the payload references.

That third column is the work. You can check it by subscribing to the referenced state topic and watching for a few minutes, or by checking whether the device appears in Zigbee2MQTT or Z-Wave JS as a live node. A retained discovery payload whose state topic is silent is a candidate for retirement. A retained discovery payload whose state topic is active is live and should be left alone.

Triage: live, retired, or unknown

The audit output should be a labeled inventory, not a purge list. Three labels are enough:

Live. The publisher exists and is publishing. The retained discovery payload is doing its job. Leave it.

Retired. The device was removed, re-paired under a new ID, or replaced. The publisher is gone. The retained payload is decommissioning debt. This is the category the audit exists to find.

Unknown. You cannot tell whether the publisher exists. Maybe the device is offline for a reason unrelated to retirement. Maybe the payload references a topic you don’t recognize. Do not delete unknown topics on the first pass. The payload may be the only record of what a device was and how it was wired. Investigate before clearing.

The temptation is to treat every silent topic as garbage. Resist it. A device that is simply powered down for the season looks identical to a device that was retired, and deleting the retained discovery payload for a seasonal device means it will not reappear when it comes back online unless it re-announces itself. Some devices do; some don’t. The audit is cheap to re-run, so the cost of leaving an unknown topic in place for another cycle is low. The cost of deleting a live one is a support call.

Removal: zero-byte retained publish

There is exactly one protocol-level way to delete a retained message: publish a zero-byte payload with the retained flag set to the same topic. HiveMQ states this directly. A new retained message on the same topic overwrites the old one, but that is replacement, not deletion. If you want the topic gone, the payload must be empty and the retain flag must be set.

You can do this from the command line with mosquitto_pub. A community post on the Home Assistant forum shows the pattern:

mosquitto_pub -h 127.0.0.1 -p 1883 -t "homeassistant/binary_sensor/garden/config" -m ''

You can also do it from Home Assistant’s own mqtt.publish service, which avoids handling broker credentials on your workstation. The service call takes the topic, an empty payload, and retain: true. A first-person homelab post describes using exactly this approach to clear ghost entities. Either method works; the protocol requirement is the same.

Clear one topic at a time, and re-subscribe to confirm it is gone before moving to the next. A wildcard subscription after the clear should no longer replay that topic. If it does, the publish did not take — check that the retain flag was actually set and that you targeted the exact topic path, including the /config suffix.

Verification: restart and check the registry

Clearing the retained topic is necessary but not sufficient. The verification step is a Home Assistant restart followed by a check that the entity did not return. This is the step that catches the case where the retained topic was cleared but the entity registry still holds a stale entry, or where a second retained topic you missed is recreating the entity under a different path.

After the restart, check two places: the entity list for the entity you expected to disappear, and the MQTT integration view for any lingering unavailable entries. If the entity is gone from both, the clear worked. If it is gone from the entity list but still shows in the integration view, you are in the older-behavior territory the community reports describe, and a manual removal step may be needed. I have not verified the current release’s behavior here, so treat this as a check to perform rather than a prediction.

The part the audit does not cover: automations still pointing at the ghost

Removing a ghost entity can silently break automations that still reference it. Home Assistant’s automation troubleshooting docs are clear that the system does not update automations when an entity ID changes, and that actions referencing missing entities log a warning like “Referenced entities light.kitchen are missing or not currently available.” The same applies when an entity is removed entirely. The automation does not fail loudly at edit time; it fails at run time, or it stops at a condition that can never be true.

So the audit has a second half. Before you clear a retired topic, search your automations, scripts, and dashboards for references to the entity the payload would create. The entity ID is derived from the payload’s default_entity_id if set, or generated from the device and component names otherwise. If you cannot determine the entity ID from the payload alone, create a test instance, let the entity appear, note its ID, then clear the topic. That is more work, but it is the only way to be sure you are not about to orphan an automation.

Once you have the entity ID, the search is textual: grep your automations.yaml, your packages, your Lovelace resources, and any Node-RED flows if you run them. Every hit is a reference that will need updating or removing. This is the step that turns a broker cleanup into a maintenance task with a real scope.

Graceful degradation: what happens when the ghost is gone

An automation that references a removed entity does not crash. It evaluates. A state trigger on a removed entity simply never fires. A condition on a removed entity evaluates as unknown or unavailable, depending on the condition type, and the automation stops at that condition. A template that references the entity’s state renders as unknown or unavailable, and any downstream logic that assumes a numeric value gets a string instead.

This is the graceful-degradation story, and it is not a happy one. The system keeps running, which means the failure is quiet. The lights that were supposed to turn on at dusk do not turn on, and nothing in the logs says why unless you go looking. The audit’s value is that it surfaces the reference before the failure, not after.

If you cannot update the automation immediately, the safer intermediate step is to disable it rather than leave it running against a ghost. A disabled automation is visible in the UI and does not produce misleading traces. A running automation against a removed entity produces traces that look like condition failures, which sends you looking at the condition instead of the missing entity.

Making it recurring

A one-time audit is a cleanup. A recurring audit is maintenance. The difference is a written decommission checklist that includes “clear retained discovery topics” as a step, right after “remove the device from Zigbee2MQTT or Z-Wave JS” and before “update automations that referenced it.” Without that step, the next teardown leaves the same debt, and the next audit finds it again.

The schedule is yours to choose. Thirty days is a reasonable cadence for a small residential or small-office install because it is long enough for a decommissioned device to stop being memorable and short enough that the audit output is still small. It is not a research finding and I am not presenting it as one. Run it quarterly if your install is stable; run it after any device replacement if it is not.

The output of each audit should be a dated inventory with the three labels — live, retired, unknown — and a note of what was cleared. That file is the record that tells you, six months from now, whether the topic you are staring at is new debt or old debt you already decided to leave alone.

When a plain switch is the better answer

Not every install needs this. If your MQTT footprint is a handful of devices that never get replaced, the audit will find nothing and cost you an hour. If your automations are simple and your device churn is low, a mechanical timer and a plain switch will outperform a smart stack on reliability and cost, and they will never leave a retained payload on a broker. The audit is worth running when you have enough MQTT-discovered devices that you cannot hold the inventory in your head, and enough device churn that retirement is a recurring event. Below that threshold, the procedure is overhead.

For a broader look at how to schedule this kind of maintenance alongside the rest of a small system, see How to Audit the Small Systems That Quietly Run Your Week. The retained-topic audit is one entry in that larger practice, not a replacement for it.

FAQ

Does clearing the retained discovery topic remove the entity from Home Assistant automatically?

Not reliably, based on the sources available. Older community reports describe the entity lingering in the MQTT integration view until a restart, and one report describes needing a manual “Remove Entity” step afterward. I have not verified the current release’s behavior. Plan for a restart and a registry check as separate steps.

Can I just delete the entity in the Home Assistant UI?

No. The broker is the source of truth for MQTT-discovered entities. If the retained discovery payload is still on the broker, the next restart re-delivers it and the entity returns. The UI deletion is necessary only if the registry holds a stale entry after the broker clear; it is not sufficient on its own.

How do I list all retained messages on my broker?

MQTT has no standard feature for listing retained messages. The practical method is to subscribe to the discovery prefix with a wildcard and capture what the broker replays. Home Assistant’s docs show mosquitto_sub -h 127.0.0.1 -v -t "homeassistant/#" as the pattern for reading all messages under the default prefix.

What if I clear a topic for a device that was only temporarily offline?

The device will not reappear in Home Assistant until it re-announces itself via a new discovery payload. Some devices do this on reconnect; some do not. This is why unknown topics should be investigated before deletion rather than cleared on the first pass.

Do retained messages expire?

MQTT 3.1.1 has no expiry mechanism for retained messages. MQTT 5 adds a message expiry interval, but whether it is honored depends on the broker. Home Assistant requires an MQTT 5 broker, but that does not mean your devices are publishing with an expiry set. Do not assume a retained payload will age out on its own.

Is MQTT Explorer required?

No. MQTT Explorer is one option for browsing the retained tree visually, and community posts recommend it. The command-line approach with mosquitto_sub and mosquitto_pub works without a third-party client, and Home Assistant’s mqtt.publish service works without broker credentials on your workstation. Use whichever fits your access model.