Skip to content

Find the messages it could not read

A message no format in Settings.Reads can read never reaches a handler, so a handler's logs will not show it. What it becomes is in What happens to a message nothing can read. This page is how to notice one and what to do next.

Watch the counter

st, ok := bus.Stats("orders")
if ok && st.Unreadable > 0 {
    // messages this subscription's formats cannot read are arriving
}

Unreadable counts attempts, not messages. Under AtLeastOnce one unreadable message is declined at each delivery until it dead-letters, so it adds MaxDeliver to the count. Under AtMostOnce it adds one. The bus does not count these in Accepted, Exhausted or Refused: it never saw them.

That is how to tell a format mismatch from a handler that keeps failing: a failing handler raises Exhausted, an unreadable message raises Unreadable and leaves Exhausted alone.

Read the log

Give Settings.Logger a handler; unset, it discards. Each unreadable delivery logs at warn level:

messaging-rabbitmq: a message on the queue could not be read  subscription=orders queue=orders-worker error=...

The error is envelope.Read's: no format claimed the message, more than one did, or one claimed it and could not decode it. It is the first thing to read.

If declining the message fails, a second warning follows: messaging-rabbitmq: declining a message that could not be read failed. A closed channel is the usual cause, and the source ends with its cause recorded.

Recover the messages

Under AtLeastOnce, each unreadable message reaches the dead-letter exchange on its last attempt and is in the dead-letter queue the operator bound to it. The adapter does not republish or inspect them, and leaves them there.

  1. Work out why from the log: a missing format in Reads, a producer writing something that matches two formats, or a body that is not what its content type says.
  2. Fix the reader (add the format to Reads, see Choose a wire format) or the producer.
  3. Move the messages from the dead-letter queue back to the exchange with whatever tool your operators use. That is outside this module.

Under AtMostOnce there is nothing to recover. The broker acknowledged the message when it delivered it, and the adapter dropped it. The counter and the log are all that remain.