The alert fired at 02:14. By 02:15, the RCA bot had posted a detailed breakdown into the incident channel, listing latency spikes and packet loss. I watched the thread, assumed the remediation agent was working, and went back to sleep. The home-automation service stayed down for six hours.
The telemetry had the answer the whole time, but it was buried in a metric label that contradicted the alert’s summary. The per-pod recording rule explicitly tagged the fired window with reason=init_container_crash_loop. The alert summary, however, screamed network_unreachable. The network was fine. The init container was crash-looping on a code defect.
This mismatch broke the remediation pipeline before it even started. The pipeline received the alert within half a minute. It ran its agent graph, consulted the fingerprint catalog, and asked the LLM planner for a fix. The planner, fed the misleading "network" summary, proposed a network reset. The fail-closed safety gate refused this action because it didn’t match the registered remediation for a network alert, and certainly not for a crash loop.
The pipeline declined. It did not retry. It did not escalate. It recorded the refusal in a metrics store that no human ever queries.
Meanwhile, the self-healing component that could touch the pod stood down correctly. Its predicate logic is strict: it fixes network strands. Recreating the pod would have re-run the same broken init script, resulting in the same crash loop. It was right to do nothing. But because the remediation pipeline had already "handled" the alert by declining it silently, the system appeared active while the service remained dead.
I woke up to a still-down service and a chat thread full of confident, useless analysis. The visible activity was a lie. The RCA bot was advisory; it posts text. The remediation pipeline was operational; it takes actions. They shared nothing but the alert object. When the operational side declined, it left no trace on the human surface.
I spent the morning reading the decline row chain out of the store. I traced the path: Alert -> Fingerprint -> Agent -> Gate -> Metrics. The gate source code confirmed the behavior: unknown actions are refused, and refusals are logged as action_declined{reason="no_registered_action"}. That’s it. No human-readable comment. No ticket update.
The fix wasn’t more automation. It wasn’t a smarter LLM. It was honest text.
I shipped a change to the alert generation logic. The alert summary now names both potential causes: network_unreachable or init_container_crash_loop. It includes a field disposition=manual_review_required when the remediation pipeline declines. This field is rendered in the alert summary itself, not just in the backend logs.
I also modified the RCA bot. It now checks the remediation pipeline’s status before posting. If the pipeline declined, the bot posts: "Remediation declined: no registered action for this fingerprint. Manual intervention required."
The change was merged, applied, and proven by the byte-exact test gate. The next time a similar alert fires, the operator won’t see a confident analysis of a network problem that doesn’t exist. They will see a refusal. They will see the reason. They will act.
An alert pipeline’s visible activity is not its action path. This stack runs advisory RCA alongside a separate remediation pipeline. When the remediation side declines, it must say so loudly. A decline with no recorded reason is unauditable. Persist the refusal reason at the point of refusal, and make it visible to every consumer, human and LLM alike.
In any multi-pipeline alerting stack, before asking "why did nobody act," ask which pipeline owns action and where its declines surface. Make the decline a first-class, human-visible artifact. An alert whose text misattributes its own cause will mislead every consumer downstream of it.