March 7, 2026

Case Study: Monitoring Model Fallbacks

One of the subtle failure modes in a multi-bot AI setup is that the system keeps working while quietly doing the wrong thing.

That happened to me this week.

Update, July 2026: The monitoring principle below still holds, but the routing architecture has evolved. Model selection now comes from a central role/provider policy rather than versioned model names copied into each deployment. Fleet updates preserve each agent's fallback policy, and live smoke tests verify the provider, model, reasoning mode, and whether fallback was used.

The runtime monitor described below still provides ongoing fallback detection. The central policy and smoke tests add proactive verification during configuration and fleet rollouts rather than replacing it.

I had several OpenClaw-based assistants configured to use OpenAI subscription-backed auth as their primary path, with Anthropic as a fallback. Operationally, this looked sensible: if OpenAI was unavailable, the bots would keep responding instead of failing closed.

The problem was that the fallback was silent.

The bots would continue answering messages, but some of them were doing so on Anthropic API keys instead of the intended OpenAI subscription path. From the user's perspective, nothing obviously broke. From an infrastructure and cost perspective, it was exactly the kind of failure you want to know about immediately.

The Failure Pattern

There were actually several overlapping causes:

1. OAuth expiry and drift. The OpenAI subscription auth used by some bots had expired or was no longer being read from the place the agent expected.
2. Tool profile resets during onboarding. Re-running onboarding to fix auth changed unrelated settings, which made some sessions appear "broken" in ways that were easy to misdiagnose.
3. Silent fallback. When the primary path failed, OpenClaw did exactly what it had been configured to do: move to the fallback model and continue serving requests.

That last one is the key design problem.

If you allow fallback, you need observability. Otherwise you haven't built resilience, you've built a leak.

The Desired Behavior

What I actually wanted was very specific:

• Use the intended OpenAI path by default
• Allow fallback so the bots stay available
• Alert immediately when fallback starts
• Keep alerting every 30 minutes while the degraded state persists
• Send a recovery alert when the primary path returns
• Distinguish between "fallback happened" and "primary auth is broken"

In other words: fallback should be allowed, but never silent.

The Monitoring Architecture

I implemented the monitor as a small external script rather than modifying prompts or relying on the assistants to self-report.

That distinction matters.

An agent can be confused about which tools it has, which model it is on, or why something failed. A host-level monitor doesn't have that problem. It can inspect the runtime state directly.

The monitor does three things:

1. Read configured intent

It reads the bot's OpenClaw config to determine:

• the configured primary model
• the allowed fallback models
• where the bot's state and logs live

This establishes the expected state.

2. Read actual runtime behavior

It then checks what actually happened, using two sources:

• Session transcripts — OpenClaw records model_change events and assistant messages with provider and model
• Gateway startup logs — the gateway logs the model it boots with, which is useful after reboots when the latest transcript may still reflect an older degraded run

This distinction turned out to be important. My first version of the monitor looked only at transcripts. After a reboot, two bots correctly came back on OpenAI, but the monitor still saw older Anthropic transcript entries and incorrectly alerted that fallback was active. The fix was to treat a clean post-restart primary startup as authoritative when it was newer than the last transcript model event.

3. Read auth failure signals

The monitor also scans logs for specific primary-path failure patterns, such as:

• OAuth refresh failures
• rejected refresh tokens
• missing OpenAI auth in the agent's auth store
• unknown model IDs

That lets it distinguish:

• primary
• fallback
• auth_failed

Alerting Behavior

Once the state machine exists, alerting becomes straightforward:

• fallback -> send a red-alert Telegram message immediately
• degraded state persists -> repeat every 30 minutes
• primary after degradation -> send a green recovery message
• auth_failed -> send a specific auth-failure alert

This is deliberately simple. No dashboards, no metrics backend, no separate queue. Just local state, log inspection, transcript inspection, and a Telegram message.

For this kind of personal infrastructure, that is enough.

Credential Boundaries Matter

Another lesson from this work is that "fallback" is not just a model-routing concern. It is also a credential-routing concern.

Different bots may have different ownership boundaries:

• some share one billing source
• some should use an isolated key with a spending cap
• some are operator-maintained but belong to someone else

That means the monitoring and fallback design has to account for who pays when a fallback occurs, not just which model gets selected.

I ended up standardizing on:

• one primary OpenAI subscription path
• one cheaper Anthropic fallback model
• separate Anthropic credential sources for different ownership groups
• one reusable monitor implementation deployed per host

The code stays shared. The policy stays local.

Why I Did Not Patch OpenClaw

I considered adding custom runtime hooks directly into OpenClaw to emit fallback events.

I didn't need to.

Because OpenClaw already records model changes in transcripts and logs gateway startup model selection, I could build the monitor entirely outside the runtime. That made the solution:

• easier to deploy
• safer to maintain across OpenClaw updates
• portable across multiple hosts

The only logic bug I had to fix was my own: after reboot, stale transcripts are not a reliable source of current state unless you compare them with current startup events.

The Broader Lesson

This is a good example of a pattern I see often with AI infrastructure:

the hard part is not making the system work once.

The hard part is making the failure modes visible.

Silent success is often worse than loud failure. If a bot stops responding, you investigate immediately. If it keeps responding on the wrong provider, wrong key, wrong cost path, or wrong tool profile, you may not notice until much later.

So the rule I would generalize is:

Any fallback that changes billing source, trust boundary, or operational semantics must produce an alert.

Fallback is fine. Hidden fallback is not.

← Back to Journal