*How jev-oncall puts a model on the paging path without letting it drop pages, and the two bugs that nearly did.*
Every on-call engineer knows the two ways paging goes wrong. The noisy way: a disk at 81% pages you at 3am, the fifth alert about the same outage wakes you again, and after a month you stop trusting the pager. The quiet way: something that mattered was labeled a warning, went to a ticket queue, and nobody saw it until a customer did.
It's tempting to put a language model in front of the pager and let it sort the noise from the signal. The worry is just as obvious. A model that's wrong one time in a hundred will eventually decide that a real outage is noise, and nobody will know until the postmortem.
[jev-oncall](https://github.com/mingleiw/jev-oncall) is my attempt to get the first benefit without the second risk. It's open source, it uses [Jev](https://typesafe.ai), TypeSafe's decision model, and it's built around one rule: **the model judges, plain code decides.** Jev never pages anyone. It answers questions with probabilities, and a few dozen lines of ordinary, testable code turn those into actions.
This post walks through the design, and two bugs I hit while building it that would each have silenced a real page.
## One call, four typed questions
Every production alert gets exactly one Jev call. The call asks four questions, and each has a typed answer rather than free text:
- **`actionable`**, yes or no: does a human need to intervene?
- **`severity`**, a score from SEV4 up to SEV1, returned as a probability for each level.
- **`team`**, a choice among your teams, which you describe in a config file.
- **`duplicate_of`**, a choice among recent alerts that could have caused this one, or `none`.
Typed answers matter more than they look. If a model writes "this looks like a SEV2, probably the database team", the code downstream has to parse prose and guess how sure it was. If it returns `{SEV1: 0.08, SEV2: 0.61, SEV3: 0.27, SEV4: 0.04}`, the code can reason about uncertainty directly. The number the policy cares about is
```
P(page) = P(SEV1) + P(SEV2)
```
That's the probability that this alert deserves to wake someone up, using the whole distribution rather than the single most likely label.
## The policy is a few lines of code
Here is the heart of it, lightly trimmed from `route_standalone()` in `triage.py`:
```python
if p_page >= policy.page_bar: # 0.80
action = "PAGE_NOW" if sev["SEV1"] >= sev["SEV2"] else "PAGE"
elif p_page > policy.no_page_bar: # 0.20
action = "REVIEW" # 15 min to ack, or it pages
elif p_actionable <= policy.drop_bar: # 0.05
action = "DROP"
else:
action = "TICKET"
```
Three things about it are deliberate.
**The bars are asymmetric.** Paging needs P(page) of 0.80. Not paging needs 0.20 or less. Everything in between is a REVIEW. And DROP, the only outcome no human ever sees, needs both a low chance of paging *and* a 95% chance that nobody needs to act. The most certainty is required exactly where a mistake would be invisible.
**When the answers disagree, the more urgent one wins.** If severity says "page" and actionable says "no human needed", the alert pages and the disagreement is logged.
**It's plain code, so it's testable and tunable.** Every probability Jev returns is stored, so `evaluate.py --sweep` can replay a history of alerts under different thresholds without calling the model again. You choose the bars from your own data instead of trusting mine.
Two rules run before the model is ever called. Non-production alerts are logged by rule and never cost a call. And if Jev errors, times out, or returns something malformed, the alert is routed by the severity it was configured with: critical pages, warning becomes a ticket, info is logged. That's exactly what would have happened without jev-oncall, so a Jev outage degrades to your current setup rather than to silence.
## The review clock
The middle band is where the design earns its keep, and where it's easiest to get wrong. A REVIEW that nobody looks at is just a ticket with a different name. So every REVIEW starts a clock: if nobody acks it within 15 minutes, it escalates to a page. Being unsure costs a person a glance, never an outage.
![[review-clock.png]]
*A REVIEW ends in one of three ways. Only the ack and a genuine resolve stop it from paging.*
### Bug one: the clock that never ran out
When I added support for Prometheus Alertmanager, I found the first bug that could have silenced a page. Alertmanager groups alerts, and whenever a group changes it re-sends every alert in it, every five minutes by default. jev-oncall treated each resend as a fresh REVIEW, and adding a REVIEW reset its deadline.
So an unacked review, re-sent every five minutes against a fifteen-minute clock, would never escalate. The safety net had a hole exactly the size of a default setting.
The fix had two parts. Resends are now recognized by a stable id, built from the alert's fingerprint and start time, so they aren't judged again. And, more generally, a REVIEW added again for the same alert keeps its *first* deadline, so no provider's resend pattern can push a page back. The same change stopped "resolved" notifications from being triaged as new alerts, which could have paged someone about an alert that had just cleared. A resolve now cancels the pending review instead.
## Dedup as a graph
The fifth page about one outage is the noisy failure. jev-oncall handles it by asking Jev which recent alert, if any, caused this one. The candidates are chosen by code, not the model: production alerts that started up to 30 minutes before (or 2 minutes after, for delivery jitter), limited to the service and its upstream dependencies when you've described your topology. Jev only picks among them.
Each confident answer, 0.70 or higher, becomes an edge in a graph. Loops are broken at whichever alert started first, since a cause can't start after its symptom. Each connected group becomes one incident, and its root takes the most urgent action of anything in the group. Alerts owned by the root's team are marked DEDUP. An alert owned by *another* team gets a REVIEW instead, so a wrong link can delay that team by the ack window but can never silence it.
A function called `check_invariants()` runs after every batch and flags any linked alert that needs more urgency than its root got, and anything dropped without a model judgment. A batch run exits with an error; the server returns the violations with its decisions.
### Bug two: the fix that silenced a page
The second bug came from the first real run of the Docker demo with a Jev key. The webhook server crashed.
Alertmanager groups by service, so alerts from different services arrive in separate deliveries. Jev linked an alert in one delivery to its cause in an earlier one, which is exactly what dedup is for. But the server routed each delivery on its own, and the cause's decision lived in an earlier batch. Looking it up raised a `KeyError`.
The first fix passed in the earlier decisions and, since a decision already sent can't be changed, let an earlier root keep whatever it got. That stopped the crash. It also opened a quieter hole: if the database alert had been judged a TICKET, and the checkout alert clearly needed an immediate page, the checkout alert became DEDUP under a ticket. Nobody would be paged. And `check_invariants()` said nothing, because it skipped links to alerts outside the current batch.
The real fix asks one question before linking to an earlier alert: did it already get an action at least as urgent as everything that would join it? If yes, link as usual. If not, the links to it are cut, and the new alerts group among themselves in this delivery, where the normal escalation applies. The cost is one extra page instead of zero, and the cut is written into the alert's reasons so it can be audited.
> [!tip] Lesson
> **The lesson I took from both bugs:** on a paging path, every fix needs one extra question, "can this make a page disappear?", and a test that fails on the old code.
>
> The cross-delivery fix landed with seven new tests. Four of them fail on the first fix, all four on the silent-page case.
## What the numbers say, and what they don't
I ran 300 synthetic alerts through the pipeline: incident clusters with a root cause and downstream symptoms, plus noise. 233 reached Jev; the other 67 were non-production and never called the model.
![[benchmark.png]]
*The dashboard for that run. Each dot is an alert placed at its P(page), against the 0.20 and 0.80 bars.*
- **Latency per call:** p50 418 ms, p95 1,477 ms, max 1,849 ms.
- **Cost:** $0.0128 for the whole run, about $0.04 per 1,000 alerts.
- **Outcomes:** 63 flagged to page, 86 for review, 33 linked to an incident that already paged, 118 ticketed, logged or dropped.
Two of those numbers matter more than the median. The slowest call came within 151 ms of the 2-second timeout, so on a slower network some alerts would take the fail-open path. That's the path working as designed, but it means fail-open is load-bearing, not theoretical. And 37% of judged alerts landed in REVIEW. That's the band doing its job, and also a sign that my default bars are too wide for this alert mix. That's what the threshold sweep is for.
What this run doesn't measure is whether the routing was *right*. Synthetic alerts carry their author's guesses as labels, which proves nothing. Accuracy and calibration need a few hundred of your own historical alerts, labeled with the severity each one turned out to have. `evaluate.py` scores exactly that, side by side with routing on configured severity alone, and I'd trust those numbers over mine.
## Try it, and help build it
The fastest way to see it is the Docker demo. It starts Prometheus, Alertmanager and jev-oncall, fires a staged incident over the first minute, and shows every decision on a live dashboard:
```bash
git clone https://github.com/mingleiw/jev-oncall
cd jev-oncall/demo
docker compose up --build
# then open http://localhost:8090/dashboard
```
It runs without a key, routing everything by configured severity. That's your baseline. Add a Jev key and replay it to see what changes.
jev-oncall judges and records decisions today. It doesn't send pages, Slack messages or tickets yet. The next step is **shadow mode**: run next to your current paging and report what it would have done, so a team can compare before trusting it. After that come Slack with an Ack button, durable state, and outbound paging. The [roadmap](https://github.com/mingleiw/jev-oncall/issues/31) has the order, and there are [good first issues](https://github.com/mingleiw/jev-oncall/issues?q=is%3Aopen+label%3A%22good+first+issue%22) if you'd like to help.
If you run it on your own alerts, I'd like to hear what it got wrong.
---
[View jev-oncall on GitHub](https://github.com/mingleiw/jev-oncall) ยท [See the architecture](https://mingleiw.github.io/jev-oncall/architecture.html)