Appearance
Webhooks
merido can push events to your own HTTP endpoints the moment they happen — a request failing upstream, a provider rate-limiting you, a budget rejecting a call — so you can alert, page, or auto-remediate without polling the API. Deliveries are signed (HMAC-SHA256, Standard Webhooks compatible), retried on failure with backoff, and fully visible in a per-endpoint delivery log.
Endpoints are managed in the dashboard (Webhooks page) or via the /api/webhooks/* admin API: create/update/delete, a per-endpoint signing secret (shown once, rotation supported), per-endpoint event-type subscriptions, and a choice of payload format (standard, slack, or discord).
Quickstart
- Create an endpoint. Dashboard → Webhooks → add your receiver URL, pick the event types you care about, and choose the
standardformat (useslack/discordonly for chat notifications — see Formats). - Save the signing secret. The
whsec_…secret is shown once at creation (and again only on rotate). Store it where your receiver can read it — you need it to verify deliveries. - Receive a test event. Use the endpoint's Send test action (or
POST /api/webhooks/endpoints/{id}/test) to get a first signed event delivered, then confirm your receiver verifies the signature and answers2xx.
From then on, every subscribed event is delivered to your URL as a POST with a JSON body.
Event catalog
Phase 1 covers the data plane — the request outcomes you'd otherwise watch the Logs page for:
| Event type | Fires when | error.code |
|---|---|---|
request.failed | A request fails with an upstream 5xx. | upstream_error |
request.rate_limited | The upstream answers 429. | rate_limited |
request.budget_rejected | A request is rejected with 402 (budget exhausted). | budget_exhausted |
The catalog extends past the data plane: Phase 2 adds control-plane state changes (budgets, provider/account reliability, and opt-in management lifecycle events), and Phase 3 adds reports and a high-volume request-level firehose. The envelope's api_version stays v1 for all of it — additive events don't bump the version, so a receiver that routes on type keeps working as the catalog grows.
Debounce
request.* events are debounced per (provider, status) in a 60-second window — a provider melting down produces one event per minute per status, not one per failed request. The payload carries data.dedup_key ("provider:status", e.g. "openai:503") so your receiver knows which stream an event belongs to and can interpret gaps: more failures of the same kind may have occurred inside the window.
Budgets
Emitted from the budget-enforcement path (see Budgets & spend caps) whenever a budget's spend crosses a threshold or its hard limit is reached:
| Event type | Fires when |
|---|---|
budget.threshold_crossed | A budget's spend crosses 85% or 95% of its hard limit. |
budget.crossed | A budget's hard limit is reached and a request is rejected with 402 — the control-plane sibling of request.budget_rejected. |
budget.projected_exceeded | merido projects the budget will exceed its limit before the window resets. Only fires when a projection is available — a budget without enough spend history doesn't get one. |
Dedup: once per budget (+ threshold) per day
Budget events are deduped per (event type, budget_id, threshold_percent) for 24 hours — a budget that stays pinned above 95% doesn't retrigger on every request that hits it. The window is visible in the payload as dedup_key, so a gap in your alerts doesn't mean the budget recovered, just that it's still muted.
Provider & account reliability
Emitted from the router's health tracking — circuit breaker transitions, cooldowns, and outage detection:
| Event type | Fires when |
|---|---|
provider.circuit_opened | The router's circuit breaker trips for a provider/account (too many consecutive failures) and stops routing to it. |
provider.circuit_closed | The breaker resets and the provider/account is routable again. |
account.cooldown_started | An account is put on cooldown — either the upstream sent a Retry-After header, or the circuit breaker applied its own cooldown. |
account.quota_exhausted | An account's provider-reported quota (rate limit or spend cap) is exhausted and the router skips it. |
provider.outage | A provider's non-2xx/4xx error rate crosses an outage threshold: ≥5 errors in a 60-second window is minor, ≥10 is major. Deduped per provider for 10 minutes. |
Management lifecycle (opt-in)
Bridged from the audit log — every key and user create/rotate/delete is already recorded there; these events push the same entries to your webhook endpoints in real time.
| Event type | Fires when |
|---|---|
key.created | A new API key is created. |
key.rotated | An API key's secret is rotated. |
key.deleted | An API key is deleted. |
user.created | A new dashboard/API user is created (cloud/multi-tenant). |
user.deleted | A user is removed. |
These are opt-in — see the callout below.
Webhook system events
| Event type | Fires when |
|---|---|
webhook.endpoint_auto_disabled | An endpoint that has been failing continuously for 5 days is automatically disabled, so a dead receiver doesn't grow the delivery queue forever. |
webhook.endpoint_auto_disabled is delivered to your other enabled endpoints — the disabled one obviously can't receive a notification about itself — so you find out from merido, not from silence. Re-enable it from the dashboard's Webhooks page (toggle it back on) or PUT /api/webhooks/endpoints/{id} with "enabled": true; re-enabling resets the failure streak, so it isn't immediately auto-disabled again on the next blip.
Reports & firehose
| Event type | Fires when |
|---|---|
report.weekly_digest | Once a week: the same savings-and-opportunities summary as the dashboard Advisor, pushed to your endpoint instead of only living in the UI. |
spend.tracked | Opt-in, high volume. A batched stream of every completed request's spend — build your own cost pipeline off it instead of polling reports. |
Default-off events
An endpoint with an empty event-type list subscribes to "all" — but "all" does not include the noisy or opt-in families: key.created, key.rotated, key.deleted, user.created, user.deleted, and spend.tracked. Subscribe to these explicitly (pick them in the dashboard's event-type picker, or list them in event_types on POST /api/webhooks/endpoints) — everything else in the catalog above is on by default.
Payload reference
The envelope (standard format)
Every event uses one envelope:
json
{
"id": "evt_0198f2d4-7b1a-7c3e-9f42-8a6b5c4d3e2f",
"type": "request.failed",
"created_at": "2026-07-08T12:34:56.789Z",
"api_version": "v1",
"org_id": null,
"data": { "…": "event-specific object" }
}| Field | Meaning |
|---|---|
id | evt_<uuidv7> — the event's identity. Stable across retries and redeliveries; use it as your idempotency key. |
type | Dotted event type from the catalog above. Route on it. |
created_at | When the event occurred — RFC 3339, UTC, millisecond precision. |
api_version | Payload schema version (v1). Bumped only on breaking changes. |
org_id | Organization id in multi-tenant (cloud) mode; null on the local profile. |
data | The event-specific payload (below). |
data for request.* events
json
{
"request_id": "req_…",
"session_id": "…",
"occurred_at": "2026-07-08T12:34:56.702Z",
"api_key_label": "prod-ci",
"api_key_id": 42,
"provider": "openai",
"account_id": "acct-3",
"model": "gpt-4o",
"virtual_model": "merido/auto",
"status": 503,
"error": { "message": "upstream unavailable", "code": "upstream_error", "retryable": true },
"latency_ms": 1234,
"tokens": { "prompt": 812, "completion": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "total": 812 },
"cost": { "cost_usd": 0.0021, "charge_usd": 0.0021 },
"free_pool": false,
"dedup_key": "openai:503"
}| Field | Meaning |
|---|---|
request_id | The gateway request id — look it up in Logs (GET /api/logs/{request_id}) for the full attempt timeline. |
session_id | Session attribution, when the caller sent a session header. |
occurred_at | When the request completed (RFC 3339 UTC). |
api_key_label | The gateway key's label — never the key material. |
api_key_id | The gateway key's numeric id (null for local/unauthenticated calls) — a stable primary key for mapping the request to its caller. |
provider / account_id | Which upstream connection failed — alias/id only, never credentials. |
model / virtual_model | The resolved upstream model and, if routed, the virtual model requested. |
status | The HTTP status the gateway returned (5xx / 429 / 402). |
error.message | Human-readable error summary. |
error.code | Normalized class: upstream_error | rate_limited | budget_exhausted. |
error.retryable | Whether retrying the request could succeed. |
latency_ms | End-to-end request latency. |
tokens | Token counts by class: prompt, completion, cache_read, cache_write, reasoning, total. |
cost | cost_usd (upstream cost) and charge_usd (what the caller was charged — differs from cost only with a multi-tenant markup). |
free_pool | true when the request was served from the free token pool. |
dedup_key | The debounce stream key, "provider:status" (see the debounce note above). |
No prompt or response content — ever
Webhook payloads carry identifiers and telemetry only. Message content, prompts, and completions are never included, so a webhook receiver is not a data-exfiltration surface. Fetch detail through the API (with its own auth) if you need it.
data for budget.* events
json
{
"budget_id": 12,
"scope": "key",
"entity_id": 7,
"entity_label": "prod-ci",
"window": "monthly",
"window_start": "2026-07-01T00:00:00.000Z",
"window_end": "2026-08-01T00:00:00.000Z",
"limit_usd": 100.0,
"spend_usd": 95.4,
"remaining_usd": 4.6,
"percent_used": 95.4,
"threshold_percent": 95,
"triggering_request_id": "req_…",
"occurred_at": "2026-07-08T12:34:56.789Z",
"dedup_key": "12:95"
}| Field | Meaning |
|---|---|
budget_id | The budget's id — look it up on the Budgets page. |
scope | What the budget applies to: org, virtual_model, or key. |
entity_id / entity_label | The scoped entity's id and human label (the key/virtual-model/org name). |
window | The budget's reset cadence: hourly, daily, weekly, monthly, yearly, or lifetime. |
window_start / window_end | The current window's bounds (RFC 3339 UTC) — absent for a lifetime budget, which never resets. |
limit_usd | The budget's configured hard limit. |
spend_usd | Spend recorded so far in the current window. |
remaining_usd | limit_usd − spend_usd (can go negative once the limit is exceeded). |
percent_used | spend_usd / limit_usd × 100. |
threshold_percent | The crossed threshold (85 or 95) — present only on budget.threshold_crossed. |
triggering_request_id | The request that pushed the budget over the threshold/limit, when known. |
occurred_at | When the crossing was detected (RFC 3339 UTC). |
dedup_key | The 24-hour dedup stream key, "{budget_id}:{threshold_percent}" (see the dedup note above). |
data for provider.circuit_opened / provider.circuit_closed
json
{
"provider": "openai",
"account_id": "acct-3",
"consecutive_failures": 6,
"cooldown_until": "2026-07-08T12:40:00.000Z",
"occurred_at": "2026-07-08T12:34:56.789Z"
}| Field | Meaning |
|---|---|
provider / account_id | The upstream connection whose circuit changed state (alias/id only). |
consecutive_failures | Consecutive failures that tripped the breaker — present on circuit_opened. |
cooldown_until | When the breaker will next allow a probe request through — present on circuit_opened. |
open_for_ms | How long the circuit stayed open before closing — present on circuit_closed instead of consecutive_failures / cooldown_until. |
occurred_at | When the transition happened (RFC 3339 UTC). |
data for account.cooldown_started
json
{
"provider": "anthropic",
"account_id": "acct-1",
"retry_after_seconds": 30,
"cooldown_until": "2026-07-08T12:35:26.789Z",
"source": "retry_after",
"occurred_at": "2026-07-08T12:34:56.789Z"
}| Field | Meaning |
|---|---|
provider / account_id | The account put on cooldown. |
retry_after_seconds | The cooldown duration, in seconds. |
cooldown_until | The absolute time the cooldown lifts (RFC 3339 UTC). |
source | What applied the cooldown: retry_after (the upstream sent a Retry-After header) or circuit_breaker (the breaker's own cooldown). |
occurred_at | When the cooldown was applied. |
account.quota_exhausted carries the same provider / account_id / occurred_at fields without the cooldown-specific ones — the account is skipped, not cooled down, until the provider reports quota again.
data for provider.outage
json
{
"provider": "openai",
"severity": "major",
"error_count": 11,
"window_seconds": 60,
"first_error_at": "2026-07-08T12:33:56.789Z",
"last_error_at": "2026-07-08T12:34:56.789Z",
"dedup_key": "openai:major"
}| Field | Meaning |
|---|---|
provider | The provider whose error rate crossed the outage threshold. |
severity | minor (≥5 errors/60s) or major (≥10 errors/60s). |
error_count | Non-2xx/4xx upstream errors counted in the window. |
window_seconds | The aggregation window (always 60). |
first_error_at / last_error_at | Bounds of the counted errors (RFC 3339 UTC). |
dedup_key | The 10-minute dedup stream key, "{provider}:{severity}". |
data for key.* / user.* events
json
{
"action": "key.rotate",
"actor": "user:3",
"target": "key:12",
"org_id": null,
"at": "2026-07-08T12:34:56.789Z",
"details": { "label": "prod-ci" }
}| Field | Meaning |
|---|---|
action | The audit-log action string, e.g. key.create, key.rotate, key.delete, user.create, user.delete. |
actor | Who performed it — a stable subject identity (e.g. user:3). |
target | What it was done to — resource id/label (e.g. key:12); null if not applicable. |
org_id | Organization id in multi-tenant mode; null on the local profile. |
at | When the action was recorded (RFC 3339 UTC). |
details | Canonical JSON event detail — secrets already redacted, the same guarantee as the audit log itself. |
data for webhook.endpoint_auto_disabled
json
{
"endpoint_id": "ep_…",
"url": "https://example.com/hooks/merido",
"failing_since": "2026-07-03T12:34:56.789Z",
"disabled_at": "2026-07-08T12:34:56.789Z"
}| Field | Meaning |
|---|---|
endpoint_id | The auto-disabled endpoint's id. |
url | Its receiver URL, so you can identify it without a lookup. |
failing_since | When its failure streak started (RFC 3339 UTC). |
disabled_at | When it was auto-disabled (RFC 3339 UTC). |
data for report.weekly_digest
json
{
"period_start": "2026-07-01T00:00:00.000Z",
"period_end": "2026-07-08T00:00:00.000Z",
"total_usd_saved": 42.17,
"total_tokens_saved": 812345,
"top_opportunities": [
{
"title": "Route 'claude-3-opus' is silently losing its prompt cache — fix the prefix to save ~$18.40/mo.",
"est_monthly_usd": 18.4,
"confidence": "observed",
"kind": "fix_cache_miss",
"apply_action": null,
"apply_hint": "advice-only — apply manually"
}
],
"headline": "You saved $42.17 this period; $18.40/mo more available across 1 opportunity."
}| Field | Meaning |
|---|---|
period_start / period_end | The digest window's bounds. |
total_usd_saved / total_tokens_saved | Realized savings this period — already booked, not projected. |
top_opportunities | Up to 5 future opportunities, same shape as the dashboard Advisor's recommendations, sorted by est_monthly_usd descending. |
top_opportunities[].apply_action | A concrete action POST /api/advisor/apply can run, or null when the opportunity is advice-only. |
headline | A one-line, human-readable summary — good for a Slack/Discord digest post. |
data for spend.tracked
Opt-in, high volume. Unlike every other event type, spend.tracked batches: one envelope carries an array of per-request records, flushed every ~5 seconds or at 512 records, whichever comes first.
json
{
"events": [
{
"request_id": "req_…",
"occurred_at": "2026-07-08T12:34:56.789Z",
"provider": "openai",
"model": "gpt-4o",
"virtual_model": "merido/auto",
"status": 200,
"latency_ms": 842,
"tokens": { "prompt": 512, "completion": 128, "cache_read": 0, "cache_write": 0, "reasoning": 0, "total": 640 },
"cost": { "cost_usd": 0.0041, "charge_usd": 0.0041 },
"api_key_label": "prod-ci",
"api_key_id": 42,
"org_id": 9,
"account_id": 3,
"session_id": "sess-9",
"free_pool": false
}
],
"count": 1
}| Field | Meaning |
|---|---|
events | Array of per-request spend records — same field shapes as the request.* events above, minus the error object (every record here is a completed, non-error request). |
api_key_id / org_id / account_id | Numeric ids for the gateway key, owning org, and upstream account (null when not applicable). Stable primary keys — map a spend record to the key/org/account without parsing the display api_key_label. |
count | Number of records in this batch (events.length). |
Queue it, don't process it inline
spend.tracked is the highest-volume event type by design — a busy key can produce a batch every few seconds. Push each batch onto a queue (SQS, a local channel, whatever you already run) and process it out of band — the same "answer fast, process async" discipline as request.* events, but more important here given the volume.
Verifying signatures
Deliveries follow the Standard Webhooks spec, so off-the-shelf verification libraries work as-is. Each POST carries:
webhook-id: evt_0198f2d4-… same as the body's "id"
webhook-timestamp: 1751972096 unix seconds — when this attempt was sent
webhook-signature: v1,g0hM9SsE+OTPJ…= one or more space-separated signatures
content-type: application/json
user-agent: merido-webhooks/<version>The signature is computed over the string {id}.{timestamp}.{raw_body}:
webhook-signature: v1,<base64( HMAC-SHA256( secret_bytes, "{id}.{timestamp}.{raw_body}" ) )>where secret_bytes is the base64-decoded part of the secret after the whsec_ prefix. During a secret rotation, deliveries carry multiple space-separated signatures (old and new secret) — verify against each and accept if any matches, and rotation is zero-downtime.
Four rules for a correct verifier:
- Verify over the raw request body bytes. Never re-parse and re-serialize the JSON — any formatting difference breaks the HMAC.
- Compare in constant time (
timingSafeEqual/hmac.compare_digest). - Reject stale timestamps: refuse if
|now − webhook-timestamp|exceeds 5 minutes (replay protection). - Dedupe on
id. Delivery is at-least-once, and retries and manual redeliveries reuse the same eventid— process each id once.
Node.js
js
const crypto = require("node:crypto");
function verifyWebhook(secret, headers, rawBody) {
const id = headers["webhook-id"];
const ts = headers["webhook-timestamp"];
if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) return false; // 5-min tolerance
const key = Buffer.from(secret.slice("whsec_".length), "base64");
const expected = crypto
.createHmac("sha256", key)
.update(`${id}.${ts}.${rawBody}`) // rawBody: the exact bytes received
.digest("base64");
// Multiple space-separated signatures may be present during secret rotation.
return headers["webhook-signature"].split(" ").some((part) => {
const sig = part.split(",")[1] ?? "";
return (
sig.length === expected.length &&
crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected))
);
});
}Python
python
import base64, hashlib, hmac, time
def verify_webhook(secret: str, headers: dict, raw_body: bytes) -> bool:
msg_id = headers["webhook-id"]
ts = headers["webhook-timestamp"]
if abs(time.time() - int(ts)) > 300: # 5-min tolerance
return False
key = base64.b64decode(secret.removeprefix("whsec_"))
signed = f"{msg_id}.{ts}.".encode() + raw_body # raw_body: exact bytes received
expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()
# Multiple space-separated signatures may be present during secret rotation.
return any(
hmac.compare_digest(part.partition(",")[2], expected)
for part in headers["webhook-signature"].split()
)Test vector
Feed this fixed input through your verifier to check it end-to-end (it's the Standard Webhooks reference vector — skip the timestamp-tolerance check for this one, the timestamp is old):
secret whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw
id msg_p5jXN8AQM9LWM0D4loKWxJek
timestamp 1614265330
body {"test": 2432232314}
expected v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE=Delivery: retries, ordering, idempotency
- At-least-once. An event may be delivered more than once (retries, manual redelivery). Dedupe on the envelope
id. - No ordering guarantee. Events can arrive out of order — use
created_at/occurred_atif sequence matters to you. - Success = any
2xxwithin 15 seconds. Anything else (non-2xx, timeout, connection error) counts as a failed attempt. - Backoff schedule. After the immediate attempt fails, delivery is retried at +5 s, +5 m, +30 m, +2 h, +5 h, and +10 h; when the last retry fails the delivery is marked dead.
- Delivery log. The dashboard shows every delivery's attempts, status, and last error. Failed and dead deliveries can be redelivered manually (button, or
POST /api/webhooks/deliveries/{id}/redeliver).
Answer fast, process async
Return 2xx as soon as you've stored the event, and do the real work out-of-band. A receiver that does slow work inline risks tripping the 15-second timeout and burning the retry schedule on deliveries that actually arrived.
Metrics
Two Prometheus counters track the delivery pipeline itself — useful for alerting on the pipeline independently of any one endpoint's delivery log (see Prometheus metrics for the full exposition):
| Counter | Meaning |
|---|---|
merido_webhook_deliveries_total{status} | Delivery attempts by outcome (delivered / failed / dead). A rising failed/dead rate for one endpoint means its receiver is unhealthy. |
merido_webhook_endpoints_auto_disabled_total | Endpoints auto-disabled after 5 continuous days of failure (see webhook.endpoint_auto_disabled above). Non-zero means a receiver went dark long enough to get cut off. |
Formats
| Format | Body | Use for |
|---|---|---|
standard | The signed JSON envelope above. | Machine consumption — your own services, alerting pipelines, workflow engines. |
slack | The same event rendered as a one-line text message: {"text": "…"}. | Posting straight to a Slack incoming webhook. |
discord | The same event as {"content": "…"}. | Posting straight to a Discord webhook. |
The chat formats replace the envelope with a rendered message (there is no standard payload to verify inside it), but deliveries still carry the standard webhook-* headers. Use them for human notification channels; use standard whenever software consumes the event.
Managing endpoints via the API
Everything the dashboard Webhooks page does is available on the admin API:
| Endpoint | Methods | Purpose |
|---|---|---|
/api/webhooks/endpoints | GET, POST | List / create endpoints (URL, event types, format). The signing secret is returned once in the create response. |
/api/webhooks/endpoints/{id} | PUT, DELETE | Update / delete an endpoint. |
/api/webhooks/endpoints/{id}/test | POST | Deliver a signed test event. |
/api/webhooks/endpoints/{id}/rotate-secret | POST | Rotate the signing secret (new secret shown once; old + new signatures sent during the rotation window). |
/api/webhooks/deliveries | GET | Delivery log — filter by endpoint_id, status, limit. |
/api/webhooks/deliveries/{id}/redeliver | POST | Re-queue a failed or dead delivery. |
Security & retention
- URL validation. Endpoint URLs are validated on create/update: private, loopback, link-local, and cloud-metadata addresses are rejected (SSRF protection). Cloud deployments require
https://URLs; plainhttp://is allowed only on the local profile for dev receivers. - Secrets. Shown once on create/rotate; verify every delivery with them — an unverified webhook receiver will happily process forged events.
- Payload hygiene. No prompt/response content, no key material, no credentials — labels and ids only (see the payload reference above).
- Retention. Delivered rows are pruned from the delivery log after 7 days, dead deliveries after 30 days. Redeliver anything you care about before it ages out.
Legacy alert webhook
The pre-existing alert_webhook_url / alert_webhook_format environment knobs keep working unchanged — a simple, unsigned, fire-and-forget alert POST. New integrations should use webhook endpoints instead: they add signing, retries, a delivery log, per-event subscriptions, and multi-endpoint support.
Related
- Monitoring — the Live feed, request Logs, and Prometheus metrics the data-plane events and Metrics counters point back into.
- Budgets & spend caps — what triggers
request.budget_rejectedand thebudget.*events. - Audit log & SIEM export — what the
key.*/user.*lifecycle events bridge from. - Usage & the Advisor — the recommendations
report.weekly_digestsummarizes. - API endpoints — the full control-plane API surface.