Production that runs itself.

The autonomous system for production that diagnoses, fixes, and prevents issues across your environment.

Today’s issues3

Checkout failing 0.9% of requests since deploy v3.2.1

ISS-2

0.9% of checkout requests have failed since deploy v3.2.1 rolled out at 02:00, up from 0.2% before it. That is about 1,800 failed checkout requests an hour. A rollback PR is open. This signal was recorded in checkout-svc, with supporting evidence available for review before the next action.

Three web apps exposed to a Next.js middleware bypass

ISS-3

web-gateway, storefront and admin run next 14.1.0, which CVE-2027-1042 affects. A crafted header can skip the middleware auth check on account and admin routes. Edge logs from the last 7 days show no requests carrying it. This signal was recorded in web-gateway, with supporting evidence available for review before the next action.

SharePoint site picker fails for orgs without Graph credentials

ISS-6

The SharePoint site picker on echelon-api returns HTTP 500 to orgs that never stored Microsoft Graph credentials. Four failures in 90 seconds tripped the production 5xx monitor, and 34 requests failed across 3 orgs over the prior 7 days. This signal was recorded in echelon-api, with supporting evidence available for review before the next action.

Today’s improvements2

Checkout autoscaler near its replica max

IMP-16

At last Friday’s peak, checkout-svc ran 11 of its 12 maximum replicas at 74% CPU. A peak about 10% higher would pin it at the max. The proposed change targets checkout-svc and includes a reviewable implementation plan with projected impact.

Staging cluster overprovisioned by 3.2×

IMP-1

About $840 a month at on-demand list price goes to idle staging nodes. The cluster runs 16 m5.xlarge nodes, and its 30-day peak used the equivalent of 5. The proposed change targets staging-cluster and includes a reviewable implementation plan with projected impact.

Checkout failing 0.9% of requests since deploy v3.2.1

HighProduction22m agocheckout-svc4 alerts

Checkout 5xx rose to 0.9% after deploy v3.2.1 restarted the prod-db read-replica at 02:00. Pooled connections to the old replica time out and retry while request volume holds steady.

Related events

Sparse · 4 events, last 22m ago

Time	Source	Event
22m	Service health check	Flagged a 5xx rise on checkout-svc and opened ISS-2
24m	Datadog	[P1] checkout-svc · 5xx ratio above 0.5% for 5m
41m	Kubernetes	checkout-svc scaled 8 → 6 replicas; prod-db read replica restarted
42m	GitHub	Deploy v3.2.1 rolled out to production

Causal graph

Deploy v3.2.1 restarted the prod-db read replica at 02:00, and checkout-svc kept stale pooled connections to the old replica. Requests on those connections time out and retry, which pushes 5xx to 0.9%. The replica floor cut from 8 to 6 removed headroom but did not start the errors.

Findings

Changes

Drafting a fix…

Trusted by engineering teams at

Antimetal automates production engineering so your team can focus on building.

Models your environment
Antimetal builds a World Model from your code, infrastructure, and observability data, refining it with every change, investigation, and team decision.
01World model
Reading your code…
IssueImprovementHealthy
Building to now · Cloning 42 repositories from acme
Triages incoming signals
Antimetal ingests tickets, alerts, deployments, and other signals, grouping related events and filtering out noise.
02Signal triage
TimeSourceEventLinked findingSeverity
22m+5Checkout failing 0.9% of requests since deploy v3.2.16 related events · 6 sources · Turned into an IssueISS-2
42mGitHubDeploy v3.2.1 rolled out to productioncheckout-svc · PR #3761
Shows what needs attention
Antimetal distills those signals into a prioritized list of Issues to address and Improvements to make your system healthier.
03Priorities
Today’s issues9
Checkout fails during deploys when database connections run out
ISS-11
Each checkout-worker rollout pushes prod-db past its 400 max_connections for up to 90 seconds.
+1−1Steady · 4 events
Checkout failing 0.9% of requests since deploy v3.2.1
ISS-2
Checkout 5xx rose after deploy v3.2.1 rolled out at 02:00. Request volume holds steady.
+1−1Sparse · 3 events
Three web apps exposed to a Next.js middleware bypass
ISS-3
web-gateway, storefront and admin run next 14.1.0, which CVE-2027-1042 affects.
+1−1Sparse · 1 event
Today’s improvements10
Checkout autoscaler near its replica max
IMP-16
At last Friday’s peak, checkout-svc ran 11 of its 12 maximum replicas at 74% CPU.
+1−1Steady · 7 events
No p95 latency monitor on search-api
IMP-4
Peer services already have latency coverage that search-api still lacks.
+1−1Sparse · 2 events
Fixes and verifies
Antimetal generates code and configuration fixes for review, then verifies approved changes in production to confirm the problem is resolved and the fix holds.
04Fix & verify
+1−1
Recycle checkout-svc database connections after 60 seconds
In a single PR, recycle checkout-svc database connections after 60 seconds.
services/checkout/database.ts+1−1
- const pool = new Pool({ connectionString });+ const pool = new Pool({ connectionString, maxLifetimeSeconds: 60 });
CIChecks runningContinuous integration monitoring
buildRunning
lintRunning
unit-testsRunning
e2e-smokeRunning
4 running
Antimetal is watching. Nothing to do until they report.
View checks on GitHub
Apply fix
World modelObserver added automatically
web-gatewaycheckout-svcprod-db
OBS-12Checkout 5xx over 0.3%Waiting
For 48h after merge. Reopens ISS-2 if it trips.

See Antimetal in action.

Get Started

“Antimetal helps us stay focused on product by resolving issues as they come in.”
Mihir PenugondaHead of Engineering, Pocket
45%
fewer recurring incidents

Patrol finds what your alerts don’t.

Powered by the Antimetal World Model, Patrol agents continuously investigate your system to find problems, improve performance and reliability, and reduce costs.

Put Patrol to work

LedgerCloud & LLM costs412 checks
Find idle capacity and spend that isn’t earning its keep.
Capacity in use22% idle
IMP-1Improvement
Right-size idle staging capacity
3.2× more capacity than the workload needs.
~$840/mo in projected savings
Scope
Cloud accounts

Understand how your environment works.

Replica rollout mergedDeployment diff

Database replica restartedRollout events

Checkout timeouts risingError traces

Likely cause · 3 systemsStale database connections

ISS-2checkout-svc

Connects the dots.

Links code changes, infrastructure events, and runtime behavior to show what’s behind a problem.

Recycle stale database connectionsPR #3775 · checkout-svc

PTPayments team

CODEOWNERSOwner

SIShreyas

Recent GitHub reviewsReviewer

Review requested2 people

Knows who owns what.

Uses code ownership and review history to identify service owners and the right people to review a fix.

Your team’s decision

Keep checkout at eight replicas. We need the capacity for peak traffic.

World Model updatedcheckout-svc

Improvement · Next patrol runReduce idle capacity. Preserve checkout.

minReplicas 8Your team’s constraint

Remembers your decisions.

Keeps past investigations, fixes, and team decisions in the World Model to guide future recommendations.

Your system, at your command.

Ask questions. Direct the work.

Ask Antimetal to investigate a problem, explain what changed, or generate a fix. Add context and steer it as it works.

#production-alerts

Shreyas Iyer10:42 AM

Checkout errors are rising. @Antimetal can you investigate?

Antimetal APP10:43 AM

Stale database connections caused this. I’ve opened a fix for review.

Here’s the fix: #3775 · Recycle stale connections

Works where you work.

Antimetal works directly within your team’s conversations in Slack or Microsoft Teams to help resolve issues and make decisions faster.

“Compared to our internal bot, Antimetal is more comprehensive and connects the dots across our data sources.”
Armando CanalsApplied ML, Finch Legal
82%
less time spent investigating issues

“You may be able to build software rapidly, but if people can’t depend on it, if you can’t figure out what breaks, well, people aren’t going to use it.”

Ben LutchVP of Engineering

Put Antimetal to work.

Antimetal works alongside your team to understand your environment, keep it running, and make it better.

Try Now

From the research log

View all articles

Time

Source

Event

22m

Service health check

Flagged a 5xx rise on checkout-svc and opened ISS-2

24m

Datadog

[P1] checkout-svc · 5xx ratio above 0.5% for 5m

41m

Kubernetes

checkout-svc scaled 8 → 6 replicas; prod-db read replica restarted

42m

GitHub

Deploy v3.2.1 rolled out to production