Monitoring and recovery

Alarms

CloudWatch alarms email hello@oncoeverything.com through SNS.

Alarm	Fires when
`oncoeverything-api-5xx`	The API returns more than 5 server errors in 5 minutes
`oncoeverything-database-cpu`	Database CPU is above 80% for 15 minutes
`oncoeverything-database-storage`	Database free storage drops below 2 GB
`oncoeverything-database-connections`	More than 60 database connections

The monthly budget is $120. It emails when the forecast passes 80%, and when actual spend passes 100%.

Logs

The API logs to CloudWatch in the log group /ecs/oncoeverything-api. Follow them live:

AWS_PROFILE=onco aws logs tail /ecs/oncoeverything-api --follow

Web app logs are in the Vercel dashboard, under the project’s Logs tab.

Health

curl https://d2u3osjg8h1sl1.cloudfront.net/health

A 200 means the API is up and its migrations ran.

Run a command inside AWS

The database accepts connections only from inside the VPC. infra/run-task.sh starts a one-off task there, on the same image the API runs, and prints its output.

AWS_PROFILE=onco infra/run-task.sh bun run apps/api/scripts/sql.ts "select count(*) from requests"

Use it for seeds and read-only queries.

Restore the database

RDS keeps backups for 14 days. You can restore to any minute in that window, to a new instance.

Restore to a new instance

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier oncoeverything \
  --target-db-instance-identifier oncoeverything-restore-test \
  --restore-time "<UTC time, e.g. 2026-09-23T14:00:00Z>" \
  --db-subnet-group-name oncoeverything \
  --no-publicly-accessible

It takes about 12 minutes.

Check it

Run a count with run-task.sh, pointed at the new instance’s address.

Delete it, or switch to it

For a drill, delete the new instance. For a real recovery, point the API at it. Then import it into Terraform.

Record each drill in docs/operations/restore-drill.md. The last one was on September 18, 2026.

Common problems

You see	Look at
Nobody gets email	SES sandbox status, and the API logs for `[mail]` errors
Sign-in links do not arrive	The same. Also check spam.
Buy this report opens a form	The Stripe keys are missing from the app secret
Downloads fail	The CloudFront key in the app secret, and `CLOUDFRONT_KEY_PAIR_ID` on the task
A copy never arrives	The API logs for the delivery runner. The sweep retries every five minutes.
The site shows old content after a Vercel variable change	Push a commit. Variables apply only on a new deploy.

Start here

Requests

Reports

Billing

People

Site

Architecture

Operations

Reference