> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oncoeverything.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring and recovery

> Alarms, logs, the budget, database restores, and running a command inside AWS.

## Alarms

CloudWatch alarms email [hello@oncoeverything.com](mailto:hello@oncoeverything.com) through SNS.

| Alarm | Fires when |
| - | - |
| `oncoeverything-api-5xx` | The API returns more than 5 server errors in 5 minutes |
| `oncoeverything-database-cpu` | Database CPU is above 80% for 15 minutes |
| `oncoeverything-database-storage` | Database free storage drops below 2 GB |
| `oncoeverything-database-connections` | More than 60 database connections |

The monthly budget is \$120. It emails when the forecast passes 80%, and when actual spend passes 100%.

## Logs

The API logs to CloudWatch in the log group `/ecs/oncoeverything-api`. Follow them live:

```bash theme={null}
AWS_PROFILE=onco aws logs tail /ecs/oncoeverything-api --follow
```

Web app logs are in the Vercel dashboard, under the project's Logs tab.

## Health

```bash theme={null}
curl https://d2u3osjg8h1sl1.cloudfront.net/health
```

A 200 means the API is up and its migrations ran.

## Run a command inside AWS

The database accepts connections only from inside the VPC. `infra/run-task.sh` starts a one-off task there, on the same image the API runs, and prints its output.

```bash theme={null}
AWS_PROFILE=onco infra/run-task.sh bun run apps/api/scripts/sql.ts "select count(*) from requests"
```

Use it for seeds and read-only queries.

## Restore the database

RDS keeps backups for 14 days. You can restore to any minute in that window, to a new instance.

**1. Restore to a new instance**

```bash theme={null}
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier oncoeverything \
  --target-db-instance-identifier oncoeverything-restore-test \
  --restore-time "<UTC time, e.g. 2026-09-23T14:00:00Z>" \
  --db-subnet-group-name oncoeverything \
  --no-publicly-accessible
```

It takes about 12 minutes.

**2. Check it**

Run a count with `run-task.sh`, pointed at the new instance's address.

**3. Delete it, or switch to it**

For a drill, delete the new instance. For a real recovery, point the API at it. Then import it into Terraform.

Record each drill in `docs/operations/restore-drill.md`. The last one was on September 18, 2026.

## Common problems

| You see | Look at |
| - | - |
| Nobody gets email | SES sandbox status, and the API logs for `[mail]` errors |
| Sign-in links do not arrive | The same. Also check spam. |
| **Buy this report** opens a form | The Stripe keys are missing from the app secret |
| Downloads fail | The CloudFront key in the app secret, and `CLOUDFRONT_KEY_PAIR_ID` on the task |
| A copy never arrives | The API logs for the delivery runner. The sweep retries every five minutes. |
| The site shows old content after a Vercel variable change | Push a commit. Variables apply only on a new deploy. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.