# Crash Recovery

EntryTarget provides a built-in mechanism to reconcile transactions after an unexpected server crash or restart.

## The Problem

If the server crashes, some transactions may have been committed to the database but the HTTP response was never delivered to your system. If the server is down long enough for the idempotency window to expire, you would have no way to know which transactions were persisted.

## The Solution

On every startup, **before accepting any HTTP traffic**, the system identifies the last committed batch and copies it to a recovery table. This data persists until the next restart — no time limit.

### Why the Last Batch?

All records in the same committed batch share the same database timestamp (generated by PostgreSQL inside the transaction). The system selects all records with the latest timestamp.

Every earlier batch must have received a response — the server had to be alive long enough to commit a subsequent batch. Only the last batch could have been persisted but not delivered before the process stopped.

## Using the Recovery Endpoint

```bash
curl http://<host>:8080/recovery \
  -H "X-Api-Key: ak_..." \
  -H "X-Api-Secret: sk_..."
```

### Response

```json
{
  "records": [
    {
      "idempotency_key": "payment-abc-123",
      "request": { ... },
      "response": { ... },
      "created_at": "2026-01-15T10:30:00.123Z"
    }
  ],
  "count": 1
}
```

### Interpreting the Response

| Result                          | Meaning                                            |
| ------------------------------- | -------------------------------------------------- |
| Your key is in the response     | Transaction **was** persisted. Do **not** retry.   |
| Your key is NOT in the response | Transaction was **not** persisted. Safe to retry.  |
| Empty response (`count: 0`)     | Previous shutdown was clean. Nothing to reconcile. |

## Reconciliation Flow

After a server restart, your reconciliation process should:

1. **Check `GET /recovery`** — see if any of your pending transactions are in the last batch
2. **For present keys** — mark as successful, do not retry
3. **For absent keys** — the transaction was not persisted; safe to retry with the same idempotency key
4. **Empty response** — the previous shutdown was clean; no reconciliation needed

## Recovery vs. Idempotency

| Feature         | `GET /idempotency/:key`         | `GET /recovery`                       |
| --------------- | ------------------------------- | ------------------------------------- |
| **Scope**       | Single key lookup               | All records from last batch           |
| **Time limit**  | Configured TTL (e.g., 1 hour)   | No time limit (until next restart)    |
| **When to use** | Normal operation, quick lookups | After crash / extended downtime       |
| **Returns**     | Single record or 404            | All records from last committed batch |

### When to Use Which

* **During normal operation:** Use `GET /idempotency/:key` to check individual transaction status
* **After a crash with short downtime (within TTL):** Either endpoint works
* **After a crash with long downtime (TTL expired):** Use `GET /recovery` — idempotency records may have been cleaned up

## Edge Cases

### Multiple Batches with Same Timestamp

If two batches happen to share the same wall-clock second (rare but possible), both are included in the recovery data. This is conservative and safe — you may see a record you already confirmed, but a false positive is harmless. A false negative would be dangerous.

### Clean Shutdown

If the server was shut down gracefully (SIGTERM), all in-flight batches complete before exit. The recovery endpoint will show the last committed batch, which was successfully delivered. The response may be non-empty, but all records in it were already acknowledged.

### First Startup (Empty Database)

On the first-ever startup, there are no idempotency records. The recovery table stays empty, and the startup log indicates a clean state.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://entrytarget.gitbook.io/docs/security-and-integrity/crash-recovery.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.