> ## Documentation Index > Fetch the complete documentation index at: https://docs.vexlo.ai/llms.txt > Use this file to discover all available pages before exploring further. # Retention > Shape is bounded by a day count and values by a byte budget. From about 1.24 traces per second the default 20 GiB budget runs dry before the 7-day shape count, and a trace keeps its shape long after the bytes behind its slots are reclaimed. Vexlo bounds shape and values on two separate clocks in two different units. Shape (traces, steps, attempts, tags) is kept for a number of days, 7 by default. Values (the captured bytes behind each slot) are kept against a byte budget, 20 GiB by default. Nothing ties the two together, so they do not run out together. Values go first, always. The byte budget only ever removes values, and a trace keeps its shape until its own day count passes, so a trace can lose every value it was captured with and still answer every other question about itself. The default configuration surprises people, so here is the arithmetic of it. The default 20 GiB covers a 7-day shape count only below about 1.24 traces per second: `vexlo check` prints 168h58m of value retention at 1.23 traces/s and 167h37m at 1.24. Above that rate the budget runs dry first, and the check exits 1 rather than let the pair pass. At the 2 traces/s it assumes by default, the budget holds 103h55m, a little over four days against a seven-day shape count. At the 30 traces/s of a busy service it holds 6h55m, and then a trace from yesterday is still searchable, still has every step, every timing and every status, and answers `expired` for the input you wanted to read. A reader who was told "7-day retention" and goes looking for yesterday's inputs is right to read that as data loss. Flags are spelled with a single dash, as `-h` prints them; the binary accepts both spellings. ## The two clocks | What | Bound | Default | Flag | | - | - | - | - | | Shape: traces, steps, attempts, tags | Day count, from the arrival of the trace's first record | 7 days | `-shape-retention-days` | | Values: the bytes behind every slot | Byte budget across the whole value store | 20 GiB (`21474836480`) | `-value-budget-bytes` | | Function index: observed callers and callees per function | Day count since that pair was last observed | 90 days | none | `vexlo serve` prints the first two on startup, and they are the only retention settings it prints: ```text theme={"system"} time=2026-11-06T22:48:22.667-08:00 level=INFO msg="serve: starting" addr=127.0.0.1:18422 requires_tls=false no_auth=false header_trust=false db_path=data-default/vexlo.db value_store_dir=data-default/values shape_retention_days=7 value_budget_bytes=21474836480 ... 3 lines omitted ``` One 10-minute tick runs three passes, each bounded to 1,000 rows per table or 1,000 value files so ingest is never stalled behind it. The shape sweep deletes a trace and its children once its first record reached the server more than the day count ago, measured on the server's own clock rather than on the `started_at` the SDK declared: the trace stops being searchable and its id returns `404`. The same pass applies the function index's own 90 days. It takes one chunk per tick rather than looping until a table is drained, so at 1,000 rows a table it clears about 26 traces a tick at the 38 slot rows the captured trace below carries, and a trace past its day count stays searchable until a tick reaches it. The byte-budget pass removes whole oldest hour segments of the value store until usage is back under the budget, never a partial segment and never a single file. A value's segment is the hour its trace was first seen, so one trace's values go together. The third pass deletes value files no live slot row points at any more, skipping any file written less than an hour ago: that is how the values of a trace the shape sweep has already deleted leave the disk. Both of the passes that free value bytes run on that one tick, so between two ticks nothing frees any, and "oldest first" describes the byte-budget pass rather than the whole story. Ingest checks the same byte budget on every value it commits, and once its own measurement has reached that budget it commits the slot and discards the bytes. Between two ticks, then, the values being discarded are the newest ones. That measurement is a cached directory walk refreshed at most once every 5 seconds, so the discarding can also outlast by that much a pass that has already freed the space. The 1,000-file bound is what the byte-budget pass runs into first. It takes a segment whole or not at all, so once the oldest hour holds more than 1,000 value files that pass frees nothing: it reports the segment as stuck, logs a warning, and leaves the store over budget. No flag moves the bound. The trace captured further down put 28 value files on disk by itself, so an hour holding more than 35 traces like it is already past the bound, and 2 traces/s is 7,200 traces an hour. What a reader meets on such a server is not oldest-first reclamation. The store stays over budget, ingest discards the bytes of every value it commits, and `GET /v1/health` answers `503` with `mode: "shape_only"`. The pass that can still thin a stuck segment is the unreferenced one, and it follows the day count rather than the budget, because a value file stops being referenced when the shape sweep deletes the trace that held it. It deletes at most 1,000 files a tick, 144,000 a day, against the 4.8 million a day that 2 traces/s at 28 files a trace would write. Ingest refuses a `trace-open` whose declared `started_at` is older than the configured shape retention, so the same day count also bounds how late a backfilled trace can arrive. The byte budget is one number for the whole server, not one per team. One team's traffic spike shortens every other team's value horizon on the same instance. ## What the byte budget buys `vexlo check` resolves the configuration `vexlo serve` would run with, then prints the arithmetic. It opens neither the database nor the value store, so it has no traffic to measure and takes the rate as a flag. `-expected-write-rate` defaults to 2 traces/s: ```text theme={"system"} $ vexlo check -db-path data/vexlo.db -value-store-dir data/values config: OK api_addr: 127.0.0.1:8080 (requires_tls=false) no_auth: false header_trust: false db_path: data/vexlo.db value_store_dir: data/values shape_retention_days: 7 value_budget_bytes: 21474836480 free disk (value_store_dir): 94841954304 bytes free, 21474836480 bytes configured budget retention arithmetic (at 2 traces/s): shape bytes/trace: 2300 value bytes/trace: 28700 total bytes/trace: 31000 configured shape retention: 168h0m0s effective value retention: 103h55m26.071080139s OVERCOMMITTED: values would run out (103h55m26.071080139s) before shape retention expires (168h0m0s) — every trace would lose its values well before it ages out of search vexlo: check: value store is overcommitted against the configured shape retention (-shape-retention-days/-value-budget-bytes/-expected-write-rate); lower -shape-retention-days, raise -value-budget-bytes, or pass -allow-overcommit to proceed anyway ``` The effective figure is the budget divided by the value bytes one trace costs times the rate: `21474836480 / (28700 x 2)` seconds, the 103h55m printed above against a configured 168h. Of the 31,000 bytes a fully valued trace costs, 28,700 are its values, which is why the budget is denominated in bytes and why value sampling is the lever that moves it. Pass a production rate and the gap widens: ```text theme={"system"} $ vexlo check -db-path data/vexlo.db -value-store-dir data/values -expected-write-rate 30 ... 9 lines omitted retention arithmetic (at 30 traces/s): shape bytes/trace: 2300 value bytes/trace: 28700 total bytes/trace: 31000 configured shape retention: 168h0m0s effective value retention: 6h55m41.738072009s OVERCOMMITTED: values would run out (6h55m41.738072009s) before shape retention expires (168h0m0s) — every trace would lose its values well before it ages out of search ... 1 line omitted ``` Just under 7 hours of values against a week of shape. Five things about that number matter before you plan against it: * **The per-trace figure is an assumption, not a measurement.** The 2,300 and 28,700 byte figures are fixed constants in the binary, a worked example of a 40-step trace with 2.2 valued slots per step and a 900 B mean value after gzip and deduplication. `vexlo check` never measures your traffic, and it has no flag for a measured average. * **The rate is whatever you pass.** `-expected-write-rate`'s own help text says `vexlo check` has no live traffic to measure it from, so the horizon is only as good as the rate you supply. * **The arithmetic assumes every trace keeps its values.** There is no sampling input. Lowering `value_sample_rate` through a [capture policy](https://docs.vexlo.ai/server/capture-policy.md) stretches the real horizon past what the figure says, and nothing reconciles the two for you. It stretches it by excluding traces, and an excluded trace's `ok` attempts read `expired` on a fetch like a reclaimed value. * **The arithmetic counts bytes and never files.** `vexlo check` knows nothing about the 1,000-file bound each reclaim pass runs under, and on a server whose hour segments hold more than that, the bound is what decides what happens when the budget fills. * **`vexlo serve` applies no overcommit check of its own.** It starts on the configuration `vexlo check` rejects, and prints nothing about it beyond the startup line above. `GET /v1/health` does not close that gap. Its `value_store.effective_retention_estimate_hours` field reads `0`, because the route has no write-rate input to compute an estimate from, and its `shape_store.effective_retention_days` field reads `7` whatever `-shape-retention-days` is set to. The run below is a server started with `-shape-retention-days 3650`: ```json theme={"system"} { "status": "healthy", "mode": "normal", "shape_store": { "disk_used_bytes": 4096, "effective_retention_days": 7 }, "value_store": { "bytes_used": 1678, "byte_budget": 21474836480, "effective_retention_estimate_hours": 0 }, "stubs_swept_24h": 0, "services": [ { "service": "delivery-service", "records_lost_24h": 0, "records_shed_24h": 0, "last_seen_at": "2026-11-07T06:48:24.717Z" } ] } ``` `bytes_used` is a fresh walk of the value store on every call, and `byte_budget` is the configured flag echoed back. Neither retention field is computed from anything, so `vexlo check` is the only place either horizon is worked out. ## What a read returns once the bytes are reclaimed The byte budget reclaims bytes. It never touches the shape store, and it never rewrites a slot's recorded state. Those two facts together are what a reader runs into. Below is one step of a trace read twice from one server storage directory, before and after the byte-budget pass reclaimed that trace's 28 value files. Nothing was reingested and no shape write happened in between. Forcing that pass took one change, and it is worth naming rather than implying a week passed: the server was stopped and restarted on the same storage directory with `-value-budget-bytes 1`, which puts every byte already on disk over the budget, and the first retention tick ten minutes later removed the lot. The step below is the one a debugger would open, the one whose input produced the error: ```json Values before theme={"system"} { "values": [ { "path": "AACAAAI", "slot": "error", "state": "stored", "encoding": "json", "value": "{\"message\":\"no eligible candidate in list\",\"stack\":[\"select_eligible_dasher.py:50\"]}", "bytes_count": 84, "truncated": false }, { "path": "AACAAAI", "slot": "input", "state": "stored", "encoding": "json", "value": "{\"candidate_ids\":[\"D-101\",\"D-102\",\"D-103\",\"D-104\",\"D-105\"],\"radius_km\":5,\"requested_time\":\"19:30\",\"zone\":\"SF-94107\"}", "bytes_count": 116, "truncated": false } ], "unknown_paths": [] } ``` ```json Values after theme={"system"} { "values": [ { "path": "AACAAAI", "slot": "error", "state": "expired" }, { "path": "AACAAAI", "slot": "input", "state": "expired" } ], "unknown_paths": [] } ``` The shape read of that same step is byte-identical across the two reads. It still reports `stored`, the size the bytes were, their encoding, and the 64-code-point preview: ```json theme={"system"} { "path": "AACAAAI", "parent_path": "AACA", "depth": 2, "name": "delivery.dispatch.SelectEligibleDasher", "parent_name": "delivery.dispatch.AssignDasher", "attempt_grouping": "single", "duration_us": 3020, "attempts": [ { "attempt": 0, "started_us": 118420, "duration_us": 3020, "status": "error", "error_class": "builtins.ValueError", "slots": { "error": { "state": "stored", "bytes_count": 84, "truncated": false, "encoding": "json", "preview": "{\"message\":\"no eligible candidate in list\",\"stack\":[\"select_elig" }, "input": { "state": "stored", "bytes_count": 116, "truncated": false, "encoding": "json", "preview": "{\"candidate_ids\":[\"D-101\",\"D-102\",\"D-103\",\"D-104\",\"D-105\"],\"radi" } } } ] } ``` That is not a stale cache. A trace-detail read answers from the shape store alone and never looks in the value store, and `preview` was computed at capture time and travels on the `step-attempt` record rather than with the bytes. So the state on a shape read is the state the SDK reported, and `GET /v1/traces/{id}/values` is the one read that reports `expired` at all. One other route reaches the bytes, and it reports their absence differently. `GET /v1/functions/examples?include=values` reads the same value store, and on a miss it returns that example's handle as the shape store recorded it, with no `value` field beside it. It never answers `expired`. Below is the same step read as one example of its own function, either side of one byte-budget pass, from a second run of the scenario above on its own port and its own storage directory. The first read carries no `preview`, because the route clears it wherever it sets `value`; the `output` handle is empty on both because that step has no output slot. ```json Example before theme={"system"} { "function": { "service": "delivery-service", "name": "delivery.dispatch.SelectEligibleDasher" }, "examples": [ { "trace_id": "0f1a2b3c4d5e6f708192a3b4c5d6e7f8", "path": "AACAAAI", "attempt": 0, "started_at": "2026-11-07T07:32:32.139Z", "status": "error", "input": { "state": "stored", "bytes_count": 116, "truncated": false, "encoding": "json", "value": "{\"candidate_ids\":[\"D-101\",\"D-102\",\"D-103\",\"D-104\",\"D-105\"],\"radius_km\":5,\"requested_time\":\"19:30\",\"zone\":\"SF-94107\"}" }, "output": { "state": "", "truncated": false } } ] } ``` ```json Example after theme={"system"} { "function": { "service": "delivery-service", "name": "delivery.dispatch.SelectEligibleDasher" }, "examples": [ { "trace_id": "0f1a2b3c4d5e6f708192a3b4c5d6e7f8", "path": "AACAAAI", "attempt": 0, "started_at": "2026-11-07T07:32:32.139Z", "status": "error", "input": { "state": "stored", "bytes_count": 116, "truncated": false, "encoding": "json", "preview": "{\"candidate_ids\":[\"D-101\",\"D-102\",\"D-103\",\"D-104\",\"D-105\"],\"radi" }, "output": { "state": "", "truncated": false } } ] } ``` The `state` is the same word in both reads. On that route the missing `value` field is the only signal, so do not read a handle's `state` there as an answer about the bytes. Plan on all of that: shape says `stored` until the day count deletes the step, and a fetch through the values route is the only way to learn whether bytes are there. ## `expired` and `missing` Both are handle states a reader sees only at read time; neither is ever emitted by the SDK. They describe different events: | State | What it asserts | | - | - | | `expired` | Bytes were stored and the byte budget reclaimed them. | | `missing` | A value record was expected and never arrived, so no bytes ever reached this server. | Telling them apart takes a store that records a value record's arrival. The shipped SQLite store keeps no such record: the bytes are the only evidence a slot ever had any. Every slot row carrying a content id is read against the value store, and a miss resolves to `expired` whatever state the SDK reported, so this store never returns `missing` at all. Five different events reach a reader as `expired`, and only the first is the one the table above describes: 1. **The byte budget reclaimed bytes that were there.** This one leaves no trail. `GET /v1/health` reports the value store's current `bytes_used` against its `byte_budget` and keeps no record of what a pass reclaimed. On the first of the two runs behind this page, that route answered `200` with `mode: "normal"` and `bytes_used: 0` seconds after the pass had reclaimed all 28 of the trace's value files. 2. **A `value` record never arrived**, because its export failed for good after the `step-attempt` record had already shipped. No wire state records that, since a record is written once and never revised. The undelivered record is counted as loss, by the SDK in that trace's own `records_lost` or by the relay daemon in the per-service `records_lost_24h`, and neither count says which slot it was: `records_lost > 0` does not imply a missing step, attempt or value. 3. **The value arrived while the value store was already over budget**, so ingest committed the slot and discarded the bytes. Shape keeps being ingested under that pressure, and this is the one of the five the API names while it is happening: `GET /v1/health` answers `503` with `mode: "shape_only"`, captured here from a server whose budget was set to one byte. The one-byte budget is a way to force it on demand, not what makes it reachable. Ingest tests the same byte budget the byte-budget pass works against, so a spike that fills the budget between two ticks discards values for the rest of that window, and the traces reading `expired` afterwards are the newest ones rather than the oldest. 4. **No `value` record was ever owed.** Two things end there. The sampling roll excluded the trace, in which case its root attempt ships `sampled_out` and its `ok` non-root attempts keep the `pending` they shipped with, while any `error` attempt on that same trace keeps its values whatever the rate is. Or the trace's root never returned, and the SDK holds a trace's values until its root closes, so they were never shipped and never will be. 5. **The outbox evicted the bytes before the `step-attempt` record was built.** That slot ships `dropped` and gets no `value` record either. ```json theme={"system"} { "status": "degraded", "mode": "shape_only", "shape_store": { "disk_used_bytes": 147456, "effective_retention_days": 7 }, "value_store": { "bytes_used": 1678, "byte_budget": 1, "effective_retention_estimate_hours": 0 }, "stubs_swept_24h": 0, "services": [ { "service": "delivery-service", "records_lost_24h": 0, "records_shed_24h": 0, "last_seen_at": "2026-11-07T06:48:24.717Z" } ] } ``` So on this server, `expired` asserts only what those five share: the slot's own row carries a content id, and no bytes are behind it now. It does not assert that the server ever held them, and it does not mean the function returned nothing. A durable value-arrival record is what it would take to separate the first case from the rest, and the shipped store has none. Causes 2, 4 and 5 need neither a pass nor pressure, and a reader meets them on an idle server. Below is a values read of a second trace, posted from the conformance fixture that carries one step per handle state, on a server holding 1,678 bytes against the default 20 GiB budget. The root's two slots shipped `sampled_out`, `AADA`'s shipped `dropped`, `AABA`'s shipped `pending`, and `AAAQ`'s shipped `stored` with no `value` record behind them. All seven read the same thing: ```json theme={"system"} { "values": [ { "path": "", "slot": "input", "state": "expired" }, { "path": "", "slot": "output", "state": "expired" }, { "path": "AADA", "slot": "output", "state": "expired" }, { "path": "AABA", "slot": "input", "state": "expired" }, { "path": "AABA", "slot": "output", "state": "expired" }, { "path": "AAAQ", "slot": "error", "state": "expired" }, { "path": "AAAQ", "slot": "input", "state": "expired" } ], "unknown_paths": [] } ``` A shape read is where the difference survives, because a read miss rewrites the state only on the values route. The trace read of that same trace still reports `sampled_out` on the root, `dropped` on `AADA` and `pending` on `AABA`. A slot whose bytes never formed keeps its own state on both routes, since no content id sends it to the value store: `redacted`, `too_large` and `not_captured` answer as themselves. So shape separates `sampled_out` and `dropped` from a reclaimed value. It does not separate `pending`. `pending` is the state that says least. A slot the SDK reported as `pending` keeps it in the shape store for good, because a record is written once: on the first trace above, the two slots of the step at `path` `AABA` read `pending` while their bytes were there, and the values route returned the bytes anyway. After the pass both read `expired`. So `pending` says nothing about whether bytes are available, on either route. Where no `value` record was ever owed, case 4 above, `pending` is also the state the slot keeps for good. ## The function index keeps 90 days The function index (observed callers and callees per function, with counts) runs on its own 90-day clock and has no flag. Its rows outlive shape: a row is deleted 90 days after the last time that caller/callee pair was observed, so `GET /v1/functions/detail` still answers for a function last called weeks ago. What does not survive is the example trace ids on that row, which are deleted as soon as their trace is, well inside 90 days at a 7-day shape retention. An index row with no reachable example is the ordinary state of this index, not a fault. ## If you want values to last as long as shape Two flags move the two horizons: raise `-value-budget-bytes` until the arithmetic clears, or lower `-shape-retention-days` until it does. Either makes `vexlo check` exit 0 and print `OK` instead of `OVERCOMMITTED`. Both clear the byte arithmetic alone: neither moves the 1,000-file bound, so a server whose hour segments hold more than that still reaches its budget around the horizon the check prints and then stops storing new values. The cheap direction is to stop promising a week of shape you have no values for. At 2 traces/s the default budget holds 103h55m, so `-shape-retention-days 4` (96h) clears it with no extra disk: ```text theme={"system"} $ vexlo check -db-path data/vexlo.db -value-store-dir data/values -shape-retention-days 4 ... 6 lines omitted shape_retention_days: 4 value_budget_bytes: 21474836480 free disk (value_store_dir): 94841741312 bytes free, 21474836480 bytes configured budget retention arithmetic (at 2 traces/s): shape bytes/trace: 2300 value bytes/trace: 28700 total bytes/trace: 31000 configured shape retention: 96h0m0s effective value retention: 103h55m26.071080139s OK: the configured value budget covers the configured shape retention ``` Keeping the week costs disk instead. Covering 7 days of values at 2 traces/s takes `28700 x 2 x 604800` bytes, about 32.3 GiB: ```text theme={"system"} $ vexlo check -db-path data/vexlo.db -value-store-dir data/values -value-budget-bytes 34715520000 ... 7 lines omitted value_budget_bytes: 34715520000 free disk (value_store_dir): 94841331712 bytes free, 34715520000 bytes configured budget retention arithmetic (at 2 traces/s): shape bytes/trace: 2300 value bytes/trace: 28700 total bytes/trace: 31000 configured shape retention: 168h0m0s effective value retention: 168h0m0s OK: the configured value budget covers the configured shape retention ``` At 30 traces/s the same 7 days takes 15 times as much, about 485 GiB. That is more free disk than the filesystem in this run had. `vexlo check` prints that as a warning above the arithmetic and still exits 0: free disk is informational there and never fatal. ```text theme={"system"} $ vexlo check -db-path data/vexlo.db -value-store-dir data/values -value-budget-bytes 520732800000 -expected-write-rate 30 ... 7 lines omitted value_budget_bytes: 520732800000 free disk (value_store_dir): 94841327616 bytes free, 520732800000 bytes configured budget warning: free disk space on this filesystem is less than the configured value budget retention arithmetic (at 30 traces/s): shape bytes/trace: 2300 value bytes/trace: 28700 total bytes/trace: 31000 configured shape retention: 168h0m0s effective value retention: 168h0m0s OK: the configured value budget covers the configured shape retention ``` Half a terabyte to keep a week of values at 30 traces/s is the real price of the default day count, and it is why lowering `value_sample_rate` is the third lever. Value sampling is decided once per trace, on the trace id, and only ever excludes the values of a trace that closes `ok`: an attempt whose own `status` is `error` keeps its values whatever the rate is. So the saving depends on how many of your traces close `ok`, the figure `vexlo check` prints does not account for it, and a sampled deployment has to redo the division itself. That lever is paid for on reads rather than on disk. On an excluded trace, the root's slots and every slot of its `ok` attempts answer `expired`, in the same word a reclaimed value gets, so a lower rate buys horizon by making more traces read as though their values had been taken away. The last option is to accept the two horizons. `-allow-overcommit` makes `vexlo check` exit 0 on an overcommitted configuration and changes none of the arithmetic it prints: the budget fills at the effective figure, shape lasts until the day count, and a fetch in between reads `expired`. What that costs depends on the file bound. Below it the store reclaims oldest first, so the traces reading `expired` are the oldest ones. Above it the store stops storing values instead, so the traces reading `expired` are the ones arriving now, and the operator accepting the overcommit is accepting a shape-only server from the moment the budget fills. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.