Benchmarking

crab bench services/web/markdown read --cases cases.jsonl --runs 10

crab bench <directory> <tool> builds the service and runs explicit workloads on a temporary local host. It measures WebAssembly execution, excluding compilation and deployment, without uploading the image. It reports the tool's own CPU, rows and peak linear memory, elapsed time, and whole-chain execution cost at Cloud's rates, before plan inclusions.

Cases

The cases file is JSON Lines: one object per line. For example:

{"name":"guide","args":{"url":"https://example.com/guide"},"http":[{"method":"GET","url":"https://example.com/guide","status":200,"headers":{"content-type":"text/html"},"body_file":"guide.html","delay_ms":100}]}

• name identifies the case; args is the tool's argument object.
• http supplies responses matched by method and exact URL. Each response has status, optional headers, and either body as text or body_file. File paths are relative to the cases file. An omitted body is empty.
• delay_ms simulates time waiting for that response. Memory reservations are billed while waiting, so immediate fixtures do not estimate a real upstream's latency.
• expected_error can name a substring of an expected tool error, such as an input-limit refusal. Unexpected errors, unarranged HTTP requests and runtime traps prevent a budget recommendation. A stopped call does not measure what finishing would require.

Benchmark calls use the unchecked local identity local through direct host execution, including for internal tools. Each case starts with a fresh migrated database; repetitions of that case share its state. The host has no production data or credentials, and runs only the selected image. Calls needing another deployment cannot reach it here.

Use representative and largest supported inputs, including error paths. The command measures the supplied workload; it does not generate worst-case inputs or establish coverage for all inputs.

Live HTTP

crab bench services/web/markdown read --cases live.jsonl --runs 10 --live-http

With --live-http, cases omit http and send real requests. Every repetition executes the tool again, including its HTTP operations. Fixture mode sends no upstream requests.

Results and budgets

Cost reflects the current budget declaration and any in-code reductions. Rerun after changing the budget.

The report includes median and maximum measurements, each run's outcome, the image digest, and a budget suggestion with 1.5× headroom over measured maxima. Memory measurements are in bytes; the suggested crab.toml budget converts them to MiB and rounds up. Tool budgets use the entry invocation's own resources; the cost estimate includes the whole call.

Ceiling checks use runtime defaults, not limits queried from Cloud. A suggestion above a ceiling is reported without clipping it. Check the destination's limits before deployment and repeat the workload under the chosen budget; see Budgets.

--json returns the report as JSON; redirected output uses JSON automatically. A completed benchmark exits 2 when a workload failed, a measurement was incomplete, or the suggested budget exceeds a default ceiling. Build and setup errors exit 1.

The selected context supplies Cloud's rates. --offline uses SDK default rates; unavailable Cloud rates also fall back to those defaults, labeled in the report. Local CPU timing varies across machines. Fixture latency and local timing make costs estimates; production usage remains the source for actual consumption.