Provider translation
Send OpenAI format. CrabLLM translates to Anthropic, Gemini, Bedrock, and Azure automatically.
Docs · providers/overview →Open source · Rust
One API. Every model.
Route requests to OpenAI, Anthropic, Gemini, Azure, Bedrock, or Ollama through one OpenAI-compatible endpoint. Sub-millisecond overhead. Single binary. No runtime.
cargo install crabllm crabctl What you get
Six capabilities, out of the box
Provider translation, routing, streaming, auth, caching, budgets — the gateway handles them so your client doesn’t.
Routing & fallback
Weighted random selection across providers. Exponential backoff retry. Automatic failover.
Docs · features/routing →Streaming first-class
SSE proxied without buffering. Per-chunk extension hooks. Keep-alive pings.
Docs · features/streaming →Virtual keys & auth
Per-key model access control. Rate limiting, usage tracking, and budget enforcement.
Docs · features/auth →Caching & rate limits
SHA-256 response cache. Per-key RPM and TPM limits. Sliding-window enforcement.
Docs · features/extensions →Budget enforcement
Per-key spend limits in USD. Automatic cost tracking from token usage and pricing config.
Docs · features/extensions →Performance
Not the bottleneck
Gateway overhead at 5,000 concurrent requests per second — identical limits (2 CPUs, 512 MB) against a mock backend with instant responses. Full results →
Latency
| Gateway | P50 | P99 |
|---|---|---|
| CrabLLM | 0.26ms | 0.54ms |
| Bifrost | 0.61ms | 1.26ms |
| LiteLLM | 159ms | 227ms |
Memory
| Gateway | Peak RSS |
|---|---|
| CrabLLM | 34.9 MB |
| Bifrost | 171.7 MB |
| LiteLLM | 541.8 MB |