kPool
Control plane for multiple LLM providers - virtual keys, budgets, account rotation, one binary.
The problem
Once more than one tool talks to more than one LLM provider, key management degenerates into a spreadsheet (ask me how I know): which key belongs to which tool, which account is near its quota, who spent what. kPool puts one gateway in front of Anthropic, OpenAI (including the Codex WebSocket transport), and xAI, and makes usage something you can budget, observe, and audit.
Architecture
kPool is a Bun workspace - server, shared contracts, embedded React admin - that compiles into a single cross-platform binary (as tools should) with the dashboard baked in. It speaks Anthropic Messages, OpenAI Responses and Chat Completions, and an Ollama-compatible facade, and can translate between dialects: an Anthropic Messages request can be served over OpenAI Responses upstream with tools, reasoning, images, and streaming events mapped back into the original shape. Clearly provider-owned models sent to the wrong endpoint are rejected instead of silently forwarded.
Virtual keys carry a model allowlist, a budget ceiling, and RPM/TPM limits (requests and tokens allowed per minute), all enforced before the provider call - with in-memory pre-call reservations (quota held before an upstream request starts) closing the race between concurrent quota checks. Every request lands in an append-only ledger with token totals and costs (CSV export included), Prometheus (a metrics system that scrapes and stores time-series data) metrics, and an audit log for admin and key mutations.
kPool preserves conversation continuity across account scheduling. Keys
like previous_response_id and session IDs bind follow-up turns to the
account that originated them; rotation to another account happens only on
retryable failures and only before a response is committed, never
mid-conversation. Selection weighs backoff state, quota-pressure bands, and
in-flight load, with reserved thresholds that protect five-hour and weekly
allowance windows. Bounded admission control (a gate that caps work entering the system)
sets a default ceiling of 256 active requests, and excess inference fails
immediately with 503 rather than queueing invisibly.
For Codex WebSockets, the gateway proxies the upgrade, injects upstream credentials into the handshake, and runs keyed frames through the same policy, guardrail, budget, and token-accounting checks as HTTP traffic. Frames it cannot parse fail closed. Persistence is Postgres in production and embedded PGlite (Postgres that runs in-process without a separate server) for zero-config local development.
Boundaries
kPool is infrastructure for accounts and keys you already have - it adds control and observability, not capacity. Client tokens are stripped at the boundary and upstream credentials never reach clients; discovered local credentials only join the routing pool after an explicit, audited import. Requests are metered at the gateway; what providers do on their side is governed by their terms, not by kPool. And because it terminates credentials, it is designed to run inside your own trust boundary, not as a public service. Deliberately out of scope, for now: SSO, managed hosting, SDK packaging, and supervising provider CLIs as child processes.