kRouter
Embeddings, decision models, and OCR served once, from my own hardware - one control plane, supervised workers, and an API that does not care which box answers.
The problem
Small models do the boring work (the work nobody brags about) in my stack. kReview and kCortex both need embeddings (a vector of numbers that places similar text close together) for search, and every tool that runs its own model server means another copy of the same weights, another update procedure, and no idea which one is slow today. Hosted embedding APIs solve that by sending every chunk of code and every memory to someone else, which I just do not want for this data. That is a preference, not a threat model.
kRouter serves those models once, the way a real service would: keys, usage, health, routing, and updates, with the model processes treated as things that crash. Nobody writes a blog post about their embedding server. If kRouter does its job, nobody has a reason to.
Architecture
Two parts. Control is one process with PostgreSQL behind it: panel, public API, accounts, routing, fleet management. Control is management - it never runs a model, it just tells everyone else where the work goes. Inference nodes on Linux and Windows each supervise a private llama-server and an OCR worker as child processes, and accept work only from control, by a per-node token they store as a hash, or from a local admin. A node's own config holds its identity and which control to trust; control polls every node every five seconds and pushes all tuning. The nodes are deliberately dumb, which makes them easy to replace.
The API is deliberately small. /v1/embeddings is OpenAI-compatible. A
decision model (a small model that answers fixed-choice questions with a probability per option) endpoint uses llama.cpp's structured question format. /extract
turns documents into text, using a PDF's text layer when it is usable and
PP-OCRv6 on OpenVINO when it is not. Keys are stored as hashes and scoped to
embeddings, decisions, or documents, and usage lands per request: tokens,
pages, bytes, duration.
Routing sends an embedding request to the node expected to answer soonest, counting in-flight tokens and penalizing a node that would have to load the model first. Batches of 4096 estimated tokens or more are split across nodes and merged back in input order, with vector shape and model checked on the way. Each node has a circuit breaker (a switch that stops sending work to a failing dependency) - open after three failures, one probe after ten seconds - and a request is retried elsewhere only when the work provably never started. Once a response streams, there is no retry: sending half an answer twice helps nobody.
Supervision assumes failure (pessimism, but scheduled). Crashed children restart on a 5 s, 15 s, 60 s, 300 s ladder, and callers get a 503 with Retry-After while it climbs. llama-server listens on loopback only, with a per-launch key and a memory cap, and a settings change that fails to start brings back the old engine. Updates are signed by control with Ed25519 (a fast elliptic-curve signature scheme) and verified by nodes against the key they received at enrollment, then rolled out one node at a time: drain, swap, health check, rollback on failure. A failed node stays on its old release, because outdated beats offline.
Swapping the OCR runtime had to pass a parity gate first: the OpenVINO candidate against an ONNX Runtime control and native PaddleOCR as the reference, scored against human-reviewed ground truth, with model and input hashes recorded.
Boundaries
There is one control process and no failover. One box to watch, one box to blame. If control is down, nothing answers: nodes refuse unauthenticated work by design. There are no rate limits or quotas, only size and byte budgets, and usage logging is accounting, not billing - during a database outage, inference keeps going and the usage buffer drops its oldest events once full.
kRouter ships no models; a fresh install starts with an empty catalog. Bring your own weights, it will not pick them for you. Its llama.cpp builds are CPU or Vulkan, x64 only. It does not serve chat models. And the parity gate covers three fixtures (three is a number, technically): it proves the runtime swap did not break those, not that the OCR is right for your documents.