I should not be allowed to rent GPUs
The listing said $5.73 an hour for four 96 GB GPUs, 800 GiB of disk included, and the math wrote itself. A large open mixture-of-experts (a model that activates only a subset of its weights for each token) model, my own endpoint, no per-token bill, as many agents as the cards can feed. I did that math. I felt very clever. Then I measured what actually happened and had to do it again, slower, with less self-respect.
The hourly price is a rate, not a cost. The cost is the rate times every billed minute, and most of my billed minutes were spent on things that were not generating tokens. Mostly me.
Eleven rentals in four days
My remote-access tool still lists the receipts: eleven targets named after GPU rentals, all created between the 3rd and the 6th of November, six of them on the first day. That is not a measured experiment. That is someone pressing the rent button like it dispenses snacks.
Every one of those starts empty. Model storage does not survive between rentals on a marketplace box, so each rental downloads the model weights again - hundreds of gigabytes for the big one - verifies them, loads them, and warms up the engine before the first request is worth anything. On the one session I actually measured properly, the meter ran for 21 minutes before the first successful request.
And twenty-one minutes was the good case. One cold start died because the container's SSH files had the wrong ownership and the remote-access broker's key was refused. The logs said so in plain words; the meter kept running while I read them. Another round went into reading the offer listing wrong: the RAM figure is already the allocated share, and I multiplied it by the GPU fraction anyway, because reading is apparently optional when there is a shiny GPU on the other end. None of that is the platform's fault. All of it is billed.
What one real session cost
Here is the session that went well - the one I get to brag about, so calibrate accordingly. Agents with real work queued, pointed at the rented endpoint through my gateway, measured from the gateway's own ledger:
- Duration of real work: 49 minutes, 1,559 successful requests.
- Input: 281.7 million tokens, 98.2 % of them served from the prompt cache (input the server has already processed and can reuse at a lower price).
- Output: 1.43 million tokens. That is 197 input tokens for every token generated.
- Rental cost from boot to the last successful request: $6.74.
- Same traffic at the API rates for the same model: $8.15.
So the rental won, by 17 %. That is the best number in this note, and I cooked it: the $6.74 stops at the last useful request. The tuning and idle time after it were billed too, and I conveniently left them out of the figure that makes me look good. Best case, minus the evidence against it. (classic me)
The break-even is a utilization number
At those API rates, my traffic mix works out to about $5.72 per million output tokens, input included. Divide the hourly rate by that and the rental has to average roughly 278 output tokens per second - about one million an hour - across every billed minute, cold start and idle included, just to tie. Not win. Tie.
My own measurements put that in context. A single warm stream decoded around 281 tokens per second; five concurrent streams reached about 624 in aggregate, twelve about 959. Those are synthetic runs with a 32k prompt. Real long-context agent traffic never proved it could hold 959, and I am not going to pretend the ceiling is the average, however much I wanted to while I was paying for it.
The break-even is not "is the GPU fast", it is "is the GPU busy". Finding work is never my problem; I do not rent without a queue. The problem is every minute the cards spend not working that queue, and I am remarkably good at producing those minutes.
Agents broke my estimate, in the rental's favor
I started from the usual assumption of about four input tokens per output token, no caching. At the same API rates that prices a million output tokens at about $0.93, input included, and puts the rental's break-even near 1,700 output tokens per second. On that estimate renting is hopeless, and I would have stopped there - which, looking at the list above, would have been the smartest thing I did all week.
Agent workloads look nothing like it. They re-read the same files, the same instructions, the same conversation, over and over: my real mix was 197 to 1, and even with 98 % of it cached, all that input makes each output token cost about six times what the generic estimate says. That is the only reason the break-even came down to 278 - which is reachable, and only while the cards are busy.
So the summary is uncomfortable for everyone involved, and I am everyone involved. The generic estimate would have talked me out of a rental that could win. The measured one says it does win - as long as the meter only runs while the queue is being worked, which is exactly the part I keep ruining.
Where the money actually goes
The work is always there. When I rent, the queue is full before the meter starts - evaluations, batch jobs, agent backlogs. Demand was never the problem. Everything around the work is:
- Cold starts I pay for again and again. Eleven rentals means eleven downloads, eleven verifications, eleven warm-ups. The queue waits; the meter does not.
- Setups that break on the clock. The SSH permissions were one. Every one of those is me debugging with a taxi meter running.
- Tuning on rented time. Once a box finally works, I cannot leave it alone. Different speculative decoding settings, different slot counts, another benchmark round. That is the "tuning and idle" the winning number conveniently excludes, and it is mine, all of it.
- Keeping it alive. I set myself a two-hour limit per rental and then dropped it. I told my agent not to stop a running box automatically. I said "do not destroy it yet" like it was a free sentence. It is an hourly decision (billed by the hour, every time).
Full disclosure, because the rest of this note is too polite: the enemy in my rentals was never the platform, the model, the engine, or a lack of work. It was me treating a rented machine like a lab bench. A lab bench does not bill by the hour.
The rule I use now: rent only when the setup is boring and scripted and the box dies when the queue does; if I am still tuning, I am paying hourly to learn, and the API is cheaper.