Prefill + decode on 14B
4.3s
Time to first token
$11.8k
Prefill cost · 1M requests
Icehouse vault · cacheinfer.property
Cheaper & Faster Inference
By using frontier research on KV cache transfer between models, cut your inference costs and time by 20%. Prefill runs on a small model, and decode on a large one.
Time to first token · Qwen3 4B → 14B
Two icehouses. Same river. Different bill.
Prefill cost scales with the size of the model doing the reading, which means today you pay large-model rates before the large model has produced a single word. On long inputs, that's most of the bill.
Prefill Qwen3 4B · decode 14B
1.7s
Time to first token · 2.5× faster
$4.6k
Prefill cost · 1M requests · 61% cheaper
Published pair on this page: 4.3s → 1.7s · $11.8k → $4.6k. Cut cost and time by 20% is the on-page claim for the method as a whole.
Rotary strip, head-wise project
We prefill on a small model instead.
Its per-layer K/V tensors are stripped of source rotary embeddings, projected head-wise into the target model's attention space, and re-embedded at target positions. The large model enters decode against a populated cache it never computed.
Strip rotary
Per-layer K/V tensors leave the small model. Source rotary embeddings are shaved off the ice before the chute.
Head-wise project
Tongs align each head into the target model's attention space. Layers keep their order down the chute.
Re-embed & decode
Re-embedded at target positions. The large model enters decode against a populated cache it never computed.
Prefill small and decode large. Transfers are gated on held-out quality and latency for every model pair. Ordinary prefill takes over whenever a pair doesn't clear, so the target model uses its usual inference path.
Private lockers
Serviced open-weight models
Private, and we set it up. Independent benchmark leaderboards increasingly place leading open-weight models at frontier-level quality across common reasoning, coding, and knowledge tasks. We handle the GPUs and the serving. You change one line of config.