InferencevLLM’s split serving stack makes predictable inference the real prize
Separating prompt processing from generation changes the capacity question: how much work can finish within a useful deadline?
Inference
How models reason and run: serving speed, memory, inference-time compute and cost per useful result.
Serving & runtime · Compression & efficiency · Inference-time reasoning
4 articles
Inference
InferenceSGLang’s smaller cache could make long-context inference cheaper—but accuracy sets the limit
The latest cache-compression results expose two distinct gains: faster generation and room for more live sessions.
InferenceDFlash 2 makes faster inference a question of better guesses
Inco AI’s parallel drafter targets more accepted tokens per model pass, with gains that depend on the workload it serves.
InferencePrismML’s 5.9 GB model tests whether compression can move inference off the server
Bonsai 2 27B makes deployment location a choice to evaluate, while aggregate benchmark retention leaves important questions unanswered.
No articles in this topic yet
Explore the other topics in this desk.