Measured recipes
A recipe is a launch line the console offers as one click. Its gain is output tokens per second against the control named in its rows, on the same GPU and the same traffic; the judge measures candidate and control in paired runs with a calibrated noise floor, and refuses a change that slows the first token or fails the quality check. Each recipe links to the rows behind it; each row keeps the launch line, the load generator and the hardware price. Raw run artifacts for any row are available on request at team@velobyte.ai.
Recipes (10)
| Model and hardware | Workload | Output | Gain | First token p90 | Measured |
|---|---|---|---|---|---|
| Qwen3-8B-FP8 on 1× H100 Qwen/Qwen3-8B-FP8 |
Coding agents 4 agents · AgentX replay · SemiAnalysis coding-agent traces at 44k-128k context, 150 s window, paired samples |
57 → 83 tok/s | 1.46× +46% output at 4 agents versus the BF16 model-card launch |
1,456 ms | 2026-10-14 stock · tuned · stock · tuned |
| Qwen3-Coder-30B-A3B-Instruct-FP8 on 1× H100 Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 |
Coding agents 32 agents · AgentX replay · SemiAnalysis coding-agent traces |
72 → 252 tok/s | 3.53× +253% output at 32 agents versus the fp8 KV + 16k prefill chunk + async scheduling line |
18,617 ms | 2026-10-12 hosttier · hosttier · hosttier |
| Qwen3-30B-A3B-Instruct-2507-FP8 on 1× H100 Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 |
Chat and API 128 concurrent · Random 2000 in / 256 out · 512 prompts |
3,476 → 4,744 tok/s | 1.36× +36% output at 128 concurrent versus the default launch line |
825 ms | 2026-10-12 tuned · tuned |
| Qwen3-32B-FP8 on 1× H100 Qwen/Qwen3-32B-FP8 |
Chat and API 128 concurrent · Random 2000 in / 256 out · 512 prompts |
975 → 1,172 tok/s | 1.20× +20% output at 128 concurrent versus the default launch line |
12,698 ms | 2026-10-12 tuned · tuned |
| gpt-oss-120b on 1× H100 openai/gpt-oss-120b |
Reasoning 64 concurrent · Random 2000 in / 1024 out · 256 prompts |
443 → 490 tok/s | 1.11× +11% output at 64 concurrent versus the default launch line |
2,272 ms | 2026-10-12 tuned · tuned · tuned |
| Llama-3.3-70B-Instruct-FP8-dynamic on 2× H100 RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic |
Chat and API 128 concurrent · Random 2000 in / 256 out · 512 prompts |
1,024 → 1,139 tok/s | 1.11× +11% output at 128 concurrent versus the default launch line |
13,687 ms | 2026-10-12 tuned · tuned |
| Qwen3-Coder-480B-A35B-Instruct-FP8 on 8× H100 Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 |
Coding agents 8 agents · AgentX replay · SemiAnalysis coding-agent traces |
77 → 77 tok/s | 1.00× | 1,217 ms | 2026-10-12 stock · stock · stock |
| gpt-oss-120b on 2× H100 openai/gpt-oss-120b |
Reasoning 128 concurrent · Random 2000 in / 256 out · 512 prompts |
1,370 tok/s | measured, no control | 966 ms | 2026-10-12 stock · stock · stock · stock |
| GLM-5.3-Flash on 8× H100 zai-org/GLM-5.3-Flash |
Chat and API 64 concurrent · Random 2k in / 256 out · 256 prompts · ignore_eos |
1,308 → 2,567 tok/s | 1.96× +96% output at 64 concurrent versus the model-card launch |
2026-10-11 · · |
|
| GLM-5.3-Flash on 8× H100 zai-org/GLM-5.3-Flash |
Coding agents 24 agents · AgentX replay · SemiAnalysis coding-agent traces · 68k avg context |
421 tok/s | measured, no control | 1,420 ms | 2026-10-11 · · |
Rows behind the recipes (29)
| Model and hardware | Variant | Workload | Output | First token | Per user | Checked |
|---|---|---|---|---|---|---|
| Qwen3-8B 1x H100 SXM · RunPod · $3.99/h · vLLM 0.30.0 |
stock control: the BF16 model-card launch at 128k (YaRN rope scaling); 17,540 GPU-s per 1M output tokens (median of the pairs) launch line--max-model-len 131072 --gpu-memory-utilization 0.90 --hf-overrides {"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768,"rope_theta":1000000}} |
AgentX replay: SemiAnalysis coding-agent traces at 44k-128k context, 4 agents, 150 s window (Fleet judge, paired samples) 4 agents |
57 tok/s | 2026-10-14 Fleet judge: paired samples against the control, A/A noise floor 0.07%, chat and quality holdouts; artifacts kept |
||
| Qwen3-8B-FP8 1x H100 SXM · RunPod · $3.99/h · vLLM 0.30.0 |
tuned confirmed 1.4563x (CI95 1.4545-1.4581, 2 pairs): 12,044 GPU-s per 1M output tokens; first-token and per-agent figures from the quality-holdout sample of this line launch line--max-model-len 131072 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --max-num-batched-tokens 8192 --long-prefill-token-threshold 2048 --block-size 32 --max-num-seqs 256 --hf-overrides {"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768,"rope_theta":1000000}} |
AgentX replay: SemiAnalysis coding-agent traces at 44k-128k context, 4 agents, 150 s window (Fleet judge, paired samples) 4 agents |
83 tok/s | 524 ms p90 1,456 ms |
130 tok/s | 2026-10-14 Fleet judge: paired samples against the control, A/A noise floor 0.07%, chat and quality holdouts; artifacts kept |
| Qwen3-8B 1x H100 SXM · RunPod · $3.99/h · vLLM 0.30.0 |
stock control for the chat holdout: 402 GPU-s per 1M output tokens (median of the pairs) launch line--max-model-len 131072 --gpu-memory-utilization 0.90 --hf-overrides {"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768,"rope_theta":1000000}} |
vLLM bench serve: 256 chat prompts at 64 concurrent, holdout of the Fleet judge (paired samples) 64 concurrent |
2,488 tok/s | 2026-10-14 Fleet judge: paired samples against the control, A/A noise floor 0.07%, chat and quality holdouts; artifacts kept |
||
| Qwen3-8B-FP8 1x H100 SXM · RunPod · $3.99/h · vLLM 0.30.0 |
tuned chat holdout 1.137x (CI95 1.094-1.180, 2 pairs): 354 GPU-s per 1M output tokens, first-token p90 inside the 3000 ms gate on every sample launch line--max-model-len 131072 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --max-num-batched-tokens 8192 --long-prefill-token-threshold 2048 --block-size 32 --max-num-seqs 256 --hf-overrides {"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768,"rope_theta":1000000}} |
vLLM bench serve: 256 chat prompts at 64 concurrent, holdout of the Fleet judge (paired samples) 64 concurrent |
2,825 tok/s | 2026-10-14 Fleet judge: paired samples against the control, A/A noise floor 0.07%, chat and quality holdouts; artifacts kept |
||
| Qwen3-Coder-30B-A3B-Instruct-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
hosttier tuned + 200 GiB host KV tier (native offloading, pinned host memory) launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --kv-offloading-size 200 --kv-offloading-backend native |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
2,140 tok/s | 1,187 ms p90 10,305 ms |
58 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-Coder-30B-A3B-Instruct-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
hosttier tuned + 200 GiB host KV tier (native offloading, pinned host memory) launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --kv-offloading-size 200 --kv-offloading-backend native |
AgentX replay: SemiAnalysis coding-agent traces, spread arrivals, 420 s window (ranking run) 16 agents |
235 tok/s | 599 ms p90 2,106 ms |
56 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-Coder-30B-A3B-Instruct-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
hosttier tuned + 200 GiB host KV tier (native offloading, pinned host memory) launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --kv-offloading-size 200 --kv-offloading-backend native |
AgentX replay: SemiAnalysis coding-agent traces, spread arrivals, 420 s window (ranking run) 32 agents |
252 tok/s | 7,589 ms p90 18,617 ms |
28 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-30B-A3B-Instruct-2507-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--enable-auto-tool-choice --tool-call-parser hermes --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
3,088 tok/s | 645 ms p90 1,218 ms |
55 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-30B-A3B-Instruct-2507-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--enable-auto-tool-choice --tool-call-parser hermes --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 512 prompts 128 concurrent |
4,744 tok/s | 277 ms p90 825 ms |
40 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-32B-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --default-chat-template-kwargs {"enable_thinking":false} --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
1,188 tok/s | 4,248 ms p90 7,537 ms |
28 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-32B-FP8 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --default-chat-template-kwargs {"enable_thinking":false} --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 512 prompts 128 concurrent |
1,172 tok/s | 3,418 ms p90 12,698 ms |
11 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned 16k prefill chunk + async scheduling (KV stays bf16: sinks attention) launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
561 tok/s | 256 ms p90 1,549 ms |
9 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned 16k prefill chunk + async scheduling (KV stays bf16: sinks attention) launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 512 prompts 128 concurrent |
516 tok/s | 4,317 ms p90 5,687 ms |
6 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 1x H100 SXM · SF Compute · $3.33/h · vLLM 0.30.0 |
tuned 16k prefill chunk + async scheduling (KV stays bf16: sinks attention) launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 1024 out, ignore_eos, 256 prompts 64 concurrent |
490 tok/s | 257 ms p90 2,272 ms |
8 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Llama-3.3-70B-Instruct-FP8-dynamic 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--enable-auto-tool-choice --tool-call-parser llama3_json --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
1,069 tok/s | 3,102 ms p90 5,752 ms |
21 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Llama-3.3-70B-Instruct-FP8-dynamic 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
tuned fp8 KV + 16k prefill chunk + async scheduling launch line--enable-auto-tool-choice --tool-call-parser llama3_json --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --async-scheduling |
Random 2000 in / 256 out, ignore_eos, 512 prompts 128 concurrent |
1,139 tok/s | 5,423 ms p90 13,687 ms |
11 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-Coder-480B-A35B-Instruct-FP8 8x H100 SXM · SF Compute · $27/h · vLLM 0.30.0 |
stock model-card launch, TP8 launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
1,423 tok/s | 1,310 ms p90 1,799 ms |
24 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-Coder-480B-A35B-Instruct-FP8 8x H100 SXM · SF Compute · $27/h · vLLM 0.30.0 |
stock model-card launch, TP8 launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder |
AgentX replay: SemiAnalysis coding-agent traces, spread arrivals, 420 s window (ranking run) 8 agents |
77 tok/s | 567 ms p90 1,217 ms |
93 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| Qwen3-Coder-480B-A35B-Instruct-FP8 8x H100 SXM · SF Compute · $27/h · vLLM 0.30.0 |
stock model-card launch, TP8 launch line--enable-auto-tool-choice --tool-call-parser qwen3_coder |
AgentX replay: SemiAnalysis coding-agent traces, spread arrivals, 420 s window (ranking run) 24 agents |
89 tok/s | 67,537 ms p90 100,615 ms |
31 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
stock model-card launch on two H100 (TP2), to compare with one launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai |
Random 2000 in / 256 out, ignore_eos, 128 prompts 32 concurrent |
647 tok/s | 163 ms p90 342 ms |
21 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
stock model-card launch on two H100 (TP2), to compare with one launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai |
Random 2000 in / 256 out, ignore_eos, 256 prompts 64 concurrent |
969 tok/s | 180 ms p90 488 ms |
17 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
stock model-card launch on two H100 (TP2), to compare with one launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai |
Random 2000 in / 256 out, ignore_eos, 512 prompts 128 concurrent |
1,370 tok/s | 252 ms p90 966 ms |
11 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| gpt-oss-120b 2x H100 SXM · SF Compute · $6.67/h · vLLM 0.30.0 |
stock model-card launch on two H100 (TP2), to compare with one launch line--reasoning-parser openai_gptoss --enable-auto-tool-choice --tool-call-parser openai |
Random 2000 in / 1024 out, ignore_eos, 256 prompts 64 concurrent |
1,238 tok/s | 63 ms p90 525 ms |
28 tok/s | 2026-10-12 InferOptimal run; artifacts kept |
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
Baseten workload: random 2k in / 256 out, ignore_eos, 256 prompts, speculation off (model card ships MTP3 on) 64 concurrent |
2,567 tok/s | 914 ms | 53 tok/s | 2026-10-11 InferOptimal run; paired on the same node within 40 minutes; artifacts kept |
|
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
Baseten workload: random 2k in / 256 out, ignore_eos 128 concurrent |
2,262 tok/s | 753 ms | 2026-10-11 InferOptimal run; artifacts kept |
||
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
Baseten workload: random 2k in / 256 out, ignore_eos 32 concurrent |
934 tok/s | 431 ms | 2026-10-11 InferOptimal run; artifacts kept |
||
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
AgentX replay: SemiAnalysis coding-agent traces, 68k-token average context, spread arrivals, 300 s window 24 agents |
421 tok/s | 730 ms p90 1,420 ms |
130 tok/s | 2026-10-11 InferOptimal run; artifacts kept |
|
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
AgentX replay: SemiAnalysis coding-agent traces, 70k-token average context, spread arrivals, 300 s window 48 agents |
372 tok/s | 3,840 ms p90 15,090 ms |
22 tok/s | 2026-10-11 InferOptimal run; artifacts kept |
|
| GLM-5.3-Flash 8x H100 SXM · SF Compute · $25/h · |
AgentX replay: SemiAnalysis coding-agent traces, 87k-token average context, spread arrivals, 300 s window 8 agents |
126 tok/s | 710 ms p90 780 ms |
326 tok/s | 2026-10-11 InferOptimal run; artifacts kept |