Evidence · methods and raw numbers
Methods and raw numbers
My launch claims are about speed and quality. This page shows what I measured, how, on which hardware, and every number as a file you can download. I measured all of it myself on 11 October 2026; no one else has checked it.
The speed and quality numbers are my own measurements. The two leaderboard results come from public boards. JevBench is run by the founder of Decision Models. Decision Models offerings are not listed on JevBench.
Claims
Claims and where the numbers are
| Claim | Number | Details |
|---|---|---|
| Perplexity Decider v1.1 was first on Decision Index 0.3.1 (board of 10 October 2026) | 62.8; Jev 60.1 | Leaderboards |
| Highest capability score on JevBench among systems inside its latency cap | 82.8; Jev 77.1 | Leaderboards |
| s1-pro speed, same GPU, same harness, Decision Index traffic | Production (NVFP4): 21.64 to 21.83 decisions/s, about 4.4x. FP8 build: 14.5, about 3.0x. Author reference code: 4.88. | s1-pro |
| No measurable quality loss | Every aggregate 95% interval includes 0; 0 answer flips in the 297-pair c1 repeat | Methods |
| Blink (s1-fast, s1-vision): deterministic configuration | 94.0 to 97.1 decisions/s; stock vLLM 138 but 6 of 300 answers flip | Blink |
| Quyet-1.0-Large (coming soon) | FP8 4.48x, NVFP4 6.62x the stock server (5.5441 decisions/s) | Quyet |
Public boards
Leaderboard results
These two results come from public boards and describe the open models, not the Decision Models service.
Decision Index. Perplexity Decider v1.1 (27B), the model behind s1-pro, scores 62.8 (62.75) on the Full score of Decision Index 0.3.1. On the board as updated on 10 October 2026 it was first; Jev scores 60.1 (60.11). The board was updated again on 12 October 2026 with 32 new entries; the model keeps its 62.8 and is now tied for second place. The Decision Index is a community board on Hugging Face that someone else runs. Open the Decision Index →
JevBench. On JevBench v1.6.1 the same model has a capability score of 82.8 (82.79); Jev scores 77.1 (77.13). Capability is the mean of JevBench's Intelligence and Calibration scores. This is the highest capability score among the systems inside JevBench's published median-latency cap. It is outside the cost cap, so it is not first on the default ranked view, and I do not claim a first rank there. Slower systems outside the latency cap score higher. Open JevBench v1.6.1 →
JevBench is run by the founder of Decision Models. Decision Models offerings are not listed on JevBench.
s1-pro · Perplexity Decider v1.1 27B
Speed and quality of s1-pro
Reference is the model author’s own code in BF16, which I ran on the same GPU. It is not Perplexity’s hosted service.
Production state. s1-pro runs the NVFP4 build today: 21.64 to 21.83 decisions/s on Decision Index traffic, about 4.4x the author reference code (4.47x in the same-pod comparison). The FP8 build it replaced reached 14.5 decisions/s, about 3.0x.
Same pod, same session
All four builds ran back to back on one RTX PRO 6000 Blackwell, with the same harness and the same rows.
| Build | Decision Index traffic decisions/s | JevBench P300 decisions/s | Speed vs reference Decision Index / P300 | Decision Index score delta points [95% CI] | P300 capability delta points [95% CI] | c1 answer flips |
|---|---|---|---|---|---|---|
| Author reference code, BF16 | 4.88 | 8.465 | reference (1.00x) | reference | reference | not re-run |
| FP8 build previous build | 14.291 | 19.355 | 2.93x / 2.29x | +0.90 [−0.26, +2.29] | −1.23 [−1.91, +0.34] | not re-run |
| NVFP4, earlier build (middle layers only) earlier test build | 20.35 | 27.548 | 4.17x / 3.25x | +0.70 [−1.04, +2.44] | +0.69 [−1.28, +1.96] | not re-run |
| NVFP4, expanded build in production | 21.825 | 29.794 | 4.47x / 3.52x | −0.34 [−2.38, +1.63] | +0.45 [−1.78, +1.93] | 0 of 297 |
NVFP4 here means mostly 4-bit: edge attention layers and the readout keep their existing precision. In the one-at-a-time test the NVFP4 build is not faster for a single request: median latency is 86.9 ms against 85.2 ms for FP8. It raises throughput under load.
Production node image
The image that would serve production traffic, measured in a separate session on the same GPU model. That session has no author-reference run, so the ratio below divides by the 4.88 from the same-pod comparison. Treat it as an estimate across sessions.
| Build | Decision Index traffic decisions/s | JevBench P300 decisions/s | Speed vs reference Decision Index / P300 | Decision Index score delta points [95% CI] | P300 capability delta points [95% CI] | c1 answer flips |
|---|---|---|---|---|---|---|
| FP8, production node image previous build | 14.544 | 20.464 | 3.0x / n/a other session | −0.57 [−1.64, +0.73] | −0.92 [−1.93, +0.48] | not re-run |
| NVFP4 (middle layers only), production node image earlier test build | 20.29 | 28.623 | 4.2x / n/a other session | +0.26 [−1.26, +1.97] | +0.15 [−1.82, +1.58] | 0 |
| NVFP4 expanded, production node image in production | 21.644 | 29.316 | 4.4x / n/a other session | −0.58 [−2.50, +1.37] | +0.84 [−1.53, +1.81] | 0 |
The 21.64 figure is the production image; 21.83 is the same-pod research build. The c1 repeat found 0 answer flips on both the engine and the node.
s1-fast · s1-vision · Blink v0.3 26B-A4B NVFP4
Blink: determinism over raw speed
Both ids are answered by the same model. Stock vLLM is faster, but it is not deterministic: 6 of 300 answers changed between two identical one-at-a-time passes. I only accept configurations with 0 flips, so Blink runs batch-invariant.
The tuned batch-invariant configuration raised throughput on Decision Index traffic from 94.0 to 97.1 decisions/s (256 sequences and 16,384 batched tokens instead of 64 sequences). That is 70.1% of the stock server’s speed. I am not claiming Blink is faster than stock serving.
| Build | Decision Index traffic decisions/s | JevBench P300 decisions/s | Decision Index score delta points [95% CI] | P300 capability delta points [95% CI] | c1 answer flips |
|---|---|---|---|---|---|
| Stock vLLM (not batch-invariant) | 138.19 | 53.49 | reference | reference | 6 of 300 |
| Batch-invariant, 64 sequences (before) | 93.983 | 38.41 | −0.48 [−2.19, +1.54] | +0.15 [−2.99, +4.52] | 0 of 300 |
| Batch-invariant, 256 sequences, 16,384 batched tokens (after) | 97.118 | 38.99 | −0.09 [−1.88, +1.95] | +0.15 [−2.99, +4.52] | 0 of 300 |
Decision Index traffic for the two batch-invariant builds is the mean of two alternating fresh-server repeats (first passes: 93.26 and 96.82). Stock is a single run.
Quyet-1.0-Large · by Chinh Nguyen
Quyet-1.0-Large
Reference is the stock Hugging Face server for the model.
Status. Quyet-1.0-Large is available on the API.
| Build | Decision Index traffic decisions/s | JevBench P300 decisions/s | Speed vs reference Decision Index / P300 | Decision Index score delta points [95% CI] | P300 capability delta points [95% CI] | c1 answer flips |
|---|---|---|---|---|---|---|
| Stock Hugging Face server (author reference) | 5.5441 | 6.6267 | reference (1.00x) | reference | reference | not re-run |
| vLLM, FP8 | 24.8468 | 17.3419 | 4.48x / 2.62x | +0.40 [−1.11, +1.84] | +1.02 [−0.90, +2.95] | 0 of 300 |
| vLLM, calibrated NVFP4 (all linear layers), BF16 KV cache | 36.7087 | 26.2594 | 6.62x / 3.96x | −0.12 [−2.52, +1.66] | −1.38 [−4.08, +1.32] | 0 of 300 |
The aggregate intervals include 0 for both builds. The NVFP4 build lowers one sub-score: the Decision Index Language score is 3.34 points lower (95% CI −6.74 to −0.33, which excludes 0). Passing the aggregate gate is not proof that every component is unchanged. NVFP4 needs a further check before it serves traffic.
Methods
How I measured
Everything below is self-measured. No third party audited these runs.
- Hardware. One NVIDIA RTX PRO 6000 Blackwell (96 GB class) per session. In the same-pod, Blink and Quyet comparisons every build ran on the same GPU in the same session; the production-image session is separate. The same-pod Decider runs used the Workstation Edition (97,887 MiB, driver 595.71.05, CUDA 13.0, torch 2.14.0+cu130); the Blink and Quyet runs used Server Edition cards with vLLM 0.30.0.
- Decisions per second. The number of decisions the server answers per second while I keep it saturated. For each build and workload I raise the number of simultaneous requests until throughput stops rising and report the best tested point. It measures one GPU’s throughput, not the latency of one customer request and not an end-to-end service figure.
- Decision Index traffic. A fixed sample of 1,000 states (5,129 decisions: every fifth of 5,000 fixed IDs) from public Decision Index tasks.
- JevBench P300. The 300 public JevBench v1.6 P300 items (297 valid for Decider; three identical over-context failures in every arm).
- Quality gate. A build passes only if the paired bootstrap 95% interval (1,000 resamples) of the score difference against the reference build, on the same rows, includes 0 or is positive, for both the Decision Index sample and the P300 capability score. It must also repeat the 300 P300 items one at a time twice with 0 answer flips (c1 determinism).
- What the gate cannot show. The intervals are wide, several points in places, so a small loss is not excluded. “No measurable quality loss” means exactly this and nothing stronger. Output probabilities can still differ slightly between batch sizes.
- Models. perplexity-ai/pplx-decider-v1.1-27b (revision 6195aa5), PixilabAI/Blink-v0.3-26B-A4B-NVFP4 (revision 08caad2), chinhnc/Quyet-1.0-Large (revision d997d58). The inference engine code used for these runs is not public yet, so outsiders cannot rerun the speed numbers today.
- Dates. All serving measurements were made on 11 October 2026. Leaderboard figures were read on 10 and 11 October 2026.
Raw numbers
Download the table
Every row used on this page, with the exact values. No prompts, customer data or private repository links.
Columns: build, Decision Index and P300 throughput, speed ratios, paired score deltas with 95% bounds, c1 answer flips. The JSON also carries the leaderboard figures and definitions.