The same open-weight model can score differently depending on who serves it. Moonshot says so itself. Its Kimi Vendor Verifier announcement explains that third-party APIs drift from the official one, and that users can't tell a weak model from a bad deployment.
We are not on that table yet. So on 16 November 2026 we ran KVV against our own endpoint in three rounds: a smoke test, the full API pre-flight suite, then three benchmarks on random samples at max thinking effort. The sampled scores came out level with Moonshot's published numbers.
Disclosure: Wallaby Token sells API access to Kimi K3 and K3.1 Pro, and the endpoint tested is ours. These are samples, not a full KVV run, and Moonshot has not reviewed them.
What Kimi Vendor Verifier is
KVV is Moonshot's open-source test kit for anyone serving Kimi models, MIT-licensed on GitHub. It has two halves:
- Pre-flight checks. Pytest suites that call the chat-completions API directly: sampling-parameter constraints, tool-call argument schemas, malformed tool calls, K3 features such as
response_format,tool_choiceand thinking effort, and reportedprompt_tokens. - Benchmarks. OCRBench, MMMU Pro Vision and Kimi Tool Call on the inspect-ai harness, plus BEAM, a 1M-token memory test, and DeepSWE, a coding-agent benchmark.
The README publishes benchmark scores per provider, one table for K3 and one for K3.1 Pro, each at max effort. The K3 table lists nine providers, Moonshot first. If you are choosing between Kimi providers, it is the only like-for-like accuracy comparison out there. Our Kimi K3 access routes page covers the other half of that choice: price, payment and who sells what.
Round 1: a 20-request smoke test
We installed KVV at commit 7b938c3 in an isolated Python environment with only the seven packages the tests need, and used one test key on our live gateway. Twenty requests covered the basics: plain requests with thinking on and off, a 128-character tool name, eight tool-call schema cases in both streaming modes, and a few prompt-token cases.
It billed $0.056. That told us the full pre-flight suite would cost a couple of dollars, not tens.
Round 2: the full pre-flight suite
The full run took 32 minutes and 768 requests on kimi-k3, and billed $2.54.
| Check | Result | What it means for your code |
|---|---|---|
| Tool-call arguments vs JSON schema, 221 schemas, streamed and not | 423 / 426 | Tool calls come back parseable and schema-valid |
response_format, JSON Schema strict and non-strict |
All pass | Structured output behaves as documented |
tool_choice: auto, none, required, named |
All pass | You can force or forbid tool calls |
stop sequences and basic thinking-effort cases |
Pass | low, high and max are accepted |
| Fixed sampling values | Applied, not rejected | Off-spec temperature or top_p gets Kimi's required value; Moonshot's API returns a 400 instead |
Tools declared inside messages |
Not supported | Declare tools in the top-level tools field |
The three tool-call misses were all streamed runs where max-effort thinking used the whole 2,048-token output cap before the call finished. The same cases passed without streaming. In your own agent, give max effort room: a larger max_tokens, or high effort for short tool steps. The K3.1 Pro thinking cost post shows what each effort tier adds to a bill.
KVV's README advises running benchmarks once pre-flight passes. Ours didn't pass every item, and the last two rows are why. Neither touches a request that sends Kimi's required sampling values and declares tools at the top level, which is how the benchmarks call the API, so we went on.
Round 3: three benchmarks on samples
We kept KVV's own task definitions, solvers and graders, and changed only which items ran: a fixed-seed random sample instead of the full set. Settings followed the table: reasoning_effort: max and nothing else for thinking, temperature 1.0, top_p 0.95, output limits as the README specifies.
| Benchmark | Model | Our sample | Moonshot on the KVV table | Other providers on the table |
|---|---|---|---|---|
| OCRBench | kimi-k3 | 58 / 60 = 0.967 | 0.89 | 0.878 to 0.897 |
| MMMU Pro Vision, 10 options | kimi-k3 | 98 / 120 = 0.817 | 0.82 | 0.804 to 0.82 |
| Kimi Tool Call | kimi-k3.1-pro | 30 / 30 = 1.0 | 0.972 | 0.972 to 1.0 |
Table figures come from the KVV README as of 16 November. The Tool Call row sits in the K3.1 Pro table, the only one that carries that benchmark.
MMMU Pro is the row that matters most. It is a reasoning test over images, and reasoning is where a cut-down deployment usually shows first. Our first 30 items scored 22, which looked low. We drew 90 more with a new seed, excluding the first batch, and they scored 76. Across all 120 the score is 0.817, with a 95% interval of 0.738 to 0.876. If our endpoint truly scored 0.82, a result of 98 or lower would turn up half the time.
No measurable gap.
The items came from the first shard of the official dataset: 865 rows covering all 30 subjects. None of the 120 answers was truncated, errored or came back empty.
OCRBench came out high. 0.967 on 60 items is about three standard errors above Moonshot's 0.89. We don't read that as beating Moonshot. A difference in grading setup on our side is the likelier cause, and 60 items is a small sample.
Tool Call went 30 for 30. The benchmark replays production failure cases where the model must fix a single tool call deep in a long context. Inputs in our sample ran from about 3,000 to 252,000 tokens, and every one of the 30 came back with the right call. That is the long-context agent shape tools like Codex CLI and Claude Code produce; our Kimi K3 CLI hub lists the tools that run on this endpoint.
The benchmark round billed $10.27, trial batches included. All three rounds together came to about $12.90 at our retail prices.
What the same scores cost
Scores only settle half the choice. Here is what the providers on the KVV tables charge, per million tokens, from their own pricing pages on 18 November 2026:
| Provider | Kimi K3, in / cached / out | Kimi K3.1 Pro, in / cached / out |
|---|---|---|
| Moonshot list price | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Fireworks | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Together | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Baseten | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Modal | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Databricks, at $0.07 per DBU | $3.00 / $0.30 / $15.00 | $2.00 / $0.20 / $10.00 |
| Verda | not listed | $2.00 / $0.20 / $10.00 |
| AWS Bedrock, global | $3.00 / $0.30 / $15.00 | Highspeed only: $4.00 / $0.20 / $20.00 |
| Wallaby | $2.55 / $0.255 / $12.75 | $1.80 / $0.18 / $9.00 |
Every provider that publishes a price charges Moonshot's list or more. Regional pinning costs extra: Together's US geo-pinned K3 is $3.75 in and $18.75 out, Bedrock's US route $3.30 and $16.50, and Fireworks adds 10% for US-only. We are 15% under list on K3 and 10% under on K3.1 Pro. DigitalOcean and Nebius didn't show a Kimi price on a page we could read that day.
Cheaper K3 endpoints do exist off the table, many of them labelled fp4 or similar quantization. None of them has KVV scores published, so you would be taking their accuracy on trust.
What we have not tested
DeepSWE and BEAM. DeepSWE runs on a separate agent platform, and BEAM feeds million-token conversations, so both cost far more than a sample. The scores above cover three of KVV's five benchmarks.
If you pick a Kimi provider on benchmark scores, ask any provider for the same receipts: which KVV tasks, how many items, what effort, what it scored. For uptime and latency rather than accuracy, our public status page has 90 days of history. If you are deciding between the two models, K3 vs K3.1 Pro covers what changes in your integration.
Try it
New Wallaby accounts start with $0.50 in free credit, no card needed. Kimi K3 is $2.55 in, $0.255 cached and $12.75 out per million tokens, 15% off the official list. Kimi K3.1 Pro is $1.80 in, $0.18 cached and $9.00 out, 10% off. Create an account.
Sources: Fireworks, Together, Baseten, Modal, Databricks, Verda and AWS Bedrock pricing pages, read 18 November 2026; Kimi Vendor Verifier README and results tables, github.com/MoonshotAI/kimi-vendor-verifier at commit 7b938c3, read 16 November 2026; Moonshot's KVV announcement on kimi.com; our smoke, pre-flight and sampled benchmark runs against api.wallabytoken.com, 16 November 2026; Wallaby gateway log for the same runs. Provider scores change as models and deployments update; check the KVV table before you buy.