Kimi K3.1 Pro scored 90.9% on Japanese idioms and 99.5% on Turkish paraphrase selection. We ran 5,315 source-dataset entries across Japanese, German and Turkish to see where those strengths hold, what they do not prove, and why the difference matters for business AI.
German conversational meaning reached 87.7%. Turkish figurative-versus-literal usage reached 99.0%. The complete run returned 1,551,208 tokens and cost an estimated $6.169655 at list price, with every response, prediction and score preserved for review.
Disclosure: Wallaby builds open-weight inference infrastructure. We ran this evaluation on Moonshot's official API on 2 November 2026. It is not a Wallaby production benchmark or a customer result.
The numbers worth talking about
Japanese OpenMWE asks whether a marked phrase is an idiom or literal language. Kimi K3.1 Pro answered 270 of 297 cases correctly.
German MultiPragEval tests conversational meaning. The model answered 263 of 300 cases correctly.
The two Turkish idiom tasks were stronger still: 199 of 200 on paraphrase selection and 195 of 197 on figurative-versus-literal usage.
Those are useful results for teams building terminology checks, document classification and local-language workflows. They are also deliberately specific. Each percentage belongs to a named dataset, not to an entire language.
What 5,315 entries actually include
The full set covers public conversational, idiom and workplace tasks, plus nine small custom tasks for structured extraction and context reversal.
| Dataset | Entries | What it measures |
|---|---|---|
| German MultiPragEval | 300 | Conversational meaning |
| German InQA+ | 384 | Indirect answers |
| Japanese OpenMWE | 297 | Idiom versus literal meaning |
| Japanese JaSocial | 10 | Public speech-act samples |
| Japanese JWCAS | 30 | Open workplace scenarios |
| Turkish ImplicaTR | 3,888 | Conversational implication |
| Turkish Idioms | 397 | Paraphrase and figurative usage |
| Custom tasks in three languages | 9 | Extraction and context reversal |
| Total | 5,315 | — |
Turkish makes up 80.7% of the entries because ImplicaTR alone contributes 3,888. That is why we do not average the six task scores into a single multilingual number. A large test set should not hide which tasks dominate it.
How we counted responses and prompts
The 5,315 source-dataset entries contain 5,311 unique prompts. Four repeated prompts remain separate source-dataset rows. The 60 InQA+ supplement samples are part of the 384-entry InQA+ set, not an extra 60. We recorded 5,368 successful API responses in total. The difference comes from earlier duplicate sampling. Duplicate responses count toward execution cost, not toward the scored total. Every recorded response returned HTTP 200 and passed a content-hash check.The more useful finding sits between the high scores
Turkish idiom selection reached 99.5% and 99.0%. Turkish ImplicaTR scored 2,179 of 3,888, or 56.0%.
German conversational meaning reached 87.7%. German InQA+ scored 256 of 384, or 66.7%. The label breakdown is more informative than the average:
Label 4, the non-answer category, reached 85.5%. Label 6, insufficient context, reached 0 of 22. A team using the model to decide when a conversation lacks enough information should keep an explicit review step for that decision.
The same caution applies to Japanese workplace writing. We generated 30 JWCAS open-ended responses, but they still need native-speaker review. JaSocial covered 10 public samples with keyword-based function checks, not a naturalness score. We did not turn either one into a made-up accuracy number.
What this means for local-language products
Japanese: idiom and literal-meaning classification is a promising fit for structured document workflows. Open-ended business writing needs a separate native review process.
German: conversational-meaning tasks are useful, but indirect answers and missing-context decisions need task-specific tests and escalation rules.
Turkish: terminology and idiom workflows show strong results. Open-ended implication should be evaluated on representative customer text rather than inferred from idiom accuracy.
For a product team, the practical question is not “Is this model multilingual?” It is “Does it work on this document, in this language, under these rules?” That is the evaluation we would run before choosing a model for a customer workflow.
What the evaluation cost
The run returned 1,551,208 tokens. Our list-price estimate is $6.169655, including duplicate sampling and different execution modes.
| Execution stage | Estimated cost |
|---|---|
| Initial 4,207 entries | $4.701157 |
| Supplementary 90 entries | $0.348890 |
| Professional-set direct calls | $1.119608 |
| Total | $6.169655 |
The first 1,000 Batch calls were estimated at 60% of standard price; the other stages used standard list rates. This is an evaluation-cost estimate, not an invoice, and it excludes taxes and any separate cache-write charges.
Evidence and sources
Download the evaluation artifacts:
Public datasets: MultiPragEval, Multilingual-IQA, OpenMWE, ImplicaTR, Turkish Idioms, JaSocial, JWCAS.
We used kimi-k3.1-pro with low reasoning effort and the same output-format requirements within each task. Objective answers were scored against their labels; open-ended generations were left unscored. We did not run a same-prompt, same-condition Claude comparison, so this article makes no Kimi–Claude superiority claim.
Try your own local-language workflow
Wallaby provides OpenAI-compatible access to curated open-weight models, itemized usage billing and automatic provider failover. Kimi K3 is live on Wallaby today; the K3.1 Pro evaluation above does not imply that K3.1 Pro is available on Wallaby.
New accounts receive $0.50 in free credit, with no card required until top-up. You can start your first API call, compare it with your own documents and see every token itemized. Self-serve top-ups start at $20.
For the commercial details, see the current Kimi K3 pricing and our broader open-weight API pricing comparison. If you are estimating a longer evaluation, our agent context cost audit shows how repeated context changes the bill.