Earlier results (Oct 2026)
Archived on Oct 26, 2026. These results compare an earlier version of jgram with Kimi K3 and are kept here unchanged. Current results are on the jgram page.
How does jgram compare with Kimi K3?
Both models use the same graders. Head-to-head rows are judged blind by GPT-4.1, in both answer orders.
| Benchmark | jgram | Kimi K3 | Result |
|---|---|---|---|
| Real-world use | |||
| WildChat-1000real user chats, head-to-head | 279 wins | 180 wins | Win |
| Arena-250real user prompts, head-to-head | 50 wins | 44 wins | Win |
| Instruction following | |||
| IFBench300 instructions | 249/300 | 218/300 | Win |
| IFEval200 prompts, strict prompt-level scoring, both models run Oct 19, 2026 | 188/200 | 183/200 | Win |
| Coding | |||
| LiveCodeBench175 contest problems, one attempt each, run Oct 20, 2026 | 157/175 | 141/175 | Win |
| Terminal-Bench87 command-line tasks | 64/87 | 60/87 | Win |
| TerminalBench 4.0Same 14 TerminalBench 4.0 tasks, one run each, with a 90-minute limit per task instead of TerminalBench's standard 8-hour limit. Kimi K3.1 Pro run in the standard Mini-SWE-agent harness; jgram Max measured in our test harness, with a live-API rerun on Tuesday. | 64.3%jgram Max | 42.9%Kimi K3.1 Pro | Win |
| BigCodeBench298 library-code tasks | 145/298 | 142/298 | Tie |
| SWE-bench Verifiedall 500 real GitHub bug fixes, both models run Oct 19, 2026 | 427/500 | 408/500 | Win |
| Aider Polyglot120 code edits, 2 tries | 116/120 | 114/120 | Win |
| Knowledge and math | |||
| SimpleQA500 factual questions, jgram with web search | 441/500 | 210/500 | Win |
| GPQA Diamond198 science questions | 179/198 | 175/198 | Win |
| MMLU-Pro200 questions | 183/200 | 179/200 | Win |
| MATH-500500 math problems | 499/500 | 492/500 | Win |
| AIME 202530 competition problems | 28/30 | 27/30 | Win |
| Agents and EQ | |||
| AutomationBench50 sales and support workflows | 20/50 | 20/50 | Tie |
| EQ-Bench 4emotional intelligence, judged | 105 wins | 14 wins | Win |
Results as of Oct 1, 2026, except SWE-bench Verified and IFEval, which were run on Oct 19, 2026, and LiveCodeBench, which was run on Oct 20, 2026 (raw results: lcb/graded.jsonl md5 2d833887b92df9bf0ab97e94017df722, lcb/summary.json md5 35116aea642d36faa658952a3fc0e9e2). Both models were run on the same harness and graded with the same grader. A row is a tie when the two scores are within 1 point. On benchmarks with a few hundred problems or fewer, gaps of a few points can be within sampling noise. SWE-bench Verified: all 500 tasks, mini-swe-agent scaffold with a 100-step limit, official SWE-bench grader, run in batches across the day; jgram leads by 3.8 points (95% interval +0.8 to +6.8). LiveCodeBench: the same 175 problems, one attempt each, run on Oct 20, 2026; jgram leads by 9.1 points (95% interval +4.6 to +14.3). IFEval: the same 200 prompts as before, strict prompt-level scoring; jgram leads by 2.5 points (95% interval -1.5 to +6.5, which includes zero); a repeat jgram run scored 185/200. jgram routes each request across several models: in the SWE-bench Verified run about 91% of jgram's model calls were served by DeepSeek V4.1 Flash and about 8% by Kimi K3. TerminalBench 4.0 compares jgram Max with Kimi K3.1 Pro (not Kimi K3) and is not counted in the win, tie and loss tally above. Same 14 TerminalBench 4.0 tasks, one run each, with a 90-minute limit per task instead of TerminalBench's standard 8-hour limit. Kimi K3.1 Pro run in the standard Mini-SWE-agent harness; jgram Max measured in our test harness, with a live-API rerun on Tuesday. Raw results: handover/TB_K31_TBRULES_vs_MINISWE_2026-10-26.json md5 507d560243fa411c541956393df3495b, run Oct 25–26, 2026.