What was delivered
- Hardware: Mac mini, Apple M5 Pro (15-core CPU, 16-core GPU), 24 GB unified memory, macOS 27. Numbers below were measured on 17–18 October 2026.
- Text: oMLX serves the models through an OpenAI-compatible API. The Qwen3.8-9B Distill is the everyday default at about 54 tokens a second, with a 32K window.
- Contest: Bonsai 2 27B (Qwen3.8-27B squeezed to ternary weights) won with 85 of 100 at 23 tokens a second; the 9B distill was second with 80; Gemma 4 12B and Ornith-1.5-9B 73; the official Qwen3.5-9B 72; MiniCPM5-2B 50 at 172 tokens a second.
- Coding: every program was run. Only Bonsai and Gemma wrote a tic-tac-toe program that never loses; in a warehouse stock task Bonsai scored 79 and Gemma 71, the 9B models 28 and 0.
- Games: the models wrote Connect Four AIs for a 112-game tournament, then Bonsai played Connect Four itself, reading the board as text each turn: 3 wins, 3 losses.
- Vision: a photo of 38 poker chips in four rows. No model counted every colour right; Bonsai came closest with 37, and Gemma and Qwen3.5-9B were off by one or two in most rows.
- Images: Qwen-Image 2.1 Turbo in ComfyUI, about 70 seconds per 1024 × 1024 image, with Chinese and English text drawn correctly.
- Music: MiniMax Music 3 in audio.cpp composed a 4-minute Chinese song from hand-written lyrics in about 30 minutes; ACE-Step 1.5 re-sang the lyrics over the melody of my own Suno song, 5 minutes of music in under 3 minutes; Qwen3-ASR listened back to check the words.
How it is built
- Four engines share one pool of unified memory, so one heavy job runs at a time: oMLX for text, a separate MLX runtime for Bonsai 2, ComfyUI for images, audio.cpp for music and speech.
- Claude Code ran the whole lab from a phone through Remote Control: installing, downloading, writing the test harnesses, running each model's code and grading the answers.
- Third-party loader code was read before it was run and kept in its own Python environment.
Who scored, and how
- The judge was Claude, running in Claude Code. The models never graded themselves or each other.
- Ten questions: each had a known correct answer. Claude read every answer and gave 0 to 10, taking points off for wrong facts, invented names or sources, and unsafe advice; an answer that never reached a conclusion scored 0.
- Code was not graded by reading it. An automatic referee ran it: every possible tic-tac-toe game, 31 hidden tests for the warehouse task, and rule, tactics and tournament tests for Connect Four. Only the documentation part was judged by Claude.
- Connect Four games were refereed by a program; Bonsai's results are wins and losses, not opinions.
- Images were rated by Claude looking at each one against its prompt. For the songs, Qwen3-ASR transcribed each one and the words were compared with the lyrics; whether it sounds good is left to your ears.
- One judge and one run per question: a hands-on comparison on one machine, not a formal benchmark.
What went wrong, and the fix
- oMLX's safe memory mode thought only about 2 GB was free and refused to load a 5 GB model. Fix: a fixed 12 GB limit.
- The distilled model's MTP speed-up accepted 0 of 270 draft tokens and made it slower. Fix: leave MTP off.
- The same model silently dropped images: its file has no vision part. Images go to Gemma or Bonsai instead.
- The 9B models often thought in circles until they ran out of tokens. Bonsai reasoned best but invented names and sources in Chinese history.
- Bonsai's moves were cut off before it named a column. Fix: a larger budget per move and a word limit on its analysis.
- Qwen-Image failed at the first step: the GGUF text encoder lacked its vision part, so ComfyUI misread the model. Fix: the official int8 encoder.
- MiniMax Music 3 in ComfyUI ran on the CPU, then hit an operation the Mac GPU does not support: two hours for 90 seconds of music. Fix: audio.cpp on Metal, more than ten times faster.
- Long-document caches had grown to 26 GB on the SSD. Fix: cleared, and capped at 10 GB.
Why it matters
A small desktop now answers questions, writes code, draws images and makes songs with nothing leaving the room. The tests also show where the limits are: local models make small mistakes in almost every program, so the code has to be run and checked, and a measured contest, not a model card, is the way to choose which model does which job.