We have been fine-tuning Gemma 4, Google's open-weights model, to write Hyperlambda. Today we ran the sixth training round through our full test suite. It wrote working code for 298 of 401 tests (74.3%). Our GPT-4.1-mini fine-tune, trained on the same corpus, scored 247 of 401 (61.6%) on the same tests, judged on the same backend, by the same judge, on the same day.
At a glance: this model is not released. The Hyperlambda Generator you use in Magic today still runs on our GPT-4.1-mini fine-tune. The Gemma model runs on our own hardware, and we will release it when it is ready. The 74.3% is what the model writes. The generator wrapper around it, as it stands, still rejects some valid long answers, and delivers 57.4%. That is a bug we are fixing, explained below.
Who ran this: my colleague Barnabas generated all 401 answers on his own machine, because the model is not exposed to the internet. Claude, the AI model I work with, then judged every answer from both models on one Magic backend, under my supervision.

The model
- Base model: Gemma 4 26B-A4B, Google's open-weights mixture-of-experts model, released under Apache 2.0. It has 26 billion parameters, of which about 4 billion are active for each token, which is what makes it cheap to run.
- Fine-tune: round 6 of our adapter, trained on the Hyperlambda corpus, which has grown past 600,000 snippets.
- Status: internal. Not downloadable and not in Magic yet.
The interesting part is not that a fine-tune beats its base model at a language the base model has never seen. It is that a model you can run on hardware you own now beats our best fine-tune of a hosted OpenAI model, by 51 tests.
The numbers
| Model | Tests passed | Pass rate |
|---|---|---|
| Gemma 4 26B-A4B, our fine-tune (round 6, unreleased) | 298 / 401 | 74.3% |
| GPT-4.1-mini, our fine-tune | 247 / 401 | 61.6% |
Head to head, test by test:
| Tests | |
|---|---|
| Both models pass | 224 |
| Only Gemma passes | 74 |
| Only GPT-4.1-mini passes | 23 |
| Neither passes | 80 |
Gemma wins 74 tests that GPT-4.1-mini fails, and loses 23 that GPT-4.1-mini passes. The gap is not noise in a handful of tests.
How the suite works
The suite has 401 tests in 24 groups, from database CRUD and endpoint generators to file formats, HTTP calls, cryptography, sandboxing, background tasks, MIME, error handling and live web scraping.
- Plain-language prompts. Every test is a task described in English, the way an AI agent or a developer would ask for it. No slot names, no code in the prompt.
- Held out. I wrote the tests separately from the training corpus, and none of them is in the training data. The training data teaches patterns, never the test prompts themselves.
- Executed, not eyeballed. Every answer is run on a live Magic backend, against real SQLite databases, files, HTTP endpoints and a mail server. A test passes when the code returns the right result or leaves the right side effects: the rows in the database, the file on disk, the request that was sent. Code that looks right but returns the wrong thing fails.
- One judge for both models. Both models' answers were judged by the same version of the judge, on the same backend, on November 16, 2026.
Four details, so nobody has to guess:
- Rejected answers. For 116 tests, the generator wrapper on Barnabas' machine rejected all three attempts and returned no code, mostly for long answers. For those tests we judged the model's first answer, at temperature 0. 64 of them pass. Most of them are valid code that our own static checker accepts, which is why I count the rejections as a wrapper bug and not as model failures. Counted the way the wrapper delivers them today, the score is 230 of 401 (57.4%).
- Missing answers on the other side. GPT-4.1-mini's generator returned no code for 19 tests, and we have no first answer to judge for those, so they count as failures. If every one of them had passed, GPT-4.1-mini would have reached 266, still 32 tests behind.
- One port number. Barnabas runs his backend on port 4444 and the judge runs on port 5000, so in the 16 answers that call the test server we changed the port. Nothing else in any answer was touched.
- Identical answers. 67 of Gemma's answers are character-for-character identical to the reference solution, and the judge passes those without executing them, since the reference solution passes by definition.
How this compares to Python
There is no benchmark where a model writes Hyperlambda on one side and Python on the other, so there is no apples-to-apples comparison, and I won't pretend there is. What the public numbers do show is that these two models are not weak at code to begin with:
| Model | Public Python benchmark | Score |
|---|---|---|
| Gemma 4 26B-A4B (the base model) | LiveCodeBench v6, from Google's model card | 77.1% |
| GPT-4.1-mini (the base model) | BigCodeBench-Instruct, official leaderboard | 48.9% |
Those are different tests from ours, and different from each other, so read them as context: both base models are competent Python programmers, and after fine-tuning they write a language that barely exists on the public web at a comparable level.
The very best models are well ahead of that in Python today. DeepSeek reports 93.5% on LiveCodeBench for its top model, and SWE-bench Verified leaderboards show 95–97% for the best coding agents, most of it self-reported and with tools and reasoning in the loop. Our numbers do not beat that, and this article does not claim they do.
What this does not prove
- It is our suite. I wrote the tests. They are held out from training, but an outside benchmark would be more convincing, and turning a set of public benchmark tasks into Hyperlambda tasks is on our list.
- It is one run. One training round, one run of 401 tests, judged on one backend.
- It is not what you get today. The model is unreleased, and the generator wrapper around it still rejects valid answers. Until both are fixed, Magic users get the GPT-4.1-mini fine-tune.
- Hyperlambda makes tasks shorter. One slot such as
http.getordata.selectdoes what takes many lines of Python, so there are fewer places to be wrong. That is a real advantage of the language, but it also means our tests are not directly comparable to Python tests of the same task.
Frequently asked questions
Can I use the Gemma 4 Hyperlambda model today?
No. It is unreleased and runs on our own hardware. The Hyperlambda Generator in Magic currently runs on our GPT-4.1-mini fine-tune.
How accurate is the new model?
It wrote working code for 298 of 401 held-out tests (74.3%), against 247 (61.6%) for our GPT-4.1-mini fine-tune. Every answer was executed on a live Magic backend and judged by its result.
Why does a 4-billion-active-parameter model beat GPT-4.1-mini here?
Both are fine-tuned on the same Hyperlambda corpus, and Gemma 4 is a newer and stronger base model for code: Google's model card gives it 77.1% on LiveCodeBench v6. With only about 4 billion parameters active per token it is also cheap to run on your own hardware.
Does this mean the model is better at code than Claude or GPT?
No. The best models score above 90% on public Python benchmarks. Our result is about one fine-tuned open model writing one language designed for AI.
When will it be released?
When the generator stops rejecting valid answers and the score is where we want it. We at AINIRO believe we will pass 90% on this suite in less than a month.
Sources
- Gemma 4 model card, Google
- BigCodeBench leaderboard
- LiveCodeBench, Codesota
- SWE-bench Verified, Vals.ai
What comes next
Two things: fixing the wrapper so it stops throwing away valid answers, and training the next round on the mistakes this run exposed. Round 6 still mixes decimal and floating-point numbers, sometimes forgets to declare typed arguments, and occasionally invents a slot that does not exist. Those are exactly the kind of mistakes the next round is trained to stop.
We at AINIRO believe we will make it to 90% or more in less than a month.