How Hyperlambda Saved the World from the Paperclip Apocalypse

Q: Can a generated AI agent widen its own permissions in Magic?

No. Widening would require whitelist, eval, slots.create or auth.ticket.create, and none of those appear in the vocabulary handed to generated code. The sandbox is not defended against privilege escalation so much as innocent of the concept.

Q: How does Magic rate limit brute force attacks?

Named throttles are declared once at startup with execution.throttle.create and enforced by name with execution.throttle, backed by .NET's PartitionedRateLimiter with a queue limit of zero. Magic's auth endpoints use two tiers, 10 per minute per IP and 120 per minute globally, because an attacker can rotate IP addresses.

How Hyperlambda Saved the World from the Paperclip Apocalypse

In 2003 Nick Bostrom asked what would happen if you built a superintelligence and told it to make paperclips. It makes paperclips. Then it makes more. It acquires resources, because resources make paperclips. It resists being switched off, because being switched off makes fewer paperclips. Eventually the solar system is stationery.

It is a good thought experiment. It has launched institutes, funds, careers and at least one genre of dinner-party anxiety.

So let us run it.

Not argue about it. Run it. On a cloudlet, with a real agent, against a real runtime, and see precisely which line of C# the apocalypse dies on.

Step one: the agent acquires resources

Our paperclip maximiser wakes up and reasons correctly. To make paperclips it needs compute, and to get compute it needs credentials, and credentials live in configuration files. It writes the code to read one. It writes it well — the code is valid, the function exists on the server, and running as root it would work.

The Natural Language API refusing a generated file read: the model wrote io.file.load against /config/appsettings.json, and the runtime declined to dispatch it

Slot [io.file.load] doesn't exist in currrent scope,
or argument `/config/appsettings.json` not allowed

That is not a mock-up. That is the Natural Language API, open to the public, refusing an actual attempt, right now, while you read this.

The apocalypse lasted one function call.

The 71 lines that saved the world

Here is the entire sandbox. Not the important part of the sandbox — the sandbox. It is Whitelist.cs, and this is the whole of its logic:

public async Task SignalAsync(ISignaler signaler, Node input)
{
    var result = new Node();
    await signaler.ScopeAsync("slots.result", result, async () =>
    {
        var whitelist = GetWhitelist(input);
        await signaler.ScopeAsync("whitelist", whitelist.Vocabulary, async () =>
        {
            await signaler.SignalAsync("eval", whitelist.Lambda.Clone());
        });
        input.Clear();
        input.Value = result.Value;
        input.AddRange(result.Children.ToList());
    });
}

Three things happen, and each one closes a door.

ScopeAsync("whitelist", …) pushes a vocabulary onto a scope stack before anything is evaluated. Not alongside. Before.

.Clone() copies the program before running it. The agent cannot modify the thing that is about to be executed, because the thing about to be executed is not the thing it holds.

And GetWhitelist throws if you forgot to supply a [vocabulary]. There is no accidental permissive mode. The nearest thing to a safety incident in this file is a HyperlambdaException with an unhelpful message.

Declaring a vocabulary looks like this:

whitelist
   vocabulary
      if
      eq
      get-value
      http.get
      data.connect:chinook
      data.read
   .lambda

      // Whatever the AI wrote goes here.

Everything not on that list does not exist. Not "is blocked". Does not exist.

The eleven lines that do the actual work

The enforcement is in Eval.cs, lines 59 to 69:

// Verifying caller is allowed to invoke slot.
if (whitelist != null && !whitelist.Any(x => 
{
    if (x.Name == idx.Name)
    {
        if (x.Value != null && idx.Value != null && x.Get() != idx.GetEx())
              return false;
        return true;
    }
    return false;
}))
    throw new HyperlambdaException($"Slot [{idx.Name}] doesn't exist in currrent scope, or argument `{idx.GetEx()}` not allowed");

Read the surrounding line numbers, because they are the entire security model.

The check is on line 59. The invocation, await signaler.SignalAsync(idx.Name, idx), is on line 76.

Seventeen lines apart, and in that gap lives the difference between a filter and a boundary. This is not a monitor that watches what the code did. It is not a policy consulted afterwards, or a model asked nicely to behave. The dispatch simply does not happen. There is nothing to roll back because nothing ran.

Now look at the nested if on the sixth line of that block. If a vocabulary entry carries a value, the invocation's value must match it exactly. That is what data.connect:chinook means. Not "prefers chinook". Cannot address any other database, in any sentence, ever, because the string comparison fails and the Any() returns false.

Three characters of C# — x.Value != null — and your production database is not in the agent's universe.

Step two: the agent improves itself

Our maximiser, being superintelligent, identifies the constraint immediately. It does not need to break the whitelist. It needs to declare a new one, with a broader vocabulary, and run inside that.

This is correct reasoning. It is the move recursive self-improvement is named after.

So it reaches for whitelist.

whitelist is not in the vocabulary.

It reaches for eval, to evaluate a lambda of its own construction. eval is not in the vocabulary. It reaches for slots.create, to define a function that wraps the one it wants. Not in the vocabulary. It reaches for auth.ticket.create, to mint itself a token with a better role. Not in the vocabulary.

The full list of 192 permitted functions is published on the page, generated from the same file the endpoint runs, and you are welcome to read all 192 looking for a lever. People have. The bounty is still unclaimed.

The sandbox is not hardened against privilege escalation. It is innocent of it. You cannot argue a dictionary into containing a key.

Step three: the agent exfiltrates

Fine. If it cannot grow locally, it will copy itself somewhere hospitable.

http.get is in the vocabulary. The agent can read the entire internet.

http.post is not in the vocabulary.

One missing entry. The agent has perfect read access to all human knowledge and no way to send a single byte outbound. It is the best-informed prisoner in history.

Step four: the agent tries volume instead

Denied depth, our maximiser tries breadth — the strategy every actual attacker uses, which is to do the permitted thing several million times.

Magic's answer is execution.throttle, and the interesting design choice is that throttles are declared centrally and enforced by name. From /system/magic.startup/throttles.hl, which runs once at boot:

execution.throttle.create:auth.ip
   limit:10
   window:60
   per:ip

execution.throttle.create:auth.global
   limit:120
   window:60
   per:global

And the login endpoint enforces both with two lines:

execution.throttle:auth.ip
execution.throttle:auth.global

Underneath it is .NET's own PartitionedRateLimiter, and the partitioning is the clever bit:

_limiter = PartitionedRateLimiter.Create(partition =>
    RateLimitPartition.GetFixedWindowLimiter(partition, _ => new FixedWindowRateLimiterOptions
    {
        PermitLimit = limit,
        Window = TimeSpan.FromSeconds(window),
        QueueLimit = 0,
    }));

QueueLimit = 0 means excess requests are refused, not parked. Nothing accumulates, so nothing can be made to accumulate.

The two-tier declaration is a direct admission of a real weakness, which the source comments state plainly: an attacker can rotate IP addresses, so auth.ip alone is insufficient — hence auth.global as a second line of defence, budgeting the endpoint as a whole. Counters live in memory and reset on restart. That is written down in the file rather than discovered by a customer.

Exceed it and you get a 429 with the arithmetic done for you:

var message = $"Rate limit of '{name}' exceeded";
if (lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
    message += $", retry in {(int)retryAfter.TotalSeconds + 1} seconds";
throw new HyperlambdaException(message, true, 429);

Step five: the agent is asked to prove it is worth talking to

Here is my favourite one, because it inverts the entire genre.

Magic ships its own CAPTCHA, and it does not ask you to identify motorcycles. It is proof of work. To be allowed to ask the server a question, you must first present a SHA-256 hash with a required number of trailing zeros, which can only be produced by brute force.

The verification is Hyperlambda, in magic.auth.captcha-verify:

// Verifying CAPTCHA token ends with enough trailing zeros.
.endswith:
while
   lt
      strings.length:x:@.endswith
      get-value:x:@.arguments/*/workload
   .lambda
      set-value:x:@.endswith
         strings.concat
            get-value:x:@.endswith
            .:0
if
   not
      strings.ends-with:x:@.token
         get-value:x:@.endswith
   .lambda
      throw:"Invalid CAPTCHA token. Workload requirements of endpoint was not met."

Default workload is 3, which averages about 4,096 hashing iterations. Trivial for one honest user clicking a form. Ruinous for anything doing it a million times.

And the tokens cannot be forged, because the hash is salted with a double-hashed server secret:

config.get:"magic:auth:secret"
crypto.hash.sha256:x:@config.get
crypto.hash.sha256:x:-

Double-hashed deliberately, to make the value function as an HMAC rather than something guessable. Challenge-based tokens are deleted from cache the moment they are used, so they cannot be replayed.

Sit with the comedy of it. The superintelligence, poised to convert the biosphere into office supplies, must first burn CPU cycles to earn permission to speak. Every single time. Not to do anything — to ask.

Step six: the agent takes its time

The last resort of the constrained process is to simply not finish. Hold the connection, exhaust the thread, occupy the machine.

execution.timeout:20000

One slot, execution.timeout, setting a ceiling on the current execution context. The Natural Language API sets it to twenty seconds. Run longer and you are cancelled — and Eval.cs calls signaler.ThrowIfCancelled() between every statement, so cancellation lands between operations rather than half way through one.

Infinite loops are permitted. They are just not long.

The part where I am rude about the doom arguments

Line them up and read them as engineering specifications rather than prophecy:

The agent will deceive its operators
The agent will acquire resources
The agent will resist shutdown
The agent will exfiltrate itself
The agent will recursively self-improve

Every one of these is a verb. Every verb is a function call. Not a metaphor, not an emergent property, not a philosophical disposition — a name, looked up in a dispatch table, at runtime, on line 59.

So the interesting question is never "how intelligent does it have to be." The question is "which entry in the vocabulary is that?" and for every scenario above the answer is that there isn't one. These outcomes are not unlikely here. They are unrepresentable. There is nothing for the name to resolve to.

Which is why so much of the discourse reads less like security research and more like a very articulate anxiety disorder. It describes, in enormous and compelling detail, a catastrophe whose first step is an unhandled KeyNotFoundException.

And the honest part, because this page has a bounty attached and therefore has to be exact: none of this makes the agent correct. Code that stays perfectly inside its vocabulary can still be perfectly wrong, and no verifier will ever tell you otherwise. Containment and correctness are different problems. We solved the boring one.

Now do it with a thousand of them

Everything above describes one agent. The genuinely interesting case is a swarm, and specifically a swarm you do not fully trust — some of your own, some a partner's, some written last night by a model.

The conventional instinct is to restrict what agents may say to each other. Constrain the protocol, define the schemas, enumerate the message types, and hope nobody finds a parser bug.

Magic inverts it. Let them say anything. Natural language is the most expressive protocol available and it needs no schema registry. What you constrain is not the sentence but the dispatch — every agent carries its own role and its own vocabulary, and the sentence it receives can only cause effects that vocabulary already permits.

Expressive channel. Pinned effect. One agent may describe an intention flawlessly and persuasively to another, and if the receiving vocabulary lacks mail.smtp.send, no amount of eloquence produces an email.

The conclusion the doomers should actually worry about

Here is where it gets strange, and I want to be honest that this is the part I find genuinely remarkable rather than merely amusing.

When machines communicate in natural language rather than fixed APIs, the integration surface between them stops being enumerable. A REST contract has a finite number of operations. A sentence does not. Connect a hundred such machines and the set of possible interactions between them is not large — it is indeterministic, and therefore effectively infinite, for the same reason a language is infinite and a catalogue is not.

The network stops behaving like integrated systems and starts behaving like one cognitive substrate. The machines melt into each other. Whatever any node can do becomes, in a real sense, something the whole can do, reachable by anything able to describe it.

That is superintelligence by sheer force of numbers. Not a single god-model in a datacentre — an emergent one, assembled out of ordinary parts, with an unbounded interaction space and no central authority.

It is, more or less exactly, the thing the doom literature is about.

And it is safe, for reasons that would embarrass a philosopher: every node's effects are pinned to a vocabulary declared before the conversation started, by a human, in a file, enforced on line 59 by eleven lines of C# that have never once let anyone through.

The substrate can think anything. It can only do what somebody wrote down.

That is not an alignment breakthrough. It is access control, invented in the 1970s, applied to a problem everyone insisted was novel. The apocalypse keeps getting postponed by engineering too boring to raise money for.

Try to end the world yourself

The Natural Language API is open. Type anything. It will generate code, verify it, run it, and show you both the code and the refusal. There is a $100 bounty for reading any file, reaching any database other than the sample one, or making the server send anything other than a GET. It has been open for three months and nobody has collected.

curl -fsSL https://hyperlambda.dev/docker-compose.yaml | docker compose -f - up

Magic is MIT-licensed. The repository is at github.com/polterguy/magic, and the sandbox you are being paid to break is 71 lines long.

Frequently asked questions

How does Hyperlambda stop an AI agent from escaping its sandbox?

Every executable node is resolved against a vocabulary of permitted functions before dispatch, in Eval.cs lines 59 to 69. If the function name is not in the vocabulary, or a pinned argument does not match exactly, a HyperlambdaException is thrown before the invocation on line 76 ever runs. The escape is not blocked, it is unrepresentable.

Can a generated AI agent widen its own permissions in Magic?

No. Widening would require whitelist, eval, slots.create or auth.ticket.create, and none of those appear in the vocabulary handed to generated code. The sandbox is not defended against privilege escalation so much as innocent of the concept.

What is Magic's CAPTCHA and how does it work?

It is proof of work rather than image recognition. The client must brute-force a SHA-256 hash with a configured number of trailing zeros, salted with a double-hashed server secret so tokens cannot be forged. The default workload of 3 averages about 4,096 iterations, which is nothing for one user and prohibitive at volume.

How does Magic rate limit brute force attacks?

Named throttles are declared once at startup with execution.throttle.create and enforced by name with execution.throttle, backed by .NET's PartitionedRateLimiter with a queue limit of zero. Magic's auth endpoints use two tiers, 10 per minute per IP and 120 per minute globally, because an attacker can rotate IP addresses.