Sparra's evaluator earns its keep by actually running the artifact, which means running commands a model chose. That is also a security surface, and Sparra's own adversarial evaluator kept finding ways through it. I started the safe way I think most people would, by deny-listing dangerous binaries like rm. The evaluator's probe walked straight around it. find -delete and perl -e mutate the filesystem without ever naming a blocked command, and shell expansion like rm${IFS}victim.txt slipped past every substring check because it never spells rm out. Even allowlisting a build tool was not enough on its own, since npm version patch is an allowed binary running a subcommand that quietly rewrites package.json.
So I inverted the whole posture to allowlist-by-default. Instead of blocking known-bad commands, the executor now runs only known build and test runners, and only their safe subcommands: test, run <script>, and the like. Interpreter-eval flags are rejected, mutating verbs like cargo publish or npm version are refused, any shell metacharacter or expansion token is rejected before spawn, and the command is tokenized and launched with no shell at all. What used to be "everything runs unless it looks dangerous" is now "nothing runs unless it is a shape I recognize as safe."
The subtler fix was closing a laundering hole the evaluator caught. A verify command the executor refuses to run could previously just be skipped while the verdict stayed green, which means an artifact could "pass" a check that never happened. Now an unsafe verify command bounces back into contract negotiation at probe time, exactly like a command with a typo would, and at rerun it carries a dedicated unsafe status that demotes the pass and names the command. The executor still only vets the shape of a command, not the contents of a project's own recipe like make test, and that residual stays inside the worktree boundary. The part I keep enjoying about Sparra is that none of this came from me imagining attacks. It came from the tool's own evaluator attacking its own executor, round after round, and me hardening it until the attacks stopped landing.