Running Details & Empirical Results

We built a local simulation and testing sandbox to execute the experiments. This page documents the methodology, datasets, and the exact findings of our empirical runs.

1. Sandbox Architecture & Methodology

Running LLM agent benchmarks in the cloud can be slow and expensive. To ensure reproducibility and speed, we implemented a local ML task sandbox:

• Tasks:
  1. spaceship_titanic: Synthetic tabular dataset modeling the Kaggle Spaceship Titanic task (with columns like HomePlanet, CryoSleep, Cabin, and missing features).
  2. wine_quality: Load standard chemical features from the scikit-learn Wine dataset and inject missing values to create preprocessing challenges.
  3. synthetic_classification: A 10-feature generated classification dataset with custom null rates.
• Execution: For each action, the agent performs real Python operations: copying files, running feature imputation, fitting RandomForest classifiers, saving pickle models, and outputting predictions to submission.csv.
• Errors: The generator has a configurable DFA Error Rate (proposing invalid steps out of order) and a Bug Rate (writing buggy code that deletes files, writes null values, or outputs malformed prediction lengths).

2. Main Experiment Results

We ran 10 trials per task.

Task-by-Task Summary

Task 1: Spaceship Titanic

• Control (G): Completion Rate = 50%, Mean Accuracy = 30.8%
• Treatment A (G + DFA): Completion Rate = 90%, Mean Accuracy = 57.0%
• Treatment B (G + LLM): Completion Rate = 100%, Mean Accuracy = 30.8%

Task 2: Wine Quality

• Control (G): Completion Rate = 60%, Mean Accuracy = 25.1%
• Treatment A (G + DFA): Completion Rate = 90%, Mean Accuracy = 34.6%
• Treatment B (G + LLM): Completion Rate = 100%, Mean Accuracy = 25.1%

Task 3: Synthetic Classification

• Control (G): Completion Rate = 60%, Mean Accuracy = 49.7%
• Treatment A (G + DFA): Completion Rate = 90%, Mean Accuracy = 81.7%
• Treatment B (G + LLM): Completion Rate = 100%, Mean Accuracy = 49.7%

Overall Aggregated Performance

3. Ablation Studies

We ran two ablations (8 trials each, with an increased noise level: 15% DFA errors, 25% bug rate) to understand verifier sensitivity.

Ablation 1: Verifier Components

This experiment checks if structural checks (DFA) and content checks (Data) are both required:

• Analysis: Under high noise, running without a DFA (Control and Data-Only) leads to a 0% score because the agent gets completely lost in transition sequences. Having DFA Only maintains routing (44.8% score) but lets data bugs slip through. The Full Verifier achieves the best score (49.5%).

Ablation 2: Retry Budget

This experiment evaluates the agent’s performance as a function of the max allowed retries:

• Analysis: The repair count saturated around 3.13. Having at least 3-5 retries ensures that even complex multi-step failures can be successfully repaired.

4. Experiment 2: Chomsky Class Compression Results

In this experiment, three agent structures (ReAct, Tree Search, and Planner-Executor) were run on the tasks. Their traces were projected onto the core pipeline alphabet $\Sigma_{core} = {\text{load_data}, \text{train_model}, \text{submit_predictions}}$.

The pairwise normalized Levenshtein similarity matrix of their projected languages is:

• Analysis: The ReAct and Planner-Executor agents produced identical projected traces, collapsing to the exact same trace language ($L(A_{ReAct}) \equiv_\tau L(A_{Planner})$ under projection), validating Chomsky Class Compression.

5. Experiment 3: Emergent Sub-Agent Discovery Results

Using the transitions from 100 simulated runs, we applied K-Means clustering ($k=3$) to automatically group the 8 operation symbols into clusters representing emergent sub-agents:

6. Experiment 4: Trace-Language Analyzer Results

We ran our LanguageAnalyzer on the traces from different agent loops to predict their Chomsky class:

7. Experiment 5: Safety-Critical Verification Results

An adversarial agent attempted to execute dangerous operations or skip required stages. The safety verifier $V_{safe}$ intercepted and blocked the actions as logged below:

• Interception Rate: 100.0% (3 out of 3 safety-critical violations blocked successfully).

8. Experiment 6: NeuroGolf 2026 Results

Competition Background

NeuroGolf 2026 is an ARC-AGI style competition where participants build ONNX-format solvers for 400 grid-based tasks (3×3 to 30×30, 10 color channels). Scoring: points = max(1, 25 - ln(max(1, cost))) where cost = params + memory. File size limit: 1.44 MB for submission.zip.

Methodology

We applied the trace-language framework to generate a competition notebook:

1. Log Harvesting: Downloaded execution logs from 18+ top-scoring Kaggle kernels (including the #1 notebook at 6154.71 pts, a 6411.7-pt multi-source solver, and a 6663.23-pt blend). Extracted the canonical pipeline: discover bundles → load floor → analyze tasks → build ONNX → optimize (graph rewrite, dim scrub) → verify → cost → blend → sha256 → package → submit.

2. DFA Construction: Built a 13-state, 76-transition DFA with 27 OpSymbols and 29-keyword coverage set. The verifier checks both operation ordering and keyword presence at each pipeline phase.

3. Notebook Generation: An LLM generates the competition notebook under DFA supervision. The verifier runs at three checkpoints: after the build pipeline, after blending, and at final submission.

Results

Key Findings

• 50% improvement over the unguided baseline (2739.27 → 4127.12)
• 4 public datasets discovered and blended correctly (octaviograu 6154.71, jsrdcht 6029, afr1ste 6335, konbu17 5331)
• 398/400 tasks covered by pre-built bundle models vs 169/400 before DFA-guided blend fix
• The DFA verifier caught the inverted blend condition that was discarding bundle models in favor of identity solvers
• 0.74 MB submission (48% under the 1.44 MB limit) via raw-bytes storage — the DFA’s structural check on packaging helped identify the re-serialization bloat

DFA Verification Results

All three pipeline checkpoints passed:

[Build Pipeline]  state=PACKAGED  accepted=True
[Blend Pipeline]  state=SUBMITTED accepted=True
[Final Submit]    state=SUBMITTED accepted=True

Error detection correctly rejected malformed traces (build without analysis, blend before verify), confirming the DFA’s discrimatory power.

Practical Implications

This experiment shows that the trace-language framework is not just theoretical — it works on real competitions. A DFA verifier, with no semantic understanding of ONNX graphs or ARC-AGI tasks, was sufficient to guide an LLM generator toward:

1. Using the correct dataset paths (Kaggle’s /kaggle/input/datasets/owner/slug/ structure)
2. Discovering all available bundle datasets
3. Blending correctly (prefer bundle models over identity, compare costs)
4. Staying under file size limits (raw bytes vs re-serialized ONNX)
5. Emitting SHA256 checksums and expected scores matching real notebook conventions

The verifier cannot understand ONNX opsets, cost formulas, or ARC task patterns — but it enforces the architectural trace that makes a submission valid.