Skip to content

Use the Harness#

The harness runs a mission scenario from start to end. A planner gets an instruction, plans the fields, and drives the vehicle through skills. The harness records the run, scores it and can replay it. The planner can be a fixed script, a language model, or your own code.

Before you start

How a Run Works#

  1. The scenario gives the instruction, the fields, the start, the limits, the success rules and the events (for example a truck that comes onto a lane).
  2. The planner reads the instruction and the observations. It calls tools: plan, dispatch (a skill), read, look and note.
  3. The run-time library drives each skill with the vehicle software. The plan checker, the trajectory checker and the shield check each step.
  4. The harness writes the run folder and scores the run against the success rules.
Mode The Planner Gets
classical No model. Fixed rules plan and recover.
text Text observations
visual Text and pictures of the map and the LiDAR
checked Text, pictures and the result of the plan checker before each plan

Run a Scenario#

  1. Look at the scenario Harness/scenarios/t1_01_one_field.json.

    Harness/scenarios/t1_01_one_field.json
    {
 "schema": "acres-harness-scenario-1",
 "start": "H",
 "id": "t1_01_one_field",
 "tier": "T1",
 "title": "One Field",
 "instruction": "Scout field F53. Then come home.",
 "fields": ["F53"],
 "deadline_s": 600.0,
 "success": {"scouted": ["F53"], "home": true, "before_deadline": true, "no_collision": true}
}

  2. Run it with the scripted planner. It needs no network and no key.

    python -m acres_harness run --scenario Harness/scenarios/t1_01_one_field.json \
    --model scripted --mode checked --out runs/first

    Expected result

    The run takes approximately 2 minutes. The vehicle scouts F53 and comes home before the deadline.

    scenario         mode     profile  model     success  end       scouted  time_s  charged_s  fuel_mj  codes   ...
t1_01_one_field  checked  vehicle  scripted  yes      finished  1/1      254.3   28.0       3.2      done:2  ...
1 of 1 runs with success

time_s is the mission time. charged_s is the planner time that the harness adds to the mission clock. codes counts the result codes of the skills.

Read the Run Folder#

The folder runs/first/t1_01_one_field__checked__scripted has these files.

File Content
run.jsonl One row for each event: the scenario, each model call, each tool call and its result, the poses, the score
episode.mcap The episode log of the vehicle. Refer to Replay.
episode.interface.mcap The messages between the planner and the vehicle
frames/ The pictures that the planner got
Command Function
python -m acres_harness score runs/first The score table of all runs below a folder
python -m acres_harness validate <run folder> Checks a run folder against the recording contract
python -m acres_harness replay <run folder> Runs the world again with the recorded replies and compares it
python -m acres_harness view <run folder> A web page on 127.0.0.1 with the map, the log and the pictures
python -m acres_harness video <run folder> --out run.mp4 A video of the world and the panes side by side

The ten scenarios in Harness/scenarios go from one field to deadlines, fuel limits, a truck on the lane and a person in the field. Harness/scenarios/maps has a scenario on a map variant.

Use a Language Model#

A model file in Harness/acres_harness/model_configs describes a model: the provider, the model name, the time limits, the call limits and the prices.

python -m acres_harness models

Expected result

claude-haiku-5-5-low provider anthropic  model claude-haiku-5-5   in use: the live model of the harness checks
claude-opus-5-5      provider anthropic  model claude-opus-5-5    template: not in use yet (design decision)
local-gemma          provider openai     model REPLACE-WITH-A-NAME-FROM-GET-V1-MODELS   template: a local open model ...
local-qwen           provider openai     model REPLACE-WITH-A-NAME-FROM-GET-V1-MODELS   template: a local open model ...
openai-template      provider openai     model REPLACE-WITH-A-MODEL-NAME   template: not in use yet (design decision)
scripted             provider scripted   model scripted-rules-0   in use: tests and the check of the pipeline

To run a scenario with a model, put its key into the environment and give the model file:

export ANTHROPIC_API_KEY=<your key>
python -m acres_harness run --scenario Harness/scenarios --model claude-haiku-5-5-low --mode visual \
    --out runs/haiku --workers 2

Keys

Keep a key in the environment only. Do not write a key into a file of the repository or into a log.

A local model server with the OpenAI protocol (Ollama, llama.cpp or vLLM) works with the templates local-gemma and local-qwen. Set the model name first. python -m acres_harness models --config <name> checks one server.

Hook Your Own Code#

A plugin adds a local planner or a recovery mode without a change to ACRES. Put the module on the Python path and name it in ACRES_HARNESS_PLUGINS.

my_plugin.py
"""A plugin: a local planner that drives goto with the example skill slow_pass, and a recovery mode."""
from acres_harness import plugins
from acres_harness.recovery_modes import register
from acres_harness.recovery_modes.menu import MenuMode

plugins.local_planner("slow", skills_dir="Skills", drive_as={"goto": "slow_pass"})


@register("my_method")
class MyMethod(MenuMode):
    """The menu mode of the harness with the flexible options."""
    flexible = True

Copy Harness/scenarios/t1_01_one_field.json to slow_one_field.json, change id to slow_one_field and add the key "local_planner": "slow". Then run it:

export PYTHONPATH=$PYTHONPATH:$PWD    # the folder of my_plugin.py
export ACRES_HARNESS_PLUGINS=my_plugin
python -m acres_harness run --scenario slow_one_field.json --model scripted --mode classical \
    --recovery-mode my_method --out runs/plugin

Expected result

slow_one_field  classical  vehicle  none   yes      finished  1/1      234.5   0.0        3.2      done:2  ...
1 of 1 runs with success

drive_as maps a built-in skill to a skill of the folder. The two skills must have the same target. goto needs a place skill, scout a field skill and return_home a home skill. A built-in skill with no map entry uses the default driver. The classical planner of this example calls scout and return_home only. A planner that calls goto drives with slow_pass.

Train a Learned Skill makes a skill folder. The Recovery Interface gives the stuck report, the menu and the executor.

Next Steps#