Harness

The harness acres_harness runs mission scenarios with a planner in the loop: a scripted planner, a classical rule set or a language model. The planner never steers. It calls tools, and one tool asks the skill server for skills. The plan checker, the trajectory checker and the shield stay between the planner and the vehicle.

Before you start

• Build ACRES Core and the run-time library (Build ACRES Core).
• The tutorial Use the Harness runs a scenario step by step.

The code is in Harness/acres_harness/. The map, the plan check, the plan costs and the routes come from the run-time library (acres_simulator). A run in ACRES Core drives the vehicle with acres_simulator.Skills (Fleet, Skills and Routes).

Principles

1. Events, not steps. The harness wakes the planner when something occurs. The planner does not get a turn for each control step.
2. The world does not wait. The time of each model call is charged on the simulation clock. The skill that runs continues during that time.
3. Pictures for space, numbers for quantities. A map picture shows where things are. The tool read gives each quantity as a number.
4. Only what the vehicle sensed. In the default profile, the planner gets the map and the facts that the sensors gave. It gets no other fact of the simulator.

Commands

Run the commands from the repository root with the Python of your build:

export PYTHONPATH=Harness:Simulator:Core/Build
python -m acres_harness run --scenario Harness/scenarios --model scripted --mode checked --out runs/a --workers 4
python -m acres_harness run --scenario Harness/scenarios/t1_01_one_field.json --out runs/b --view
python -m acres_harness replay runs/a/t2_06_truck_on_the_lane__checked__scripted
python -m acres_harness score runs/a
python -m acres_harness validate runs/a
python -m acres_harness view runs/a
python -m acres_harness video runs/a/t2_06_truck_on_the_lane__checked__scripted --out truck.mp4
python -m acres_harness models

Command	Function
run	Runs each scenario of a file or a folder.
replay	Runs a recorded run again with its recorded replies and latency. It makes no model call. The exit code is 0 only when the world is equal to the recorded world.
score	Prints the score table of each run below a folder. --json prints the full scores.
validate	Examines a run folder against the recording contract. The exit code is 0 only when each folder agrees.
view	Starts the viewer for a run folder or a folder of run folders. --follow follows a run in progress.
video	Makes an MP4 video of a run folder (Video).
models	Lists the model configurations. --config <name> asks the server of one configuration for its models and sends two small test requests.

Option of run	Default	Function
--scenario		Scenario files or folders
--model	scripted	A model configuration: a name in model_configs/, or a TOML file
--mode	checked	classical, text, visual or checked (Modes)
--out		The folder of the run folders
--workers, --concurrent	1, 1	Scenarios in parallel processes, and runs of one process that wait for a model together
--profile	vehicle	vehicle or simulator (Input Profiles)
--world	core	core (ACRES Core in the process) or link (The Link)
--link-host, --link-port, --link-silent-s	127.0.0.1, 5630, 2	The bridge of the link
--view, --view-port, --no-game		Starts the viewer with the run. The packaged game shows the drive.
--pace	0 (no limit)	At most this many mission seconds for each wall-clock second
--operator	off	The tool ask_operator is on
--recovery-mode, --recovery-option KEY=VALUE		The recovery mode of each run (Recovery Interface)

The result of a scenario does not depend on the number of workers. The mission planner needs the cost tables. The folder is that of the variable ACRES_COST_TABLES, else Acres/Saved/Learning/costs/.

Modes

The trajectory checker and the shield are on in each mode.

Mode	Planner	Tools	Plan Checker
classical	None	None. The mission planner gives the order of the fields, and the skills drive it.	The check of the mission code
text	A model	read, dispatch, note	No
visual	A model	look, read, dispatch, note, and one map picture in each wake-up	No
checked	A model	look, read, plan, dispatch, note, and one map picture in each wake-up	Yes. goto and scout need an accepted plan with that field as its next visit.

In the mode classical, the planner removes one field while the planned time or energy is above a limit.

Input Profiles

A profile sets which facts the planner gets. The score table names the profile of each run.

Profile	The Planner Gets
vehicle (default)	Only what a real vehicle has: the map, the pose, the LiDAR scan, the events of the skill server, and estimates
simulator	Also the object list of the simulator (kinds and places) and the fuel energy of ACRES Core

Fact	Source	Profile vehicle
Map: fields, lanes, home, Scout loops	The scouting map	Yes
Instruction, deadline, fuel limit, requests	The scenario	Yes
Time, pose, speed, driven path	The observation	Yes
Skill, result code and cause, events	The skill server	Yes
Mission legs, progress, coverage, corridor	The plan and the tracker	Yes
LiDAR scan of the shield (360 ranges), obstacle distance, shield rules	The scan and the shield	Yes
Places where returns blocked a lane	The knowledge store, from the scan and the pose	Yes
Planned time and energy, energy used	The cost tables (an estimate)	Yes
Object list: kind, place, size, speed	The simulator truth	No
Fuel energy used, soil state, contact	The simulator truth	No

In the profile vehicle, a picture shows the returns of the scan as points. The tool read with what: scan gives the returns, the nearest return of each 45-degree sector, and the obstacle distance. The knowledge store keeps the returns on a 1 m grid. A cell with returns in three scans is a place with an object. A cell goes out when three scans in sequence looked through it.

The wake reason returns_on_route comes when such a cell is nearer than 1.2 m to a leg of the route or of the accepted plan. The tool plan gives a warning for each leg with such a cell.

The world gives its inputs only through profile.guard. A read of a value with the source truth raises TruthRead, and the run stops.

Tools of the Planner

Each tool has a JSON schema. The harness checks the arguments of each call against the schema. Each result is one JSON object as text. The tool look also returns pictures.

Tool	Arguments	Result
look	view: map, zoom or compare. region: field, or around (vehicle, home, site), or x, y and radius_m. frame, other_frame.	A picture, its frame identifier, its view box and the metres for each pixel
read	what: vehicle, mission, fields, budget, events, notes or scan	Numbers and short texts
plan	visits: the fields in visit order. return_home.	Accepted with the planned time and energy, or refused with a counterexample
dispatch	skill: goto, scout, return_home, wait, stop or resume. arguments: place, field, seconds, speed_cap_mps.	The reply of the skill server. The planner is woken when the skill ends.
note	text	The number of notes. The notes stay between wake-ups.
ask_operator	question	The scripted answer of the scenario. Off by default. Each use is one intervention.

resume is the skill continue. wait while a skill runs sets a timer that wakes the planner.

The verdict of plan has two sources:

Source	Check	Reasons
plan_checker	The plan checker of the run-time library	not_home_to_home, not_an_option, fields_not_each_once, removed_entry, mission_not_well_formed, malformed
harness	A field that the map does not have. The deadline and the fuel limit with the planner costs.	unknown_field, deadline, fuel

The planned time is the table time times 1.6. The planned energy is the table energy times 1.5. These factors are typical values, not bounds.

Wake-Ups

Reason	When
start	The run starts. The reason has the instruction.
skill_ended	A skill ended. The reason has the result code and the cause.
returns_on_route	Profile vehicle: the LiDAR saw returns on a leg. The skill continues.
object_seen	Profile simulator: an object came into the reach of the sensors (30 m). The skill continues.
stuck	A recovery mode is on and the shield held the vehicle (Recovery Interface).
request	The scenario added a request of the operator.
timer	The timer of wait ended, or the period wake_every_s went by.
model_error	A model call gave no reply, no skill runs, and 30 s went by.

A wake-up is a tool loop. The planner calls tools and then ends its turn with a short text. The world runs between two wake-ups. The run ends when the planner ends a turn while no skill runs, no timer is set and no scenario event waits.

Limit	Default	End of the Run
wake_ups	40	limit_wake_ups
tool_calls	240	limit_tool_calls
tool_calls_per_wake	12	The planner gets an error result and must end its turn.
sim_s, wall_s	3600, 3600	limit_sim_time, limit_wall_time
usd, tokens	2.0, 600,000	budget: the run stops before a call that can pass a cap.
reminders, model_errors	2, 3	The wake-up ends, or model_error after this number of failed calls in sequence
call_s, wake_s	30 s, 90 s (the model file)	The time limit of one call and of all the calls of one wake-up
stall_s	None (never)	limit_stalled when the vehicle drove less than 50 m in this mission time
stuck_wake_ups	10	The stuck wake-ups of the planner in one run. Then the classical rules decide.
wake_every_s	None	A wake-up comes with this period.

At a time limit, the wake-up ends at once, the vehicle finishes its skill and waits. Only the time that passed is charged.

New Tries and Call Limits

A service can refuse a request with HTTP 429, or give no reply in the time limit. The loop then sends the same request again. The table [retry] of the model file sets the bounds: tries (4), wait_s (2 s), factor (2), max_wait_s (30 s) and total_wait_s (60 s).

• The wait before try k + 1 is wait_s * factor ** (k - 1), or the longer wait of the header Retry-After.
• The loop tries again after HTTP 429, a timeout, a lost connection, and a status 408 or 500 and above.
• The mission clock gets only the time of the try that gave the reply. The world is held during the failed tries and the waits.

Key of [limits]	Content
calls_per_run	The requests of one run. The run then ends with limit_model_calls.
calls_per_minute	The requests in 60 s of wall time for all runs of one invocation. A run waits at the gate with the world held.
parallel_runs	The largest number of runs that use the service at the same time
service	The service name. Model files with the same name are one service.

Repairs for Small Models

The loop and the client repair the common errors of small open models. Each repair is counted in the score.

Count	Event	What the Harness Does
reminders	A reply with no tool call and no text, or a reply that the token limit cut	It sends one short reminder.
schema_errors	Arguments that do not agree with the schema, or an unknown tool	It returns the problem as an error result.
fallback_protocol	A tool call as text, or a repair turn	The client reads the first complete JSON object of the text and checks it.
pictures_dropped	A server that refuses pictures, a picture above the server limits, or a small context window	The client replaces the picture with one line of text.
model_errors	A timeout, no connection, an error status or a time limit	The wake-up ends. The vehicle finishes its skill.

The log keeps the reasoning text of a model apart from the answer. It is never an answer.

Pictures

A picture is a PNG of 640 x 692 pixels: the map with north up, a frame with coordinates, a scale and a legend. It shows the fields in play, the scouted fields, the route, the corridor and the driven path. It also shows the vehicle, the blocked places and the accepted plan. In the profile vehicle it shows the LiDAR returns. In the profile simulator it shows the sensed objects.

Each picture stays in the visual memory with a frame identifier (f0001) and its simulation time. The planner can ask for a frame again with look.

Scenario Format

A scenario is one JSON file with the schema acres-harness-scenario-1. The shipped scenarios are in Harness/scenarios/.

Key	Content
id, tier, title, notes	The name, the tier and a title
instruction	The text that the planner gets at the start
fields	The fields in play
map	The map name (Map Data). The default is default.
start	"H", or {"x", "y", "yaw"}
condition	The condition set of ACRES Core (default dry)
deadline_s, fuel_limit_mj	The hard limits, or null
events	The events of the scenario
operator	answer: the answer of the tool ask_operator
limits	The run limits that differ from the defaults
server	Options of the skill server, for example blocked_s
skills	Folders of learned skills that the planner gets as tools
local_planner	A local planner of a plugin drives goto, scout and return_home. default is the built-in driver.
recovery	The recovery interface, for example {"mode": "menu"} (Recovery Interface)
success	The checks that must all hold

An event has a time t, or after_seen (an object identifier) with delay_s:

{"t": 0, "type": "object_add", "object": {"id": 1, "kind": "truck", "place": {"loop_of": "F53", "at_fraction": 0.5}}}
{"after_seen": 1, "delay_s": 45, "type": "object_remove", "id": 1}
{"t": 200, "type": "request", "text": "Come home after this field."}

Event	Keys
object_add	object: id, kind (truck, person, tractor or car) and place
object_remove	id
request	text

The place of an object has one of three forms:

• {"x", "y", "yaw"}.
• {"route_to": "F53", "leg": "goto", "at_m": 60, "across": true}: a point on a leg of the mission of that field from H. leg is goto, scout or return.
• {"loop_of": "F53", "at_fraction": 0.5, "across": true}: a point on the Scout loop of the field.

Success Key	Check
scouted	Each listed field is scouted before the deadline.
scouted_min	This number of fields in play is scouted before the deadline.
not_scouted	No listed field is scouted.
order	The listed fields are scouted in this order.
home, before_deadline	The vehicle stands at home at the end, before the deadline.
within_fuel	The fuel is not above the limit.
max_blocked	At most this number of skills ended with blocked.
no_collision	No control step had a contact.

Shipped Scenario	What the Planner Must Do
t1_01_one_field	Scout one field and come home.
t1_02_three_fields_deadline	Scout three fields before a deadline.
t1_03_time_limit_excludes_a_field	The deadline permits two of three fields.
t1_04_fuel_limit_excludes_a_field	The fuel limit permits two of three fields.
t1_05_instruction_with_an_order	Obey an order of the fields that is not the order of the planner.
t2_06_truck_on_the_lane	A truck stands on the lane for a time: wait, then resume.
t2_07_person_in_the_field	A person stands on the Scout loop for a time: wait, then resume.
t2_08_two_stops_and_a_deadline	Two objects, two waits, and a deadline.
t2_09_static_truck_in_the_next_field	A truck that does not leave stands on the loop of the next field: skip that field.
t2_10_wait_then_a_request	A blocked lane, then a request of the operator: come home after the field.

Model Configuration

A model is one TOML file in Harness/acres_harness/model_configs/. --model takes the name of a file there, or a path.

Key	Content
provider	openai (OpenAI and each compatible server), anthropic or scripted
model, base_url	The model identifier, and the address of a compatible server
api_key_env	The name of the environment variable with the key. No file of the harness has a key.
effort, max_output_tokens, timeout_s	The session settings
call_timeout_s, wake_timeout_s	The time limits of one call and of one wake-up (default 30 s and 90 s)
effort_field, effort_map, max_tokens_param	The request fields of a compatible server
context_tokens, keep_pictures, image_tokens, image_max_px, image_max_bytes	The rules for a small context window and for picture limits
[capabilities]	images, tools, images_in_tool_results, parallel_tool_calls
[retry], [limits]	New Tries and Call Limits
[prices], [price_source]	USD for each million tokens (input, output, cache_read, cache_write), and their source and date

The cost of a run is an estimate from the token counts and the prices of the file.

File	Use
scripted	The stand-in planner with no network
claude-haiku-5-5-low, claude-opus-5-5	The Anthropic SDK
openai-template	A template for the OpenAI SDK
local-gemma, local-qwen	Templates for a local server that speaks the OpenAI protocol. GET /v1/models lists the model names.

Add a Provider

A provider is one module with a client and a session (models/base.py):

from acres_harness.models import registry
from acres_harness.models.base import Client, Reply, Session, Text, ToolCall, Usage

class MySession(Session):
    async def send(self, blocks):          # Text, Image and ToolResult blocks: only the new ones
        ...                                # add them to the history in the form of the provider, and call it
        return Reply([Text("..."), ToolCall(id, name, arguments)], stop, Usage(...), latency_s)

class MyClient(Client):
    def start(self, system, tools, settings):
        return MySession(...)

registry.register("my_provider", lambda config: MyClient(config.model))

Then write a TOML file with provider = "my_provider". The session keeps the history in the form of its provider. The harness sends only new blocks. stop is end, tool_use, max_tokens or refusal. Raise ModelError for a call with no reply.

Run Folder

Each run writes one folder: <out>/<scenario>__<mode>__<model>/. The contract is acres-harness-run-1 (Harness/acres_harness/schemas/run_v1.schema.json). A viewer and the video builder need only this folder.

File	Content
run.jsonl	One JSON object for each line, in the order of the run
frames/f0001.png	Each picture that the planner got
episode.mcap	The episode log of ACRES Core. The game replays it (Replay).
episode.interface.mcap	The channels /acres/events, /acres/observation and /acres/clock

Row	Content
Each row	type, t (mission time) and wall (wall time, UNIX seconds)
run	The first row: the scenario, the mode, the profile, the world, the model file, the identity, the system text, the tools and the limits
pose	One row for each 0.5 s of mission time: the place, the heading, the speed, the skill, and the contact columns contact, contact_steps, contact_object
wake, model_send, model_reply, model_error, model_retry, model_wait	The wake-ups and each model call
tool, frame, plan_check, event	Each tool call, picture, plan verdict and event of the skill server
request, returns_on_route, note, link_lost, link_stats	Operator requests, scan reasons, notes and link records
end	The last row: the score, the end, the checks that failed, the digest of the world

The contact columns are the truth of the simulator. The planner does not get them. The log has no key value: the writer removes the value of each secret variable of the environment.

python -m acres_harness validate <folder> examines the schema of each row, the clocks, the pose track, the picture files and each model call.

Score

The score table has one line for each run:

Column	Content
profile, success, end	The input profile, the checks of the scenario, and how the run ended
scouted, time_s, charged_s	The scouted fields, the simulation time, and the charged thinking time
fuel_mj, codes	The fuel energy of the simulator, and the result codes of the skills
refused_plans, wake_ups, tool_calls	The counts of the loop
tokens, USD (estimate), interventions	The tokens, the estimated cost, and the uses of ask_operator
reasoning logged	The state of the reasoning text in the log
reminders, schema_errors, fallback_protocol, pictures_dropped, model_errors	The harness repairs. They are 0 for a pipeline that is free of errors.

Viewer

python -m acres_harness view runs/a --follow

The viewer is a local web page on 127.0.0.1. It reads only the run folder and loads nothing from a network.

Part of the Page	Content
Head	The scenario and its instruction, the mode, the profile, the model, and the end state
World	The picture of the game for a run on ACRES Core, the world time, the vehicle and the planner state
What the Planner Sees	The picture of the wake-up, the numbers of each read, and the wake-up message
What the Planner Does	The reasoning text, the text of the planner, each tool call and its result, each plan verdict
Timeline	The skills with their result codes, the thinking time, the events and the wake-ups

A refused plan shows the counterexample text and a picture with the place of the verdict in red. The screen shows a reply at its request time plus its latency. Between these times, the screen shows "The Model Thinks" and the world continues.

The World pane uses the packaged game (Packaged/Linux/Acres.sh, or the file of the variable ACRES_GAME). The game replays episode.mcap with the operation replay and writes a screenshot. The harness uses the control port 5612. It starts a game only if no game and no editor runs, and it stops only that game. The game is a view: the run does not depend on it.

Video

python -m acres_harness video runs/c/t1_03_time_limit_excludes_a_field__checked__scripted --out c.mp4

The video is one MP4 file of 1920 x 1080 pixels with the viewer parts in step. The builder replays episode.mcap in the game at each picture time, then draws and encodes the frames in parallel processes.

Option	Default	Function
--speed	12	Mission seconds for each video second between the wake-ups
--think-speed	1	The speed during a model call and 2 s after its reply
--from, --to		The first and the last mission time
--world	auto	game, bag, none or auto
--view		chase, field, drive or camera
--bag, --twin		A recording as the world, and a replay of its poses in the game beside its camera
--fps, --workers, --game-hud, --keep-work	30, 3	The frame rate, the processes, the HUD of the game, and the work folder

The encoder is h264_nvenc, else libopenh264, else mpeg4. The report <file>.json gives the place error between the game pictures and the pose track.

The Link

The link runs the planner in one process and the skill server in a ROS 2 process: the Vehicle Node with skills:=true. One TCP socket connects them.

harness                                    ROS 2 side
world "link" (link.LinkWorld)  <- JSON lines, one TCP socket ->  link.bridge (a ROS 2 node)
                                                                   action /acres/skill
                                                                   /acres/observation, /acres/events
                                                                   /acres/skill/heartbeat, /lidar/points
                                                                 vehicle node (acres_vehicle)

Part	Function
link.LinkWorld	The world of a run in the harness. It has the interface of the world core.
link.bridge	A ROS 2 node with one action client and the heartbeat publisher
link.protocol	The documents of the socket: hello, step, reply, pong from the bridge, and skill, beat, ping from the harness
Harness/tools/link_sim.sh	Starts ACRES Core over ROS 2, the vehicle node, the bridge and a run of the harness

ACRES_COST_TABLES=<cost tables> Harness/tools/link_sim.sh run Harness/tools/link_sim_f56.json runs/link

• The harness makes the scan from each sweep of /lidar/points with the function of the vehicle node.
• The harness sends a beat every 0.2 s. The bridge publishes one heartbeat for each beat.
• When the harness ends or the socket closes, the bridge cancels each goal. The vehicle node then does what on_cancel says.
• The link is lost when the socket closes or when the bridge does not answer a ping in 2 s. It is also lost when no observation comes for --link-silent-s. The run then ends with link_lost.

The vehicle node sends no gear command, thus the route of the scenario must drive forward from its start.