Map Data#

This page lists each input of a map, tells you which file has it, which program makes the file and which program reads it. It also tells you how a map is one data folder and how the generator makes a variant of the default map. The last section tells you what the packaged game needs to show a variant.

A Map Is One Folder#

A map folder has the layout of Acres/Content/Simulation. The default map is that folder of the repository. A variant is the folder Acres/Content/SimulationMaps/<name> of the repository, or <ACRES_MAPS_DIR>/<name> when you set the variable ACRES_MAPS_DIR.

Program	How It Gets the Map
ACRES Core	`FAcresWorldOptions::ContentDir` is the map folder.
The library `acres_runtime`	Each function that makes a world has the option `content_dir`. `acres_runtime_map_folder(root, name, ...)` gives the folder of a name. `acres_runtime_identity_map` and `acres_runtime_map_hash_folder` give the identity of a folder. `acres_runtime_baseline_run` has the option `content_dir`, and `acres_runtime_baseline_tables_map` drives the cost tables of a folder.
The package `acres_simulator`	`acres_simulator.maps.use("<name>")`, or the variable `ACRES_MAP`, selects the map of the process. A `Core`, a `Vehicle`, the planner of `plans` and the identity then use that map. The commands `acres_simulator.baseline` and `acres_simulator.tables` have the option `--map`.
The harness	The key `map` of a scenario is the name of the map. The scenario `Harness/scenarios/maps/m1_01_one_field_on_a_variant.json` is an example.
The packaged game	The switch `-MapData=<folder>` selects the map folder of the process (see "The Game").
The vehicle node	The node uses the default map of its tree. A parameter for the map folder is not made.

With no name and no folder, each program uses the default map. One process has one map at a time, because one process has one pool of threads and one world. To run more than one map in one run, use acres_simulator.maps.run. It starts one process for each map, with a limit on the number of processes. The results come back in the order of the names.

python -m acres_simulator.maps                         # the maps with their hashes
python -m acres_simulator.maps --tables all --threads 4   # the cost tables of each map, into the cache
python -m acres_simulator.maps --run default,acre-split --jobs 2 --out <folder> -- \
    python -m acres_simulator.baseline --out {out} --threads 2 --fields F56,F04

The cost tables of a map are in the cache folder Acres/Saved/Learning/costs/<map_hash>/v<sim_version>/.

The Files of a Map Folder#

The column "Drive" tells you if a simulated drive of ACRES Core or of the library reads the file. Such a file is a part of map_hash.

Input	File in the Map Folder	Made By	Read at Run Time By	Drive
Heights of the tile	`ACRE/heights.f32` (1001 x 1001 values, 1.524 m), with the grid in `ACRE/site.json`	`Tools/ACRE/Map/acre_map.py` and the edit scripts (`field_edits.py`, `building_models.py`) from the LiDAR survey	ACRES Core (`AcresCoreTerrain.cpp`), the game (`AcresVehicle.cpp`, `AcresSoilVisuals.cpp`, `AcresFarmRuntime.cpp`)	Yes
Site: grid, roads for the cars, buildings, silos, georeference	`ACRE/site.json`	`Tools/ACRE/Map/site_edits.py`, `refresh_site_roads.py`	ACRES Core, the library (`sensor_georef.cpp`), the game (road graph, HUD, sensors), the map build	Yes
Field boundaries	`ACRE/fields.json` (polygons in survey cells)	`Tools/ACRE/Map/fields_recut.py`, `field_edits.py`. The land cover of each field is the instruction of the farm user	ACRES Core (`LoadFieldsAndCrops`), the library (`sensor_crop_mask.cpp`), the game (`AcresFieldSetup.cpp`), the map build	Yes
Field raster and cover raster	`ACRE/field_ids.u8`, `ACRE/cover.u8`, `ACRE/field-grid.json` (382 x 382 cells, 4 m)	`Tools/ACRE/Map/fields_recut.py`, `field_edits.py`	ACRES Core (`LoadSoilsAndFarm`), the game (`AcresFarmRuntime.cpp`, `AcresSensors.cpp`)	Yes
Surfaces (asphalt, concrete, gravel, dirt, grass lane, grass)	`ACRE/surface_polygons.json`	`Tools/ACRE/Map/game_layer.py`	ACRES Core (`LoadSurfaceMap`), the game (`AcresFieldSetup.cpp`), the map build	Yes
Surfaces as a raster (older)	`ACRE/surfaces.u8`, `ACRE/grass-lanes.u8`	`Tools/ACRE/Map`	The game (HUD map, ground visuals). ACRES Core reads `surfaces.u8` only when the polygon file is not there	Yes (`surfaces.u8`)
Soil	`ACRE/soils.json`, `ACRE/soil_units.u8` (1000 x 1000), `farm.json` (hydrology and crop parameters)	SSURGO data of USDA-NRCS, `Tools/ACRE/Map/expand_drive_region.py`. `farm.json` by hand	ACRES Core, the game	Yes
Crops: kind and planting date	`ACRE/crops_<season>.json`	`Tools/ACRE/Map/game_crops.py`	ACRES Core, the game (`-CropSeason=`)	Yes
Crops: the planted patches	`ACRE/crop_patches.bin` and `.json`, `ACRE/vegetation.json` (the headland of each planted field)	`Tools/Scene/export_level_crop_patches.py` reads the level file. `vegetation.json` from `Tools/ACRE/Tools`	ACRES Core. The game has the patches as actors of the level and reads none of the two patch files	Yes
Harvested state	No file: a field with the crop `none` in the season file has no standing crop. A harvest in a run is a state of the game	-	-	-
Places: home and spawn places, landmarks, yards	`ACRE/places.json`	`Tools/ACRE/Map/places.py` from labels that a person set	ACRES Core (`LoadPlaces`), the game (`AcresPlaces.cpp`, `-SpawnAt=`), the map build (the home pose is the place `spawn-icsc-garage`)	Yes
Regional ground round the tile	`ACRE/DriveRegion/` (`region.bin`, `fields.json`, `field_ids.u8`, `crops.*`, and files for the game)	`Tools/ACRE/Map/regional_*.py`, `expand_drive_region.py`	ACRES Core, the game (`AcresDriveRegion.cpp`)	Yes
Obstacles for the map build and for collisions	`ACRE/Scouting/obstacles.json`	`Learning/acres_learn/tasks/scouting/obstacles.py` from the scene export of the Unreal editor and the layers of `Calibration/Map/Layers`	ACRES Core (`FAcresScene::LoadObstacles`), the game (HUD map), the map build	Yes
Scouting map: outlines, loops, checkpoints, entry points (gates), the route graph, the home pose	`ACRE/Scouting/scouting_map.json`	`Learning/acres_learn/tasks/scouting/build_map.py`	The library (routes, planner, corridor), ACRES Core (zones), the game (`AcresRouteGuide.cpp`), the harness	Yes
Ground classes (0.5 m)	`ACRE/Scouting/ground_classes.u8`	`build_map.py`	ACRES Core, the library	Yes
Named places for a planner	`named_places.json`	`python -m acres_learn.maps.describe <folder> --write`. The generator writes it for a variant	The harness map and knowledge tools	No
The record of a variant	`map.json` (seed, recipe, edits, rejected draws, crops, wet patches), `Sources/lanes.geojson`	The generator	The generator, a person	No
Scene identity	`ACRE/scene_identity.json`	The scene export	ACRES Core compares it with the scene file	Yes
Road and grass masks	`road_mask.bin`, `grass_mask.json`	Older tools	The game on an older level only	No
Weather defaults	`environment.json`	By hand	The game	No

Files that are in the folder but are not the map: polaris.json, tractor.json, sensors.json, sensors_polaris.json, polaris_lidar_self.f32, soil_library.json, npc_cars.json, implements/, rigs/. They are the vehicle, the sensors and the actors. A variant folder has copies of them, because each reader takes one folder. The generator makes the copies again at each run.

Inputs That Are Not in the Map Folder#

Input	File	Made By	Read By
Obstacle geometry for the LiDAR and for collisions: buildings, trees, parked cars	`Core/Data/acre_scene.json`, `acre_scene.triangles.f32`	`Core/Scripts/build_scene.py` from `Tools/Scene/export_photo_scene_core.py` (it runs in the Unreal editor on the level `V03ACRE`)	ACRES Core (`FAcresScene`), the map build (the crowns of low trees)
Lane center lines	`Calibration/Map/Layers/lanes.geojson` (a variant: `Sources/lanes.geojson` in its folder)	`Tools/ACRE/Map/lanes.py`, `lane_edits.py` from aerial images and RTK drives	The map build
US 52, lot furniture, regional surfaces and center lines	`Calibration/Map/Layers/us52.json`, `lot_furniture.json`, `regional_*.geojson`	`Tools/ACRE/Map`	The map build
Parameters of the map build	`Learning/acres_learn/configs/scouting_v1.json`, section `map`	By hand	The map build

What the Unreal Level Has Baked In#

The level V03ACRE.umap has these parts as placed actors and cooked assets. The game finds them by a tag and does not make them from data:

The terrain mesh (ACRETerrainVisual).
The pavement, the buildings, the silos, the trees and the parked cars (ACREPavementVisual, ACREBuildingVisual, ACRESiloVisual, ACRETree, ACREParkedCar).
The ground cover.
The crop patches of the planted fields (actors with the tags corn, soybean, potato and the field name).
The materials and the textures.
The cooked chunks of the regional ground.

The game makes these parts from the data files at run time:

The wheel and ground physics. The height of the ground is from heights.f32, not from a collision mesh.
The ruts, the field work and the water.
The HUD map and the roads of the cars.
The crop patches of fields that the level does not have.
The regional crops and markings.

What `map_hash` Covers#

map_hash is the first 16 hexadecimal digits of a SHA-256. The hashed text has two parts:

The scouting map: its sources, its parameters, the size of the graph, each field, each edge and the home pose.
One line with the SHA-256 of each file with "Yes" in the column "Drive" that is in the folder. The list is in Runtime/Source/identity.cpp (DriveFiles), with each file ACRE/crops_<season>.json.

Part 2 makes the hash change when the heights, the crops or the soil change. Thus old cost tables are not used for a changed map. The hash of the default map is 1cf52d0db9a57710.

The hash does not cover Core/Data/acre_scene.json directly. It covers ACRE/scene_identity.json, and ACRES Core refuses a scene file that does not agree with that file. The hash does not cover the files of the vehicle. The regression test finds a change of a drive from those files, and sim_version records it.

The Generator#

python -m acres_learn.maps.variants makes each variant of Learning/acres_learn/configs/map_variants_v1.json. A variant has a name, a seed and a recipe (the count of each edit). The generator copies the default map folder, changes the sources of the terrain tile, and makes the scouting products with the map build. The site outline, the heights, the buildings, the trees and the regional ground stay as they are.

Edit	What Changes
`merge`	Two fields become one field. A grass lane between them goes away, with its center line.
`split`	A new grass lane across a field makes two fields.
`dead_end`	A new grass lane goes from the edge of a field to its middle and stops there.
`remove_lane`	A grass lane goes away: its center line, and its surface but for the two ends. Routes must go a different way.
`move`	One side of a field moves in by 4 m to 10 m. The strip becomes mown grass.
`one_gate`	A field keeps one entry point only, and not always the first one.
Crops	Each field with a crop gets a crop kind, a planting date (the growth) or the state "harvested".
`wet`	Wet patches: circles of the soil raster get the soil unit with the highest field capacity.

The rules of a valid site:

Each field that the generator changed is one polygon of 0.4 ha or more.
A new lane is 5.99 m wide: the vehicle (1.59 m), the lane clearance of the map build on each side (2.0 m), and 0.2 m on each side.
The tile has 59 fields at most, because ACRES Core has the field numbers 1 to 59. A split uses a number that a merge set free.
After the build, each field of the tile is a task field. It has an entry point and a path from the home pose, and its loop covers its checkpoints.
Each spawn place is 40 m or less from the graph of the home pose.

A draw that breaks a rule is rejected, and the generator draws again with the next attempt number. map.json records each rejected draw. The random numbers come from one generator of the seed and the attempt number, thus one seed gives one folder. Learning/tests/test_map_variants.py makes a variant two times and compares each file.

The generator is Python and is in Learning/acres_learn/maps. The reason: it changes the inputs of the map build, and the map build is offline Python with 2,400 lines (shapely, scipy). No code of a drive is in the generator. The functions that load a map and give its identity are in the library.

The build of one variant uses 7.2 GB of memory, thus the generator makes one variant at a time.

Limits of a Variant#

The game has the data of the variant with the switch -MapData=. The terrain, the pavement and the trees of its picture stay those of the level (see "The Game").
surfaces.u8, grass-lanes.u8 and the files of the game in DriveRegion are those of the default map. ACRES Core does not read them when surface_polygons.json is there.
A variant has no level crop patches. ACRES Core lays the patches of each field on its grid.
The obstacle scene is that of the default map. A new lane does not go round a tree. The map build keeps the loop and the graph clear of each obstacle. A draw whose field has no path is rejected.
The effect of a wet patch on a drive is that of the soil unit. It was not measured.

The Variants#

The seeds and the recipes are in Learning/acres_learn/configs/map_variants_v1.json. The folders are in Acres/Content/SimulationMaps , in Git through LFS, and the packaged game stages them with Content/Simulation (DirectoriesToAlwaysStageAsNonUFS). The game takes a variant by name, -MapData=acre-mixed, as ACRES Core does, or by folder. Learning/acres_learn/configs/map_variants_digest_v1.json has the digest of each map. These commands make the folders, the cost tables and the digest:

cd Learning && PYTHONPATH=. python -m acres_learn.maps.variants        # the folders, 4.5 minutes for each one
python -m acres_simulator.maps --tables all --threads 4               # the cost tables, 3.5 to 5 minutes for each map
python -m acres_learn.maps.digest --map <name> --out <folder>          # one time for each map
python -m acres_learn.maps.digest --merge <folder>                     # the digest file

These historical measurements precede the release-number reset. The first five variants were made in the morning. acre-sparse and acre-maze are the hard test maps: they were added because only acre-deadend differed strongly from the default map.

Map	Seed	Map Hash	Tile Fields	Lanes	Lane Length (m)	Plans	Sample	Tables (s, 4 Threads)
`default`	-	`1cf52d0db9a57710`	59	117	14629	129 of 129	6 of 6	-
`acre-merged`	101	`d9bc6fe8bdf3812f`	54	116	14636	124 of 124	6 of 6	231.8
`acre-split`	202	`696352ec09874d38`	59	121	14981	129 of 129	6 of 6	243.3
`acre-detour`	303	`c33d16be365d97fb`	58	115	13789	128 of 128	6 of 6	243.0
`acre-deadend`	404	`0f81d42d5c72d280`	58	120	14380	128 of 128	6 of 6	236.2
`acre-mixed`	505	`7b1eea70256491de`	59	125	14604	129 of 129	6 of 6	262.0
`acre-sparse`	606	`c2c486bb34a1c03d`	55	108	12721	125 of 125	6 of 6	203.5
`acre-maze`	707	`c458b8f8472f89de`	57	118	13320	127 of 127	6 of 6	203.8

"Plans" is the count of single-field missions with a plan, of the task fields of the map. "Sample" is the count of complete missions of six single-field missions with the default chain in the condition set dry. The sample has the fields that the variant changed first. The least distance to an obstacle in each sample is 2.04 m. Each variant was valid at the first draw.

Map	Edits	Graph Nodes	Graph Edges	Junctions	Dead Ends	Tile Gates	Tile Fields with One Gate	Detour Mean	Detour Largest	Mean Route (m)
`default`	-	845	853	336	481	212	5	1.893	4.558 (F28)	2330
`acre-merged`	6 merge, 2 move, 2 one_gate, 1 split	814	822	325	462	190	6	1.877	4.558 (F28)	2300
`acre-split`	1 dead_end, 3 merge, 3 move, 3 split	822	831	319	474	200	5	1.953	4.558 (F28)	2342
`acre-detour`	2 merge, 3 one_gate, 5 remove_lane, 1 split	822	827	327	469	191	9	1.883	4.558 (F28)	2367
`acre-deadend`	4 dead_end, 1 merge, 2 move, 8 one_gate, 2 remove_lane	846	848	340	478	205	12	2.212	14.511 (F46)	2522
`acre-mixed`	2 dead_end, 4 merge, 4 move, 5 one_gate, 3 remove_lane, 4 split	822	826	321	476	194	10	1.934	4.525 (F27)	2306
`acre-sparse`	3 dead_end, 4 merge, 2 move, 12 one_gate, 14 remove_lane	779	779	312	444	163	18	1.879	3.982 (F46)	2442
`acre-maze`	8 dead_end, 2 merge, 3 move, 16 one_gate, 8 remove_lane	782	784	310	450	162	20	1.85	4.603 (F28)	2497

"Lanes" is the count of lane lines in the route graph. "Tile Gates" is the count of entry points of the tile fields. "Detour" is the route on the graph from the home pose to the nearest gate of a tile field, divided by the straight distance. "Mean Route" is the mean length of the reference route of the planner for the single-field missions of the tile. The digest file has the length for each mission (path_length_m), thus an evaluation can weight a success by the path length.

How the variants are different:

acre-merged has five fields less, 31 graph nodes less and 22 gates less. Its routes are a little shorter.
acre-split has four new lane lines and 350 m more lane. Its mean detour is larger (1.95).
acre-detour has 840 m less lane, and nine fields with one gate. Its mean route is 37 m longer. The removed lanes are those that routes use most, but the graph has a second way for most of them.
acre-deadend has twelve fields with one gate and four dead-end lanes. Its mean detour is 2.21 and its largest is 14.5 (field F46: its one gate is on the far side). Its mean route is 190 m longer.
acre-mixed has each edit: ten fields with one gate, six new lanes and three removed lanes.
acre-sparse has 14 removed lanes: 1,900 m less lane, 74 graph edges less and 18 fields with one gate.
acre-maze has eight dead-end lanes and 20 fields with one gate. One of its edges (116.5 m) is driven at 0.42 m/s and has a time-out in its cost tables: the table drive reports it. The planner does not use that edge. The cause was not examined.

The differences of acre-merged, acre-split and acre-mixed from the default map are small in these numbers. The lane network of the site is dense, and one removed lane seldom makes a long detour.

The Split of the Maps#

Learning/acres_learn/configs/map_splits_v1.json has the split. It is a new file beside splits_v1.json, because splits_v1.json is the split of the fields of one map and a script makes it.

Set	Maps
Training	`default`, `acre-merged`, `acre-split`
Validation	`acre-mixed`
Test (no training sees them)	`acre-detour`, `acre-deadend`, and the hard maps `acre-sparse`, `acre-maze`
Unseen test with the camera	The Unreal map `V03ACRE` in the game. Its data is the default map. The pictures are the part that no training in ACRES Core saw

A New Location#

A person must give these inputs for a new site:

The heights: a grid of 1001 x 1001 values with a step of 1.524 m. The origin of the grid is in a projected coordinate system.
The field boundaries, with the land cover of each field.
The surfaces as polygons with a class (asphalt, concrete, gravel, dirt, grass lane, grass).
The center lines of the lanes and roads, with a surface and a role.
The obstacles: outlines of buildings, trunks and crowns of trees, keep-out areas.
The places: one home place and the spawn places, each with a heading.
The soil units as a raster, with the table of their properties (SSURGO in the United States).
The crop of each field with its planting date.

Three gaps are known:

The obstacle geometry of ACRES Core comes from the Unreal scene export. Core/Data/acre_scene.json and Scouting/obstacles.json are made from the level in the Unreal editor. A new site without a level needs a tool that makes the two files from the outlines and heights of item 5. The file is also not in the map folder: the world options of the library take it from <root>/Core/Data.
Cost tables need a simulated world of the map. The planner needs the tables, and the tables are drives in ACRES Core on that map (python -m acres_simulator.maps --tables <name>). A map that ACRES Core cannot load has no tables.
The georeference is written for the ACRE site. The origin ORIGIN_FT and the projection EPSG:2968 are constants in Learning/acres_learn/tasks/scouting/build_map.py and Learning/acres_learn/acre.py. sensor_georef.cpp of the library has the definitions of UTM 16N and of EPSG:2968. It reads only the origin from site.json. The size of the grids (1000, 1001, 382) is a constant in ACRES Core and in the game. The limit of 59 fields is also a constant.

The Game#

The packaged game reads each data file from one folder. AcresMapData::Dir() (Acres/Source/Acres/AcresMapData.h) gives that folder, and each reader of the game makes its path with it. The default is Content/Simulation of the game. Acres/Config/DefaultGame.ini stages that folder as loose files (DirectoriesToAlwaysStageAsNonUFS), thus a data folder can change with no new cook.

Packaged/Linux/Acres.sh -VehicleDemo -Vehicle=polaris -MapData=$PWD/Acres/Content/SimulationMaps/acre-deadend

The switch -MapData=<folder> selects a map folder. A relative folder is from the folder where the game starts. A folder with no ACRE/site.json stops the game with a fatal error. The game reads the folder one time and keeps it for the process. The menu keeps the switch when it starts a session.
What the game has from the folder: the physics (heights, surfaces, soil), the fields, the HUD map, the route guide, the roads for the cars and the places. It also reads the vehicle files and the sensor files from that folder, as ACRES Core does. A variant folder has copies of these files from the time of its generation. Make the variant again after a change of sensors_polaris.json or of polaris_lidar_self.f32.
The crop patches. The level has the crop patches of the default map as actors. Some fields have an outline in ACRE/fields.json that is not that of the default map. For each such field the game hides the level patches and lays the patches from the outline. This is the code for a field with no level patches (AcresFarmRuntime.cpp). The log line ACRES_MAP_DATA changed fields lists these fields.
The control channel. The operation hello gives map_data, map_variant and map_changed_fields.
The check. Tools/SimControl/check_map_variant.py --map <folder> --output <folder> starts the game with the switch. It compares three values with the data of the folder. The first is the field at three places where the two maps are different. The others are the size of the route graph and the route to a field with one gate.

What stays that of the level:

The picture of the ground. The terrain, the pavement of a removed lane and the trees are actors of the level. The ground material of a new lane or of a removed lane is not made from surface_polygons.json. The run-time ground tiles for ruts in AcresSoilVisuals.cpp are the place to start. Estimate: 2 to 4 days, with game checks. The physics and the LiDAR material of the ground are those of the variant.
The rasters surfaces.u8 and grass-lanes.u8 of a variant folder are those of the default map. The HUD map and the ground visuals read them.
The obstacle scene is that of the default map (see "Limits of a Variant").
A site with a different grid size needs the constants 1000, 1001 and 382 as data. Estimate: 1 to 2 days more. The places are AcresFarmRuntime.cpp (645 to 647), AcresSoilVisuals.cpp (224, 336) and AcresSiteMap.cpp (163). A second site also needs its own level and assets.