Skip to content

Tests and Regression Checklist#

This page lists the tests of the repository and the checks that are necessary after each type of change.

Before You Start#

Run all commands from the repository root. A test fails when its exit code is not zero.

Tests without the Game#

These tests compile the engine-free models with a standard compiler. They do not start Unreal. The compiler flags promote the warnings that Unreal's clang reports as errors.

Command Scope
Tools/Terramechanics/build.sh && Tools/Terramechanics/physics_tests The Maxxum model: tyre and soil, soil library, drivetrain, fuel, energy ledger, weather model.
Tools/ImplementModel/build.sh The implement model: tool forces, hitch control, PTO and hydraulic loads, with the implement data files.
Tools/PolarisModel/build.sh The Polaris model and the drive-by-wire, with the recorded extracts in Tools/PolarisModel/Data.
Tools/LidarModel/build.sh The LiDAR radiometry (AcresLidarPhysics.h).
Tools/EpisodeLog/build.sh The episode log: CDR layout, each message type, the MCAP writer and reader.
Core/build.sh ACRES Core: the library, the Python module and their tests.
python Tools/ACRE/Map/test_map.py The map mathematics: frames, lane edges, crop rules, footprints, grid scale factors.
python Acres/Raw/farm-equipment/scripts/test_mechanics.py The mechanics data of the farm equipment.
python Tools/ACRE/Tools/check_acre_layer_shader.py Compiles the ground-layer shader include with the shader compiler of the engine.
python Tools/ui_text_check.py The UI text style.
python Learning/tests/run_all.py The learning stack. Refer to Learning.
Verification/check.sh The formal proofs and their differential tests. Refer to Verification.

Tools/ImplementModel/build.sh and Tools/PolarisModel/build.sh accept UE_CLANG=<clang++>. The scripts then compile the models with the clang of the engine as a portability check.

ROS 2 Tests#

Do these steps in a shell with the ROS 2 environment.

  1. Activate the environment and the DDS loopback fence.

    source ROS/Env/setup_env.sh

  2. Run the tests of the three packages.

    (cd ROS && colcon test --packages-select acres_sim acres_description acres_core_sim)

  3. Show the results.

    (cd ROS && colcon test-result --verbose)
Package Scope
acres_sim Stream frames, message conversions, JSON, the J1939 codec, lockstep barriers, the control services and the topic parity. The tests use a stand-in game.
acres_core_sim The georeference, the INS, the drive-by-wire path and its timeout, the stream frames, the server sockets, determinism and the rate factor.
acres_description The nodes of the Polaris description.

Two more tools compare the workspace with recorded data of the real Polaris.

Command Scope
python ROS/Tools/check_msg_defs.py <bags> --msgs <package folders> The message definitions agree byte for byte with the recorded bags.
python ROS/Tools/check_topic_parity.py --sim <bag> --vehicle <bags> The topic names, types, QoS and frame identifiers of a simulator bag agree with the vehicle bags.

Tests with the Packaged Game#

CAUTION

Start only one game at a time. Two games can use all the memory of the GPU.

Command Scope
python Tools/SimControl/lockstep_determinism.py --out <dir> --headless Two lockstep runs with the same commands give the same trajectory.
python Tools/SimControl/lockstep_determinism.py --out <dir> --game core The same check on ACRES Core. It does not use the game.
python Tools/SimControl/compare_episodes.py <a.mcap> <b.mcap> Compares two episode logs step by step.
python Tools/SimControl/render_check.py capture --out <dir> Captures the sensor camera, the GPU LiDAR and the main view at fixed poses.
python Tools/SimControl/render_check.py compare <a> <b> Compares two captures.
python Tools/Performance/benchmark_linux.py --name <name> Measures the frame rate and the peak GPU memory.
python Learning/tests/farm_marks_test.py The marks on the farm stay after resets and replays.
python Calibration/Map/game_surface_check.py, buildings_game_check.py The surfaces and the buildings in the game agree with the map layers.
Packaged/Linux/Acres.sh -RigSelfTest Each equipment rig map agrees with its test poses. The log tag is ACRES_RIG_SELFTEST.

Tools and Scripts gives the options of each tool.

Field-Day Kit Tests#

The tests of the field-day kit use a fake vehicle inside the DDS loopback fence. They do not connect to the real vehicle.

Command Scope
python Calibration/Polaris/FieldDay/Test/test_camera_solver.py The offline solution of the camera mount on synthetic captures.
python Calibration/Polaris/FieldDay/Test/test_dbw_safety.py Each abort path of the drive-by-wire tests.
python Calibration/Polaris/FieldDay/Test/test_camera_check.py The camera check.
python Calibration/Polaris/FieldDay/Test/test_policy_run.py Each stop condition of the policy run.

Field-Day Kit describes the kit.

Regression Checklist#

Do the checks of each row that applies to your change.

Change Checks
An engine-free vehicle, soil or weather model Tools/Terramechanics, Core/build.sh. Update the model page.
The Polaris model or the drive-by-wire Tools/PolarisModel/build.sh, Core/build.sh, the acres_core_sim tests, Verification/check.sh.
An implement Tools/ImplementModel/build.sh, -RigSelfTest.
A sensor model Tools/LidarModel/build.sh for the LiDAR. A short session with sensors on. render_check.py for the camera and the GPU LiDAR.
The game code Package the game. Run a short session with sensors on. Run the lockstep determinism check.
The renderer or a render tier render_check.py, benchmark_linux.py.
The episode log or a message Tools/EpisodeLog/build.sh, the acres_sim tests, check_msg_defs.py.
The ROS 2 bridge The ROS 2 tests, check_topic_parity.py.
The map test_map.py, the checks in Calibration/Map. Build the scouting map products again.
UI text ui_text_check.py.
The documentation The checks of Write Documentation.

After a short session with sensors on, examine two files in the session folder.

  1. Open sensors/episode-000/episode.json.

    Expected Result

    The file states that the episode is complete.

  2. Open session-summary.json.

    Expected Result

    The file shows no dropped rows.

Environment Variables of the Calibration Tools#

The calibration tools and some ROS 2 tools read data of the lab from folders outside the repository. An environment variable sets each folder. The default is a folder below ~/Codes/polaris/.

Name Type Unit Default Description
POLARIS_EXTRACT path ~/Codes/polaris/extract The extraction of each bag that the lab made on the vehicle PC.
POLARIS_EPISODES path ~/Codes/polaris/episodes The standard episodes and the working data of the calibration studies.
POLARIS_CACHE path $POLARIS_EPISODES/_cache Parsed runs as .npz files.
POLARIS_LIDAR_WORK path $POLARIS_EPISODES/lidar_sim The renders and tables of the LiDAR calibration.
POLARIS_FOLLOWER path $POLARIS_EPISODES/follower The logs of the lab follower and the closed-loop runs.
POLARIS_LOGS path ~/Codes/polaris/logs The local copy of the vehicle logs.
ACRE_WEATHER path ~/Codes/polaris/weather The Purdue Mesonet record of ACRE (acre_30min.parquet).
POLARIS_ENV path ~/Codes/polaris/polaris/polaris.env The file with the login of the vehicle PC. Do not commit it. Do not print it.
POLARIS_HOST, POLARIS_BACKUP_HOST host The vehicle PC and the backup PC.
FETCH_KBPS number kbit/s 6000 The download limit of run_remote_extract.sh.

WARNING

The tools that use POLARIS_HOST connect to the PC on the real vehicle. Only the lab runs them. Do not start ROS 2 nodes or send commands on the vehicle PC.