Calibrate against Real Logs#
This tutorial gives the calibration loop of the Polaris: recorded logs of the real vehicle give the parameters of the simulator.
It uses the tools in Calibration/Polaris, Tools/PolarisModel, Core/Scripts, Calibration/Weather and Calibration/Tractor.
WARNING
The workstation only reads data of the real vehicle. Do not send a command to the vehicle PC from the workstation. Only the lab operates the vehicle. Refer to Field-Day Kit.
Before You Start#
- You have the conda environment
torchenvwithnumpy,scipy,pandas,pyprojandmatplotlib. - You have
g++with C++20. - For the steps with ACRES Core: you built ACRES Core. Refer to Build ACRES Core.
- For the steps with the game: you have the packaged game. Refer to Run the Packaged Game.
The repository contains the event tables of four recorded runs. The procedures Build the Model Tools to Replay a Run in the Game thus need no data of the vehicle. The other procedures read a local copy of the log extract. These variables give the folders.
| Variable | Default | Content |
|---|---|---|
POLARIS_EXTRACT |
~/Codes/polaris/extract |
The extract of the bags: one folder for each bag |
POLARIS_EPISODES |
~/Codes/polaris/episodes |
The episodes that episodes.py builds |
POLARIS_FOLLOWER |
$POLARIS_EPISODES/follower |
The runs of the path follower of the lab |
POLARIS_LIDAR_WORK |
$POLARIS_EPISODES/lidar_sim |
The work folder of the LiDAR comparison |
ACRE_WEATHER |
~/Codes/polaris/weather |
The record of the weather station |
BUILD_DIR |
$TMPDIR/acres-polaris-model |
The binaries of Tools/PolarisModel |
The Loop#
| Stage | Tool | Output |
|---|---|---|
| 1. Read the logs on the vehicle PC | Calibration/Polaris/extract_on_device.py |
Compact topic files, sampled scans and frames |
| 2. Make an inventory | Calibration/Polaris/episodes.py, inventory.py |
inventory.md, inventory.json, the episodes |
| 3. Make event tables | Tools/PolarisModel/polaris_dbw.py extract |
Tools/PolarisModel/Data/*.csv |
| 4. Fit the parameters | Tools/PolarisModel/polaris_dbw.py fit |
fit_results.json, polaris.json |
| 5. Test the model | Tools/PolarisModel/build.sh |
44 checks |
| 6. Replay in the model and in the game | polaris_dbw.py replay, Core/Scripts/replay_logs.py, unreal_dbw_replay.py, compare_unreal.py |
Traces and metrics |
| 7. Calibrate the sensors | The lidar_* and camera_* scripts |
lidar_results.md, camera_results.md |
| 8. Test the closed loop | Calibration/Polaris/closed_loop_sim.py |
closed_loop_results.md |
| 9. Set the weather and the soil of each run | Calibration/Weather/ground_conditions.py |
ground_conditions.md, the replay files |
| 10. Test on the vehicle | The field-day kit | New logs |
Know the Logs#
The file Calibration/Polaris/inventory.md lists each recorded run: where it is on the farm, what it contains and what it can calibrate.
The file inventory.json has the same data for programs. Its block totals gives these numbers.
| Quantity | Value |
|---|---|
| Runs | 74: 57 sensor bags, 16 taught paths, 1 diagnostic bag of the path follower |
| Dates | 2025-03-05 to 2026-08-13 |
| Total duration | 3476 s, of which 1691 s in motion |
| Distance | 9249 m |
| Runs on the tile | 54 fully, 3 partly |
| LiDAR scans | 19 304, of which 1326 are in the extract |
| Camera frames | 16 907, of which 1608 are in the extract |
| Pose quality | 6 runs with a converged INS, 16 with the fix only, 49 degraded, 2 invalid, 1 without a pose |
The logs come from the vehicle PC as a compact extract. The lab or a person with its approval runs the extraction.
CAUTION
Calibration/Polaris/run_remote_extract.sh connects to the vehicle PC. Do not run it without the approval of the lab.
The extraction only reads bags. It refuses to start while a ROS process runs on the vehicle PC.
To build the inventory again from a local extract, do these steps.
-
Build the episodes. Each episode has the vehicle state at 100 Hz in the frames of the simulator.
-
Write the inventory.
Expected Result
The script writes
Calibration/Polaris/inventory.jsonandCalibration/Polaris/inventory.md.
Calibration/Polaris/geo.py gives the conversion between the fix of the vehicle and the frames of the simulator.
The ACRE Scene gives the same conversion.
The Event Tables#
The fit of the vehicle uses four runs. The folder Tools/PolarisModel/Data contains their event tables.
| Run | Duration | Content |
|---|---|---|
dbw_direct_test_01 |
20.2 s | Drive-by-wire commands and reports. A ULC command of 0.5 m/s and a steering command. The vehicle was stationary in L. |
grass_diag_20260731_174757 |
41.5 s | The path follower of the lab on grass: ULC speed and steering angle commands, odometry at 100 Hz. No drive-by-wire reports. |
human_20260813_172445 |
4.5 s | Steering report, IMU and odometry. The vehicle was stationary. |
human_20260813_192910 |
2.3 s | The same signals. The vehicle was stationary. |
Each run has two files.
| File | Columns | Topics |
|---|---|---|
<run>_commands.csv |
t_s, topic, v0 to v7 |
enable, steer, throttle, brake, ulc, gear_command, driver_gear, driver_steer, driver_throttle, driver_brake, drive_mode |
<run>_reports.csv |
t_s, topic, v0 to v5 |
steer, throttle, brake, gear, ulc, velocity, odom, imu, follower, safety_stop |
Tools/PolarisModel/DbwLog.h gives the meaning of the values v0 to v7 for each topic.
To make the tables again from a local extract, run this command. It overwrites the files in Tools/PolarisModel/Data.
Build the Model Tools#
-
Build the tools and run the checks.
Expected Result
The script builds
polaris_testsandlibpolaris.soand then runs 44 checks. The end of the output is:== 9. Replay regressions on the vehicle's logs == [PASS] dbw_direct_test_01 steering: steering wheel angle RMSE 0.750 deg over the 30 deg step (479 reports), firmware reference RMSE 0.196 deg [PASS] dbw_direct_test_01 ULC: throttle percent_cmd RMSE 0.370 % (77 reports), vel_ref RMSE 0.020 m/s (227 reports) [PASS] dbw_direct_test_01 brake: line pressure RMSE 0.339 bar (264 reports: 20 % stop hold and release) [PASS] dbw_direct_test_01 standstill and timeout: vehicle never moves (max wheel speed 1.5e-12 m/s; recorded 0 with 15.9 % pedal in L); steering disengages at 15.850 s (recorded 15.843 s) [PASS] grass_diag follower on grass: speed RMSE 0.106 m/s (OxTS, 595 samples), path curvature RMSE 0.00128 1/m against a recorded RMS of 0.00664 1/m ... 44 passed, 0 failed
The tools compile the model sources of the game: AcresUtvModel.cpp, AcresVehicleModel.cpp, AcresSimModel.cpp, AcresSoilModel.cpp and AcresPowerModel.cpp.
The test stand UtvBench.h replaces the chassis of the game with a rigid chassis that has yaw, heave, pitch and roll.
| File | Function |
|---|---|
Tools/PolarisModel/build.sh |
Builds the tools and runs the checks. BUILD_DIR sets the output folder. |
Tools/PolarisModel/polaris_tests.cpp |
The 44 checks: statics, turns, power, brakes, actuators, time step, energy, drive modes, replays, command timing. |
Tools/PolarisModel/polaris_replay.cpp |
The C interface of libpolaris.so: load, set a parameter, replay a log. |
Tools/PolarisModel/UtvBench.h |
The test stand and the reader of polaris.json. |
Tools/PolarisModel/DbwLog.h |
The reader of the event tables and the replay loop. |
Tools/PolarisModel/polaris_dbw.py |
The extraction, the fit and the replay from Python. |
Fit the Vehicle Parameters#
The fit runs the C++ model itself. polaris_dbw.py loads libpolaris.so, replays the recorded commands and minimises the difference to the recorded reports.
The method is a least-squares fit with bounds (scipy.optimize.least_squares).
The standard errors come from the Jacobian. The script increases them for the autocorrelation of the residuals.
-
Run the fit.
Expected Result
The script prints the RMSE before and after the fit for each actuator and writes
Tools/PolarisModel/Data/fit_results.json. -
To put the fitted values into
polaris.json, add the option--write.CAUTION
The option
--writechangesAcres/Content/Simulation/polaris.json. RunTools/PolarisModel/build.shafter it.
Each actuator has its own fit on its own signals. The table gives the results of Tools/PolarisModel/Data/fit_results.json.
| Fit | Signal | Parameters | RMSE Before | RMSE After |
|---|---|---|---|---|
| Steering | Steering wheel angle, one step of 30° at standstill, 479 reports | Delay, lag | 2.46° | 0.75° |
| Brake | Line pressure, stop hold and release, 264 reports | Delay, increase, release lag, release exponent | 1.92 bar | 0.34 bar |
| ULC throttle | Pedal command during the start at standstill, 77 reports | Offset, acceleration gain, integral gain | 11.64 % | 0.37 % |
| Grass, longitudinal | Speed of the odometry, 596 samples | Speed scale, proportional gain of the ULC | 0.376 m/s | 0.106 m/s |
| Grass, lateral | Path curvature, 456 samples | Steering ratio, steering centre | 0.00526 1/m | 0.00128 1/m |
| Timeouts | The time from the last command to the release | Command timeout, actuator timeout of the ULC |
The value before the fit uses the priors of the script. The priors are values that an engineer assumes without the logs. Polaris Ranger Dynamics and Drive-by-Wire and ULC give the fitted values with their standard errors.
The lateral fit shows only the product of the steering ratio and the wheelbase: 34.4 ± 2.4 m (95 %).
The file fit_results.json tests four wheelbase values at the installed ratio 16.
| Hypothesis | Wheelbase | Curvature RMSE | Rejected at 95 % |
|---|---|---|---|
| Manufacturer, 113 in | 2.8702 m | 0.00200 1/m | Yes |
| Installed planner | 2.8448 m | 0.00197 1/m | Yes |
| Frames of the OxTS | 2.6000 m | 0.00169 1/m | Yes |
| Configuration of the path follower | 2.0400 m | 0.00131 1/m | No |
The model keeps the wheelbase of the manufacturer and the fitted ratio 11.98.
Replay a Run in the Model#
-
Replay one run on the test stand and write the trace.
Expected Result
The file has one row for each physics step, 2426 rows for this run, and these columns:
-
Replay all four runs in ACRES Core and on the test stand.
python Core/Scripts/replay_logs.py --out /tmp/acres-core-replay --figures /tmp/acres-core-replay/figuresExpected Result
The script prints two tables. The first table compares ACRES Core with the test stand.
| Run | Steps | Speed max / RMS (m/s) | ... | Path end / max (m) | | dbw_direct_test_01 | 2426 | 0 / 0 | ... | 0 / 0 | | grass_diag_20260731_174757 | 4986 | 0.0117 / 0.00147 | ... | 0.00337 / 0.00343 | | human_20260813_172445 | 543 | 0 / 0 | ... | 0 / 0 | | human_20260813_192910 | 272 | 0 / 0 | ... | 0 / 0 |The second table compares the two models with the recorded reports.
| Run | Metric | Core | Test Stand | Reference | | dbw_direct_test_01 | swa_rmse_deg | 0.7497 | 0.7497 | | | dbw_direct_test_01 | throttle_cmd_rmse_pct | 0.3697 | 0.3697 | | | dbw_direct_test_01 | brake_rmse_bar | 0.3392 | 0.3392 | | | grass_diag_20260731_174757 | speed_rmse_mps | 0.107 | 0.1062 | | | grass_diag_20260731_174757 | curvature_rmse_per_m | 0.001275 | 0.001281 | |
The two models use the same source file for the vehicle. They differ only in the chassis. On the moving run their paths differ by 3.4 mm after 41.5 s.
Replay a Run in the Game#
Core/Scripts/unreal_dbw_replay.py plays the commands of an event table into the game through the vehicle bridge.
Core/Scripts/compare_unreal.py then drives ACRES Core with the same commands and compares the two simulators.
-
Examine the client without the game.
-
Start the game with the Polaris, the vehicle bridge and the session log.
-
Play the recorded commands in a second terminal.
python Core/Scripts/unreal_dbw_replay.py --run dbw_direct_test_01 --port 5556 \ --place spawn-icsc-garage --out $HOME/acres-sessions/core_cmp/replayExpected Result
The folder
replaycontainscommands_sent.csv,reports.jsonlandreplay.json.commands_sent.csvhas the physics time at which the game received each command. -
Compare ACRES Core with the session of the game.
To replay the path of a recorded run with a session driver, use the replay options of the game. Refer to Replay.
Calibration/Polaris/replay_tracking.py then gives the tracking metrics of that session.
python Calibration/Polaris/replay_tracking.py --replay <episode>/replay.csv --session <session folder>
| Metric | Meaning |
|---|---|
reached_fraction |
The share of the real track that the simulated vehicle reached |
cross_track_rms_m, cross_track_p95_m, cross_track_max_m |
The distance from the simulated position to the real track |
speed_rmse_mps |
The difference between the simulated speed and the real speed at the same position on the track |
real_track_on_road_fraction |
The share of the real track on road cells of the map. It examines the registration of the track. |
Calibrate the LiDAR#
The LiDAR calibration compares the recorded scans of the RoboSense Helios with simulated scans at the same poses.
LiDAR gives the model. The results are in Calibration/Polaris/lidar_results.md.
| Step | Command | Output |
|---|---|---|
| 1. Label the real scans | python Calibration/Polaris/lidar_stats.py label |
Surface labels from the camera |
| 2. Build the real tables | python Calibration/Polaris/lidar_stats.py tables |
One record for each sampled scan |
| 3. Compute the real statistics | python Calibration/Polaris/lidar_stats.py stats |
lidar_stats.json, Figures/lidar_*.png |
| 4. Fit the radiometry | python Calibration/Polaris/lidar_model.py fit |
lidar_model_fit.json |
| 5. Select the poses | python Calibration/Polaris/lidar_sim.py select |
poses.csv |
| 6. Render the simulated scans | python Calibration/Polaris/lidar_sim.py render --label after --overlay <overlay> |
Simulated scans |
| 7. Build the simulated tables | python Calibration/Polaris/lidar_sim.py tables --label after |
The same records for the simulator |
| 8. Compare | python Calibration/Polaris/lidar_sim.py compare |
lidar_compare.json, Figures/lidar_cmp_*.png |
Step 6 starts the packaged game. Only one game can run on a workstation at a time.
lidar_compare.json scores 11 metrics against thresholds. Before the calibration 1 metric was in its limit. After the calibration 9 metrics are in their limits.
| Metric | Before | After | Threshold |
|---|---|---|---|
| Largest elevation error of a beam | 6.16° | 0.00° | 0.05° |
| Return rate for each ring, mean absolute error | 10.8 points | 4.9 points | 5.0 points |
| Points for each scan, relative error | 0.195 | 0.077 | 0.10 |
| Returns from the vehicle, intersection over union | 0.00 | 0.99 | 0.90 |
| Ground detection, mean absolute error | 0.026 | 0.016 | 0.05 |
| Ground detection, largest error | 0.411 | 0.152 | 0.15 |
| Range histogram, Jensen-Shannon divergence | 0.262 | 0.124 | 0.15 |
| Intensity median, absolute error | 6 | 2 | 3 |
| Intensity histogram, Jensen-Shannon divergence | 0.899 | 0.465 | 0.20 |
Two metrics stay out of their limits: the largest ground detection error and the intensity histogram.
Calibrate the Camera#
The camera calibration has two parts: the mount of the camera on the vehicle and the comparison of real and simulated images.
Camera gives the model. The results are in Calibration/Polaris/camera_results.md.
| Step | Command | Output |
|---|---|---|
| 1. Solve the mount | python Calibration/Polaris/camera_extrinsic.py |
camera_extrinsic.json, figures |
| 2. Select the real frames | python Calibration/Polaris/camera_compare.py select |
frames.csv, poses_grid.csv |
| 3. Render at the poses | python Calibration/Polaris/camera_render.py --poses poses_grid.csv --out renders/ |
One image for each pose |
| 4. Refine the poses | python Calibration/Polaris/camera_compare.py refine |
poses_refined.csv |
| 5. Render at the refined poses | python Calibration/Polaris/camera_render.py --poses poses_refined.csv --out renders/ |
One image for each frame |
| 6. Evaluate | python Calibration/Polaris/camera_compare.py evaluate |
metrics.json, figures. The repository has a copy, camera_metrics.json. |
The mount solve uses the calibration bag of the lab: a marker target that the LiDAR and the camera see at the same time.
The solved mount is the position (2.745, 0.606, 1.887) m and the angles (-3.5°, 16.5°, -2.6°) from base_footprint.
Calibration/Polaris/reolink_lens.py contains the lens model. Calibration/Polaris/camera_mount_check.py examines the mount after each field day.
The comparison puts the segmentation network of the lab on real frames and on simulated frames of the same pose and time.
camera_metrics.json gives these results for 133 frames.
| Metric | At the Logged Pose | At the Refined Pose |
|---|---|---|
| Drivable area, intersection over union, median of the frames | 0.456 | 0.840 |
| Drivable area, intersection over union, all frames | 0.549 | 0.790 |
| Pixel accuracy | 0.598 | 0.748 |
| Agreement of the planner selection | 0.436 | 0.947 |
The refinement selects the best of many pose candidates for each frame. Its result is thus optimistic. The held-out estimate of the median for the drivable area is 0.824.
Test the Closed Loop with the Path Follower of the Lab#
The lab drove the real Polaris with its own path follower. Calibration/Polaris/closed_loop_sim.py lets the same follower drive the simulated Polaris on the same routes.
The plan is Calibration/Polaris/closed_loop_plan.md. The results are in Calibration/Polaris/closed_loop_results.md.
The source of the path follower belongs to the lab and is not in the repository.
-
Build the path follower from its local read-only copy.
-
Make the run configurations from the real runs.
-
Drive the configurations of one vehicle hypothesis in the game.
Add
--coreto use ACRES Core in place of the game. -
Change the bags into arrays.
-
Compute the metrics, the tables and the figures.
Steps 2 and 5 use torchenv. Steps 3 and 4 use the ROS 2 environment.
The test has three vehicle hypotheses and three settings of the follower.
| Hypothesis | Wheelbase | Steering Ratio | Option of the Game |
|---|---|---|---|
| H1, as fitted | 2.8702 m | 11.98 | None |
| H2 | 2.15 m | 16 | -PolarisSet=geometry.wheelbase_m=2.15;steering.ratio=16 |
| H3, the installed ratio | 2.8702 m | 16 | -PolarisSet=steering.ratio=16 |
| Follower Setting | Wheelbase Parameter | Ratio Parameter |
|---|---|---|
deployed |
2.04 m | 16 |
L_true |
The wheelbase of the vehicle | 16 |
L_ratio_true |
The wheelbase of the vehicle | The ratio of the vehicle |
Calibration/Polaris/follower_logs.py builds the real runs from the logs of the follower.
Calibration/Polaris/pp_kinematic.py gives a kinematic reference of the same control law.
Calibration/Polaris/closed_loop_analysis.py compares the simulated runs and the real runs at the same route index, not on a time axis.
The file Calibration/Polaris/closed_loop_results.md gives these results for hypothesis H1 with the deployed follower.
The path follower drove 15 simulated runs (7.7 km) without manual input.
| Metric | Real | Simulation | Result |
|---|---|---|---|
| Lap time, run A (2 m/s, route ACRE_CIRCLE) | 425 s | 357 s | 16 % short. Limit 10 %. Fail. |
| Lap time, run B (2 m/s, circle route) | 269 s | 237 s | 12 % short. Fail. |
| Lap time, run D (speed caps and stops of the real run) | 572 s | 602 s | 5 % long. Pass. |
| Distance to the nearest route vertex, run A | 0.65 m median, 2.85 m at 90 % | 0.25 m median, 1.13 m at 90 % | Paired median difference 0.31 m. Limit 0.4 m. Pass. |
| Mean signed offset from the route | 0.58 m right | 0.25 m right (A), 0.37 m right (B) | Same side. |
| Steering command above 280° | 9.3 % of the logged lines | 2.7 % | Fail. |
| Departures from the route | 1 (run C) | 0 | The simulation does not show the real departure. |
| Speed that the ULC holds against the command | 1.03 | 1.03 | Pass. |
The simulated Polaris follows the route more accurately than the real Polaris. The simulation agrees with the kinematic reference (lap 366 s, 0.71 m at 90 %). The results file states that the cause of the larger real error is not in the model. Hypotheses H1 and H2 give the same closed loop. The real runs reject hypothesis H3.
Set the Weather and the Soil of a Run#
The weather and the soil water of each recorded run come from the record of the Purdue Mesonet station at ACRE. Set Soil and Weather shows how to use the result in a session.
-
Download the station record into
$ACRE_WEATHER. -
Compute the conditions of each run.
Expected Result
The script writes
ground_conditions.md,ground_conditions.jsonand the replay files inCalibration/Weather/Replay.
| Output | Content |
|---|---|
Calibration/Weather/ground_conditions.md |
The rain, the soil water, the ground class and the command line for each run |
Calibration/Weather/Replay/env_<date>.json |
The weather of one day for the option -EnvConfig=. The repository has 33 days. |
Calibration/Weather/Replay/fields_w<index>/field-setup.json |
The soil water for the option -FieldSetup= |
Calibration/Weather/soil_state_cli.cpp |
A tool that evaluates the soil library of the game at a given water content |
ground_conditions.json covers 269 runs: 57 bags, 16 taught paths, 1 diagnostic bag and 195 runs of the path follower.
The ground class is dry for 57 runs, moist for 114 runs and wet for 98 runs.
The script does not use the probe readings as absolute water contents. It changes each reading into a wetness index: 0 at the wilting point, 1 at field capacity and 2 at saturation. The simulator then sets the water content of each soil unit from this index. Soil Water gives the model.
Calibration/Tractor/soil_weather_analysis.py shows the effect of these conditions on the Maxxum with a chisel plow.
The file Calibration/Tractor/soil_weather_results.json gives these steady-state values on silt loam.
| Quantity | Dry, Index 0.58 | Wet, Index 1.64 | Saturated, Index 2.0 |
|---|---|---|---|
| Cone index | 2.14 MPa | 0.63 MPa | 0.48 MPa |
| Rear wheel slip | 2.9 % | 3.4 % | 4.5 % |
| Draft | 21.1 kN | 21.7 kN | 22.6 kN |
| Rut depth | 1.2 cm | 1.9 cm | 3.5 cm |
| Fuel for each hectare | 9.08 L | 9.40 L | 9.95 L |
| Tractive efficiency | 0.791 | 0.775 | 0.750 |
Calibration/Tractor/soil_weather_results.md gives the full tables and the comparison with ASABE D497.
Results Summary#
| Result | Value | Source File |
|---|---|---|
| Steering wheel angle of the replay | RMSE 0.75° | Tools/PolarisModel/Data/fit_results.json |
| Line pressure of the replay | RMSE 0.34 bar | Tools/PolarisModel/Data/fit_results.json |
| Pedal command of the ULC | RMSE 0.37 % | Tools/PolarisModel/Data/fit_results.json |
| Speed on grass | RMSE 0.106 m/s | Tools/PolarisModel/Data/fit_results.json |
| Path curvature on grass | RMSE 0.00128 1/m, recorded RMS 0.00664 1/m | Tools/PolarisModel/Data/fit_results.json |
| Command timeout | 0.100 ± 0.008 s (95 %) | Tools/PolarisModel/Data/fit_results.json |
| LiDAR metrics in their limits | 9 of 11 | Calibration/Polaris/lidar_compare.json |
| Camera, drivable area at the refined pose | 0.840 median | Calibration/Polaris/camera_metrics.json |
| Closed loop with the path follower, lap time | 16 % short, 12 % short, 5 % long (three runs) | Calibration/Polaris/closed_loop_results.md |
| Closed loop with the path follower, paired route distance | 0.31 m median difference | Calibration/Polaris/closed_loop_results.md |