Sensors and Rigs#

This tutorial shows the sensor rig of each vehicle and how to change a mount pose and a rate. It also shows how to examine the result in a recording, and what the equipment rig does.

Before You Start#

Build the packaged game. Refer to Build the Game.
Do the tutorial First Recorded Dataset. It shows the files of a sensor episode.
Use Python 3. The examples use only the standard library.
Keep 700 MB of the disk free. The three recordings of this page use 540 MB.

Do all steps in the repository root.

The Sensor Rig of Each Vehicle#

A sensor rig is the set of sensors of a vehicle with their mounting poses. The Maxxum uses Acres/Content/Simulation/sensors.json. The Polaris uses the same file with the overlay sensors_polaris.json on top. Each key of the overlay replaces the key of the base file. Objects merge key by key.

The sensor rigs of the Maxxum and the Polaris, with the mount values of the two configuration files. Open the diagram

A mount position is in body FLU metres: forward, left, up. A mount rotation is roll, pitch and yaw in degrees. The rotation is about the fixed body axes in the sequence roll, pitch, yaw. A positive pitch points the sensor down. The key mount_frame gives the origin of the positions.

`mount_frame`	Origin	Used By
`chassis`	The chassis origin. It is 0.4 of the wheelbase in front of the rear axle, at the height of the centre of gravity.	Maxxum
`base_footprint`	The middle of the rear axle, on the ground.	Polaris

The chassis origin of the Maxxum is 1.057 m in front of base_footprint and 1.100 m above it. The chassis origin of the Polaris is 1.148 m in front of base_footprint and 0.655 m above it.

Maxxum#

The sensors of the Maxxum are generic sensors on the roof of the cab. The values are not a survey of a real installation.

Sensor	Key of the Position	Position	Rotation	Rate	Main Settings
Front camera	`camera.position_flu_m`	[0.6, 0, 1.65]	[0, 0, 0]	10 Hz	896 × 512 pixels, 90° horizontal field of view
LiDAR	`lidar.position_flu_m`	[0, 0, 1.9]	[0, 0, 0]	10 Hz	8 rings × 180 columns, 60 m
GNSS antenna	`gnss.antenna_flu_m`	[0, 0, 1.7]	-	10 Hz	RTK receiver model
IMU	`imu.position_flu_m`	[0, 0, 0]	-	120 Hz	Constant bias and white noise
CAN	-	-	-	20 Hz	Two frames in `can.jsonl`

Polaris#

The sensors of the Polaris are the sensors of the real vehicle of the Purdue lab.

Sensor	Key of the Position	Position	Rotation	Rate	Main Settings
RoboSense Helios LiDAR	`lidar.position_flu_m`	[2.5, 0, 2.039]	[-1.518, 5.317, 0]	10 Hz	32 rings × 1800 columns, 161.3 m
Reolink camera	`camera.position_flu_m`	[2.745, 0.606, 1.887]	[-3.49, 16.52, -2.63]	10 Hz	896 × 512 pixels, calibrated lens
OxTS antenna	`gnss.antenna_flu_m`	[1.1, 0, 2.15]	-	10 Hz	RTK receiver model
OxTS IMU	`imu.position_flu_m`	[0.95, -0.45, 0.7]	-	120 Hz	Constant bias and white noise
OxTS INS output	`ins.position_flu_m`	[0, 0, 0]	-	100 Hz	Fix, IMU, velocity and UTM odometry at `base_footprint`
CAN	-	-	-	off	The Polaris uses drive-by-wire reports

The rotation keys are lidar.rpy_deg and camera.rpy_deg. The pages Camera, LiDAR, INS and GNSS and Wheel and CAN Signals give the models.

Record with the Default Rig#

Record the Maxxum with the default sensor rig.

Packaged/Linux/Acres.sh -VehicleDemo -RenderOffscreen -VehicleTest=turn -VehicleTerrainTest \
  -SensorRecord -SensorOutput="$PWD/out/rig-default"

Expected Result

The game stops after approximately 35 s. The log on the terminal contains these lines.

ACRES_SENSOR_PROFILE profile=tractor-default validated=0 changes=none
ACRES_SENSOR_START episode=/home/user/ACRES/out/rig-default/episode-001 camera=1 lidar=8x180 gpu_lidar=1 proxies=16582 classified_components=8379
ACRES_SENSOR_STOP episode=/home/user/ACRES/out/rig-default/episode-001 truth=2040 imu=2038 gnss=170 lidar=169 camera=169 can=680 ins=0 lidar_points=203127 dropped_rows=0 camera_missed=1 camera_busy=0 lidar_busy=1 errors=0

Write this program into the file show_rig.py.

import json
import sys

m = json.load(open(sys.argv[1]))
print(m["state"], m["sensor_profile"], m["sensor_profile_detail"]["changes"])
print(m["counts"])
for name, position in m["extrinsics_flu_m"].items():
    print(name, [round(v, 3) for v in position])
print("rpy", m["mount_rotations"]["lidar_rpy_deg"], m["mount_rotations"]["camera_rpy_deg"])
c = m["config"]
print("seed", c["seed"], "gnss", c["gnss"]["hz"], "Hz, camera", c["camera"]["hz"], "Hz")

Show the sensor rig of the recorded episode.

python3 show_rig.py out/rig-default/episode-001/episode.json

Expected Result

complete tractor-default []
{'truth': 2040, 'actions': 2040, 'gnss': 170, 'imu': 2038, 'lidar': 169, 'camera': 169, 'can': 680, 'ins': 0, 'lidar_points': 203127}
imu [0, 0, 0]
gnss [0, 0, 1.7]
lidar [0, 0, 1.9]
camera [0.6, 0, 1.65]
base_footprint [-1.057, 0, -1.1]
rpy [0, 0, 0] [0, 0, 0]
seed 42 gnss 10 Hz, camera 10 Hz

The options have these functions.

Option	Function
`-VehicleTest=turn`	Drives a fixed program of 17 s and then stops the game. The program turns to the left from 5 s to 11 s.
`-VehicleTerrainTest`	Keeps the vehicle on the ACRE terrain. Without it, a test program uses a flat test area.
`-SensorRecord`	Records the sensors from the start of the session.
`-SensorOutput=`	Sets the folder of the episodes. Give an absolute path.

The file episode.json is the record of the sensor rig. The object extrinsics_flu_m gives all positions from the chassis origin. The count of the camera and of the LiDAR can differ by one or two samples between two runs. A recording of 17 s gives 2040 physics steps, 170 GNSS epochs and 340 CAN samples of two frames each.

Note

The state is complete_with_errors when a recording stops while a LiDAR scan is in work on the GPU. The array errors of episode.json then gives the cause. The other streams are complete.

Change a Mount Pose and a Rate#

The mount poses have no command-line option. Change them in a copy of the configuration file. The option -SensorConfig= then replaces sensors.json with the copy. The copy must be a complete file.

Write this program into the file make_sensors.py. It moves the camera, points it 10° down and sets the GNSS rate to 5 Hz.

import json
import sys

source, target = sys.argv[1], sys.argv[2]
with open(source) as f:
    config = json.load(f)
config["camera"]["position_flu_m"] = [0.9, 0.0, 1.70]
config["camera"]["rpy_deg"] = [0.0, 10.0, 0.0]
config["gnss"]["hz"] = 5
with open(target, "w") as f:
    json.dump(config, f, indent=2)

Make the changed configuration file.

python3 make_sensors.py Acres/Content/Simulation/sensors.json my_sensors.json

Record with the changed file. Set a different seed and camera rate, and set the LiDAR off.

Packaged/Linux/Acres.sh -VehicleDemo -RenderOffscreen -VehicleTest=turn -VehicleTerrainTest \
  -SensorRecord -SensorConfig="$PWD/my_sensors.json" -SensorSeed=7 -SensorNoLidar -SensorCameraHz=5 \
  -SensorOutput="$PWD/out/rig-changed"

Expected Result

The log contains these lines.

ACRES_SENSOR_PROFILE profile=tractor-default validated=0 changes=camera.hz,camera.mount.rpy_deg,camera.position_flu_m
ACRES_SENSOR_STOP episode=/home/user/ACRES/out/rig-changed/episode-001 truth=2040 imu=2038 gnss=85 lidar=0 camera=85 can=680 ins=0 lidar_points=0 dropped_rows=0 camera_missed=0 camera_busy=0 lidar_busy=0 errors=0

Show the sensor rig of the new episode.

python3 show_rig.py out/rig-changed/episode-001/episode.json

Expected Result

complete custom ['camera.hz', 'camera.mount.rpy_deg', 'camera.position_flu_m']
{'truth': 2040, 'actions': 2040, 'gnss': 85, 'imu': 2038, 'lidar': 0, 'camera': 85, 'can': 680, 'ins': 0, 'lidar_points': 0}
imu [0, 0, 0]
gnss [0, 0, 1.7]
lidar [0, 0, 1.9]
camera [0.9, 0, 1.7]
base_footprint [-1.057, 0, -1.1]
rpy [0, 0, 0] [0, 10, 0]
seed 7 gnss 5 Hz, camera 5 Hz

The camera is at the new position, and its pitch is 10°. The GNSS receiver and the camera have 85 samples each. The sensor profile is custom, because the camera data are different from the data of the shipped profile. The list changes contains only camera and LiDAR settings. The switches and the rates of the other sensors do not change the profile. A row of camera.jsonl shows the pose of the camera in the world for each image.

CAUTION

The game stops with an error message when a value is out of its range. For example, a LiDAR range of 210 m stops the game with the message LiDAR range 1-200 m; elevation window within +/-90.

Record the Polaris Rig#

The Polaris gets its sensor rig from the overlay. No option is necessary.

Record the parked Polaris for 17 s.

Packaged/Linux/Acres.sh -VehicleDemo -RenderOffscreen -Vehicle=polaris -VehicleTest=coast \
  -VehicleTerrainTest -SensorRecord -SensorOutput="$PWD/out/rig-polaris"

Show the sensor rig.

python3 show_rig.py out/rig-polaris/episode-001/episode.json

Expected Result

complete_with_errors polaris-calibrated []
{'truth': 2042, 'actions': 2042, 'gnss': 171, 'imu': 2040, 'lidar': 170, 'camera': 170, 'can': 0, 'ins': 1701, 'lidar_points': 7391766}
imu [-0.198, -0.45, 0.045]
gnss [-0.048, 0, 1.495]
lidar [1.352, 0, 1.384]
camera [1.597, 0.606, 1.232]
ins [-1.148, 0, -0.655]
base_footprint [-1.148, 0, -0.655]
rpy [-1.518, 5.317, 0] [-3.49, 16.52, -2.63]
seed 42 gnss 10 Hz, camera 10 Hz

The file episode.json gives the positions from the chassis origin for both vehicles. Subtract the row base_footprint to get the values of the overlay. For example, the LiDAR gives [2.5, 0, 2.039].

The INS point is base_footprint. The INS writes 100 rows for each second into ins.jsonl. The sensor profile is polaris-calibrated. This shows the validated profile without changes. The state of this run is complete_with_errors, because the game stopped while the last LiDAR scan was in work. Refer to the note above.

To change the rig of the Polaris, copy sensors_polaris.json, change the copy and give it with -PolarisSensorConfig=. The copy can contain only the keys that are different from sensors.json.

Note

Do not use -VehicleTest=turn with the Polaris. The program uses 70 % throttle, and the Polaris turns over in the turn.

The menu changes the sensor settings that a user changes frequently. It does not change the mount poses.

Start the game without options.
```
Packaged/Linux/Acres.sh
```
Select New Simulation.
Select the tab Sensors.
Change the settings. The table below gives the groups.
Select Start.

Group	Settings
Camera	Front Camera, Frame Rate, Warm-Up Frames, Width, Height, Horizontal Field of View, Exposure, ISO, Shutter, Aperture, Sensor Noise, Lens Distortion, Rolling Shutter, Readout Time
LiDAR	LiDAR, Scan Rate, Channels (Rings), Points per Ring, Maximum Range, Lowest Beam, Highest Beam, Range Noise, Returns
GNSS	GNSS Receiver, Rate, Receiver Start, Elevation Mask, RTK Correction Age Limit, Use Galileo, Use GLONASS
IMU	IMU, Rate, Accelerometer Noise, Gyro Noise
CAN	CAN Bus, Rate
Recording	Delivery Latency, Noise Seed

The menu writes a session folder below Packaged/Linux/Acres/Saved/Sessions. The folder contains a sensors.json with the settings. The menu then starts the session with -SensorConfig= for this file. With one sensor on, it adds -SensorRecord. The episodes of the session go into the folder sensors of the session folder.

The mode Pilot has no Sensors tab and records no sensors. The Menu Settings page gives each setting with its range and its key.

CAUTION

The menu permits a LiDAR range of 250 m and a camera field of view of 150°. The game accepts a maximum of 200 m and 120°. Keep the two settings at or below these limits.

Sensor Options#

A command-line option replaces the value of the configuration file.

Name	Type	Unit	Default	Description
`-SensorRecord`	flag		off	Records the sensors from the start of the session. The key `F6` starts and stops a recording in the game.
`-SensorOutput=`	path		`<vehicle output>/sensors`	The folder of the episodes.
`-SensorConfig=`	path		`sensors.json`	A complete configuration file that replaces the shipped file.
`-PolarisSensorConfig=`	path		`sensors_polaris.json`	The overlay of the Polaris.
`-SensorSeed=`	integer		42	The seed of all noise streams.
`-SensorDelay=`	number	s	0.05	The modelled latency of the rows.
`-SensorGnssHz=`, `-SensorImuHz=`	number	Hz	10, 120	The rate of the GNSS receiver and of the IMU.
`-SensorLidarHz=`, `-SensorCameraHz=`	number	Hz	10, 10	The rate of the LiDAR and of the camera.
`-SensorCameraWidth=`, `-SensorCameraHeight=`	integer	pixel	896, 512	The size of the camera image.
`-SensorCameraHfov=`	number	deg	90	The horizontal field of view of the camera.
`-SensorCameraWarmup=`	integer	frame	2	The renders before each camera sample. Range 0 to 16. 0 renders on each frame.
`-SensorCameraNoHistory`	flag			Makes one render for each sample, without temporal history.
`-SensorCameraExposure=`	string		`volume`	The exposure mode: `volume`, `physical` or `copy`.
`-SensorCameraIdeal`	flag			Sets the lens, rolling shutter and noise models of the camera off.
`-SensorLidarColumns=`, `-SensorLidarRings=`	integer		180, 8	The layout of the LiDAR.
`-SensorLidarRange=`	number	m	60	The maximum range of the LiDAR.
`-SensorNoCamera`, `-SensorNoLidar`, `-SensorNoGnss`, `-SensorNoImu`, `-SensorNoCan`, `-SensorNoIns`	flag			Sets one sensor off.
`-SensorNoNoise`	flag			Sets the noise of all sensors to zero.

All rates have the range 1 Hz to 120 Hz. The CAN rate and the INS rate have no option. The pages of the sensor models give the options of each model, for example -SensorGnssOutage=. The Command-Line Options page lists all options of the game. A client can also move the camera and the LiDAR while a session runs. Refer to the request set_vehicle_shift on the Simulator Control Channel page.

The Equipment Rig#

The equipment rig is a different thing from the sensor rig. It is the skeleton and the controls of a vehicle mesh or an implement mesh. The physics state moves the equipment rig in each frame. The sensors do not use it: a sensor mount is rigid on the chassis.

Vehicle	Rig Map	Bones	Controls from the Physics State
Maxxum	`Acres/Content/Simulation/rigs/maxxum_150.json`	15	Wheel spin, steering angle, front axle oscillation, hitch lift angle
Polaris	`Acres/Content/Simulation/rigs/polaris.json`	21	Wheel roll, steering wheel, suspension travel, pedals

A rig map contains the bones, the controls and the driver expressions of the Blender source file. The class UAcresRigComponent reads the map and sets the bone poses from the control values. For example, the rig gets one steering angle and divides it between the two front wheels by the Ackermann rule. Each implement has its own rig map in the same folder.

Examine all rig maps against the test poses of Blender.

Packaged/Linux/Acres.sh -VehicleDemo -nullrhi -RigSelfTest

Expected Result

The log contains one line for each of the 10 rig maps and one summary line. These are four of the lines.

ACRES_RIG_SELFTEST chisel_plow PASS poses=7 bones=10 max_position_error_mm=0.00011 max_angle_error_deg=0.000004
ACRES_RIG_SELFTEST maxxum_150 PASS poses=7 bones=15 max_position_error_mm=0.00039 max_angle_error_deg=0.000150
ACRES_RIG_SELFTEST polaris PASS poses=7 bones=21 max_position_error_mm=0.00047 max_angle_error_deg=0.000325
ACRES_RIG_SELFTEST_DONE passed=10 of 10

Stop the game with Ctrl+C.

The option -nullrhi starts the game without a renderer. The test needs only the files and the mathematics. The Add a Vehicle page gives the procedure to make a rig map for a new mesh.

Next Steps#

Collect Data records datasets with a drive script and with the ROS 2 tools.
INS and GNSS gives the keys of the GNSS receiver, the IMU and the INS.
Camera and LiDAR give the keys of the two scene sensors.
Sensor Stream sends the sensor data to ROS 2 while the session runs.
Add a Sensor gives the procedure for a new sensor.

Sensors and Rigs#

Before You Start#

The Sensor Rig of Each Vehicle#

Maxxum#

Polaris#

Record with the Default Rig#

Change a Mount Pose and a Rate#

Record the Polaris Rig#

Configure the Sensors in the Menu#

Sensor Options#

The Equipment Rig#

Next Steps#