Skip to content

LiDAR Dataset#

The LiDAR dataset format stores full LiDAR sweeps as range images: from ACRES Core, from the game, or from a recording of the Polaris. The package acres_data in Data/ writes the format, reads it and makes the data sets. A model that predicts scans, or that learns from scans, reads the data through this package.

Before you start

  • The data is not in Git. The default folder is ~/data/acres/lidar. The variable ACRES_DATA sets a different folder.
  • The tutorial Collect Data makes a simulated data set step by step.

Format#

The stored form is the full cloud of the sensor as a range image. The RoboSense Helios of the Polaris has 32 rings and 1800 columns for each sweep, at 10 Hz. One cell is one beam: the ring r and the column c. The format does not store a top view.

<data root>/<set>/index.json                      the sequences of the set
<data root>/<set>/<sequence>/meta.json            source, identity, calibration, split tag, layers, counts
<data root>/<set>/<sequence>/scans.npy            one row for each scan: time, pose, quality, motion, counts
<data root>/<set>/<sequence>/range-00000.npy      uint16, shape (scans, 32, 1800)
<data root>/<set>/<sequence>/intensity-00000.npy  uint8, shape (scans, 32, 1800)
<data root>/<set>/<sequence>/body_mask.npy        bool, shape (32, 1800)
<data root>/<set>/<sequence>/labels/              layers and tables that only a simulator has
Item Value
Schema acres-lidar-dataset-1
Cell order Ring (0 to 31, the ring order of the driver), then column (0 to 1799).
Column 0 Straight ahead. The columns go clockwise, 0.2 degrees for each column.
range uint16. Range in meters = value x 0.0025. The value 0 is a cell without a return.
Largest stored range 163.8375 m. The sensor gives 161.3 m at most.
Range error 1.25 mm at most. The Helios packet has the same step of 2.5 mm: the range of a real scan has no loss.
intensity uint8, the reflectivity byte of the sensor (0 to 255). 0 in a cell without a return. Absent when the source has no intensity.
Shard 256 scans in one .npy file. The last shard of a sequence can be shorter.
Sequence One continuous drive, or one section of a drive, with one split tag.
Split tag train, val, test, guard, excluded or none.

The range is the distance from the lens center along the beam, as the driver of the sensor and ACRES Core define it. The lens center is 0.035 m from the spin axis. A point from the stored range and the beam table is 1.5 mm (root mean square) from the point of the source cloud. The largest such distance in the drive of 2 October is 0.114 m. It comes from the beam table and not from the range.

Each shard is a plain array file. The reader maps it into memory, and a read of one scan is one part of one file. The files have no compression: a random read costs no decompression.

The sensor layers are range and intensity. They are the same for a real and for a simulated sequence. The labels of a simulator are in the folder labels, apart from the sensor layers. The reader gives them only on request.

Label Form Source
material uint8 for each cell, in shards ACRES Core, the game. The codes are in meta.json.
class uint8 for each cell, in shards The game.
objects float32 (scans, 8, 10): present, kind, x, y, yaw, length, width, height, vx, vy ACRES Core: the true objects on the path.
commands float32 (scans, 4): curvature, speed, gear, shield flag ACRES Core: the command of the default driver.

Scan Table#

scans.npy has one row for each scan.

Field Type Unit Description
t float64 s The time of the scan on the clock of the pose. A real scan: Unix time. A simulated scan: time from the start.
stamp_ns int64 ns The time stamp of the source message.
east, north, up float64 m The position of base_footprint in the frame of the ACRE tile.
qw, qx, qy, qz float32 The attitude: the axes of base_footprint (x forward, y left, z up) to the axes of the map.
speed, lateral_speed float32 m/s The velocity in the body axes.
yaw_rate, pitch_rate, roll_rate float32 rad/s The angular rates in the body axes.
pose_quality uint8 0 none, 1 degraded, 2 ins, 3 rtk, 4 truth.
flags uint8 Bit 0: a gap. The time from the scan before is not one period.
dropped_columns uint16 The columns in which no ring has a return.
returns, body_returns uint32 The cells with a return, and those that the body rule marks.
source_index uint32 The number of the scan in its source.

A value that is not known is NaN. The motion between two scans is known only when the quality of the two poses is ins or better. A sequence with a degraded pose keeps its position to some meters, and the reader marks its motion as not valid.

The time of a real scan is the time of its last column. The rule for column c is in the calibration.

The returns of the vehicle body stay in the data. The body rule is that of the scan filter of the vehicle. A return is from the body when its cell is in body_mask.npy, or when its point is in the body box of meta.json.

Calibration#

Each meta.json has the calibration of its sensor.

Key Description
elevations_deg, azimuth_offsets_deg One value for each ring. A beam has the azimuth of its column plus the offset of its ring.
clockwise, lens_center_m The column order and the distance of the lens center from the spin axis.
sweep_s, last_column_at_stamp The time of one sweep. Column c fires (1799 - c) / 1800 x sweep_s before the time of the scan.
ring_firing_s The firing time of each ring in its column.
mount_xyz_m, mount_rpy_deg The sensor frame in base_footprint.
min_range_m, max_range_m The limits of the sensor.

A sweep_s of 0 shows a scan from one instant: all beams have one pose. A set made with --sweep has a sweep_s of 0.1: ACRES Core traced each column from the pose at its firing time.

Size#

Layers Bytes for Each Scan Bytes for Each Hour at 10 Hz
range 115,200 4.15 GB
range and intensity (a real scan) 172,800 6.22 GB
range and material (ACRES Core with no options) 172,800 6.22 GB
range, intensity and material (ACRES Core with the options) 230,400 8.29 GB
range, intensity, class and material (the game) 288,000 10.37 GB

The scan table and the object table add 0.4 kB for each scan.

Data Sets#

A data set is one folder below the data root. The tools make these kinds of data sets:

Kind Tool Content
ACRES Core missions python -m acres_data.generate_core The scans of single-field missions with objects on the path. Each sequence is one mission: a field, a soil condition, an object scenario and a seed.
ACRES Core twin of a route python -m acres_data.replay_core An ACRES Core scan at the pose of each recorded scan of a drive
Game sensor episode python -m acres_data.convert_game The GPU LiDAR scans of one sensor episode of the game
Polaris recording python -m acres_data.convert_polaris, python -m acres_data.convert_extract The scans of a recorded bag or of a bag extract

Each sequence of ACRES Core states the map hash, the simulation version and the hash of the sensor file. The split tag of a sequence (train, val, test) is that of its field in configs/splits_v1.json.

Reader#

LidarSet, Sequence#

LidarSet reads the index of a set. Sequence reads one sequence through memory maps. The two classes need numpy only.

Name Description
LidarSet.open(name, splits=None) A set below the data root. splits keeps the sequences with these tags.
Sequence.range_m(index) The range in meters, float32. NaN is a cell without a return. index is one number, a slice or a list.
Sequence.intensity(index) The intensity bytes, or None.
Sequence.label(name, index) A label of a simulator.
Sequence.scans The scan table.
Sequence.calibration.points(range_m, frame) The points in the frame "sensor" or "base".
Sequence.body_mask The cells that see the vehicle body. 
from acres_data import LidarSet
from acres_data.transforms import TopView, body_cells

ds = LidarSet.open("core-v3", splits=("train",))
seq = ds["S001"]
r = seq.range_m(100)                       # (32, 1800)
xyz = seq.calibration.points(r, "base")    # (32, 1800, 3)
body = body_cells(seq.calibration, r, seq.body_mask)
grid = TopView()(seq.calibration, r, drop=body)

TopView#

The top view is a transform of the reader. It gives a grid in base_footprint: x forward, y left.

Name Default Description
x_min, y_min -25.6 m The corner of the grid.
cell_m 0.2 m The side of a cell.
cells_x, cells_y 256 The number of cells. The grid is 51.2 m square, with the vehicle in the center.
z_min, z_max -1 m, 3 m The height band above base_footprint.

The result has two channels. Channel 0 is 1 where a cell has a return. Channel 1 is the largest height in the cell.

from acres_data.transforms import TopView

grid = TopView(x_min=0.0, y_min=-12.8, cell_m=0.1)
image = grid(seq.calibration, seq.range_m(slice(0, 8)))
image.shape    # (8, 2, 256, 256)

ScanWindows#

acres_data.torch_dataset.ScanWindows is the PyTorch data set. It is the only module that needs torch. One sample has K past scans, the next scans, and the motion of the vehicle between them. A sample has no action. The commands are labels.

Key Shape Description
past (K, 32, 1800) or (K, 2, 256, 256) The range in meters with 0 for no return, or the top view.
future (F, ...) The next scans in the same form.
motion (K + F, 3) x forward, y left and yaw of each scan in the frame of the last past scan.
motion_valid (K + F) False where a pose is degraded or absent.
simulated, set, sequence, anchor The origin of the sample.

A window does not cross a gap or the end of a sequence. The argument match applies the sensor-matching step to simulated sequences only.

import torch
from acres_data.match import SensorMatch
from acres_data.torch_dataset import ScanWindows

ds = ScanWindows(["core-v3"],
                 splits=("train",), past=8, future=1,
                 view="top", match=SensorMatch.load())
loader = torch.utils.data.DataLoader(
    ds, batch_size=16, shuffle=True, num_workers=8)

The reader with 8 worker processes gives about 7,000 range images, or about 1,500 top views, for each second. Run python -m acres_data.speed to measure your computer.

Sensor Differences#

The simulated sensor is the block lidar of Acres/Content/Simulation/sensors_polaris.json. The table compares it with the RoboSense Helios of the Polaris.

Item Real Sensor ACRES Core
Rings and columns 32 and 1800, clockwise, column 0 ahead The same
Beam angles Read from the sensor at its start A table measured from recorded scans, within 0.007 degrees
Range 0.2 m to 161.3 m, step 2.5 mm 0.2 m to 161.3 m, noise 2.2 mm
Column time 55.56 µs for each column: a sweep of 0.1 s An option: each column traced from the pose at its firing time. Off: all beams at one instant.
Returns for each beam One, the strongest One
Intensity A byte, 0 to 255 An option of Batch.lidar: the same byte
Vehicle body In the scan From a table measured on real scans
Crops and ground cover In the scan An option: plant cells in the crop patches of the map, and a grass layer

The game and ACRES Core share the crop canopy model (AcresLidarCanopy.cpp) and the crop state of a day (AcresLidarDayState.cpp). The game option -LidarCropState=<path> and the Python call Batch.set_lidar_crop_state load a daily crop state for the LiDAR. The page LiDAR gives the model and its validation.

Sensor-Matching Step#

The step acres_data.match.SensorMatch makes a simulated scan statistically like a real scan. It is a transform of the reader for simulated scans. It does not change stored data. It has a seed: the same scan gives the same result.

Part Function Fitted Values
Sweep Draws the scan of one instant as a sweep of 0.1 s. Each column gets the direction and the range for the pose at its firing time. None
Keep A return stays with a probability for its ring and its range bin (32 rings, 21 bins). 672 values
Cell A return stays with a probability for its ring and its group of 4 columns: the real pattern of missing cells. 14,400 values
Noise A normal error of the range on ground returns, one value for each ring. 32 values, 0 to 70 mm
Chop Removes the returns beyond one range. A mode of its own. 150 m
File Use
Data/acres_data/configs/sensor_match_v1.json Scans of one instant with no crops and no intensity
Data/acres_data/configs/sensor_match_v2.json Scans with crops, intensity and the sweep. The sweep part is off.
Data/acres_data/configs/sensor_match_game_v0.json A first keep table for the GPU LiDAR of the game

The step removes returns and adds none. Its keep table has no class, thus it also removes near crop returns. For a model that must see the crop near the vehicle, use the scans with crops and no step, or the noise part only.

Make a Data Set#

Run the commands in the repository root with the Python of your build. Each tool has the option --workers. The result does not depend on the number of workers.

  1. Set the paths:

    export PYTHONPATH=Data:Simulator:Learning:Core/Build

  2. Make a simulated data set with crops, intensity and the sweep:

    python -m acres_data.generate_core --set core-v3 --crops --sweep --seed 1 --fields-tile 20,6,10 --fields-region 20,7,12

    The run-time library drives each mission with the skills scout and return_home. --list shows the missions.

  3. Convert a sensor episode of the game:

    python -m acres_data.convert_game --run <run folder> --name <sequence>

  4. Fit the sensor-matching step on a twin of a route:

    python -m acres_data.compare fit --twin <set> --config <file>
python -m acres_data.compare show

  5. Measure the reader: python -m acres_data.speed --out <file>.

One ACRES Core scan of 57,600 beams takes about 28 ms on one thread.

Limits#

  • A simulated crop is mature and standing in each planted field, with the marks of the vehicle. It has no plant rows.
  • The crop state of the site on a recorded day is not known unless a crop-state file gives it.
  • The body rule does not remove the motion in the sweep before it tests the body box.
  • A data set from the game needs Unreal and a GPU, and only one game can run at a time.