Skip to content

First Recorded Dataset#

This page records a dataset of 20 s with the camera and the LiDAR of the Maxxum. A drive script drives the vehicle, thus the session runs unattended.

Before You Start#

  • Build the packaged game. Refer to Build the Game.
  • Use a Python environment with numpy and matplotlib, for example the conda environment torchenv.
  • Keep 300 MB of the disk free. The dataset of this page uses 210 MB.

Record the Dataset#

  1. Go to the repository root.

    cd ACRES

  2. Write the drive script into the file drive.json.

    {
  "path": [[0, 0], [200, 0]],
  "keys": [
    {"t": 0, "gear": 8, "speed_kmh": 0},
    {"t": 2, "speed_kmh": 10},
    {"t": 20, "exit": true}
  ]
}

    The script follows a straight path of 200 m in front of the spawn pose. At 2 s it sets a speed of 10 km/h in gear 8. At 20 s it ends the session.

  3. Start the session. Give absolute paths.

    Packaged/Linux/Acres.sh -VehicleDemo -RenderOffscreen -SensorRecord -SessionLog \
  -VehicleSpawnU=625 -VehicleSpawnV=477 -VehicleSpawnYaw=180 \
  -DriveScript="$PWD/drive.json" -VehicleOutput="$PWD/out/first-dataset"

    Expected Result

    The game stops after approximately 30 s. The log on the terminal contains these lines.

    ACRES_DRIVE_SCRIPT keys=3 path_points=2 file=/home/user/ACRES/drive.json
ACRES_SENSOR_START episode=/home/user/ACRES/out/first-dataset/sensors/episode-001 camera=1 lidar=8x180 gpu_lidar=1 proxies=16582 classified_components=8379
ACRES_DRIVE_SCRIPT_DONE t=20.00
ACRES_SENSOR_STOP episode=/home/user/ACRES/out/first-dataset/sensors/episode-001 truth=2402 imu=2400 gnss=201 lidar=201 camera=199 can=802 ins=0 lidar_points=230774 dropped_rows=0 camera_missed=1 camera_busy=0 lidar_busy=0 errors=0

The options have these functions.

Option Function
-VehicleDemo Starts a session directly, without the menu.
-RenderOffscreen Renders without a window. Remove it to see the session.
-SensorRecord Records the sensors from the start of the session.
-SessionLog Writes the session log of the vehicle.
-VehicleSpawnU=, -VehicleSpawnV=, -VehicleSpawnYaw= Set the start pose on the survey grid. Cell (625, 477) is on the central farm road. A yaw of 180 degrees points to the west.
-DriveScript= Gives the drive script.
-VehicleOutput= Sets the session folder. The sensor episodes go into its folder sensors.

Find the Files#

  1. List the session folder.

    ls out/first-dataset out/first-dataset/sensors/episode-001

    Expected Result

    out/first-dataset:
farm-config.json  farm-fields.csv    farm-water.bin    sensors               tractor.csv
farm-events.csv   farm-state.json    farm-water.json   session-summary.json  water-depth.f32
farm-field.bin    farm-visuals.json  farm-weather.csv  theta.f32

out/first-dataset/sensors/episode-001:
actions.jsonl  camera.jsonl  episode.json  imu.jsonl  lidar.jsonl          truth.jsonl
camera         can.jsonl     gnss.jsonl    lidar      sensors-config.json

The episode folder has these contents after the run of this page.

Path Content Count Size
camera/front_<microseconds>.png Camera images, 896 x 512 pixels, 10 Hz 199 169 MB
lidar/scan_<microseconds>.ply LiDAR point clouds, 8 rings x 180 columns, 10 Hz 201 7.0 MB
lidar/scan_<microseconds>.png A small top view of each scan 201 2.0 MB
camera.jsonl, lidar.jsonl One row for each image and each scan: time, physics step, poses, file 199, 201 0.5 MB
gnss.jsonl GNSS epochs, 10 Hz 201 0.3 MB
imu.jsonl IMU samples, 120 Hz 2400 0.6 MB
can.jsonl CAN frames 802 0.2 MB
truth.jsonl, actions.jsonl The true vehicle state and the controls at each physics step 2402 2.4 MB
episode.json The manifest: calibration, sensor parameters, counts and drop counters 1 47 KB
sensors-config.json The sensor configuration of the episode 1 7 KB

The file name of an image or a scan gives the episode time in microseconds. Session Log gives the format of each file.

A recorded camera frame

Read the Dataset with Python#

  1. Write this program into the file read_episode.py.

    import json
from pathlib import Path

import matplotlib.image as mpimg
import numpy as np

episode = Path("out/first-dataset/sensors/episode-001")

# The manifest: counts and drop counters.
info = json.loads((episode / "episode.json").read_text())
print(info["schema"], info["state"], info["sensor_profile"])
print("counts ", info["counts"])
print("dropped", info["dropped"])

# One camera row and its image.
row = json.loads((episode / "camera.jsonl").read_text().splitlines()[100])
image = mpimg.imread(episode / row["file"])
print(row["file"], image.shape, image.dtype, "t =", row["sample_time_s"], "s")

# One LiDAR row and its point cloud (binary PLY: the header gives the fields).
row = json.loads((episode / "lidar.jsonl").read_text().splitlines()[100])
data = (episode / row["file"]).read_bytes()
header, body = data.split(b"end_header\n", 1)
types = {"float": "<f4", "ushort": "<u2", "uchar": "u1"}
lines = (line.split() for line in header.decode().splitlines())
fields = [(p[2], types[p[1]]) for p in lines if p[0] == "property"]
points = np.frombuffer(body, dtype=np.dtype(fields))
ranges = np.sqrt(points["x"] ** 2 + points["y"] ** 2 + points["z"] ** 2)
print(row["file"], len(points), "points, range",
      round(float(ranges.min()), 2), "to", round(float(ranges.max()), 2), "m")

  2. Run the program.

    python read_episode.py

    Expected Result

    acres-episode-2-unreal complete tractor-default
counts  {'truth': 2402, 'actions': 2402, 'gnss': 201, 'imu': 2400, 'lidar': 201, 'camera': 199, 'can': 802, 'ins': 0, 'lidar_points': 230774}
dropped {'rows_queue_overflow': 0, 'lidar_busy': 0, 'camera_missed': 1, 'camera_busy': 0, 'camera_encode_backlog': 0}
camera/front_010108334.png (512, 896, 4) float32 t = 10.108333861 s
lidar/scan_010000001.ply 1149 points, range 6.7 to 59.63 m

The image array has four channels: red, green, blue and alpha. The points are in the sensor frame: x forward, y left, z up, in metres. The PLY header gives the names of all point fields, for example intensity, ring, class and return_index.

Read One Row#

Each line of a .jsonl file is one JSON object. This is the first row of camera.jsonl, with some keys removed.

{
  "physics_step": 13,
  "simulation_time_s": 0.108333339,
  "file": "camera/front_000100000.png",
  "width": 896,
  "height": 512,
  "pose": {
    "position_enu_m": [189.900381, 35.052070, 3.265629],
    "quaternion_flu_to_enu_xyzw": [0.0, 0.0, 1.0, -0.0]
  },
  "frame_index": 0,
  "sample_time_s": 0.100000005,
  "available_time_s": 0.150000005,
  "delivery_time_s": 0.325000017
}
Key Unit Meaning
physics_step The number of the physics step of the sample. One step is 1/120 s.
simulation_time_s s The time of that physics step.
file The image file, relative to the episode folder.
pose.position_enu_m m The camera position: east, north and up from the origin of the tile.
pose.quaternion_flu_to_enu_xyzw The rotation from the camera body frame (forward, left, up) to the world frame.
sample_time_s s The time at which the sensor took the sample.
available_time_s s The sample time plus the delivery delay of the sensor configuration.
delivery_time_s s The physics time at which the recorder released the row.

Examine the Dataset#

The key dropped of episode.json counts the samples that the recorder did not write. In this run the recorder missed one camera image: camera_missed is 1. Collect Data gives the meaning of each drop counter.

Next Steps#