Skip to content

Sensor Stream#

The sensor stream is a binary socket for the sensor data of one agent: clock, INS data, LiDAR clouds and camera images. This page gives the protocol as the game writes it and as the ROS 2 bridge reads it.

The game opens the stream with -SensorStream=<port>. The game has no default port. The ROS 2 launch files, the Core server and the tools use port 5601. The ROS 2 bridge changes the frames into ROS 2 messages: refer to Topics. The other two sockets of a session are the Vehicle Bridge and the Simulator Control Channel.

Item Type Function
-SensorStream= Option Opens the stream of agent 0.
-SensorStreams= Option Opens one stream for each agent.
Transport Protocol TCP on 127.0.0.1, one client, binary frames.
Frame Header Protocol The 32 bytes at the start of each frame.
Units and Frames Protocol The time bases, the coordinate frames and the units.
Hello (1) Frame The sensor layout of the agent as JSON.
Clock (2) Frame The time of each physics step.
INS (3) Frame One INS epoch: 40 values.
LiDAR (4) Frame One organised cloud.
Camera (5) Frame One image.
Barrier (6) Frame The end of a lockstep step.
Rates and Sizes Protocol The frame rates and the data volume.
Flow Control and Drops Protocol The queue limits and the drop counters.
Log Lines Protocol The lines that the game writes to its log.
Python Reader Example A reader for the header and the LiDAR frame.
Differences in ACRES Core Table The behaviour of the Core server.

Options#

-SensorStream=#

Opens the sensor stream of agent 0 on 127.0.0.1 at the given port. The option also starts the sensor recorder of the agent. Without -SensorRecord, the recorder writes no files.

A vehicle that has no INS in its sensor configuration gets one. The Maxxum is such a vehicle. The INS then reports the point base_footprint.

Name Type Unit Default Description
-SensorStream= integer no stream The TCP port, from 1 to 65535.
sensor_stream integer no stream The same value as a key of one agent in the file of -Agents=. The key has priority. 
Packaged/Linux/Acres.sh -VehicleDemo -Vehicle=polaris \
    -SensorStream=5601

ACRES_SENSOR_STREAM_LISTENING agent=polaris port=5601
ACRES_SENSOR_START episode=(stream only) camera=1
  lidar=32x1800 gpu_lidar=1 proxies=16582
  classified_components=8379

-SensorStreams=#

Opens one sensor stream for each agent of a session with more than one vehicle. The value is a list of ports with commas. The port at index N is the port of agent N. The game uses the list for an agent only when the agent has no port from -SensorStream= or from its key sensor_stream.

Each stream has its own sender thread, its own queue and its own sequence numbers. The stamps of all streams use one clock, the simulation time of the world.

Name Type Unit Default Description
-SensorStreams= list of integers no stream One TCP port for each agent, in the sequence of the agents. 
Packaged/Linux/Acres.sh -VehicleDemo \
    -Vehicles=polaris,maxxum \
    -SensorStreams=5601,5602

ACRES_SENSOR_STREAM_LISTENING agent=polaris port=5601
ACRES_SENSOR_STREAM_LISTENING agent=maxxum port=5602

The Python Reader on the port of the Maxxum:

hello maxxum 180 x 8
lidar seq=45 stamp=0.808 s shape=(180, 8, 4) returns=1013

Protocol#

Transport#

The stream is a TCP socket. The game listens on 127.0.0.1 only. The data does not go out of the computer.

  • One client reads the stream at a time. A second client stays in the listen queue until the first client disconnects.
  • The game accepts a client only when the sensor recorder knows its layout.
  • The first frame on each connection is the Hello frame.
  • The game does not read from the socket. The client sends no data.
  • The game queues frames only while it has a client. Data from the time before the connection is not in the stream.
  • The socket has TCP_NODELAY and a send buffer request of 16 MiB.

All numbers are little-endian. A frame is a header of 32 bytes and a payload.

import socket

sock = socket.create_connection(("127.0.0.1", 5601))

#include "acres_sim/stream.hpp"

const std::string address = acres_sim::require_loopback("127.0.0.1");
const int fd = acres_sim::connect_tcp(address, 5601, 2.0);

ACRES_SENSOR_STREAM_CONNECTED agent=polaris port=5601
  send_buffer=8388608

Frame Header#

Each frame starts with this header. PutHeader in AcresSensorStream.cpp writes it and parse_frame_header in stream.hpp reads it.

Offset Type Field Contents
0 char[4] magic The characters TSS1.
4 uint16 type The frame type, 1 to 6.
6 uint16 flags Always 0.
8 uint32 sequence A counter for all frame types of this stream.
12 uint32 payload bytes The number of bytes after the header.
16 float64 stamp The simulation time of the data in seconds.
24 float64 publish time The time of CLOCK_MONOTONIC in seconds when the game queued the frame.

The sequence increases by 1 for each frame that the game queues. A dropped frame gets no number. The counter does not start again at a new connection.

A client that reads CLOCK_MONOTONIC on the same computer can measure the transport time: its clock minus the publish time. The Python function time.monotonic() reads that clock on Linux.

The ROS 2 bridge closes the connection and connects again when the magic is not TSS1.

import struct

HEADER = struct.Struct("<4sHHIIdd")
(magic, kind, flags, sequence, size,
 stamp_s, publish_s) = HEADER.unpack(data[:32])

acres_sim::FrameHeader header;
if (!acres_sim::parse_frame_header(data, header))
    return;  // bad magic

The first frames of a session with the Polaris:

type=1 sequence=0 payload=1210    stamp=0.000000
type=5 sequence=1 payload=1376272 stamp=0.108333
type=4 sequence=2 payload=921616  stamp=0.108333
type=2 sequence=3 payload=8       stamp=0.325000
type=3 sequence=4 payload=320     stamp=0.325000

Units and Frames#

Quantity Convention
Stamp The physics clock of the world in seconds. All agents of a session use this clock.
Vehicle time The physics time of one agent. The stamp is the vehicle time plus a constant offset.
Publish time CLOCK_MONOTONIC of the computer in seconds.
Body frame FLU: x forward, y to the left, z up, in metres.
base_footprint The middle of the rear axle on the ground. The INS reports this point.
World frame ENU on the map grid of the tile: x east, y north, z up, in metres.
UTM Easting and northing in metres in the zone of the Hello frame (16 for ACRE).
Quaternion The sequence x, y, z, w. It rotates a vector from the body frame into the reference frame.
LiDAR points The sensor frame of the LiDAR (FLU) in metres.
Image Rows from the top, 8 bits for each channel, channel sequence blue, green, red.

The stream has no modelled delay. The keys delay_s and lidar.delay_s apply only to the files of the session log. A LiDAR frame and a camera frame go out when the game completes them. Their stamps give the time of the measurement.

The clock frame gives the offset between the two time bases:

offset_s = stamp_s - struct.unpack("<d", payload)[0]

The vehicle bridge reports its data in vehicle time. Add the offset to compare the two sockets.

Frame Types#

Hello (1)#

The first frame of each connection. The payload is one JSON object in UTF-8. The stamp is 0. FAcresSensorRecorder::StreamHello makes the text.

Name Type Unit Default Description
schema string acres-sensor-stream-1.
agent string The name of the agent.
vehicle string maxxum or polaris.
georeferenced boolean true when the map has a geographic reference.
mount_convention string A text that describes the mount poses.
body_origin_flu_m array of 3 m The origin of the vehicle body, seen from base_footprint.
lidar.enabled boolean true when the agent has a LiDAR.
lidar.frame_id string The frame name of the cloud.
lidar.hz number Hz The scan rate.
lidar.columns, lidar.rings integer The size of the organised cloud.
lidar.range_m number m The maximum range.
lidar.azimuth_order string cw or ccw: the direction of the column index.
lidar.position_flu_m, lidar.rpy_deg array of 3 m, deg The mount pose from base_footprint.
camera.enabled boolean true when the camera renders.
camera.hz number Hz The image rate.
camera.width, camera.height integer px The size of the image.
camera.fx, fy, cx, cy number px The intrinsics of the image (OpenCV convention).
camera.distortion_model string plumb_bob.
camera.d array of 5 \(k_1, k_2, p_1, p_2, k_3\). All 0 when the lens model is off.
camera.encoding string bgr8.
camera.position_flu_m, camera.rpy_deg array of 3 m, deg The mount pose from base_footprint.
ins.enabled boolean true when the agent sends INS frames.
ins.hz number Hz The INS rate.
ins.position_flu_m array of 3 m The INS point from base_footprint.
ins.utm_zone integer The UTM zone.
ins.datum string nad83_2011 or wgs84_g2139.
ins.imu_position_flu_m, ins.gnss_antenna_flu_m array of 3 m The mount positions of the IMU and the antenna.

The rotations are roll, pitch and yaw about the fixed FLU axes, \(R = R_z R_y R_x\). The Hello frame gives the calibrated mounts. A hardware shift from the simulator control channel does not change it.

The Hello frame of the Polaris, with shortened numbers:

{
 "schema": "acres-sensor-stream-1",
 "agent": "polaris",
 "vehicle": "polaris",
 "georeferenced": true,
 "body_origin_flu_m": [1.14808, 0, 0.65536],
 "lidar": {
  "enabled": true, "frame_id": "lidar", "hz": 10,
  "columns": 1800, "rings": 32, "range_m": 161.3,
  "azimuth_order": "cw",
  "position_flu_m": [2.5, 0, 2.039],
  "rpy_deg": [-1.518, 5.317, 0]
 },
 "camera": {
  "enabled": true, "hz": 10,
  "width": 896, "height": 512,
  "fx": 599.526, "fy": 592.378,
  "cx": 440.130, "cy": 270.805,
  "distortion_model": "plumb_bob",
  "d": [-0.55389, 0.40842, -0.0099785,
        0.0044239, -0.16673],
  "encoding": "bgr8",
  "position_flu_m": [2.745, 0.606, 1.887],
  "rpy_deg": [-3.49, 16.52, -2.63]
 },
 "ins": {
  "enabled": true, "hz": 100,
  "position_flu_m": [0, 0, 0],
  "utm_zone": 16, "datum": "nad83_2011",
  "imu_position_flu_m": [0.95, -0.45, 0.7],
  "gnss_antenna_flu_m": [1.1, 0, 2.15]
 }
}

Clock (2)#

One frame for each physics step: 120 frames in each second of simulation time. The physics thread queues it.

Offset Type Field Contents
0 float64 vehicle time The physics time of the agent in seconds.

The stamp of the header is the same time on the clock of the world. The ROS 2 bridge of agent 0 publishes the stamp on the topic /clock.

if kind == 2:
    vehicle_time_s = struct.unpack("<d", payload)[0]

type=2 sequence=3 payload=8 stamp=0.325000
  vehicle time 0.32500001695007086

INS (3)#

One INS epoch at the rate ins.hz. The payload is 40 values of the type float64: 320 bytes. The values with an index from 0 to 31 contain the errors of the INS model. The values from 32 to 39 are the exact values of the simulation. INS and GNSS gives the model.

Index Field Unit Contents
0, 1 UtmE, UtmN m UTM easting and northing.
2 UtmH m Ellipsoidal height of the fix.
3 to 6 UtmQx, UtmQy, UtmQz, UtmQw Attitude in the UTM grid: x grid east, y grid north, z up.
7 to 9 VelX, VelY, VelZ m/s Velocity in the body frame.
10 to 12 GyroX, GyroY, GyroZ rad/s Angular rate in the body frame.
13 to 15 AccelX, AccelY, AccelZ m/s² Specific force in the body frame. At rest, z is +9.81.
16, 17 Lat, Lon deg Latitude and longitude in the datum of the INS.
18 Height m Ellipsoidal height.
19 to 21 CovE, CovN, CovU m² Diagonal of the position covariance: east, north, up.
22 to 25 EnuQx, EnuQy, EnuQz, EnuQw Attitude, body frame to world frame.
26 to 28 EnuX, EnuY, EnuZ m Position in the world frame.
29 FixStatus The status of sensor_msgs/NavSatStatus: 2 for an RTK fix, -1 without a geographic reference.
30 GridRotationDeg deg UTM grid azimuth of the grid north of the map.
31 UtmZone The UTM zone.
32 to 34 TruthEnuX, TruthEnuY, TruthEnuZ m Exact position of base_footprint in the world frame.
35 to 38 TruthQx, TruthQy, TruthQz, TruthQw Exact attitude, body frame to world frame.
39 TruthSpeed m/s Exact forward speed.

Without a geographic reference, the values 0 to 6, 16 to 21, 30 and 31 are NaN.

The INS rate does not have to divide 120 Hz. An epoch is the first physics step at or after its due time. At 100 Hz the interval between two stamps is thus 1/120 s or 2/120 s, with a mean of 0.01 s.

if kind == 3:
    ins = struct.unpack("<40d", payload)
    easting_m, northing_m = ins[0], ins[1]
    latitude_deg, longitude_deg = ins[16], ins[17]
    truth_enu_m = ins[32:35]

double ins[acres_sim::kInsFields];
std::memcpy(ins, frame.payload.data(), sizeof(ins));
const double easting_m = ins[acres_sim::kUtmE];

type=3 sequence=4 payload=320 stamp=0.325000
  utm 500438.112 4479958.172 181.885
  lat lon h 40.4703010 -86.9948318 181.885
  fix 2.0 zone 16.0
  truth enu -38.709 -141.667 0.599

LiDAR (4)#

One organised cloud for each scan. A thread-pool task queues it when the scan is complete. The stamp is the end of the sweep.

Offset Type Field Contents
0 uint32 columns The number of azimuth steps. This is the height of the cloud.
4 uint32 rings The number of rings. This is the width of the cloud.
8 uint32 point step 16: the number of bytes of one point.
12 uint32 reserved 0.
16 float32[4] x columns x rings points x, y, z, intensity for each cell.
  • The cell of column \(c\) and ring \(r\) has the index \(c \cdot \text{rings} + r\).
  • x, y and z are in the sensor frame of the LiDAR in metres.
  • intensity is the reflectivity byte, 0 to 255, as a float32.
  • A beam without a return has NaN in x, y and z and the intensity 0.
  • Each beam has one return: the strongest return of the beam.

The points are in the sensor frame at the firing time of their column. The frame has no time for each point. LiDAR gives the column sequence, the ring table and the firing times.

import numpy as np

if kind == 4:
    columns, rings, step, _ = struct.unpack_from(
        "<4I", payload)
    cloud = np.frombuffer(
        payload, "<f4", columns * rings * 4, 16
    ).reshape(columns, rings, 4)
    valid = np.isfinite(cloud[:, :, 0])

type=4 sequence=2 payload=921616 stamp=0.108333
  sub-header (1800, 32, 16, 0)
  cloud (1800, 32, 4), 43624 returns,
  max range 160.2 m
  cell [0, 9]: [27.44567 -1.746628 0. 15.]

Camera (5)#

One image for each camera sample. A thread-pool task queues it after the camera model. The stamp is the simulation time of the sample.

Offset Type Field Contents
0 uint32 width The width of the image in pixels.
4 uint32 height The height of the image in pixels.
8 uint32 encoding 1: bgr8.
12 uint32 row step The number of bytes of one row: 3 x width.
16 uint8[] pixels height x row step bytes. The top row is first.

The image is the output of the camera model: lens distortion, rolling shutter and sensor noise. The Hello frame gives the intrinsics. Camera gives the model.

if kind == 5:
    width, height, encoding, row_step = (
        struct.unpack_from("<4I", payload))
    image = np.frombuffer(
        payload, np.uint8, height * row_step, 16
    ).reshape(height, width, 3)  # blue, green, red

type=5 sequence=1 payload=1376272 stamp=0.108333
  sub-header (896, 512, 1, 2688)

Barrier (6)#

The marker at the end of a lockstep step. The game sends it only in lockstep mode. When a step request of the Simulator Control Channel is complete, the game queues one barrier on each sensor stream.

Offset Type Field Contents
0 uint64 step The physics step of the world that the request reached.

All clock, INS, LiDAR and camera frames of the requested steps are in the stream before the barrier. The game waits until the sensor recorder has no scan and no image in work, and then sends the barrier. The stamp of the header is the latest physics time of the agent.

Lockstep Stepping gives the procedure.

if kind == 6:
    step = struct.unpack("<Q", payload)[0]

Three step requests of 12 steps each, in a session with the Polaris:

step=12  frames: clock 12, ins 10, lidar 1, camera 1
  barrier sequence=25 stamp=0.100000 payload=8 step=12
step=24  frames: clock 12, ins 10, lidar 1, camera 1
  barrier sequence=50 stamp=0.200000 payload=8 step=24
step=36  frames: clock 12, ins 10, lidar 1, camera 1
  barrier sequence=75 stamp=0.300000 payload=8 step=36

Before the first request, the stream contains only the Hello frame.

Behaviour#

Rates and Sizes#

Frame Rate Payload
Hello One time for each connection Approximately 1.2 kB
Clock 120 Hz 8 bytes
INS ins.hz, 100 Hz 320 bytes
LiDAR lidar.hz, 10 Hz 16 + 16 x columns x rings bytes
Camera camera.hz, 10 Hz 16 + 3 x width x height bytes
Barrier One for each step request 8 bytes

The Polaris sends 921 616 bytes for each cloud and 1 376 272 bytes for each image: approximately 23 MB in each second.

The frames are not in the sequence of their stamps. The physics thread queues the clock frame and the INS frame in their step. A cloud and an image come later, when the game completes them.

A run of 30 s with the Polaris on an RTX 5060 Ti. The transport time is the receive time minus the publish time.

clock   frames=3599 rate=120.00 Hz
        transport p50=0.05 ms max=7.77 ms
ins     frames=3000 rate=100.02 Hz
        transport p50=0.02 ms max=7.72 ms
lidar   frames= 300 rate=  9.97 Hz
        transport p50=0.41 ms max=1.33 ms
camera  frames= 302 rate= 10.00 Hz
        transport p50=0.46 ms max=3.57 ms

Flow Control and Drops#

Each stream has one sender thread and one queue. A slow client does not stop the physics, the game thread or the renderer.

Condition Result
No client The game queues no frames.
Queue plus the new frame above 96 MiB The game drops a LiDAR frame or a camera frame.
Queue plus the new frame above 192 MiB The game drops all frame types.
The send operation fails The game closes the connection and clears the queue.
The client closes the connection The game clears the queue and waits for the next client.
A new client connects The game sends a new Hello frame. The sequence and the counters of the log continue.

A dropped frame gets no sequence number, thus a client cannot find a drop from the sequence. The game counts the drops for each frame type and writes them to its log.

The statistics of the log show the frames, the megabytes and the drops of each type:

ACRES_SENSOR_STREAM_STATS agent=polaris queued_mb=0.0
  hello=1/0.0MB/drop0 clock=3503/0.1MB/drop0
  ins=2920/1.0MB/drop0 lidar=293/257.5MB/drop0
  camera=294/385.9MB/drop0

Log Lines#

Line Time
ACRES_SENSOR_STREAM_LISTENING agent= port= The socket is open.
ACRES_SENSOR_STREAM_LISTEN_FAILED agent= port= The game cannot open the port. The session continues without the stream.
ACRES_SENSOR_STREAM_CONNECTED agent= port= send_buffer= A client connected and the Hello frame went out.
ACRES_SENSOR_STREAM_STATS agent= ... Each 30 s while the stream has a client.
ACRES_SENSOR_STREAM_DISCONNECTED agent= port= reason= ... The connection closed. The reason is peer closed, send failed or hello failed.
ACRES_SENSOR_STREAM_STOP agent= ... The session ended.

The statistics text does not show the barrier frames.

ACRES_SENSOR_STREAM_DISCONNECTED agent=polaris
  port=5601 reason=peer closed queued_mb=0.0
  hello=1/0.0MB/drop0 clock=3599/0.1MB/drop0
  ins=3000/1.0MB/drop0 lidar=301/264.6MB/drop0
  camera=302/396.4MB/drop0

Python Reader#

Python Reader#

The example connects to the stream, reads the header of each frame and decodes the first LiDAR cloud. It needs Python 3 and NumPy.

  1. Start the game with a sensor stream.

    Packaged/Linux/Acres.sh -VehicleDemo -Vehicle=polaris -SensorStream=5601

  2. Run the example when the log shows ACRES_SENSOR_STREAM_LISTENING.

Expected Result

The example prints the agent of the Hello frame and the first cloud. The number of returns changes with the place of the vehicle.

For ROS 2 clients, use the ROS 2 bridge. The C++ functions of ROS/acres_sim/include/acres_sim/stream.hpp read the same frames without ROS 2.

import json
import socket
import struct

import numpy as np

# magic, type, flags, sequence, payload bytes, stamp, publish time
HEADER = struct.Struct("<4sHHIIdd")


def read_exact(sock, count):
    data = bytearray()
    while len(data) < count:
        chunk = sock.recv(count - len(data))
        if not chunk:
            raise ConnectionError("the stream closed")
        data += chunk
    return bytes(data)


def read_frame(sock):
    (magic, kind, flags, sequence, size,
     stamp_s, publish_s) = HEADER.unpack(read_exact(sock, 32))
    if magic != b"TSS1":
        raise ValueError("bad magic")
    return kind, sequence, stamp_s, read_exact(sock, size)


def lidar_cloud(payload):
    columns, rings, _, _ = struct.unpack_from("<4I", payload)
    points = np.frombuffer(payload, "<f4", columns * rings * 4, 16)
    return points.reshape(columns, rings, 4)


sock = socket.create_connection(("127.0.0.1", 5601))
while True:
    kind, sequence, stamp_s, payload = read_frame(sock)
    if kind == 1:
        hello = json.loads(payload)
        print("hello", hello["agent"], hello["lidar"]["columns"],
              "x", hello["lidar"]["rings"])
    elif kind == 4:
        cloud = lidar_cloud(payload)
        returns = np.isfinite(cloud[:, :, 0]).sum()
        print(f"lidar seq={sequence} stamp={stamp_s:.3f} s "
              f"shape={cloud.shape} returns={returns}")
        break

hello polaris 1800 x 32
lidar seq=79 stamp=0.008 s shape=(1800, 32, 4) returns=44088

Differences in ACRES Core#

The Core server core_sim of the package acres_core_sim serves the same protocol. The ROS 2 bridge reads it without a change.

Item Game Core Server
Option -SensorStream=<port>, no default. --sensor-ports <port>, one port for each agent. The launch file uses 5601.
Hello frame Sequence from the counter, stamp 0. Sequence 0, stamp at the world time. camera.enabled is false.
Camera frames Yes. No. ACRES Core has no camera.
LiDAR frames The GPU path or the CPU path of the game. The ray-cast LiDAR of ACRES Core.
Barrier After the sensors are idle. After the LiDAR scans of the steps.