Skip to content

Add a Sensor#

This page gives the procedure to add a sensor to the game, from the model to the ROS 2 topic. It also gives the rules for threads and noise that keep a recording deterministic.

Before You Start#

The examples on this page add a barometric altimeter with the name baro. The game does not contain this sensor.

The Path of a Sample#

A sample goes through these stages. Each stage has one section on this page.

Stage File Item to Add
Model Acres/Source/Acres/Acres<Name>Model.h The mathematics, without Unreal types.
Configuration AcresSensors.h, AcresSensors.cpp, sensors.json The keys, the options and the range checks.
Menu AcresSession.cpp The rows of the Sensors tab.
Schedule AcresSensors.cpp The stream, the generator and the call in PhysicsSample or GameTick.
Files AcresSensors.cpp The row of the .jsonl file, the binary files and the keys of episode.json.
Sensor stream AcresSensorStream.h, AcresSensors.cpp The frame type, the payload and the block of the hello message.
ROS 2 ROS/acres_sim The frame constant, the conversion function, the publisher and the transform.
ACRES Core ROS/acres_core_sim The same model behind the same sockets, when ACRES Core needs the sensor.
Tests and documentation Tools, ROS/acres_sim/test, Documentation Unit tests, a recording check and the model page.

Rules#

Obey these rules in all stages.

Topic Rule
Clock The physics step of 1/120 s is the clock. A sample time is a count of steps divided by 120.
Physics thread PhysicsSample runs on the physics thread. Read only FAcresSensorPhysicsSample there. Do not read the Unreal world.
Game thread Scene queries and renders run in GameTick. Record the pose and the time of the sample on the physics thread first.
Thread pool Encode large files on the thread pool. Count the tasks in EncodesInFlight.
Queues Give each queue a limit. Count each dropped sample and report the count in episode.json.
Noise Give the sensor its own FAcresPcg32 stream. Use the same number of values for each sample.
Noise off -SensorNoNoise sets the standard deviations to zero. The sensor must continue to use its values.
Units and frames Use SI units. Output world quantities in ENU and sensor quantities in body FLU.
Mounts Give a mount as FLU metres from the chassis origin and as roll, pitch and yaw in degrees.
Bad values Stop the game with checkf when a key is out of its range.

Write the Model#

Put the mathematics in a file without Unreal types when ACRES Core or a test program must use it. The files AcresGeoUtm.h and AcresLidarPhysics.h are examples: plain C++ in the headers, no CoreMinimal.h. Give a model with a .cpp file a name that ends in Model.cpp, as the other models have.

  1. Create Acres/Source/Acres/AcresBaroModel.h.
  2. Write a /// @file comment that describes the model and gives its publications.
  3. Write the functions with SI units in the argument names.

  4. Write a /// comment with Args: and Returns: for each function.

    #pragma once
#include <cmath>

/// @file
/// Barometric altimeter (example): pressure of the ISA troposphere at a height, and the altitude that a sensor
/// computes from a pressure. No Unreal types.
namespace AcresBaro
{
/// Static pressure of the standard atmosphere.
///
/// Args:
///     AltitudeM: Height above mean sea level, m.
/// Returns: The pressure, Pa.
inline double PressurePa(double AltitudeM)
{
    return 101325. * std::pow(1 - 2.25577e-5 * AltitudeM, 5.25588);
}
} // namespace AcresBaro

  5. Add the .cpp file to ACRES_MODEL_SOURCES in Core/CMakeLists.txt when ACRES Core uses the model.

Note

The receiver model AcresGnssModel.cpp and the camera and LiDAR models use Unreal types. ACRES Core does not compile them.

Add the Configuration Keys#

The structure FAcresSensorConfig holds all settings. The function LoadSensorConfig reads them.

  1. Add the fields to FAcresSensorConfig in AcresSensors.h: a switch, a rate, a mount and the noise.

    /// Barometer (example): on or off, rate (Hz), mount (FLU m), white noise (Pa).
bool Baro = false;
double BaroHz = 20, BaroStdPa = 3;
FVector BaroFlu = FVector::ZeroVector;

  2. Read the block in LoadSensorConfig. Use TryGetObjectField so that a file without the block continues to load.

    if (const TSharedPtr<FJsonObject>* Baro = nullptr; J->TryGetObjectField(TEXT("baro"), Baro) && Baro)
{
    (*Baro)->TryGetBoolField(TEXT("enabled"), C.Baro);
    Number(*Baro, TEXT("hz"), C.BaroHz);
    Number(*Baro, TEXT("std_pa"), C.BaroStdPa);
    Vec3(*Baro, TEXT("position_flu_m"), C.BaroFlu);
}

  3. Add the mount to the loop that moves the mounts of the frame base_footprint to the chassis frame.

  4. Add the options below the other FParse calls: -SensorBaroHz= and -SensorNoBaro.
  5. Set the noise to zero in the block of -SensorNoNoise.
  6. Add the rate to the check of 1 Hz to 120 Hz. Add the mount to the check of 10 m.
  7. Add the keys to FAcresSensorConfig::ToJson. The files sensors-config.json and episode.json then contain them.
  8. Add the block to Acres/Content/Simulation/sensors.json with the defaults and a provenance text.
  9. Add the values of the Polaris to sensors_polaris.json when they are different.

A session folder contains a copy of sensors.json. Thus an old session has no block for the new sensor. The block ins is the example of a block that is not mandatory. Add the mount and the settings to DataSignature only when they change the camera or LiDAR data.

Add the Menu Rows#

The Sensors tab of the menu is a table in AcresSession.cpp. One row is one setting.

  1. Add one row for the switch and one row for each setting that a user changes frequently.

    {TEXT("Sensors"), TEXT("Barometer"), TEXT("sen:baro.enabled"), TEXT("Barometer"), TEXT(""), K::Toggle, 0, 1, 1, 1,
 nullptr, nullptr, nullptr},
{TEXT("Sensors"), TEXT("Barometer"), TEXT("sen:baro.hz"), TEXT("Rate"), TEXT("Hz"), K::Integer, 1, 120, 1, 1,
 nullptr, TEXT("sen:baro.enabled"), nullptr},

  2. Add the switch to the test AnySensor in FAcresSession::Prepare.

The identifier of a row is sen: and the JSON path of the key. The menu writes the value into the sensors.json of the session folder. The page Menu Settings comes from this table. The build makes it again.

Schedule the Sensor in the Recorder#

The class FAcresSensorRecorder owns the streams, the generators and the schedule.

  1. Add the stream: increase StreamCount, add the name to StreamNames and add a value to EStream.
  2. Add a generator member, for example RngBaro.

  3. Seed the generator in Start with the next free stream number.

    RngBaro.Seed(uint64(Config.Seed) + 6 * 1009, 7);

  4. Add the state of the sensor to the resets in Start and to the block for S.ResetEpoch != LastEpoch.

  5. Add the fields that the sensor reads to FAcresSensorPhysicsSample.
  6. Fill the new fields in AAcresVehiclePawn::AsyncPhysicsTickActor, before the call of PhysicsSample.

  7. Call the model in PhysicsSample when the sensor is due.

    if (Config.Baro && Due(BaroStream, Config.BaroHz, TimeS))
{
    const FVector MountM = P + Q.RotateVector(BodyFromFlu(Config.BaroFlu));
    const double AltitudeM = OriginFt.Z * AcresGeo::UsSurveyFootM + MountM.Z;
    const double PressurePa = AcresBaro::PressurePa(AltitudeM) + Config.BaroStdPa * RngBaro.Gaussian();
    Emit(BaroStream, FString::Printf(TEXT("{\"physics_step\":%llu,\"pressure_pa\":%.3f"), S.Step, PressurePa),
         TimeS);
}

The stream numbers 0 to 5 are in use. The table gives them.

Stream Seed Sequence
GNSS errors seed 1
IMU seed + 1009 2
LiDAR seed + 2018 3
GNSS constellation seed + 3027 4
LiDAR weather seed + 4036 5
INS seed + 5045 6

Due gives the first sample at \(t = 0\) and then one sample each \(1/f\) seconds. A sensor of the game thread uses a different pattern. The LiDAR is the example.

  1. On the physics thread, put a request with the pose and the time into a queue. Use the queue mode Spsc.
  2. In GameTick, take the request and do the scene query or the render.
  3. Encode the result on the thread pool.
  4. Return false from IsIdle while a request or a task is in work.

In lockstep, the simulator control channel waits for IsIdle before it answers a step request. A sensor that does not report its work in IsIdle thus sends its data after the barrier.

Write the Rows and the Files#

Start opens one file <stream>.jsonl for each name in StreamNames. One thread writes all rows.

Function Use
Emit(Stream, Row, SampleS) A row with the modelled latency. Give the row without the closing brace. The function adds the three time keys.
Write(Stream, Line) A complete row without latency, for example a truth row.
AcresSensorIo::WriteBytes(Path, Data, Bytes) A binary file, for example a point cloud. Create its folder in Start.
  1. Write each row as one JSON object with the unit in each key, for example pressure_pa.
  2. Add physics_step to each row. A reader then joins the row with truth.jsonl.
  3. Add the count of the stream to the log line ACRES_SENSOR_STOP in Finish and to StatusLabel.
  4. Add the mount to extrinsics_flu_m in WriteManifest.
  5. Add a block with the description of the model to episode.json in WriteManifest.
  6. Add one sentence to the array limitations in WriteManifest.

The object counts of episode.json gets the new stream automatically. The loop uses StreamNames. The Session Log page gives the files of an episode.

Add the Frame to the Sensor Stream#

The sensor stream sends the data to the ROS 2 bridge while the simulation runs.

  1. Add a value to FAcresSensorStream::EFrame in AcresSensorStream.h. The next free value is 7.
  2. Describe the payload in the file comment of AcresSensorStream.h: each field with its type and unit.

  3. Send the frame from the recorder at the same place as the row.

    if (LiveStream)
{
    const double Payload[] = {PressurePa, AltitudeM};
    LiveStream->Publish(FAcresSensorStream::EFrame::Baro, S.PhysicsTimeS, nullptr, 0, Payload, sizeof(Payload));
}

  4. Add a block to StreamHello with the switch, the rate, the frame name and the mount from base_footprint.

  5. For a large frame, add the type to the test bHeavy in FAcresSensorStream::Publish.

The stamp of a frame is the physics time of the vehicle. Publish adds the offset to the world time. The counters of the stream have 8 entries, for the frame types 0 to 7. A frame type above 7 needs larger arrays. The stream drops large frames when more than 96 MB wait in its queue. An old ROS 2 bridge counts a frame of a type that it does not know and does not fail.

CAUTION

Do not change the sequence of the values in an existing payload. A bridge of a different version then reads incorrect values.

Convert the Frame in the ROS 2 Bridge#

The package acres_sim reads the frames and publishes the messages.

  1. Add the frame constant to FrameType in ROS/acres_sim/include/acres_sim/stream.hpp, with a comment for the payload.

  2. Declare a conversion function in conversions.hpp and write it in src/conversions.cpp. The function must not use a node.

    /// One barometer frame as a FluidPressure message (example).
sensor_msgs::msg::FluidPressure baro_to_message(const double* values, double stamp_s, const std::string& frame_id);

  3. Create the publisher in VehicleBridge::setup in src/vehicle_bridge.cpp. Use vehicle_qos().

  4. Add the frame type to VehicleBridge::on_frame and publish the message there.
  5. Send a large frame through the queue of the heavy frames, as the LiDAR and the camera do.
  6. Add the static transform of the mount to mount_transforms. Read the mount from the hello message.
  7. Use a standard message type when one exists. For a new type, refer to Add a ROS 2 Interface.
  8. Add the frame to ROS/acres_sim/test/fake_game.py. The integration test then gets the new topic.

In lockstep, the bridge must publish a frame that goes through a queue before the barrier of its step. The queue of the heavy frames carries the barrier for this reason. The Topics page and the Sensor Stream page must list the new topic and frame.

Add the Sensor to ACRES Core#

Do this stage only when a client of ACRES Core needs the sensor. FInsModel in ROS/acres_core_sim is the example.

  1. Use the model file of the first stage. Do not copy the equations.
  2. Read the same keys from sensors.json and from the overlay, as FInsConfig::Load does.
  3. Use a generator with the same algorithm and the same seed, as FPcg32 does.
  4. Call the model in FCoreSimServer::EmitAfterStep and send the frame with EncodeFrame.
  5. Add the block of the sensor to the hello message of the server.
  6. Compute the sample also when the server has no client. The noise then does not depend on the client.

Write the Tests#

Test Location Pattern
Model mathematics Tools/<Name>Model/ with a build.sh Tools/LidarModel, Tools/PolarisModel/polaris_tests.cpp
Conversion to messages ROS/acres_sim/test/test_conversions.cpp Conversions.InsToPolarisMessages
Frame layout ROS/acres_sim/test/test_stream.cpp Stream.ReadsEveryFrameTypeFromASocket
Bridge with a simulated game ROS/acres_sim/test/test_integration.py, fake_game.py test_maxxum_guidance_can_and_mounts
ACRES Core model ROS/acres_core_sim/test/test_core_sim_unit.cpp Ins.NoiseFreeEpochIsTheTruth, Ins.NoiseLevels
  1. Test the model against an independent implementation or against published values.
  2. Test that the output without noise is the truth.
  3. Test that the standard deviation of the output agrees with the configuration.

  4. Record two episodes with the same seed and compare the rows. They must be the same.

    Packaged/Linux/Acres.sh -VehicleDemo -VehicleTest=turn -VehicleTerrainTest -SensorRecord -SensorOutput=/tmp/baro-a
Packaged/Linux/Acres.sh -VehicleDemo -VehicleTest=turn -VehicleTerrainTest -SensorRecord -SensorOutput=/tmp/baro-b
cmp /tmp/baro-a/episode-001/gnss.jsonl /tmp/baro-b/episode-001/gnss.jsonl

    Expected Result

    cmp prints nothing. The two files of a stream of the physics thread are the same.

  5. Run the unit tests of the ROS 2 packages.

    source ROS/Env/setup_env.sh
cd ROS
colcon build
colcon test --packages-select acres_sim acres_core_sim
colcon test-result --verbose
cd ..

The option -VehicleTest=turn drives a fixed program of 17 s and then stops the game. The Tests and Regression Checklist page lists all tests.

Write the Documentation#

  1. Write a model page in Documentation/Models with the sections that Write Documentation gives.
  2. Add the page to the navigation in mkdocs.yml and to the table on the Conventions page.
  3. Give each key in a parameter table with its type, unit, default and description.
  4. Add the sensor rig values to the Sensors and Rigs tutorial.
  5. Add each new term to the Glossary.
  6. Build the site and do the checks of Write Documentation.

The build generates three pages from the code: the option index, the menu settings and the header pages. Thus the FParse calls, the menu rows and the /// comments of the new sensor appear on those pages after a build.

Checklist#

  • The game stops with a clear message for each key that is out of its range.
  • A file without the new block loads.
  • The episode has the new .jsonl file, the count in episode.json and zero dropped rows.
  • Two recordings with the same seed give the same rows.
  • -SensorNoNoise gives the truth and does not change the other streams.
  • The ROS 2 topic has the correct frame name, stamp and transform.
  • A step request in lockstep returns after the data of the step.
  • The unit tests, the language check and the strict build of the site pass.