AcresCameraModel.h#

Acres/Source/Acres/AcresCameraModel.h Generated

Photometric and geometric model of a machine-vision RGB camera, applied on the CPU to the scene-capture render.

Unreal renders an ideal pinhole image (a USceneCaptureComponent2D writing into a render target, set up in AcresSensors.cpp) that is a little wider than the output. FAcresCameraModel::Process then builds the delivered image pixel by pixel. Chain per output pixel:

distorted pixel -> undistorted ray (Brown-Conrady inverse, precomputed) -> lateral chromatic aberration (radial scale of the ray per colour) -> rolling-shutter row rotation/translation -> pinhole projection into the wider render -> bilinear sample of the CFA colour in linear light (sRGB decoded) -> vignetting -> electrons (PRNU, shot + dark + read noise) -> ADC (black level, bits) -> ISP (black subtract, Malvar-He-Cutler or bilinear demosaic, sRGB encode).

The render is display-referred (tone-mapped), so "linear" means the inverse sRGB curve of the 8-bit image, not scene radiance. Settings come from the sensors.json blocks camera.lens, camera.rolling_shutter and camera.photometric. Pure C++: no Unreal objects, and Process runs on Unreal's thread pool.

FAcresCameraModelConfig#

struct FAcresCameraModelConfig

Camera model settings (sensors.json camera.lens / camera.rolling_shutter / camera.photometric).

Normalized coordinates are x = (u - cx) / fx, y = (v - cy) / fy with the optical axes x right, y down, z forward.

Name	Type	Unit	Default	Description
`Enabled`	`bool`		`true`	Master switch. False (or -SensorCameraIdeal) delivers the ideal pinhole render with no processing. Derived in `ApplyCommandLine` as Distortion \|\| RollingShutter \|\| Sensor.
`Distortion`	`bool`		`true`	Apply the lens model: distortion, principal point offset, vignetting and chromatic aberration (camera.lens.enabled).
`K1, K2, K3, P1, P2`	`double`			OpenCV plumb_bob / Brown-Conrady coefficients on normalized undistorted coordinates: radial K1, K2, K3 and tangential P1, P2 (dimensionless).
`PrincipalOffsetXPx, PrincipalOffsetYPx`	`double`			Principal point minus image centre, pixels (x right, y down). Applied only with Distortion on.
`FxPx, FyPx, CxPx, CyPx`	`double`			Explicit output intrinsics, pixels, as a recorded camera_info gives them (OpenCV K: fx, fy, cx, cy; camera.lens.intrinsics). FxPx <= 0 (the default) derives fx = fy from the horizontal FOV and the principal point from the image centre plus the offset above.
`RadialMinSlope`	`double`		`0`	Radial extension beyond the calibrated field (camera.lens.radial_min_slope): when above 0, the polynomial r_d(r) = r (1 + k1 r^2 + k2 r^4 + k3 r^6) is used out to the radius where its slope dr_d/dr falls to this value, and continues as a straight line with that slope beyond it. A strong barrel calibration (the Purdue Reolink's k1 = -0.55) turns back before the image corners; the real lens keeps mapping wider rays there. 0 = off.
`RadialLimitR`	`double`		`0`	Derived by FAcresCameraModel::Initialize: the undistorted radius where the extension starts (0 = none).
`Vignetting`	`FString`		`TEXT("poly")`	Vignetting model: "poly" (V = 1 + a2 r^2 + a4 r^4), "cos4" (cos^4 of the ray angle) or "none".
`VignetteA2, VignetteA4`	`double`			VignetteA2, VignetteA4: "poly" coefficients, with r the distorted normalized radius.
`ChromaticAberrationPx`	`double`		`.7`	Lateral chromatic aberration at the image corner, pixels: red is pushed outward, blue inward (0-5).
`RenderScale`	`double`		`1.`	Render focal length = output fx x scale (0.5-2).
`RenderMargin`	`double`		`.02`	Extra fractional field of view around what the distorted image needs, so rolling-shutter shifts stay inside the render (0-0.2).
`RollingShutter`	`bool`		`true`	Simulate the row-by-row readout of a rolling-shutter sensor.
`ReadoutS`	`double`	s	`.025`	Time from the first to the last row, s (0-0.1).
`AssumedDepthM`	`double`	m	`15.`	Single scene depth used to turn camera translation into image motion, m (0 = rotation only).
`Sensor`	`bool`		`true`	Simulate the Bayer sensor, ADC and demosaic (camera.photometric.enabled).
`Noise`	`bool`		`true`	Add PRNU, shot, dark and read noise (off with -SensorCameraNoNoise or -SensorNoNoise).
`PixelPitchUm`	`double`		`2.8`	Pixel pitch, micrometres (reported only).
`FullWellE`	`double`		`10000`	Full-well capacity, electrons.
`ReadNoiseE`	`double`		`2.5`	Read noise, electrons RMS.
`DarkCurrentEps`	`double`		`50`	Dark current, electrons per second.
`PrnuStd`	`double`		`.01`	Pixel gain non-uniformity, fraction (sigma).
`BaseIso`	`double`		`100`	ISO at unity analog gain.
`AdcBits`	`int`		`12`	ADC resolution in bits (8-16).
`BlackLevelDn`	`double`		`256`	ADC offset added to every pixel, DN (must stay below a quarter of the range).
`Cfa`	`FString`		`TEXT("RGGB")`	Bayer layout of the top-left 2x2 block, "RGGB", "BGGR", "GRBG" or "GBRG".
`Demosaic`	`FString`		`TEXT("malvar")`	"malvar" (Malvar, He and Cutler 2004 5x5 filters) or "bilinear".

FAcresCameraModelConfig::Parse#

Reads the "lens", "rolling_shutter" and "photometric" sub-objects of the sensors.json "camera" block.

Argument	Description
`CameraBlock`	The "camera" JSON object. Missing keys keep their defaults.

void Parse(const TSharedPtr<FJsonObject>& CameraBlock);

FAcresCameraModelConfig::ApplyCommandLine#

Applies -SensorCameraIdeal, -SensorCameraNoRollingShutter, -SensorCameraNoNoise, -SensorCameraNoDistortion, -SensorCameraReadout=<s> and the global -SensorNoNoise (same as -SensorCameraNoNoise), then recomputes Enabled.

Argument	Description
`CommandLine`	The command line to read (null: the process's; "" gives the file settings with the derived values updated, as the sensor profile's reference needs).

void ApplyCommandLine(const TCHAR* CommandLine = nullptr);

FAcresCameraModelConfig::Validate#

Checks every parameter against its supported range; stops the game with a checkf message if not.

void Validate() const;

FAcresCameraModelConfig::ToJson#

Serialises the settings with the same keys as sensors.json.

Returns: A JSON object with "enabled", "lens", "rolling_shutter" and "photometric".

TSharedPtr<FJsonObject> ToJson() const;

FAcresCameraOverrides#

struct FAcresCameraOverrides

Switches used by the self-test to turn individual effects off while keeping the rest of the pipeline.

Name	Type	Unit	Default	Description
`Vignetting, Noise, Aberration`	`bool`			Vignetting, Noise, Aberration: false disables that effect for one `Process` call.

FAcresCameraModel#

class FAcresCameraModel

A calibrated virtual camera: intrinsics, precomputed undistortion table, PRNU map and the per-frame image pipeline.

The recorder builds one in Start with Initialize, sizes the scene-capture render target from RenderWidth x RenderHeight and RenderHfovDeg, and for every captured frame calls Process on a thread-pool task. After Initialize the object is read-only, so one instance can be shared by several threads.

Example

FAcresCameraModel Model;
Model.Initialize(Config.CameraModel, 1920, 1080, 90, 1. / 333, 100, 42);
TArray<FColor> Image;
Model.Process(RenderPixels.GetData(), Motion, FrameIndex, Image); // 1920 x 1080 sensor image

Name	Type	Unit	Default	Description
`FOverrides`	`using`		`FAcresCameraOverrides`	Short alias for the self-test switches.
`RenderWidth, RenderHeight`	`int`			Render target the scene capture must produce: wider FOV so distortion and rolling shutter never sample outside it. RenderWidth, RenderHeight in pixels.
`RenderHfovDeg`	`double`	°	`90`	Render horizontal FOV, degrees.
`RenderFx`	`double`		`0`	Render focal length, pixels. RenderCx,
`RenderCx`	`double`		`0`	See above.
`RenderCy`	`double`		`0`	Render principal point, pixels (the exact centre).
`Width, Height`	`int`			Output (distorted) image size in pixels.
`Fx, Fy, Cx, Cy, EffectiveHfovDeg, EffectiveVfovDeg`	`double`			Output intrinsics, OpenCV pixel-centre convention (pixels): Fx, Fy focal lengths, Cx, Cy principal point. EffectiveHfovDeg / EffectiveVfovDeg: true field of view of the distorted image through the principal point, degrees.
`PrnuSequenceBase`	`static constexpr uint64`		`0x10000`	PRNU stream sequence offset (65536), kept clear of the row-numbered frame-noise sequences.

FAcresCameraModel::FromUnrealBody#

Converts motion in the Unreal body frame (x forward, y right, z up; left-handed) to the optical frame.

The axis map is a reflection (determinant -1), so the linear velocity maps directly but the angular velocity, a pseudovector, changes sign.

Argument	Description
`AngularBody`	Angular velocity in the Unreal body frame, rad/s.
`VelocityBodyMps`	Camera point velocity in the Unreal body frame, m/s.

Returns: The same motion in optical axes.

static FMotion FromUnrealBody(const FVector& AngularBody, const FVector& VelocityBodyMps);

FAcresCameraModel::Initialize#

Computes intrinsics, render size, the undistortion table, sRGB lookup tables and the PRNU map.

fx = fy = Width / (2 tan(HfovDeg / 2)); cx = (Width - 1) / 2 + principal offset (OpenCV pixel centres). Each output pixel's undistorted ray is found by Newton iteration on the forward distortion model (at most 30 iterations, stop at 1e-12). The render is the smallest even-sized pinhole image, at focal fx x RenderScale, that contains every ray grown by the CA scale and RenderMargin.

Argument	Description
`Config`	Model settings; validated again here.
`Width`	Output image width, pixels.
`Height`	Output image height, pixels.
`HfovDeg`	Pinhole-equivalent horizontal FOV that defines fx, degrees.
`ShutterS`	Exposure time, s (dark-current integration).
`Iso`	ISO; analog gain = Iso / BaseIso.
`Seed`	Master seed for the PRNU map and the per-frame noise streams.

Note

Stops the game (checkf) if the distortion cannot be inverted to 1e-3 px, folds over inside the field of view, or needs a render larger than 8192 px.

void Initialize(const FAcresCameraModelConfig& Config, int Width, int Height, double HfovDeg, double ShutterS, double Iso, int Seed);

FAcresCameraModel::IsActive#

True when the model processes frames (false = ideal pinhole passthrough).

bool IsActive() const ;

FAcresCameraModel::Process#

Turns one render into one sensor image.

Argument	Description
`In`	RenderWidth x RenderHeight sRGB pixels (row-major, as read back from the render target).
`Motion`	Camera motion in optical axes, for the rolling-shutter warp.
`FrameIndex`	Frame counter; selects the noise stream so each frame gets fresh, reproducible noise.
`Out`	Receives Width x Height sRGB pixels, alpha 255.
`RawOut`	Optional; receives the Width x Height Bayer mosaic in ADC counts (DN).
`Ov`	Self-test switches.

Note

Thread-safe (const, no shared mutable state); requires IsActive(). Rows run in parallel with ParallelFor (Unreal's parallel loop over the task graph).

void Process(const FColor* In, const FMotion& Motion, uint64 FrameIndex, TArray<FColor>& Out, TArray<uint16>* RawOut = nullptr, const FOverrides& Ov = FOverrides()) const;

FAcresCameraModel::MapToRender#

Maps an output pixel to its sampling position in the render, for the green channel (no CA), exactly as Process does. Used by the self-test for verification.

Argument	Description
`U`	Output column, pixels.
`V`	Output row, pixels (sets the rolling-shutter time offset).
`Motion`	Camera motion in optical axes.

Returns: Render pixel coordinates (u, v).

FVector2D MapToRender(int U, int V, const FMotion& Motion) const;

FAcresCameraModel::UndistortedRay#

Undistorted normalized ray (x/z, y/z) of output pixel (U, V), from the precomputed table.

Argument	Description
`U`	Output column, pixels.
`V`	Output row, pixels.

Returns: Normalized undistorted coordinates (dimensionless).

FVector2D UndistortedRay(int U, int V) const ;

FAcresCameraModel::NoiseInitState#

PCG32 init state of the per-frame noise streams: Seed + 3027 + (FrameIndex + 1) x 1000003. The stream sequence is the row number.

Argument	Description
`Seed`	Master seed.
`FrameIndex`	Frame counter.

Returns: The init state for FAcresPcg32::Seed.

static uint64 NoiseInitState(int Seed, uint64 FrameIndex) ;

FAcresCameraModel::PrnuInitState#

PCG32 init state of the fixed PRNU map: Seed + 3027 (sequence PrnuSequenceBase + row).

static uint64 PrnuInitState(int Seed) ;

FAcresCameraModel::ManifestJson#

Describes the model for episode.json "camera_model": settings, render geometry, gain, electrons per DN and plain-text notes on the pipeline, signal domain, rolling-shutter approximation and noise seeding.

TSharedPtr<FJsonObject> ManifestJson() const;

FAcresCameraModel::RunSelfTest#

Runs synthetic checks and writes them to Dir.

Writes camera_selftest.json (intrinsics, flat-field levels with raw SHA-1 and timings, a 9 x 7 grid of geometry samples with and without a test motion), 128 x 128 raw DN crops raw_srgbNNN.u16 of six flat grey levels, and checker_motion.png. Triggered by -SensorCameraSelfTest; useful for checking the lens and sensor model after a change.

Argument	Description
`Dir`	Output folder (created if missing).

Returns: Empty string on success, otherwise an error message.

FString RunSelfTest(const FString& Dir) const;

FAcresCameraModel::Config#

The validated settings in use.

const FAcresCameraModelConfig& Config() const ;

FAcresCameraModel::ElectronsPerDn#

Electrons per ADC count at the current gain: full well at gain / (2^bits - 1 - black level).

double ElectronsPerDn() const ;

FMotion#

struct FMotion

Motion of the camera during readout, in the right-handed optical frame (x right, y down, z forward).

Name	Type	Unit	Default	Description
`AngularRadps`	`FVector`	rad/s	`FVector::ZeroVector`	Camera angular velocity, rad/s.
`VelocityMps`	`FVector`	m/s	`FVector::ZeroVector`	Camera linear velocity, m/s.