Libretro Peripheral Capture Interfaces: Sensors, Cameras, and Microphones

The libretro API covers more than buttons, video, and the audio callback. It also defines optional interfaces for environmental sensors, camera frames, and microphone capture. These interfaces let a core reproduce a game peripheral without hard-coding a platform SDK or opening host devices directly. They are not a single interchangeable input path: each has its own negotiation, data lifetime, timing, and failure contract.

This distinction matters for both accuracy and user trust. A game that polls tilt expects sensor samples in emulated input time. A camera accessory may need a frame only after a game-specific capture command. A microphone-controlled title may read samples at a requested rate and explicitly enable capture. Requesting every host capability at startup, then silently substituting zeros or stale data, hides incompatibility and may access private devices without a clear reason.

Negotiate optional support as a capability

The core asks the frontend for an interface through the environment callback. The current libretro API header marks the sensor, camera, and microphone requests as experimental environment commands. A request can fail because the frontend does not implement the command, because the user disabled the feature, or because no supported device is available. The camera contract specifically says that a successful interface query does not prove an actual camera is present. Treat success as permission to use the returned contract, not as proof of hardware availability.

The core should make these requests at the API-defined lifecycle point, inspect all returned fields, and retain a clean unsupported path. The camera request belongs in retro_load_game(). The microphone interface has a version field and a separate open operation. Sensor access provides a frontend-owned interface with operations to enable a sensor and read its current value. Each interface is optional and must not become a prerequisite for booting content that can run without it.

Do not cast one environment structure to another or infer support from the frontend name. Use the exact structure and command declared by the header version used to compile the core. A failed request should leave a deterministic fallback: for example, a neutral tilt value, an unavailable camera state, or no microphone samples. Record that fallback in diagnostics when it changes gameplay, but do not flood the log every frame.

Sensors are values queried through the input path

The sensor interface separates control from sampling. Its enable operation identifies a port, a sensor action, and a rate; the current sensor value is then retrieved by a sensor identifier. The header describes examples such as accelerometers and gyroscopes, but the set of supported sensors and the physical interpretation depend on the frontend and host device. A core should not assume that every platform has the same sensor axes, orientation, sample cadence, or calibration.

The requested rate is a policy request, not a guarantee that new physical measurements arrive at exactly that interval. Sample the value at the point the emulated device expects it, preserve the selected port and sensor ID, and keep coordinate conversion explicit. If the emulated console expects a signed tilt range while the host returns physical acceleration, document the conversion and clamp behavior. Do not silently combine gravity, orientation, and acceleration values as if they were the same signal.

Initialization and cleanup should be symmetric. Enable only the sensor required by the loaded content, stop requesting it when the emulated peripheral is detached or the game unloads, and handle a rejected enable request. If a sensor is absent, report the capability as unavailable and expose a deliberate neutral or user-configurable substitute. A disabled sensor should not be represented as a plausible but stale last reading.

Camera frames have explicit format and lifetime

The camera callback allows the core to request raw framebuffer data, an OpenGL texture, or both. The core sets capability bits and desired width and height; the dimensions are hints and the frontend may provide a different size. Raw frames use XRGB8888, with the first pixel at the top-left and pitch measured in bytes. An OpenGL texture is frontend-owned and may be used only by a core that negotiated the required hardware-rendering context.

The camera is not automatically started. The core must call the provided start function from retro_run() and stop it explicitly. The frontend invokes frame callbacks on the same thread as retro_run(), which simplifies synchronization but does not make the data permanent: the header warns that raw buffers, texture identifiers, and transform data may be invalidated when the callback returns. Copy pixels or the metadata the core needs before returning; never retain a borrowed pointer as if it were a save-state-safe image.

Camera input should be sampled at the moment the emulated device would capture it, not on every host frame by default. Keep the frontend frame arrival separate from the console’s capture trigger and transformation pipeline. If emulated hardware converts color to luminance, applies exposure or dithering, or blocks cartridge RAM while capture is active, those are core responsibilities. The host camera callback only supplies source imagery; it does not emulate the peripheral.

Microphone capture is a handle-based audio input

The microphone interface is separate from the normal libretro audio output callback. The core requests the interface, opens a microphone with optional parameters, checks the parameters actually granted, and explicitly changes the microphone’s active state before reading samples. Microphones are inactive by default. The opened handle remains valid until closed, but opening is not guaranteed to be thread-safe and the frontend may not provide any device.

Choose the requested sample rate from the emulated peripheral’s needs, then inspect the returned rate before converting samples. A host driver may provide a different rate or channel configuration. Resample deliberately and preserve a bounded buffer; do not block retro_run() waiting for a physical device. A failed read, a short read, or an inactive microphone should have a defined emulator behavior rather than leaking an uninitialized buffer into the game.

Microphone data is sensitive. Request it only for content that uses it, make activation visible to the user through frontend policy, stop capture promptly when no longer needed, and close the handle during unload. The API’s availability is not a privacy permission system by itself. Respect the frontend’s permission result and never bypass it with direct host capture APIs.

A minimal negotiation pattern

The following is deliberately schematic C, not a drop-in core implementation. It illustrates that the core must check the environment result and preserve independent fallbacks:

struct retro_camera_callback camera = {0};
camera.caps = 1u << RETRO_CAMERA_BUFFER_RAW_FRAMEBUFFER;
camera.width = requested_width;
camera.height = requested_height;
bool camera_api = environ_cb(RETRO_ENVIRONMENT_GET_CAMERA_INTERFACE, &camera);
if (!camera_api || !camera.start || !camera.stop || !camera.frame_raw_framebuffer)
    camera_available = false;

The actual callback type, environment function signature, initialization order, and unload path must match the libretro API header version used by the core. A core that chooses OpenGL must also negotiate and obey the hardware-rendering lifecycle; adding the texture capability alone is not sufficient.

Determinism, states, and replay

Host sensor and microphone readings are external inputs. They are not made deterministic merely because the CPU emulator is deterministic. For reproducible tests, record sampled values at the emulated polling boundary and replay that trace instead of consulting live hardware. Camera tests should use a fixed synthetic frame or a test image held outside copyrighted game content. Save states should serialize the emulated device state and any captured pixels needed to resume a capture, not opaque frontend pointers, camera handles, or texture names.

Separate three clocks: emulated device time, libretro retro_run() cadence, and host sensor/capture time. If a frontend skips frames or changes video refresh, a core must still avoid inventing additional device samples unless its contract says otherwise. Timestamping or buffering can help, but any mapping from host time into emulated time must be explicit and testable.

Validation and acceptance criteria

Test each optional interface under at least four conditions: frontend command unsupported, command supported but device absent, device present but activation rejected, and active data delivery. For camera support, test raw-frame pitch, dimensions that differ from the hint, one-frame callback lifetime, and both clean start/stop and content unload. For microphone support, test NULL from open, altered returned sample rate, inactive reads, short reads, and close. For sensors, test failed enable requests, unsupported sensor IDs, neutral values, axis range and device rotation.

Compare a deterministic recorded-input run with a live-device run only at the sampled boundary. Report frontend, core build, interface version, requested capabilities, actual sample format, and fallback decisions. Do not claim that a frontend supports these peripherals because it advertises a generic libretro version; test the exact environment command and behavior.

The robust design is a small adapter layer around each optional interface. It owns negotiation, capability status, bounded buffers, conversion, logging, and cleanup, while the emulated peripheral consumes stable internal values. That boundary makes unsupported devices understandable, live data safe, and regression tests repeatable without turning host hardware into hidden emulator state.

Related:

Sources: