Streaming Audio

Adamo streams audio the same way it streams cameras: you attach a named audio track to a robot, and it shows up next to its video tracks. A robot can carry any number of audio and camera tracks at once. Each audio frame carries its capture time on the fabric clock, the same clock as video, so a receiver can line speech up with the picture.

Audio works in both directions. A robot publishes its microphone and plays what operators say; a browser or an operator program does the same from the other side. Every track lives under a robot, {robot}/audio/{track}, and anyone on the organization can discover, receive and play it.

Tip

The defaults are tuned for the lowest delay: mono voice-quality audio that needs about 125 kbps of your connection. Audio keeps its quality when the network is busy; video adapts around it.

Attaching a microphone

Attach an audio track before you call run(), exactly like a camera. attach_audio captures the machine’s default input. attach_audio_test publishes a 440 Hz tone and needs no hardware, which is the quickest way to check the path end to end.

Python

import adamo

with adamo.Robot(name="my-robot") as robot:
    robot.attach_audio("mic")                                       # default input device
    # robot.attach_audio("mic", input=adamo.AudioInput.TestTone())  # 440 Hz, no hardware
    robot.run()

Rust

use adamo::{AudioOptions, Protocol, Robot};

fn main() -> adamo::Result<()> {
    let mut robot = Robot::from_env(Some("my-robot"), Protocol::Quic)?;
    robot.attach_audio("mic", &AudioOptions::default())?;
    // robot.attach_audio_test("mic", &AudioOptions::default())?;
    robot.run()
}

Enable the crate feature: --features audio (it implies video).

The track publishes on adamo/{org}/{robot}/audio/mic. Operators on operate.adamohq.com viewing the robot hear it once they interact with the page (browsers block autoplay until a click; see Operator playback).

Sources

Source	Python	Rust	Notes
Default input	attach_audio(name)	attach_audio(name, &options)	ALSA default on Linux, the default Core Audio input on macOS. On macOS the process needs microphone permission.
ALSA device	attach_audio(name, input=AudioInput.Alsa("plughw:1,0"))	attach_audio_alsa(name, "plughw:1,0", &options)	Linux. Prefer a plughw: device so ALSA resamples when the hardware cannot capture at an Opus rate.
GStreamer	attach_audio(name, input=AudioInput.Gstreamer("pulsesrc"))	attach_audio_gst(name, "pulsesrc", &options)	A capture description ending in raw audio; the SDK converts and resamples it. Needs the gstreamer feature.
Test tone	attach_audio(name, input=AudioInput.TestTone())	attach_audio_test(name, &options)	440 Hz mono, no hardware.
Your own PCM	robot.audio(name)	robot.audio(name, &options)	Returns an AudioTrack whose send takes interleaved int16 samples.

channels is a ceiling: with Stereo, a mono microphone is still sent as mono, never upmixed. A source that cannot be opened or fails while streaming, such as an unplugged microphone, ends that track only: video and the other audio tracks keep running, and run() reports the error naming the track and source when it returns. Core Audio and GStreamer sources also fail after 2 seconds without audio.

Pushing your own PCM

Python

track = robot.audio("speech")  # 48 kHz mono, like the defaults
track.send(samples)  # numpy.int16 array or little-endian bytes, any length

Rust

let track = robot.audio("speech", &AudioOptions::default())?;
track.send(&samples)?; // interleaved i16, any whole number of frames

send takes the samples as captured just now; send_at (Rust) or send(samples, capture_time_us=...) (Python) gives the capture time of the first sample in UNIX microseconds. Up to 64 chunks queue ahead of the encoder; beyond that send blocks, which paces a producer that runs faster than real time. The first send starts the robot’s runtime in Python when run() is not already driving it.

Options

Option	Default	Meaning
sample_rate	Hz48000	Audio quality: 8, 12, 16, 24 or 48 kHz.
channels	Mono	Mono or Stereo. With Stereo, a mono microphone is still sent as mono.
frame_duration	Ms10	How much audio goes into each packet: 2.5, 5, 10, 20, 40 or 60 ms. Shorter means lower delay and more packets.
bitrate_kbps	48	Total bitrate for all channels, 6 to 510.
application	LowDelay	LowDelay has the least delay. Voice and Music add a few milliseconds for better quality at low bitrates, and can add loss protection with inband_fec.
dtx	false	Send almost nothing while the microphone is silent. Receivers play the gap as silence.

Python

robot.attach_audio(
    "mic",
    input=adamo.AudioInput.Alsa("plughw:1,0"),
    channels=adamo.AudioChannels.Stereo,
    bitrate_kbps=96,
    application=adamo.AudioApplication.Voice(inband_fec=True),
    dtx=True,
)

Rust

use adamo::{AudioApplication, AudioChannels, AudioOptions};

robot.attach_audio_alsa(
    "mic",
    "plughw:1,0",
    &AudioOptions {
        channels: AudioChannels::Stereo,
        bitrate_kbps: 96,
        application: AudioApplication::Voice { inband_fec: true },
        dtx: true,
        ..AudioOptions::default()
    },
)?;

inband_fec lets Rust and Python receivers rebuild a lost packet from the one after it. It costs some bitrate and browsers do not use it.

How audio travels

Audio is sent ahead of video and with its own protection against packet loss, so a lost packet is usually rebuilt without waiting for a resend. Each track reserves its bandwidth (about 125 kbps at the defaults) before video shares out the rest, and on the hosted relay audio uses its own connection, so a congested video stream does not drag audio down with it.

Receiving audio

In Rust and Python

AudioReceiver decodes one robot’s track from any session and hands you PCM frames in order. It rebuilds lost packets when it can, waits at most max_reorder_delay (40 ms by default) for a missing one, and then fills the gap with synthesized audio so playback never stalls. Those frames are marked concealed.

Python

import adamo

with adamo.connect() as client, client.audio_receiver("my-robot", "mic") as receiver:
    while True:
        frame = receiver.recv()
        play(frame.samples)  # int16 array, shape (frames, channels)

audio_receiver takes sample_rate (an AudioSampleRate, default 48 kHz), channels (default: the sender’s), max_reorder_delay_ms (40) and max_queued_frames (50). receiver.stats() returns decoded, rebuilt, concealed, late and dropped frame counts. No frames arrive while a sender with dtx is silent.

Rust

use adamo::{AudioReceiveOptions, AudioReceiver, Protocol, Session};
use std::time::Duration;

let session = Session::connect(None, Protocol::Quic)?;
let receiver = AudioReceiver::subscribe(&session, "my-robot", "mic", AudioReceiveOptions::default())?;
loop {
    if let Some(frame) = receiver.recv(Duration::from_secs(1))? {
        // frame.samples: interleaved i16 at frame.sample_rate, frame.channels
    }
}

Playing audio on a speaker

play_audio receives a track and plays it on an output device: the machine’s default output, an ALSA device, or a GStreamer sink. It buffers only as much as the network currently needs, from 20 ms up to max_latency_ms (150), so delay never builds up. A robot uses the same call to play an operator’s microphone, which is published as a track under the robot.

Python

# On the robot: play whatever operators send to it.
for track in robot.audio_tracks():
    if track != "mic":
        playback = robot.play_audio(track)  # AudioOutput.Default()

# On an operator machine:
with adamo.connect() as client, client.play_audio("my-robot", "mic") as playback:
    print(playback.stats())

Rust

use adamo::{AudioOutput, AudioPlayback, AudioPlaybackOptions};

let tracks = session.audio_tracks("my-robot")?;
let playback = AudioPlayback::start(
    &session,
    "my-robot",
    "operator",
    AudioOutput::Default,
    AudioPlaybackOptions::default(),
)?;

audio_tracks lists the tracks a robot carries right now, its own and the ones operators publish to it.

Operator playback

The hosted operator at operate.adamohq.com plays a robot’s audio once you interact with the page, because browsers block audio autoplay until the first click or keypress; in the VR view, entering VR is that gesture. A robot without an audio track is skipped; audio never delays video. See Web Interface for the mute toggle and audio stats, and the TypeScript SDK for custom browser UIs.

Microphone hardware notes

• On a Jetson, use a USB microphone. The onboard audio path is not reliably usable for capture. Confirm the OS sees a capture device with arecord -l, and find its ALSA name (the plughw:CARD,DEVICE string) with arecord -L.
• Check that the adapter has an input. Many cheap USB audio adapters are output only.
• A charge-only USB cable keeps the device from enumerating; use a data cable.

arecord -l                    # capture-capable sound cards
arecord -L                    # PCM device names, including plughw:CARD,DEV
arecord -d 3 -f cd test.wav   # record 3 s to check the microphone

Availability

SDK	Audio
Python	Send, receive and play. Every wheel includes audio; Linux uses the system libasound.
Rust	Send, receive and play with --features audio. Builds need libopus (libopus-dev or brew install opus) and, on Linux, libasound2-dev.
C / C++	Not available yet in the 1.0 C SDK.
TypeScript	Send the browser microphone with useMicrophonePublisher, play tracks with <AudioStream>, and list them with useAudioTracks. dtx is not available in browsers.

Topics

Topic	Description
adamo/{org}/{robot}/audio/{track}	The audio packets.
adamo/{org}/{robot}/audio/{track}/meta	The track’s settings as JSON: codec, sample_rate, channels, frame_duration_us, bitrate_kbps, dtx, inband_fec, source.
adamo/{org}/{robot}/audio/{track}/alive	Present while the track streams. This is how tracks are discovered.