Skip to main content

FnVoice - End-to-End Audio Pipeline

How a spoken word travels from one client's microphone to another client's speaker, mapped to the actual classes and methods.

The pipeline has six stages across two threads on each side plus the network relay:

  1. Mic Capture - VoiceChatRecorder (Unity main thread, 50 Hz coroutine)
  2. Encoding - AudioEncodeThread (background thread) -> OpusEncoder
  3. Relay - the FnVoice server forwards the room event to peers
  4. Receive - the receiving client's transport enqueues the packet
  5. Decoding - AudioDecodeThread (background thread) -> OpusDecoder
  6. Playback - VoiceChatPlayer or VoiceChatPlayerOnFilterRead (per remote speaker)

The same code lives in all three copies (main Unity project, the Unity package, and the FigNetXR project); paths below are relative to the FnVoice module's Runtime/VoiceChat/.

Sequence diagram​

sequenceDiagram
    participant Mic as Mic Capture<br/>(VoiceChatRecorder · main thread)
    participant Enc as Encoding Thread<br/>(AudioEncodeThread)
    participant Srv as Server<br/>(FnVoice relay)
    participant Cli as Client<br/>(receiver transport)
    participant Dec as Decoding Thread<br/>(AudioDecodeThread)
    participant Play as Playback<br/>(VoiceChatPlayer / OnFilterRead)

    Note over Mic: MicrophoneReadCoroutine ticks @ 50 Hz
    Mic->>Mic: _clip.GetData(sampleBuffer) - read mic @ record rate (48k on Quest)
    Mic->>Mic: AudioResampler.ResampleAudio -> 16k (box-average decimation)
    Mic->>Mic: preprocessor.ProcessCaptureFrame (AEC/AGC/NS - identity when off)
    Mic->>Mic: VoiceDetectorFloat.Process (VAD: speak vs silence)
    Mic-)Enc: OnSampleAvailable(16k frame) - enqueue to encode thread

    Note over Enc: EncodeThread (background)
    Enc->>Enc: muted? -> skip
    Enc->>Enc: VoiceChatUtils.Compress -> OpusEncoder.Encode (Opus)
    Enc->>Enc: encrypt (if enabled)
    Enc->>Enc: build VoiceChatPacket -> FNVoice_RoomEvent
    Enc->>Srv: send RoomEvent (Unreliable / RUDP)

    Srv->>Cli: relay VOICE_CHAT_EVENT to room peers

    Cli-)Dec: AddCompressedAudio(event) - enqueue incoming

    Note over Dec: DecodeThread (background)
    Dec->>Dec: find DecoderState by session (NetworkId)
    Dec->>Dec: out-of-order? PacketId <= LastPacketId -> drop
    Dec->>Dec: decrypt (if needed)
    Dec->>Dec: OpusDecoder.Decode -> PCM (16k, 640 samples)
    Dec->>Dec: enqueue -> decodedVoipPacketsChannel

    Note over Play: Update() on the remote player (main thread)
    Play->>Dec: TryGetDecodedPacket()
    Dec--)Play: ArraySegment<float> PCM
    Play->>Play: ApplyGain
    Play->>Play: write to clip / ring buffer -> AudioSource
    Play->>Play: playDelay from FV.GetPing() (jitter buffer)

Step-by-step​

1. Capture - VoiceChatRecorder (main thread, 50 Hz coroutine)​

  • MicrophoneReadCoroutine -> ProcessMicrophoneData -> ReadAndProcessSample: _clip.GetData(...) pulls mic samples at the device record rate (Quest reports caps=16000/48000 -> records at 48 kHz).
  • AudioResampler.ResampleAudio downsamples 48k->16k via a box-average decimation (ResampleDownsample). (The windowed-sinc downsampler here garbled 16k on-device - see Notes.)
  • ProcessAudioSample runs the preprocessor (IAudioPreprocessor.ProcessCaptureFrame - AEC/AGC/NS; a no-op IdentityAudioPreprocessor when Audio Processing is off) then VAD (VoiceDetectorFloat) to decide speak vs silence (with a forceTransmit hold + a small pre-roll buffer).
  • OnSampleAvailable(buffer) fires the 16 kHz frame to the encode side.

2. Encode - AudioEncodeThread + VoiceChatUtils / OpusEncoder​

(wired by AudioPipelinePlayGround in the demo, or FigNetVoiceManager in production)

  • The encode thread dequeues the frame, skips if muted, then VoiceChatUtils.Compress -> CodecFactory.GetEncoder -> OpusEncoder.Encode (640-sample 16k frame -> ~35-55 byte Opus packet), optional encrypt, wrapped in a VoiceChatPacket.
  • FigNetVoiceManager packs it into an FNVoice_RoomEvent and sends it Unreliable (RUDP) to the server.

3. Relay - Server​

  • Forwards VOICE_CHAT_EVENT to the other members of the voice room.

4-5. Receive + Decode - AudioDecodeThread (background thread)​

  • Incoming events -> AddCompressedAudio -> ProcessAudioFrames looks up the per-sender DecoderState.
  • DecoderState.Decode: out-of-order guard (PacketId <= LastPacketId -> drop, past the first packets), decrypt if needed, OpusDecoder.Decode -> 640 PCM samples @16k, enqueue to a bounded decodedVoipPacketsChannel (DropOldest).

6. Playback - VoiceChatPlayer / VoiceChatPlayerOnFilterRead (per remote speaker)​

  • Update() pulls decoded PCM via TryGetDecodedPacket, applies gain, buffers it, and uses a small ping-based delay (FV.GetPing()) as a jitter buffer for smooth playback.
  • The two players differ only in the final hop:
    • VoiceChatPlayer - writes PCM into an AudioClip created at the codec rate (16k); Unity's engine resamples 16k->hardware on playback. (Manual resampling here drifted the volume - see Notes.)
    • VoiceChatPlayerOnFilterRead - resamples 16k->AudioSettings.outputSampleRate with the accumulator AudioResampler into a ring buffer drained by OnAudioFilterRead.

Notes - on-device gotchas (Quest 3 / arm64 / IL2CPP)​

Two C#-side issues garbled 16 kHz playback on-device while the editor was clean (the 48 kHz preset bypasses both resamplers, so it was unaffected). Both are fixed in the current code:

  • Capture downsample: the windowed-sinc ResampleWithAntiAliasing (sum/weightSum normalization) produced garbage on-device -> replaced by ResampleDownsample (box-average).
  • Playback: a per-frame VoicePcmResampler upsample drifted the playback volume up/down -> replaced by the codec-rate-clip approach (VoiceChatPlayer) and the accumulator AudioResampler (OnFilterRead).

The native libopus -ffast-math build flag was investigated and ruled out (voice is clean with the flag on or off); it is kept off only because Opus discourages it and it yields no measurable speedup for a voice codec.