Audio, Adaptive Bitrate, and Production Hardening

This is part 4 of a four-part series drawn from shipping Chromecast casting in Remotype, the app that turns your phone into a keyboard, trackpad, and remote for every computer you own.

• Part 1: Fundamentals and Launching Your Own Receiver
• Part 2: WebRTC Signaling Over the Cast Custom Namespace
• Part 3: Resolution, Aspect Ratio, and the Silent Decoder Cliffs
• Part 4: Audio, Adaptive Bitrate, and Production Hardening (you are here)

4.1 Separate tracks, and the SDP direction that broke everything

Video and audio travel as separate RTP tracks in one peer connection — H.264 for the screen, Opus for the sound. With Pion:

video, _ := webrtc.NewTrackLocalStaticSample(
    webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeH264}, "video", "screen")
audio, _ := webrtc.NewTrackLocalStaticSample(
    webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeOpus}, "audio", "screen")

Now the single most consequential line in this entire series. Do not add them with AddTrack:

// WRONG for a TV: produces a=sendrecv — "please send video back to me"
pc.AddTrack(video)

// RIGHT: we push media at the TV; the TV sends nothing.
pc.AddTransceiverFromTrack(video,
    webrtc.RTPTransceiverInit{Direction: webrtc.RTPTransceiverDirectionSendonly})
pc.AddTransceiverFromTrack(audio,
    webrtc.RTPTransceiverInit{Direction: webrtc.RTPTransceiverDirectionSendonly})

AddTrack's default produces a=sendrecv m-lines — an offer that invites the receiver to send media back. A Cast device has no camera, no microphone, and no permission to capture anything, and its WebRTC stack responded to our sendrecv offer by silently failing to generate an answer at all. Days of debugging, one word in the SDP. sendonly. Always.

The audio pipeline itself is straightforward: capture system audio (ScreenCaptureKit's audio tap on macOS), deliver interleaved stereo Int16 PCM at 48 kHz to the encoder process, chunk into exact 20 ms frames, Opus-encode, and write with a fixed duration so RTP timestamps stay honest:

frame := 960 * 2 // 20ms @ 48kHz, stereo
for len(pcmBuf) >= frame {
    pkt, _ := opusEnc.encode(pcmBuf[:frame])
    pcmBuf = pcmBuf[frame:]
    audio.WriteSample(media.Sample{Data: pkt, Duration: 20 * time.Millisecond})
}

4.2 The audio bug that wasn't a bug: check the sink's volume

Our "audio doesn't work" report survived a full pipeline audit — capture verified, Opus packets counted onto the wire, the receiver's SDP answer accepting the track (m=audio … a=recvonly). The stream was perfect. The Chromecast's own volume was 0.0. A brand-new or long-idle device commonly sits at zero, and a flawless stream into a muted sink is indistinguishable, in every log, from a broken one.

Two fixes, both worth shipping:

// 1. When starting a cast that carries audio, nudge a silent sink — once,
//    and never override a level the user actually chose.
if sinkVolume() <= 0.001 {
    setVolume(0.4)   // CASTv2: {"type":"SET_VOLUME","volume":{"level":0.4}}
}

// 2. Wire your app's volume buttons to CASTv2 SET_VOLUME so users control the
//    TV's output from the remote they're already holding.

(The sink's current volume rides along in every RECEIVER_STATUS broadcast — parse it, don't poll for it.)

4.3 Adaptive bitrate: start low, climb fast, never open at the ceiling

Screen content is brutal on rate control: mostly static, then a scroll event touches every pixel. Fixed bitrates fail in both directions — starve text or flood the link. We run Google Congestion Control (send-side BWE via Transport-Wide CC, which Pion provides as an interceptor) and feed its estimate to the encoder as a live target within a floor and ceiling.

The subtle bug worth inheriting our scar tissue for: never open at your ceiling. Our 40 Mbps preset initially set 40 Mbps as the starting rate — so the very first IDR frame of a near-4K stream was enormous, and on busy Wi-Fi it shredded before the decoder could assemble a single keyframe. The receiver sat on its splash while ICE, DTLS and RTP all reported perfect health (recognize the pattern by now?). Worse, it was intermittent: quiet link, crisp; busy link, dead.

let startBitrate = min(ceilBitrate, 8_000_000)   // open conservative…
encoder.setBitrate(startBitrate)

webrtc.onBitrateEstimate = { bps in              // …and let BWE climb to the ceiling
    let clamped = min(max(bps, 1_500_000), ceilBitrate)
    encoder.setBitrate(clamped)                  // floor keeps text legible on dips
}

The first keyframe fits through anything, the ladder reaches 40 Mbps within seconds on a clean link, and a congested network settles wherever it honestly can.

4.4 The fallback chain: degrade, never dead-end

Every failure mode in this series terminates somewhere useful:

WebRTC @ user's quality preset
  │  no answer within 45 s              → "noanswer"
  │  connected but no first frame in 12 s → "noplay"
  ▼
WebRTC @ auto (1080p-class)   ← "noplay" retries one rung down first:
  │                              if 4K-class won't decode, 1080p will
  ▼
HLS to the Default Media Receiver (CC1AD845)
  │  works on frozen-firmware first-gen Chromecasts; ~10 s latency, honestly labeled
  ▼
Named error with actionable copy — never a spinner that spins forever

The render watchdog driving the "noplay" transition is only possible because the receiver reports {kind:"playing"} (Part 2.6). One care point: gate it on having seen the receiver's accepted message — an old cached receiver that predates the reporting protocol stays silent and must not be executed for it.

4.5 Production checklist — the lessons, compressed

Infrastructure

• ☐ Serve the receiver page with Cache-Control: no-store. Chromecasts cache aggressively; during our rollout the TV ran week-old receiver code while we debugged "fixes that did nothing." The page is 5 KB and loads once per cast — caching it buys nothing and costs your sanity.
• ☐ Know your actual deploy pipeline. One of our receiver fixes sat undeployed for an hour because the site deployed via wrangler deploy, not the git push we were confidently waiting on.
• ☐ Registered App ID matches the current receiver URL; app is Published (Unlisted is fine).

Protocol

• ☐ CASTv2 library forwards custom-namespace frames without a "type" key (§2.2).
• ☐ Offer sent on the receiver's ready, resent only after long silence, hard deadline into fallback (§2.3–2.4).
• ☐ Session token pin cleared on sender connect/disconnect/stop (§2.5).
• ☐ Receiver reports accepted / staged errors / icestate / playing+decoded-size (§2.6).

Media

• ☐ sendonly transceivers — never AddTrack's sendrecv — for a receive-only sink (§4.1).
• ☐ Encode size inside H.264 Level 5.1 (983,040 MB/s) and 4096×2304 (§3.2).
• ☐ Encode at the source display's aspect ratio; no baked-in letterboxing (§3.3).
• ☐ Open at ~8 Mbps, ladder to the ceiling; QP capped so text survives dips (§4.3).
• ☐ Nudge a zero-volume sink; wire app volume to SET_VOLUME (§4.2).

Diagnostics

• ☐ --castprobe / --caststatus / --caststop CLI against real devices from day one (§1.3).
• ☐ Log every inbound CASTv2 frame during development — the silent failures in this series were only ever visible from the wire.
• ☐ Success metric = the receiver's playing report. Nothing upstream of the glass counts.

4.6 Closing: why we bothered

Remotype's promise is that your phone controls every computer you own — and "put my computer on the TV" belongs in that promise. Off-the-shelf casting meant ten seconds of HLS latency, which is unusable when the phone in your hand is also the trackpad moving the cursor on that screen. The system in this series delivers the computer's screen to a $30 Chromecast at 3546×2304/30 with latency you stop noticing, audio included, degrading gracefully all the way down to hardware from 2013.

Every bug in this series shipped against a real TV, on real Wi-Fi, and was diagnosed with the tools described here. If it saves your team the weeks it cost ours, it did its job.

Remotype is built by Custavia. The keyboard and trackpad are free forever; casting is part of Remotype Pro. If you build something with this guide, we'd genuinely love to hear about it.

---

Built by Custavia. Remotype's keyboard and trackpad are free forever; casting is part of Remotype Pro. If this series saves your team the weeks it cost ours, it did its job — and we'd love to hear what you build with it.