Resolution, Aspect Ratio, and the Silent Decoder Cliffs

This is part 3 of a four-part series drawn from shipping Chromecast casting in Remotype, the app that turns your phone into a keyboard, trackpad, and remote for every computer you own.

• Part 1: Fundamentals and Launching Your Own Receiver
• Part 2: WebRTC Signaling Over the Cast Custom Namespace
• Part 3: Resolution, Aspect Ratio, and the Silent Decoder Cliffs (you are here)
• Part 4: Audio, Adaptive Bitrate, and Production Hardening

3.1 "Connected" proves nothing about pixels

Burn this into your monitoring philosophy: ICE connected, DTLS established, RTP flowing, zero packet loss, and a session state of "casting" are all fully consistent with a black screen. A TV decoder handed a stream it cannot decode does not error, does not NACK, does not send a Picture Loss Indication. It simply never produces a frame, while your entire pipeline reports green.

We learned this chasing what looked like a haunted regression: native-resolution casting worked one day, showed an eternal splash screen the next, and every layer we instrumented was healthy. The receiver's {kind:"playing"} report was what cracked it — its absence localized the failure to the decoder itself, and the numbers did the rest.

3.2 The H.264 Level 5.1 cliff: 972,000 vs 992,250

H.264 decoders declare a Level — a hard budget on macroblocks (16×16 pixel blocks) per second. Level 5.1, the ceiling for virtually every TV SoC, allows 983,040 MB/s. Run the numbers:

One percent over the line. That was our entire "regression": an improvement from 3464×2250 to true-native 3600×2338 crossed a limit no error message would ever name.

And there's a second cliff hiding behind the first: TV SoCs also enforce a maximum coded frame size of 4096 × 2304 regardless of macroblock rate. We proved this the empirical way — 3564×2314 satisfies Level 5.1's rate budget (970,050 MB/s) and still rendered nothing; trimming ten rows to 2304 rendered instantly.

The takeaway as code — clamp to both limits, preserving aspect, and take the largest size that survives:

// Fit the encode inside every decoder limit, largest-first.
let fps = 30

// Limit 1: hardware coded-size ceiling (dimensions, independent of rate).
if eh > 2304 { ew = (ew * 2304 / eh).rounded(.down); eh = 2304 }
if ew > 4096 { eh = (eh * 4096 / ew).rounded(.down); ew = 4096 }

// Limit 2: H.264 Level 5.1 macroblock rate — 983,040 MB/s, with 1% margin.
let mbBudget = Double(983_040 / fps) * 0.99
let mbs = ((ew + 15) / 16).rounded(.down) * ((eh + 15) / 16).rounded(.down)
if mbs > mbBudget {
    let scale = (mbBudget / mbs).squareRoot()
    ew = (ew * scale).rounded(.down)
    eh = (eh * scale).rounded(.down)
}

let w = max(2, Int(ew) & ~1)   // H.264 wants even dimensions
let h = max(2, Int(eh) & ~1)

For our 3600×2338 Retina panel this lands on 3546×2304 @ 30 — 8.2 megapixels, within 1.5% of native, and confirmed by the TV itself: RECEIVER IS RENDERING: 3546x2304.

Lesson: treat the receiver's playing report — not connection state — as your definition of success. And when a resolution mysteriously fails, do the macroblock math before touching anything else.

3.3 Aspect ratio: stop double-letterboxing your users

Screens are not all 16:9. MacBooks are 16:10-ish (ours is 1.54:1); TVs are 1.78:1. The lazy pipeline — capture the screen, scale-to-fit into a fixed 1280×720 or 1920×1080 canvas — bakes black bars into the encoded video. The TV then letterboxes that to fit its own panel, and your users get bars on all four sides plus wasted bitrate encoding pure black.

Size the encode to the source display's aspect ratio and let the TV do the single, inevitable letterbox:

let bounds = CGDisplayBounds(CGMainDisplayID())
let srcAR  = bounds.width / bounds.height          // e.g. 1.54 for a 16:10 Mac

var ew = Double(longEdgeCap)                        // e.g. 1920 for the default rung
var eh = (ew / srcAR).rounded()                     // 1848 × 1200, not 1920 × 1080

In ScreenCaptureKit terms: set SCStreamConfiguration.width/height to your aspect-correct target and capture the full display into it — don't rely on scalesToFit against a mismatched canvas. A 1.54:1 source on a 1.78:1 TV will always have modest pillarboxing; that's geometry. What you're eliminating is the doubled boxing and the resolution thrown away inside it. (If your product wants true edge-to-edge, offer an explicit "fill & crop" option — never silently crop someone's desktop.)

3.4 Quality presets that map to real constraints

With the limits understood, user-facing quality settings become honest engineering rather than vibes. Ours:

switch quality {
case "high_quality":   // native pixels, clamped by Part 3's two cliffs
    ew = min(nativePixelWidth, 3840); eh = min(nativePixelHeight, 2400)
    ceilBitrate = 40_000_000
case "low_latency":    // smallest useful frame, fastest to settle
    longEdgeCap = 1280
    ceilBitrate = 6_000_000
default:               // "auto": 1080p-class, the right default
    longEdgeCap = 1920
    ceilBitrate = 12_000_000
}

Two supporting decisions worth copying:

• H.264 High profile for capable sinks, Baseline only for legacy paths. High's CABAC entropy coding and 8×8 transform are dramatically better on screen content — sharp text at the same bitrate — and every device that can run a CAF receiver decodes High. We keep Constrained Baseline only for our HLS fallback aimed at frozen-firmware first-generation Chromecasts.
• Bound the encoder's quality degradation. Rate control will happily raise QP to 50 and smear your text into soup on a busy frame. Cap it: kVTCompressionPropertyKey_MaxAllowedFrameQP = 38 keeps glyphs legible and forces the bitrate controller (Part 4) to do its job instead.

Built by Custavia. Remotype's keyboard and trackpad are free forever; casting is part of Remotype Pro. If this series saves your team the weeks it cost ours, it did its job — and we'd love to hear what you build with it.