MobileGL/tools/trace_replay/skills/mismatch-retrace-debugging.md

Mismatch retrace debugging

Use this when an apitrace fixture replays correctly on one environment but mobilegl_trace_replay, Android trace replay, or another backend produces a different image.

Goal

Find the first call where the replayed output diverges from a trusted golden environment, then reduce the problem to one of:

• vertex input, index, transform, or VS output mismatch
• fragment shader, uniform, texture, sampler, or buffer input mismatch
• framebuffer, blend, depth/stencil, scissor, color mask, sRGB, or format mismatch
• missing/incorrect resource binding or unsupported backend format/path

Do not start by guessing from the final image. First identify the earliest divergent render pass and draw call.

Variables

export REPO="$PWD"
export WORK="$PWD/.trace-debug"
export CASE="case-name"
export TRACE="$WORK/$CASE/trace.trace"
export APITRACE="$REPO/build-apitrace/apitrace"
export GOLDEN_GLRETRACE="/path/to/golden/glretrace"
export TARGET_GLRETRACE="/path/to/target/glretrace"
export MOBILEGL_REPLAY="$REPO/build-linux/tools/trace_replay/mobilegl_trace_replay"
export MOBILEGL_LIB="$REPO/build-linux/libMobileGL.so"
export WIDTH=854
export HEIGHT=480
export TARGET_CALL=0
export BACKEND=DirectGLES

The golden and target environments must use the same trace, surface size, target call numbers, and comparison crop. They do not need to report the same GL vendor, renderer, version, or extension set. The only requirement for the golden environment is that it can replay the case and pass the fixture's golden image test. Disable overlays and use a strict SSIM threshold while localizing the first divergent call unless the known golden already has small nondeterministic noise.

Prepare two replay environments

1. Golden environment: a known-good driver/backend that can replay the trace, produce snapshots, and pass the case against the committed or otherwise accepted golden PNG. Prefer upstream glretrace on desktop GL/Mesa when it is the simplest environment that satisfies that test.
2. Target environment: the backend being troubleshot. This can be another glretrace environment, mobilegl_trace_replay on Linux, or the Android trace APK.

First prove the golden environment is acceptable by comparing its target-call snapshot against the accepted golden PNG:

mkdir -p "$WORK/$CASE"
"$GOLDEN_GLRETRACE" --headless -S "$TARGET_CALL" -s "$WORK/$CASE/golden." \
  --call-nos "$TRACE" > "$WORK/$CASE/golden-retrace.log" 2>&1

If the generated snapshot does not pass the fixture golden comparison, choose a different golden environment before doing mismatch localization.

Then run the target with that accepted golden snapshot or with the fixture's committed golden PNG:

mkdir -p "$WORK/$CASE"

"$MOBILEGL_REPLAY" \
  --trace "$TRACE" \
  --golden "$WORK/$CASE/golden.$(printf '%010d' "$TARGET_CALL").png" \
  --output "$WORK/$CASE/target-id" \
  --backend "$BACKEND" \
  --mobilegl-library "$MOBILEGL_LIB" \
  --target-call "$TARGET_CALL" \
  --width "$WIDTH" \
  --height "$HEIGHT" \
  --ssim-threshold 1.0 > "$WORK/$CASE/target-retrace.log" 2>&1 || true

Keep target GL identity lines such as MOBILEGL_TRACE_GL_VENDOR, MOBILEGL_TRACE_GL_RENDERER, MOBILEGL_TRACE_GL_VERSION, and MOBILEGL_TRACE_GL_SHADING_LANGUAGE_VERSION as diagnostic context only. Do not reject a golden environment because its GL strings differ from the target.

For Android, use android-plugin/trace-replay-ci.sh with the same trace, golden, target call, width, height, crop, and SSIM threshold values. Pull result.json, actual.png, diff.png, retrace.log, and logcat.txt before continuing.

Build a call map

Dump a narrow call listing around the failing frame or target call:

"$APITRACE" dump --calls="$START_CALL-$END_CALL" "$TRACE" \
  > "$WORK/$CASE/calls-$START_CALL-$END_CALL.txt"

Mark:

• FBO/render-target changes: glBindFramebuffer, glFramebuffer*, glDrawBuffer*, glReadBuffer, glViewport, glScissor
• pass-producing calls: glClear*, glDraw*, glMultiDraw*, glDispatchCompute, glBlitFramebuffer, glCopyTex*, glGenerateMipmap
• sync/resource hazards: glMemoryBarrier, glFenceSync, glWaitSync, glFlush, glFinish
• texture/buffer mutation before the pass: glTex(Sub)Image*, glCompressedTex(Sub)Image*, glBuffer(Sub)Data, glMapBuffer*, glUnmapBuffer

Treat a render pass as a contiguous run of producer calls that writes the same draw framebuffer attachments with the same logical viewport/scissor region, ending just before the next framebuffer/attachment change, resolve/blit, mipmap generation, or swap.

Find the first divergent render pass

Use snapshots at pass endpoints rather than every draw call first. For regular glretrace, snapshot call numbers are explicit:

mkdir -p "$WORK/$CASE/pass"
CALLS="1000,1250,1430,1700"
"$GOLDEN_GLRETRACE" --headless --call-nos \
  --snapshot-prefix "$WORK/$CASE/pass/golden." \
  --snapshot "$CALLS" "$TRACE"
"$TARGET_GLRETRACE" --headless --call-nos \
  --snapshot-prefix "$WORK/$CASE/pass/target." \
  --snapshot "$CALLS" "$TRACE"

If the target is MobileGL, run mobilegl_trace_replay once per endpoint:

for call in 1000 1250 1430 1700; do
  "$MOBILEGL_REPLAY" \
    --trace "$TRACE" \
    --golden "$WORK/$CASE/pass/golden.$(printf '%010d' "$call").png" \
    --diff "$WORK/$CASE/pass/mobilegl.$(printf '%010d' "$call").diff.png" \
    --output "$WORK/$CASE/pass/mobilegl-$call" \
    --backend "$BACKEND" \
    --mobilegl-library "$MOBILEGL_LIB" \
    --target-call "$call" \
    --width "$WIDTH" \
    --height "$HEIGHT" \
    --ssim-threshold 1.0 || true
done

Binary search the pass endpoints:

• If endpoint N matches golden, every completed pass up to N is probably OK.
• If endpoint N differs, search earlier pass endpoints.
• Stop when the previous pass endpoint matches and this pass endpoint differs.

For offscreen passes, do not force backbuffer snapshots. The default retrace snapshot reads the current draw buffer, which is usually what is needed for FBO attachment debugging. Use --snapshot-force-backbuffer only when explicitly checking the final default framebuffer.

Find the first divergent draw call inside the pass

List producer calls inside the failing pass and binary search those call numbers. Include clears, blits, compute dispatches, and mipmap generation; a non-draw producer can be the first bad write.

mkdir -p "$WORK/$CASE/draw"
DRAWS="1430,1442,1459,1490,1512,1550"
"$GOLDEN_GLRETRACE" --headless --call-nos \
  --snapshot-prefix "$WORK/$CASE/draw/golden." \
  --snapshot "$DRAWS" "$TRACE"

Compare the target at the same call numbers. The first bad producer is the lowest call where:

• previous producer output matches golden
• current producer output differs from golden

Keep the previous-good call and first-bad call. Most state that matters to the bad call was established between those two calls or immediately before the previous-good call.

Dump and diff GL state

Use apitrace state dumps around the boundary:

"$GOLDEN_GLRETRACE" --headless --dump-state "$PREVIOUS_GOOD_CALL" "$TRACE" \
  > "$WORK/$CASE/state-golden-prev.json"
"$GOLDEN_GLRETRACE" --headless --dump-state "$FIRST_BAD_CALL" "$TRACE" \
  > "$WORK/$CASE/state-golden-bad.json"
"$TARGET_GLRETRACE" --headless --dump-state "$FIRST_BAD_CALL" "$TRACE" \
  > "$WORK/$CASE/state-target-bad.json"

If both environments are standard glretrace, use apitrace's helper to compare snapshots and optionally state:

python3 "$REPO/3rdparty/apitrace/scripts/retracediff.py" \
  --retrace "$GOLDEN_GLRETRACE" \
  --ref-env NAME=VALUE \
  --src-env NAME=VALUE \
  --diff-prefix "$WORK/$CASE/retracediff" \
  --snapshot-frequency "$DRAWS" \
  --diff-state \
  "$TRACE"

For MobileGL target state, inspect MobileGL logs and add temporary focused logging in the relevant state sync path. Keep logs scoped to the bad call range using retrace-side call-number markers rather than backend-specific snapshot hooks.

Classify the bad call

Start from the bad call's command and bound pipeline. Use the call dump, --dump-state, shader sources, and MobileGL backend logs.

Vertex and VS stage

Check:

• current program, vertex shader source, translated shader source, compile/link log, and reflected attribute locations
• VAO binding and every enabled attribute: size, type, normalized/integer flag, stride, relative offset, divisor, binding index, and bound VBO
• index buffer binding, index type, byte offset, base vertex, draw count, and primitive mode
• UBO/SSBO bindings consumed by the VS, including offset alignment and range size
• transform-relevant uniforms and uniform block contents
• primitive restart, provoking vertex, clip/cull distance, front-face, cull mode, depth clamp, and viewport transform

Common MobileGL failures here are attribute location remapping, integer vs float attribute path mixups, stale VAO/EBO binding, base-vertex handling, wrong buffer range offset, and translated VS precision/type differences.

Fragment and FS stage

Check:

• fragment shader source, translated shader source, compile/link log, reflected sampler/image/uniform bindings, and output locations
• all textures sampled by the FS: target, level, layer, internal format, dimensions, swizzle, base/max level, compare mode/function, border/wrap/filter, and whether the backend allocated a supported equivalent format
• sampler object binding versus texture object sampler state
• plain uniforms, UBO/SSBO ranges, image bindings, atomic counters, and texture buffer bindings
• discard/alpha-test-equivalent logic, precision-sensitive math, NaN/min/max behavior, shadow compare, and integer/unsigned sampling

Common MobileGL failures here are stale sampler binding, texture unit remap errors, depth-compare defaults, unsupported or substituted texture formats, incorrect swizzle, missing DSA texture parameter sync, and shader translation differences around precision or undefined values.

Framebuffer and composition stage

Check:

• draw framebuffer object, every color/depth/stencil attachment, attachment level/layer, backend object id, internal format, sample count, and dimensions
• draw buffer/read buffer mapping, MRT compacting/remapping, and default-FBO handling
• viewport, scissor, depth range, color mask, depth mask, stencil masks, clear values, and per-target blend enable
• blend equation/function/color, logic op, dither, sRGB enable, multisample state, sample mask, alpha-to-coverage, depth/stencil test and ops
• resolve/blit path, invalidate/discard behavior, mipmap generation, and readback orientation/channel order

Common MobileGL failures here are stale FBO sync, draw/read framebuffer confusion, wrong MRT attachment order, missing per-target blend state, format substitution, sRGB mismatch, and backend readback from the wrong buffer.

Resource validity checks

Before changing code, rule out invalid trace/backend combinations:

• Does the target backend advertise every extension used by the trace path?
• Is the internal format renderable/filterable/sampleable on the target GLES or Vulkan path?
• Is the attachment complete under target rules, especially depth/stencil, integer color, multisample, layered, and texture-buffer cases?
• Are buffer ranges aligned to the target backend's UBO/SSBO/TBO requirements?
• Are compressed textures, texture views, immutable storage, and DSA calls fully implemented in the target path?
• Is any GL error logged at or before the first bad call? The first GL error is usually more useful than the final pixel diff.

Confirm the root cause

Make one narrow diagnostic change at a time. Record it in attempts.txt when the case is already tracked there:

• what was changed
• exact case/backend/device
• first bad pass/call before and after
• final mismatchPixels
• whether the change moved, fixed, or did not affect the divergence

A real fix should make the first-bad call match, not just reduce the final mismatch. After fixing, rerun:

ctest --test-dir "$REPO/build-linux/tools/trace_replay" -V \
  -R "MobileGLTraceReplay\\.$CASE\\.$BACKEND"

For Android-only bugs, rerun android-plugin/trace-replay-ci.sh and keep result.json, actual.png, diff.png, retrace.log, and logcat.txt as the evidence bundle.