diff --git a/tools/trace_replay/skills/mismatch-retrace-debugging.md b/tools/trace_replay/skills/mismatch-retrace-debugging.md new file mode 100644 index 00000000..30e7332b --- /dev/null +++ b/tools/trace_replay/skills/mismatch-retrace-debugging.md @@ -0,0 +1,326 @@ +# Mismatch retrace debugging + +Use this when an apitrace fixture replays correctly on one environment but +`mobilegl_trace_replay`, Android trace replay, or another backend produces a +different image. + +## Goal + +Find the first call where the replayed output diverges from a trusted golden +environment, then reduce the problem to one of: + +- vertex input, index, transform, or VS output mismatch +- fragment shader, uniform, texture, sampler, or buffer input mismatch +- framebuffer, blend, depth/stencil, scissor, color mask, sRGB, or format + mismatch +- missing/incorrect resource binding or unsupported backend format/path + +Do not start by guessing from the final image. First identify the earliest +divergent render pass and draw call. + +## Variables + +```sh +export REPO="$PWD" +export WORK="$PWD/.trace-debug" +export CASE="case-name" +export TRACE="$WORK/$CASE/trace.trace" +export APITRACE="$REPO/build-apitrace/apitrace" +export GOLDEN_GLRETRACE="/path/to/golden/glretrace" +export TARGET_GLRETRACE="/path/to/target/glretrace" +export MOBILEGL_REPLAY="$REPO/build-linux/tools/trace_replay/mobilegl_trace_replay" +export MOBILEGL_LIB="$REPO/build-linux/libMobileGL.so" +export WIDTH=854 +export HEIGHT=480 +export TARGET_CALL=0 +export BACKEND=DirectGLES +``` + +The golden and target environments must use the same trace, surface size, target +call numbers, and comparison crop. They do not need to report the same GL vendor, +renderer, version, or extension set. The only requirement for the golden +environment is that it can replay the case and pass the fixture's golden image +test. Disable overlays and avoid tolerance/fuzz while localizing the first +divergent call unless the known golden already has small nondeterministic noise. + +## Prepare two replay environments + +1. Golden environment: a known-good driver/backend that can replay the trace, + produce snapshots, and pass the case against the committed or otherwise + accepted golden PNG. Prefer upstream `glretrace` on desktop GL/Mesa when it + is the simplest environment that satisfies that test. +2. Target environment: the backend being troubleshot. This can be another + `glretrace` environment, `mobilegl_trace_replay` on Linux, or the Android + trace APK. + +First prove the golden environment is acceptable by comparing its target-call +snapshot against the accepted golden PNG: + +```sh +mkdir -p "$WORK/$CASE" +"$GOLDEN_GLRETRACE" --headless -S "$TARGET_CALL" -s "$WORK/$CASE/golden." \ + --call-nos "$TRACE" > "$WORK/$CASE/golden-retrace.log" 2>&1 +``` + +If the generated snapshot does not pass the fixture golden comparison, choose a +different golden environment before doing mismatch localization. + +Then run the target with that accepted golden snapshot or with the fixture's +committed golden PNG: + +```sh +mkdir -p "$WORK/$CASE" + +"$MOBILEGL_REPLAY" \ + --trace "$TRACE" \ + --golden "$WORK/$CASE/golden.$(printf '%010d' "$TARGET_CALL").png" \ + --output "$WORK/$CASE/target-id" \ + --backend "$BACKEND" \ + --mobilegl-library "$MOBILEGL_LIB" \ + --target-call "$TARGET_CALL" \ + --width "$WIDTH" \ + --height "$HEIGHT" \ + --tolerance 0 \ + --fuzz-percent 0 > "$WORK/$CASE/target-retrace.log" 2>&1 || true +``` + +Keep target GL identity lines such as `MOBILEGL_TRACE_GL_VENDOR`, +`MOBILEGL_TRACE_GL_RENDERER`, `MOBILEGL_TRACE_GL_VERSION`, and +`MOBILEGL_TRACE_GL_SHADING_LANGUAGE_VERSION` as diagnostic context only. Do not +reject a golden environment because its GL strings differ from the target. + +For Android, use `android-plugin/trace-replay-ci.sh` with the same trace, +golden, target call, width, height, crop, tolerance, and fuzz values. Pull +`result.json`, `actual.png`, `diff.png`, `retrace.log`, and `logcat.txt` before +continuing. + +## Build a call map + +Dump a narrow call listing around the failing frame or target call: + +```sh +"$APITRACE" dump --calls="$START_CALL-$END_CALL" "$TRACE" \ + > "$WORK/$CASE/calls-$START_CALL-$END_CALL.txt" +``` + +Mark: + +- FBO/render-target changes: `glBindFramebuffer`, `glFramebuffer*`, + `glDrawBuffer*`, `glReadBuffer`, `glViewport`, `glScissor` +- pass-producing calls: `glClear*`, `glDraw*`, `glMultiDraw*`, + `glDispatchCompute`, `glBlitFramebuffer`, `glCopyTex*`, + `glGenerateMipmap` +- sync/resource hazards: `glMemoryBarrier`, `glFenceSync`, `glWaitSync`, + `glFlush`, `glFinish` +- texture/buffer mutation before the pass: `glTex(Sub)Image*`, + `glCompressedTex(Sub)Image*`, `glBuffer(Sub)Data`, `glMapBuffer*`, + `glUnmapBuffer` + +Treat a render pass as a contiguous run of producer calls that writes the same +draw framebuffer attachments with the same logical viewport/scissor region, +ending just before the next framebuffer/attachment change, resolve/blit, +mipmap generation, or swap. + +## Find the first divergent render pass + +Use snapshots at pass endpoints rather than every draw call first. For regular +`glretrace`, snapshot call numbers are explicit: + +```sh +mkdir -p "$WORK/$CASE/pass" +CALLS="1000,1250,1430,1700" +"$GOLDEN_GLRETRACE" --headless --call-nos \ + --snapshot-prefix "$WORK/$CASE/pass/golden." \ + --snapshot "$CALLS" "$TRACE" +"$TARGET_GLRETRACE" --headless --call-nos \ + --snapshot-prefix "$WORK/$CASE/pass/target." \ + --snapshot "$CALLS" "$TRACE" +``` + +If the target is MobileGL, run `mobilegl_trace_replay` once per endpoint: + +```sh +for call in 1000 1250 1430 1700; do + "$MOBILEGL_REPLAY" \ + --trace "$TRACE" \ + --golden "$WORK/$CASE/pass/golden.$(printf '%010d' "$call").png" \ + --diff "$WORK/$CASE/pass/mobilegl.$(printf '%010d' "$call").diff.png" \ + --output "$WORK/$CASE/pass/mobilegl-$call" \ + --backend "$BACKEND" \ + --mobilegl-library "$MOBILEGL_LIB" \ + --target-call "$call" \ + --width "$WIDTH" \ + --height "$HEIGHT" \ + --tolerance 0 \ + --fuzz-percent 0 || true +done +``` + +Binary search the pass endpoints: + +- If endpoint N matches golden, every completed pass up to N is probably OK. +- If endpoint N differs, search earlier pass endpoints. +- Stop when the previous pass endpoint matches and this pass endpoint differs. + +For offscreen passes, do not force backbuffer snapshots. The default retrace +snapshot reads the current draw buffer, which is usually what is needed for FBO +attachment debugging. Use `--snapshot-force-backbuffer` only when explicitly +checking the final default framebuffer. + +## Find the first divergent draw call inside the pass + +List producer calls inside the failing pass and binary search those call +numbers. Include clears, blits, compute dispatches, and mipmap generation; a +non-draw producer can be the first bad write. + +```sh +mkdir -p "$WORK/$CASE/draw" +DRAWS="1430,1442,1459,1490,1512,1550" +"$GOLDEN_GLRETRACE" --headless --call-nos \ + --snapshot-prefix "$WORK/$CASE/draw/golden." \ + --snapshot "$DRAWS" "$TRACE" +``` + +Compare the target at the same call numbers. The first bad producer is the +lowest call where: + +- previous producer output matches golden +- current producer output differs from golden + +Keep the previous-good call and first-bad call. Most state that matters to the +bad call was established between those two calls or immediately before the +previous-good call. + +## Dump and diff GL state + +Use apitrace state dumps around the boundary: + +```sh +"$GOLDEN_GLRETRACE" --headless --dump-state "$PREVIOUS_GOOD_CALL" "$TRACE" \ + > "$WORK/$CASE/state-golden-prev.json" +"$GOLDEN_GLRETRACE" --headless --dump-state "$FIRST_BAD_CALL" "$TRACE" \ + > "$WORK/$CASE/state-golden-bad.json" +"$TARGET_GLRETRACE" --headless --dump-state "$FIRST_BAD_CALL" "$TRACE" \ + > "$WORK/$CASE/state-target-bad.json" +``` + +If both environments are standard `glretrace`, use apitrace's helper to compare +snapshots and optionally state: + +```sh +python3 "$REPO/3rdparty/apitrace/scripts/retracediff.py" \ + --retrace "$GOLDEN_GLRETRACE" \ + --ref-env NAME=VALUE \ + --src-env NAME=VALUE \ + --diff-prefix "$WORK/$CASE/retracediff" \ + --snapshot-frequency "$DRAWS" \ + --diff-state \ + "$TRACE" +``` + +For MobileGL target state, inspect MobileGL logs and add temporary focused +logging in the relevant state sync path. Keep logs scoped to the bad call range +using call-number markers or a `MOBILEGL_TRACE_CURRENT_CALL_OVERRIDE` style +guard if available. + +## Classify the bad call + +Start from the bad call's command and bound pipeline. Use the call dump, +`--dump-state`, shader sources, and MobileGL backend logs. + +### Vertex and VS stage + +Check: + +- current program, vertex shader source, translated shader source, compile/link + log, and reflected attribute locations +- VAO binding and every enabled attribute: size, type, normalized/integer flag, + stride, relative offset, divisor, binding index, and bound VBO +- index buffer binding, index type, byte offset, base vertex, draw count, and + primitive mode +- UBO/SSBO bindings consumed by the VS, including offset alignment and range + size +- transform-relevant uniforms and uniform block contents +- primitive restart, provoking vertex, clip/cull distance, front-face, cull + mode, depth clamp, and viewport transform + +Common MobileGL failures here are attribute location remapping, integer vs float +attribute path mixups, stale VAO/EBO binding, base-vertex handling, wrong buffer +range offset, and translated VS precision/type differences. + +### Fragment and FS stage + +Check: + +- fragment shader source, translated shader source, compile/link log, reflected + sampler/image/uniform bindings, and output locations +- all textures sampled by the FS: target, level, layer, internal format, + dimensions, swizzle, base/max level, compare mode/function, border/wrap/filter, + and whether the backend allocated a supported equivalent format +- sampler object binding versus texture object sampler state +- plain uniforms, UBO/SSBO ranges, image bindings, atomic counters, and texture + buffer bindings +- discard/alpha-test-equivalent logic, precision-sensitive math, NaN/min/max + behavior, shadow compare, and integer/unsigned sampling + +Common MobileGL failures here are stale sampler binding, texture unit remap +errors, depth-compare defaults, unsupported or substituted texture formats, +incorrect swizzle, missing DSA texture parameter sync, and shader translation +differences around precision or undefined values. + +### Framebuffer and composition stage + +Check: + +- draw framebuffer object, every color/depth/stencil attachment, attachment + level/layer, backend object id, internal format, sample count, and dimensions +- draw buffer/read buffer mapping, MRT compacting/remapping, and default-FBO + handling +- viewport, scissor, depth range, color mask, depth mask, stencil masks, + clear values, and per-target blend enable +- blend equation/function/color, logic op, dither, sRGB enable, multisample + state, sample mask, alpha-to-coverage, depth/stencil test and ops +- resolve/blit path, invalidate/discard behavior, mipmap generation, and + readback orientation/channel order + +Common MobileGL failures here are stale FBO sync, draw/read framebuffer +confusion, wrong MRT attachment order, missing per-target blend state, format +substitution, sRGB mismatch, and backend readback from the wrong buffer. + +## Resource validity checks + +Before changing code, rule out invalid trace/backend combinations: + +- Does the target backend advertise every extension used by the trace path? +- Is the internal format renderable/filterable/sampleable on the target GLES or + Vulkan path? +- Is the attachment complete under target rules, especially depth/stencil, + integer color, multisample, layered, and texture-buffer cases? +- Are buffer ranges aligned to the target backend's UBO/SSBO/TBO requirements? +- Are compressed textures, texture views, immutable storage, and DSA calls fully + implemented in the target path? +- Is any GL error logged at or before the first bad call? The first GL error is + usually more useful than the final pixel diff. + +## Confirm the root cause + +Make one narrow diagnostic change at a time. Record it in `attempts.txt` when +the case is already tracked there: + +- what was changed +- exact case/backend/device +- first bad pass/call before and after +- final `mismatchPixels` +- whether the change moved, fixed, or did not affect the divergence + +A real fix should make the first-bad call match, not just reduce the final +mismatch. After fixing, rerun: + +```sh +ctest --test-dir "$REPO/build-linux/tools/trace_replay" -V \ + -R "MobileGLTraceReplay\\.$CASE\\.$BACKEND" +``` + +For Android-only bugs, rerun `android-plugin/trace-replay-ci.sh` and keep +`result.json`, `actual.png`, `diff.png`, `retrace.log`, and `logcat.txt` as the +evidence bundle.