From c4e6ea1f23c0f08ebe772d6790eb554dd29c394c Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Tue, 11 Aug 2026 23:24:42 -0400 Subject: [PATCH] [Docs] (tools/trace_replay): strip investigation logs from the README - usage and procedures only --- tools/trace_replay/README.md | 302 +---------------------------------- 1 file changed, 6 insertions(+), 296 deletions(-) diff --git a/tools/trace_replay/README.md b/tools/trace_replay/README.md index 5df80009..3c5f2550 100644 --- a/tools/trace_replay/README.md +++ b/tools/trace_replay/README.md @@ -324,299 +324,9 @@ Run the binary directly rather than through `ctest`, whose `ENVIRONMENT` property overrides these variables. Build with clang, not gcc: gcc rejects `GLXImpl.cpp` under `-Wchanges-meaning`. -### improved-transparency-minecraft-26.3 on the ANGLE lane - -This case has never passed on the Android DirectGLES lane. It renders correctly -everywhere else, including Android DirectVulkan on the same emulator. The -rendered frame loses the whole translucent layer - clouds and water are absent -while opaque geometry is pixel-exact - so the compositing chain never receives -the translucent content rather than blending it wrongly. - -**It is two defects in the ESSL we generate, and both are ours.** Replaying the -fixture through the recipe above and through Mesa GLES, with a -`MOBILEGL_LOG_LEVEL_DEBUG` build, gives SSIM 1.000000 on Mesa and 0.970127 on -ANGLE while the *frontend* call stream is byte-identical - 2698222 calls, same -names, same order, same arguments, the only difference being how often the app -polls `glClientWaitSync`. That comparison is what misled the earlier -investigation: it is the app-to-MobileGL direction. It says nothing about the -GLES and ESSL MobileGL emits *downwards*, which is where the two lanes part. - -Two shaders that MobileGL generates fail to compile on ANGLE and link no -program at all, so every draw that uses them is a silent no-op: - -| # | ANGLE compile error | what it is | what disappears | -| --- | --- | --- | --- | -| A | `0:2: 'GL_EXT_texture_buffer' : extension is not supported` then `'isamplerBuffer' : Illegal use of reserved word` | the cloud vertex shader emits `#extension GL_EXT_texture_buffer : require` and `uniform highp isamplerBuffer CloudFaces` unconditionally | clouds | -| B | `'[' : array indexes for fragment outputs must be constant integral expressions` | the OIT coefficient fragment shader declares `layout(location = 0) out highp vec4 coeff[2];` and writes `coeff[attachmentIndex][i]` from a loop | the whole translucent accumulation | - -Neither is an ANGLE mistranslation: - -- **A is a capability gap we do not guard.** Minecraft 26.3 builds clouds - entirely from `gl_VertexID` plus `texelFetch` on a buffer texture - (`glTexBuffer(GL_TEXTURE_BUFFER, GL_R8I, ...)`). Mesa GLES advertises - `GL_EXT_texture_buffer` and `GL_OES_texture_buffer`; the ANGLE in the local - farm advertises neither (142 extensions against Mesa's 162), and MobileGL's - reported `GL_MAX_TEXTURE_BUFFER_SIZE` drops to the 65536 default because it - cannot query one. We emit the `require` line anyway. -- **B is a GLSL ES rule we violate.** Fragment output arrays must be indexed - with constant integral expressions; SPIRV-Cross hands us a loop-variable - index and Mesa's compiler accepts it, ANGLE does not. Any strict ES driver - rejects this shader, so it is not ANGLE-specific in principle - it is - Mesa's leniency that hides it on the Linux lane. - -##### The local ANGLE is not the CI ANGLE - check before attributing - -This trap cost a full round of analysis, so check it first. The Linux farm -recipe above uses the emulator SDK's ANGLE; the Android lane uses a *pinned* -build downloaded by `apk.yml` (`MOBILEGL_TRACE_ANGLE_VARIANT`, default -`ec889e6ea831`). They are far apart: - -| | local farm ANGLE | CI lane ANGLE | -| --- | --- | --- | -| `GL_RENDERER` | `ANGLE (Mesa, Vulkan 1.4.354 (llvmpipe ...), llvmpipe-26.1.4)` | `ANGLE (Mesa, Vulkan 1.3.0 (llvmpipe ...), Mesa-25.2.4)` | -| extensions | 142 | 184 | -| `GL_EXT_texture_buffer` / `GL_OES_texture_buffer` | absent | **present**, `GL_MAX_TEXTURE_BUFFER_SIZE 134217728` | -| ES 3.2 core base vertex | no | yes | - -So **defect A cannot be what fails on CI** - it is an artefact of the older -local ANGLE. The CI frame nevertheless loses clouds *and* water with the same -signature (`ssim=0.968845` against the local farm's 0.970127, opaque geometry -exact), and defect B is the remaining named cause: the coefficient writer is in -the OIT phase shader that *both* the cloud and the translucent-terrain -programs link against, so one rejected shader empties both layers. - -That last step is **not yet directly confirmed on CI**, because the retrace APK -is built `-Pmobilegl.logLevel=MOBILEGL_LOG_LEVEL_INFO`, where `MGLOG_E` is -compiled out - the CI `mobilegl.log` carries 294 INFO lines and zero ERROR -lines, so the shader-compile diagnostics never reach the artifact. Confirming it -means replaying on an emulator with a debug-level trace APK and the pinned ANGLE -variant (`tools/trace_replay/run_android_retrace_local.py`), or promoting -shader-compile failures to a log level that survives an INFO build. - -There is therefore **no honest harness accommodation**: no -`--avoid-angle-llvmpipe-*` flag can conjure a missing extension or make an -illegal shader legal, so the fixture stays red on the Android DirectGLES lane. -Fixing it means fixing the emitters - rewriting non-constant fragment-output -indexing into a switch over constant indices, and guarding the buffer-texture -`require` on driver support - which is shared backend work, not harness work. - -Two earlier candidates remain correctly ruled out and should not be re-walked: -per-attachment blend equations are both recorded and applied correctly on ANGLE -(`glBlendEquationSeparatei` with GL_MAX on one attachment reads back as GL_MAX -and rasterizes as GL_MAX), and the GLES draw-buffer slot restriction is already -handled by `BackendFramebufferObject::RecomputeBackendColorSlots`. - -#### Where in the frame it goes wrong - -Snapshotting the same intra-frame call points on both stacks (`--target-call`, -ten replays in parallel, logs to `/dev/null`) localises it. SSIM against the -golden at each point: - -| target call | what runs there | Mesa | ANGLE | gap | -| --- | --- | --- | --- | --- | -| 2667445 | last opaque draw, into FBO 29 | 0.146477 | 0.000195 | - | -| 2667488 | **OIT composite**: fullscreen triangle, program 50, into FBO 3 | 0.976448 | 0.945409 | 0.031 | -| 2667543 | post draw, program 56 | 0.985380 | 0.954480 | 0.031 | -| 2667595 | post draw, program 64 | 0.985685 | 0.954788 | 0.031 | -| 2667619 | golden point | 1.000000 | 0.970127 | 0.030 | - -The gap opens at the composite and is then constant to four decimal places - -every pass after 2667488 contributes the same increment on both drivers. So -nothing downstream of the composite is implicated, and the GUI/post chain is -fine. The two stacks already disagree at 2667445, before the composite runs, -which points at the translucent accumulation targets the composite samples -rather than at the composite draw itself. - -#### Which attachment, and which draw - -`--dump-fbo-attachments` on both stacks pins it to a single draw. These numbers -are from the local farm, so the first divergence they show is defect A's cloud -draw; on CI, where clouds compile, the chain instead breaks one pass later at -the coefficient accumulation. Comparing the manifests at three call boundaries, -over all 30 live framebuffers: - -| call | what has just run | verdict | -| --- | --- | --- | -| 2665649 | depth blit, before the translucent chain | every OIT target identical on both stacks | -| 2666231 | all entity/particle translucent draws done | still identical: FBO 21 att0 (texture 1554, RGBA32F) reads `c1 max=178.98 mean=1.81046` on Mesa against `179 / 1.8105` on ANGLE | -| 2666232 | the cloud draw - `glDrawElementsInstancedBaseVertex(count=221706)` with `GL_TEXTURE_BUFFER` bound | Mesa moves to `c1 max=1727.87 mean=70.2899`; **ANGLE does not move at all** | - -At the golden point the same holds for the rest of the chain: texture 1555 (the -translucent colour accumulation) has mean 0.105617 on Mesa against 0.00403189 on -ANGLE, and texture 1556 (the alpha/coefficient target, FBO 26 attachment 1) is -alpha `max=1.34863 mean=0.261387` on Mesa and **identically zero** on ANGLE - -never written, because defect B linked no program for that pass. Everything -outside the translucent chain matches: the opaque terrain, the block atlas and -its mips, and the GUI targets differ only in the last float ULP. - -So the earlier reading of the SSIM bisect was right about the location and wrong -about the cause. The composite at 2667488 is innocent; it faithfully composites -accumulation buffers that two failed shader compiles left empty. - -Reproduce with: - -```sh -for stack in mesa angle; do ... --target-call 2666232 \ - --dump-fbo-attachments 2666232:out-$stack/fbos ; done -paste out-mesa/fbos/manifest.txt out-angle/fbos/manifest.txt -``` - -and read the compile errors straight out of `mobilegl.log`: - -```sh -grep -aE 'Shader compilation failed|linking failed' out-angle/mobilegl.log -``` - -### minecraft-1.21.4-fabric-iris-sundial-lite-in-world on the ANGLE lane - -This case takes the **emulator process down**, every run, on its first attempt. -It is not a timeout (it dies ~74s into a 900s budget) and it is not host memory -pressure. From the retained diagnostics of run 31552175083: - -- `host-dmesg.txt` contains no OOM, no `oom-kill`, no `Killed process`. -- `host-memory.txt` reports 11Gi of 15Gi available and 44Ki of 8Gi swap used. -- `host-dmesg.txt` contains exactly two faults, both at the same moment and the - same instruction: - - ``` - llvmpipe-1[3216]: segfault at 8 ip 00007f99296b276e error 4 - llvmpipe-0[3215]: segfault at 8 ip 00007f99296b276e error 4 - ``` - -Those are host Mesa **llvmpipe** rasterizer worker threads - the emulator's own -renderer, not SwiftShader - and their death takes the emulator with them -(`kvm [2665]` before the fault, `kvm [3554]` after the restart, matching -`pid_2665.ini` in `emulator-first-attempt.log`). The faulting instruction -decodes as `mov 0x13c0(%rsp,%rax,8),%rax` followed by `cmpl $0x0,0x8(%rax)`: -an **indexed load out of a stack pointer-table** that returned null, then a -dereference of it. That is the shape an out-of-range array index produces in a -JIT rasterizer. - -Do not raise the swap allocation to match `test.yml` - the OOM hypothesis is -dead, and doing so would only hide the question. - -The retry leg is a separate, milder failure (`statusCode 5`, "failed to make -current OpenGL context and drawable" after a complete capability probe) and is -deliberately **not** covered by the surface-lost infrastructure clause: its -`retrace.log` carries neither the `0x300b` nor the `-1000000001` marker, and its -`mobilegl.log` does reach `OpenGL ES capabilities:`. Both guards exclude it, so -the run is charged as a failure rather than retried away. - -#### Not the same root cause as improved-transparency, and it reproduces on the desktop - -Checked directly: the generated ESSL for this fixture contains **no fragment -output arrays and no dynamic output indexing**. Across its 44 shader dumps there -are zero `out ... [N]` declarations, zero `gl_FragData` references, and zero -`mg_FragColor_` replicas - so `BroadcastLegacyFragColor` never even fires here. -Every output is a plainly named location, e.g. - -``` -layout(location = 0) out highp vec4 gbufferData0; -layout(location = 0) out float iris_FogFragCoord; -layout(location = 2) out vec3 skyColorUp; -``` - -So the improved-transparency defect (non-constant index into a fragment-output -array) is **not** what kills the emulator here, and the two need separate fixes. - -The crash does reproduce locally through the ANGLE recipe above, which is much -cheaper than bisecting on an emulator. Running this fixture on both stacks: -Mesa passes at SSIM 0.996301; ANGLE exits **139** (SIGSEGV) with no result.json, -and the host kernel log shows the same fault as CI - several `llvmpipe-N` -worker threads, `segfault at 8`, `error 4`, all at one instruction. It dies -immediately after program 58 links, on a draw logged as -`Using raw depth fetch sampler on unit 13`. Debug that repro under gdb before -reaching for the 13-commit bisect. - -#### What the crash actually is - -Run the ANGLE invocation under `gdb -batch` with `handle SIGSEGV stop nopass` -(apitrace installs its own handler, so gdb has to stop first). The faulting -thread is `llvmpipe-0`, and: - -``` -rip 0x7fffea1f7a28 -> "No symbol matches $rip", in no shared object -rax 0x0 -=> cmpl $0x0,0x8(%rax) <-- fault: deref of NULL+8 - vmovdqa %ymm5,0xac0(%rsp) - je ... - mov (%rax),%rax -``` - -`rip` belongs to no library, so this is **llvmpipe's JIT-compiled fragment -shader**, not Mesa C code - which is why the backtrace is garbage (the "frames" -are SIMD shader constants: `0x3c800000` = 0.015625f, `0x42800000` = 64.0f, -`0xffc00000` = NaN). Do not chase that backtrace. - -The CI kernel log records the instruction immediately before the fault: -`mov 0x13c0(%rsp,%rax,8),%rax`. So the shape is: **index a stack-resident -pointer table by a unit number, get NULL back, then null-check-and-follow it.** -That is a per-texture-unit descriptor lookup. A sampler in this shader resolves -to a texture unit for which the ANGLE/lavapipe side has no valid descriptor, and -llvmpipe's JIT dereferences it instead of returning the (0,0,0,1) that sampling -an incomplete texture is required to give - so the crash itself is a **driver -bug**, whatever state we hand it. - -The draw is program 58 (fragment shader 60), a deferred reflection/solid -composite ending `texBuffer0 = reflectionData; texBuffer3 = solidColor;`, with -20+ samplers declared (`colortex0..7`, `depthtex0..2`, `shadowtex0/1`, -`shadowcolor0`, `noisetex`, `gtexture`, `normals`, `specular`, `gaux2`, -`transmittanceTex`) against `GL_MAX_TEXTURE_IMAGE_UNITS = 32`. The last state -operation logged before the fault is `Using raw depth fetch sampler on unit 13` -- `GetRawDepthFetchSampler()->Bind(unit)` in `DirectGLES.cpp`, which puts a -dedicated sampler object on the unit when a `sampler2D` uniform reads a -depth-format texture. Mesa GLES takes that same path 90 times in this trace -without crashing; ANGLE dies on the second one. - -Still unproven: **which** sampler/unit holds the NULL descriptor. Unit 13 is the -prime suspect because it is the last thing logged, not because it has been -shown to be the NULL entry. Nothing in the debugger points at a specific -MobileGL emission commit, so the held 13-commit bisect does **not** collapse to -a one-commit confirmation on this evidence. - -#### The sampler state is not the trigger - -A temporary probe in the sampler-unit sync (archived, never landed) logged, for -every sampler uniform of every draw, the tuple `(uniform location, uniform type, -resolved unit, bound texture id, internal format, completeness)`. Run on both -stacks and diffed: - -- **No unit is missing a texture and none is incomplete.** Across the entire - ANGLE run there are zero `tex=NONE` and zero `complete=0` tuples. The - "sampler uniform points at a unit with nothing bound" theory is dead. -- Unit 13 - the raw-depth-fetch unit, and the prime suspect from the previous - leg - holds `tex=64`, a depth format, `complete=1`. It is exonerated as a - dangling binding. -- **The crashing draw's state is identical on both stacks.** Its 14 tuples match - the passing Mesa run exactly, and Mesa executes the very tuple - `loc=35 type=GL_SAMPLER_2D unit=13 tex=64 complete=1` **45 times** without - crashing. ANGLE dies the first time it reaches it. - -So nothing in the sampler->unit->texture mapping distinguishes the crash. What -the diff did turn up is one genuine divergence, ~80 tuples upstream: ANGLE's -compiler reports an **extra live sampler uniform** that Mesa's does not, and it -aliases a unit already in use - - -``` -angle: loc=3 type=GL_SAMPLER_2D unit=0 tex=76 <- both stacks -angle: loc=14 type=GL_SAMPLER_2D unit=0 tex=76 <- ANGLE only -mesa : (no loc=14; stream continues one tuple ahead from here) -``` - -Two sampler uniforms resolving to the same unit is the subject of -`b6a2bf08 [Fix] (DirectGLES): resolve an aliased texture unit by the sampler's -type`, which sits **inside** the `8025745f..5a400e02` regression window - the -first concrete link between that window and this crash. It is a lead, not a -proven cause: it is upstream of the fault and the two runs still agree on the -crashing draw itself. - -**Fix shape: avoidance, not correctness.** Nothing here shows MobileGL in the -wrong - our state at the faulting draw is exactly the state a passing driver -handles 45 times over. Sampling is required to be safe whatever the binding, so -the NULL-deref is the driver's bug. Any change on our side is therefore working -around a driver defect and should be scoped and named as such, ideally behind -the `--avoid-angle-llvmpipe-*` family rather than in shared backend paths. The -ingredient the probe did **not** capture, and the next thing to instrument, is -the *sampler object* bound to the unit: unit 13 takes a depth-format texture -through a plain `GL_SAMPLER_2D` with the raw-depth-fetch sampler override, while -unit 12 takes another depth texture as `GL_SAMPLER_2D_SHADOW` in the same draw. +One more caveat before attributing anything: the emulator SDK's ANGLE is not +the ANGLE the Android lane runs. The CI lane uses a pinned build +(`MOBILEGL_TRACE_ANGLE_VARIANT`, default `ec889e6ea831`) whose version and +extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2 +entry points). Compare `GL_RENDERER` and the relevant extension lists on both +stacks before treating a local result as a statement about CI.