swung0x48/MobileGL
1
0
Fork 0
mirror of https://github.com/MobileGL-Dev/MobileGL synced 2026-09-08 04:08:32 +09:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] (tools/trace_replay): strip investigation logs from the README - usage and procedures only

Browse Source
This commit is contained in:
swung0x48
2026-08-11 23:25:20 -04:00
parent b4e07ce651
commit c4e6ea1f23
1 changed files with 6 additions and 296 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/trace_replay/README.md
+6 -296
View File
@@ -324,299 +324,9 @@ Run the binary directly rather than through `ctest`, whose `ENVIRONMENT`
property overrides these variables. Build with clang, not gcc: gcc rejects
`GLXImpl.cpp` under `-Wchanges-meaning`.
### improved-transparency-minecraft-26.3 on the ANGLE lane
This case has never passed on the Android DirectGLES lane. It renders correctly
everywhere else, including Android DirectVulkan on the same emulator. The
rendered frame loses the whole translucent layer - clouds and water are absent
while opaque geometry is pixel-exact - so the compositing chain never receives
the translucent content rather than blending it wrongly.
**It is two defects in the ESSL we generate, and both are ours.** Replaying the
fixture through the recipe above and through Mesa GLES, with a
`MOBILEGL_LOG_LEVEL_DEBUG` build, gives SSIM 1.000000 on Mesa and 0.970127 on
ANGLE while the *frontend* call stream is byte-identical - 2698222 calls, same
names, same order, same arguments, the only difference being how often the app
polls `glClientWaitSync`. That comparison is what misled the earlier
investigation: it is the app-to-MobileGL direction. It says nothing about the
GLES and ESSL MobileGL emits *downwards*, which is where the two lanes part.
Two shaders that MobileGL generates fail to compile on ANGLE and link no
program at all, so every draw that uses them is a silent no-op:
| # | ANGLE compile error | what it is | what disappears |
| --- | --- | --- | --- |
| A | `0:2: 'GL_EXT_texture_buffer' : extension is not supported` then `'isamplerBuffer' : Illegal use of reserved word` | the cloud vertex shader emits `#extension GL_EXT_texture_buffer : require` and `uniform highp isamplerBuffer CloudFaces` unconditionally | clouds |
| B | `'[' : array indexes for fragment outputs must be constant integral expressions` | the OIT coefficient fragment shader declares `layout(location = 0) out highp vec4 coeff[2];` and writes `coeff[attachmentIndex][i]` from a loop | the whole translucent accumulation |
Neither is an ANGLE mistranslation:
- **A is a capability gap we do not guard.** Minecraft 26.3 builds clouds
entirely from `gl_VertexID` plus `texelFetch` on a buffer texture
(`glTexBuffer(GL_TEXTURE_BUFFER, GL_R8I, ...)`). Mesa GLES advertises
`GL_EXT_texture_buffer` and `GL_OES_texture_buffer`; the ANGLE in the local
farm advertises neither (142 extensions against Mesa's 162), and MobileGL's
reported `GL_MAX_TEXTURE_BUFFER_SIZE` drops to the 65536 default because it
cannot query one. We emit the `require` line anyway.
- **B is a GLSL ES rule we violate.** Fragment output arrays must be indexed
with constant integral expressions; SPIRV-Cross hands us a loop-variable
index and Mesa's compiler accepts it, ANGLE does not. Any strict ES driver
rejects this shader, so it is not ANGLE-specific in principle - it is
Mesa's leniency that hides it on the Linux lane.
##### The local ANGLE is not the CI ANGLE - check before attributing
This trap cost a full round of analysis, so check it first. The Linux farm
recipe above uses the emulator SDK's ANGLE; the Android lane uses a *pinned*
build downloaded by `apk.yml` (`MOBILEGL_TRACE_ANGLE_VARIANT`, default
`ec889e6ea831`). They are far apart:
| | local farm ANGLE | CI lane ANGLE |
| --- | --- | --- |
| `GL_RENDERER` | `ANGLE (Mesa, Vulkan 1.4.354 (llvmpipe ...), llvmpipe-26.1.4)` | `ANGLE (Mesa, Vulkan 1.3.0 (llvmpipe ...), Mesa-25.2.4)` |
| extensions | 142 | 184 |
| `GL_EXT_texture_buffer` / `GL_OES_texture_buffer` | absent | **present**, `GL_MAX_TEXTURE_BUFFER_SIZE 134217728` |
| ES 3.2 core base vertex | no | yes |
So **defect A cannot be what fails on CI** - it is an artefact of the older
local ANGLE. The CI frame nevertheless loses clouds *and* water with the same
signature (`ssim=0.968845` against the local farm's 0.970127, opaque geometry
exact), and defect B is the remaining named cause: the coefficient writer is in
the OIT phase shader that *both* the cloud and the translucent-terrain
programs link against, so one rejected shader empties both layers.
That last step is **not yet directly confirmed on CI**, because the retrace APK
is built `-Pmobilegl.logLevel=MOBILEGL_LOG_LEVEL_INFO`, where `MGLOG_E` is
compiled out - the CI `mobilegl.log` carries 294 INFO lines and zero ERROR
lines, so the shader-compile diagnostics never reach the artifact. Confirming it
means replaying on an emulator with a debug-level trace APK and the pinned ANGLE
variant (`tools/trace_replay/run_android_retrace_local.py`), or promoting
shader-compile failures to a log level that survives an INFO build.
There is therefore **no honest harness accommodation**: no
`--avoid-angle-llvmpipe-*` flag can conjure a missing extension or make an
illegal shader legal, so the fixture stays red on the Android DirectGLES lane.
Fixing it means fixing the emitters - rewriting non-constant fragment-output
indexing into a switch over constant indices, and guarding the buffer-texture
`require` on driver support - which is shared backend work, not harness work.
Two earlier candidates remain correctly ruled out and should not be re-walked:
per-attachment blend equations are both recorded and applied correctly on ANGLE
(`glBlendEquationSeparatei` with GL_MAX on one attachment reads back as GL_MAX
and rasterizes as GL_MAX), and the GLES draw-buffer slot restriction is already
handled by `BackendFramebufferObject::RecomputeBackendColorSlots`.
#### Where in the frame it goes wrong
Snapshotting the same intra-frame call points on both stacks (`--target-call`,
ten replays in parallel, logs to `/dev/null`) localises it. SSIM against the
golden at each point:
| target call | what runs there | Mesa | ANGLE | gap |
| --- | --- | --- | --- | --- |
| 2667445 | last opaque draw, into FBO 29 | 0.146477 | 0.000195 | - |
| 2667488 | **OIT composite**: fullscreen triangle, program 50, into FBO 3 | 0.976448 | 0.945409 | 0.031 |
| 2667543 | post draw, program 56 | 0.985380 | 0.954480 | 0.031 |
| 2667595 | post draw, program 64 | 0.985685 | 0.954788 | 0.031 |
| 2667619 | golden point | 1.000000 | 0.970127 | 0.030 |
The gap opens at the composite and is then constant to four decimal places -
every pass after 2667488 contributes the same increment on both drivers. So
nothing downstream of the composite is implicated, and the GUI/post chain is
fine. The two stacks already disagree at 2667445, before the composite runs,
which points at the translucent accumulation targets the composite samples
rather than at the composite draw itself.
#### Which attachment, and which draw
`--dump-fbo-attachments` on both stacks pins it to a single draw. These numbers
are from the local farm, so the first divergence they show is defect A's cloud
draw; on CI, where clouds compile, the chain instead breaks one pass later at
the coefficient accumulation. Comparing the manifests at three call boundaries,
over all 30 live framebuffers:
| call | what has just run | verdict |
| --- | --- | --- |
| 2665649 | depth blit, before the translucent chain | every OIT target identical on both stacks |
| 2666231 | all entity/particle translucent draws done | still identical: FBO 21 att0 (texture 1554, RGBA32F) reads `c1 max=178.98 mean=1.81046` on Mesa against `179 / 1.8105` on ANGLE |
| 2666232 | the cloud draw - `glDrawElementsInstancedBaseVertex(count=221706)` with `GL_TEXTURE_BUFFER` bound | Mesa moves to `c1 max=1727.87 mean=70.2899`; **ANGLE does not move at all** |
At the golden point the same holds for the rest of the chain: texture 1555 (the
translucent colour accumulation) has mean 0.105617 on Mesa against 0.00403189 on
ANGLE, and texture 1556 (the alpha/coefficient target, FBO 26 attachment 1) is
alpha `max=1.34863 mean=0.261387` on Mesa and **identically zero** on ANGLE -
never written, because defect B linked no program for that pass. Everything
outside the translucent chain matches: the opaque terrain, the block atlas and
its mips, and the GUI targets differ only in the last float ULP.
So the earlier reading of the SSIM bisect was right about the location and wrong
about the cause. The composite at 2667488 is innocent; it faithfully composites
accumulation buffers that two failed shader compiles left empty.
Reproduce with:
```sh
for stack in mesa angle; do ... --target-call 2666232 \
--dump-fbo-attachments 2666232:out-$stack/fbos ; done
paste out-mesa/fbos/manifest.txt out-angle/fbos/manifest.txt
```
and read the compile errors straight out of `mobilegl.log`:
```sh
grep -aE 'Shader compilation failed|linking failed' out-angle/mobilegl.log
```
### minecraft-1.21.4-fabric-iris-sundial-lite-in-world on the ANGLE lane
This case takes the **emulator process down**, every run, on its first attempt.
It is not a timeout (it dies ~74s into a 900s budget) and it is not host memory
pressure. From the retained diagnostics of run 31552175083:
- `host-dmesg.txt` contains no OOM, no `oom-kill`, no `Killed process`.
- `host-memory.txt` reports 11Gi of 15Gi available and 44Ki of 8Gi swap used.
- `host-dmesg.txt` contains exactly two faults, both at the same moment and the
same instruction:
```
llvmpipe-1[3216]: segfault at 8 ip 00007f99296b276e error 4
llvmpipe-0[3215]: segfault at 8 ip 00007f99296b276e error 4
```
Those are host Mesa **llvmpipe** rasterizer worker threads - the emulator's own
renderer, not SwiftShader - and their death takes the emulator with them
(`kvm [2665]` before the fault, `kvm [3554]` after the restart, matching
`pid_2665.ini` in `emulator-first-attempt.log`). The faulting instruction
decodes as `mov 0x13c0(%rsp,%rax,8),%rax` followed by `cmpl $0x0,0x8(%rax)`:
an **indexed load out of a stack pointer-table** that returned null, then a
dereference of it. That is the shape an out-of-range array index produces in a
JIT rasterizer.
Do not raise the swap allocation to match `test.yml` - the OOM hypothesis is
dead, and doing so would only hide the question.
The retry leg is a separate, milder failure (`statusCode 5`, "failed to make
current OpenGL context and drawable" after a complete capability probe) and is
deliberately **not** covered by the surface-lost infrastructure clause: its
`retrace.log` carries neither the `0x300b` nor the `-1000000001` marker, and its
`mobilegl.log` does reach `OpenGL ES capabilities:`. Both guards exclude it, so
the run is charged as a failure rather than retried away.
#### Not the same root cause as improved-transparency, and it reproduces on the desktop
Checked directly: the generated ESSL for this fixture contains **no fragment
output arrays and no dynamic output indexing**. Across its 44 shader dumps there
are zero `out ... [N]` declarations, zero `gl_FragData` references, and zero
`mg_FragColor_` replicas - so `BroadcastLegacyFragColor` never even fires here.
Every output is a plainly named location, e.g.
```
layout(location = 0) out highp vec4 gbufferData0;
layout(location = 0) out float iris_FogFragCoord;
layout(location = 2) out vec3 skyColorUp;
```
So the improved-transparency defect (non-constant index into a fragment-output
array) is **not** what kills the emulator here, and the two need separate fixes.
The crash does reproduce locally through the ANGLE recipe above, which is much
cheaper than bisecting on an emulator. Running this fixture on both stacks:
Mesa passes at SSIM 0.996301; ANGLE exits **139** (SIGSEGV) with no result.json,
and the host kernel log shows the same fault as CI - several `llvmpipe-N`
worker threads, `segfault at 8`, `error 4`, all at one instruction. It dies
immediately after program 58 links, on a draw logged as
`Using raw depth fetch sampler on unit 13`. Debug that repro under gdb before
reaching for the 13-commit bisect.
#### What the crash actually is
Run the ANGLE invocation under `gdb -batch` with `handle SIGSEGV stop nopass`
(apitrace installs its own handler, so gdb has to stop first). The faulting
thread is `llvmpipe-0`, and:
```
rip 0x7fffea1f7a28 -> "No symbol matches $rip", in no shared object
rax 0x0
=> cmpl $0x0,0x8(%rax) <-- fault: deref of NULL+8
vmovdqa %ymm5,0xac0(%rsp)
je ...
mov (%rax),%rax
```
`rip` belongs to no library, so this is **llvmpipe's JIT-compiled fragment
shader**, not Mesa C code - which is why the backtrace is garbage (the "frames"
are SIMD shader constants: `0x3c800000` = 0.015625f, `0x42800000` = 64.0f,
`0xffc00000` = NaN). Do not chase that backtrace.
The CI kernel log records the instruction immediately before the fault:
`mov 0x13c0(%rsp,%rax,8),%rax`. So the shape is: **index a stack-resident
pointer table by a unit number, get NULL back, then null-check-and-follow it.**
That is a per-texture-unit descriptor lookup. A sampler in this shader resolves
to a texture unit for which the ANGLE/lavapipe side has no valid descriptor, and
llvmpipe's JIT dereferences it instead of returning the (0,0,0,1) that sampling
an incomplete texture is required to give - so the crash itself is a **driver
bug**, whatever state we hand it.
The draw is program 58 (fragment shader 60), a deferred reflection/solid
composite ending `texBuffer0 = reflectionData; texBuffer3 = solidColor;`, with
20+ samplers declared (`colortex0..7`, `depthtex0..2`, `shadowtex0/1`,
`shadowcolor0`, `noisetex`, `gtexture`, `normals`, `specular`, `gaux2`,
`transmittanceTex`) against `GL_MAX_TEXTURE_IMAGE_UNITS = 32`. The last state
operation logged before the fault is `Using raw depth fetch sampler on unit 13`
- `GetRawDepthFetchSampler()->Bind(unit)` in `DirectGLES.cpp`, which puts a
dedicated sampler object on the unit when a `sampler2D` uniform reads a
depth-format texture. Mesa GLES takes that same path 90 times in this trace
without crashing; ANGLE dies on the second one.
Still unproven: **which** sampler/unit holds the NULL descriptor. Unit 13 is the
prime suspect because it is the last thing logged, not because it has been
shown to be the NULL entry. Nothing in the debugger points at a specific
MobileGL emission commit, so the held 13-commit bisect does **not** collapse to
a one-commit confirmation on this evidence.
#### The sampler state is not the trigger
A temporary probe in the sampler-unit sync (archived, never landed) logged, for
every sampler uniform of every draw, the tuple `(uniform location, uniform type,
resolved unit, bound texture id, internal format, completeness)`. Run on both
stacks and diffed:
- **No unit is missing a texture and none is incomplete.** Across the entire
ANGLE run there are zero `tex=NONE` and zero `complete=0` tuples. The
"sampler uniform points at a unit with nothing bound" theory is dead.
- Unit 13 - the raw-depth-fetch unit, and the prime suspect from the previous
leg - holds `tex=64`, a depth format, `complete=1`. It is exonerated as a
dangling binding.
- **The crashing draw's state is identical on both stacks.** Its 14 tuples match
the passing Mesa run exactly, and Mesa executes the very tuple
`loc=35 type=GL_SAMPLER_2D unit=13 tex=64 complete=1` **45 times** without
crashing. ANGLE dies the first time it reaches it.
So nothing in the sampler->unit->texture mapping distinguishes the crash. What
the diff did turn up is one genuine divergence, ~80 tuples upstream: ANGLE's
compiler reports an **extra live sampler uniform** that Mesa's does not, and it
aliases a unit already in use -
```
angle: loc=3 type=GL_SAMPLER_2D unit=0 tex=76 <- both stacks
angle: loc=14 type=GL_SAMPLER_2D unit=0 tex=76 <- ANGLE only
mesa : (no loc=14; stream continues one tuple ahead from here)
```
Two sampler uniforms resolving to the same unit is the subject of
`b6a2bf08 [Fix] (DirectGLES): resolve an aliased texture unit by the sampler's
type`, which sits **inside** the `8025745f..5a400e02` regression window - the
first concrete link between that window and this crash. It is a lead, not a
proven cause: it is upstream of the fault and the two runs still agree on the
crashing draw itself.
**Fix shape: avoidance, not correctness.** Nothing here shows MobileGL in the
wrong - our state at the faulting draw is exactly the state a passing driver
handles 45 times over. Sampling is required to be safe whatever the binding, so
the NULL-deref is the driver's bug. Any change on our side is therefore working
around a driver defect and should be scoped and named as such, ideally behind
the `--avoid-angle-llvmpipe-*` family rather than in shared backend paths. The
ingredient the probe did **not** capture, and the next thing to instrument, is
the *sampler object* bound to the unit: unit 13 takes a depth-format texture
through a plain `GL_SAMPLER_2D` with the raw-depth-fetch sampler override, while
unit 12 takes another depth texture as `GL_SAMPLER_2D_SHADOW` in the same draw.
One more caveat before attributing anything: the emulator SDK's ANGLE is not
the ANGLE the Android lane runs. The CI lane uses a pinned build
(`MOBILEGL_TRACE_ANGLE_VARIANT`, default `ec889e6ea831`) whose version and
extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2
entry points). Compare `GL_RENDERER` and the relevant extension lists on both
stacks before treating a local result as a statement about CI.