Commit Graph
22 Commits
Author SHA1 Message Date
swung0x48 e02e5caa17 [Fix] (MG_State, MG_Backend/DirectGLES): join both link phases on the draw and drain paths, and refuse to bind a program whose SPIR-V never arrived 2026-08-10 06:08:30 -04:00
BZLZHH dcf918b9ee [Perf] (MG_State): adopt in-flight compile jobs across shader objects (P1 stage 6)
~21% of a shaderpack's glCompileShader calls hand different shader objects
byte-identical source; the P0b cache only helps after one finishes, so under
async two workers would run the whole pipeline twice. Now the GL thread
consults a per-context (stage, hash, length, envFingerprint) -> weak-node
map at enqueue and ADOPTS the in-flight (or completed) node instead of
posting a duplicate - a hit is honored only after a full byte comparison
(the hash never decides), a cancel-requested or settled-cancelled node is
never adopted, and no worker ever waits.

Sharing a node makes the unconditional cancel wrong, so release is now
adopter-counted: a plain GL-thread Int (every mutation site is a GL entry
point; the single-threadedness argument and the terminal-early-out that
keeps the count exact are in the header), and the cancel fires only at
count zero AND with no pending link pinning the node (the stage-4
MarkLinkReferenced precedence). Adoption also re-points the object's source
at the node's snapshot so the layer-1 memo's pointer compare stays armed -
without that, an adopter's next glCompileShader would re-enqueue the very
duplicate this stage removes. Both guards are negative-control-proven: each
removed guard fails exactly its own tests. Count discipline was proven with
a temporary hard-abort on underflow/leak across the full suite and retrace
corpus - zero hits.

18 new tests (13 GL-surface incl. shared-node re-source/delete/orphan-sweep
isolation, shared failure logs, 48-over-6 stress with a deterministic
adoption count, flag-off and KHR-suspended zero-adoption guards; 5 direct
map cases incl. fingerprint mismatch and cancelled/expired pruning).
Gates: 538/538 unit both flag states, async suites x5 no flakes, NVIDIA
DirectGLES retrace identical sets both states. Timing: 2-worker
(Android-shaped) 1-3% faster consistently on complementary and BSL;
4-worker unchanged - the win this stage exists for lands where CPU is
scarce.
2026-08-08 20:17:51 -04:00
BZLZHH bd0def6133 [Feat] (MG_Impl, MG_State, MG_Util): the GL_KHR_parallel_shader_compile surface (P1 stage 5)
GL_COMPLETION_STATUS_KHR in both object getters, reading the non-joining
node-direct state - the one query that must never block is asserted never
to reach a join gate. glMaxShaderCompilerThreadsKHR/ARB share one
implementation: a zero count suspends async FIRST and then joins every
outstanding compile and link this context owns (suspend-before-join is the
only order whose post-condition is 'nothing in flight'), a nonzero count
restores; the suspension is a process latch the extension controls, kept
distinct from the configuration flag that gates the ADVERTISEMENT - an app
that turned threading off has not made the extension disappear.
GL_MAX_SHADER_COMPILER_THREADS_KHR reports the thread count. DriverPost
gains the MobileGL-side async row (PASS/INFO naming the env knob) and an
informational host-driver row backed by a new GLES capability probe.

The extension string itself lands per backend in the two follow-up
commits, keeping this one green stand-alone.
2026-08-08 13:17:58 -04:00
BZLZHH 6f8b7fbc40 [Feat] (MG_State, MG_Util): async program linking on the job graph (P1 stage 4)
glLinkProgram with the flag on snapshots its inputs in a GL-thread prologue
(stage-sorted shaders with their compile nodes taken without joining, env,
explicit locations/fragdata/xfb, draw-buffer count), then runs the whole
link body - glslang link/mapIO, SPIR-V, reflection, routing tables - as a
ProgramLinkTask that auto-posts when its last compile dependency settles
(+1-guarded countdown; no worker ever waits on another job). The publish is
one move of the LinkArtifacts block at the join, with the second version
bump so nothing memoized during the pending window survives.

The consume-once TShader claim moved onto the shared compile node as a CAS:
two link jobs racing for one shader resolve to winner-takes-the-parse,
loser re-parses the preprocessed source against the node's own env -
identical SPIR-V pinned by test for 2 and for 12 sharing programs.

Two deliberate corrections to the design's cancel matrix, both test-proven:
attach/detach do NOT cancel a pending link (the snapshot isolates it, and
glCreateShaderProgramv's link-then-detach would otherwise discard its own
result before anyone read it); and a compile node a pending link depends on
is pinned against the orphan-name sweep - the ordinary LWJGL teardown
compile/attach/link/detach/delete used to cancel the dependency and turn a
must-pass link into GL_FALSE.

Continuations are now throw-contained per-item (a stage-3 leftover made
load-bearing by the first real continuation), and the review's deadlock
find is fixed: the dispatch loop no longer cancels a node while holding the
pool mutex, since that cancel can run OnDepSettled -> Post -> same mutex.

Explicit joins: the draw path (GetProgramForDraw, both the pipeline stage
loop and the plain-UseProgram half) and the composite-link site; destroy
paths cancel-not-join; COMPLETION_STATUS readers stay non-joining.

Gates: 506/506 unit both flag states; AsyncCompile/AsyncLink/AsyncTeardown
suites x10 repeats clean both states (teardown with 128 jobs in flight,
then re-Initialize); full NVIDIA DirectGLES retrace flag on twice - result
sets identical to flag off, zero new deltas. Compile-phase prefix-diff,
flag on vs off: complementary-reimagined 5.21s -> 2.16s, BSL 1.72s ->
0.90s - past the design's final acceptance targets before the KHR
extension is even advertised. Default remains OFF until stage 5+7.
2026-08-08 11:58:38 -04:00
BZLZHH e5fb57f7eb [Feat] (MG_State, MG_Util): async shader compilation behind the default-off flag (P1 stage 3)
glCompileShader with MOBILEGL_ASYNC_SHADER_COMPILE=1 snapshots its inputs on
the GL thread (source SharedPtr, CompileEnv, cache handle) and runs the whole
pure pipeline - preprocess, validators, extractors, glslang parse - as a
ShaderCompileTask on the worker pool, returning immediately. Every read of
compile-produced state joins through the single Compiled() gate; links stay
synchronous this stage and join their attached shaders at the top of the
body. Flag off, the path is the same code run inline.

Mechanics: the job node owns all its inputs (no back-pointer, no lifetime
tie to the shader object), so re-sourcing or deleting a pending shader is
cancel-and-drop, never a wait; glslang worker hygiene is a TLS-allocator
scope guard plus GL-thread builtin prewarm (gated on the flag, latch reset
on Destroy so re-initialization re-warms); worker-side diagnostics defer
through the job and replay on the GL thread at the join, enforced by
IsPoolThread asserts in RecordError and an empty-deferred-errors tripwire.
A body that throws publishes a COMPLETE failed compile (status false, real
info log) rather than an abandoned node, and never memoizes away the retry;
a failed enqueue (OOM) cancels the node instead of stranding the joiner -
including inside the dispatch loop, where the in-flight slot is repaid.
The pool StopAndDrains from an atexit sentinel too: workers still inside
glslang parse while exit() ran static destructors was a real 2-in-5 SIGSEGV,
reproduced and fixed (15/15 clean after).

Backend-internal shader objects (default FS, DirectVulkan blit/mipmap) are
cache-less and always compile inline - compile-and-read-in-one-breath needs
no round trip.

Gates: unit suite 488/488 with the flag off AND on (x5); AsyncCompileTest
(12 e2e cases: pending re-source/delete/recompile, byte-identical failure
logs across modes, 48-compile cache stress) x10 repeats clean both modes;
full NVIDIA DirectGLES retrace identical result sets flag off/on (zero new
deltas); compile-phase timing flat as designed (links still serial - the
parallel win arrives with stage 4's async link + stage 5's
KHR_parallel_shader_compile).
2026-08-08 10:33:50 -04:00
BZLZHH c93e5fa409 [Refactor] (MG_State, MG_Util): join-by-construction link/compile artifacts (P1 stage 2)
Still fully synchronous - EnsureLinkJoined()/EnsureCompileJoined() are empty
inline no-ops (verified to fold away at every one of the ~1200 call sites;
this project builds without LTO) - but every read of link- or compile-produced
state now goes through a private accessor the compiler enforces, so when
stage 4 moves the bodies onto pool workers, 'which reads must join' is a
type-system fact instead of a 400-line audit.

- ProgramObject: the 31 fields ResetLinkArtifacts clears plus the 5 link
  outputs it forgot (infoLog, linkedFragData{Location,Index}, the geometry
  strip-capture pair) move into a nested LinkArtifacts behind Artifacts().
  ResetLinkArtifacts is now a worker-safe pure clear; the link-observable
  version bumps (backendState/link/uboContent) move to a GL-thread-only
  BumpLinkObservableVersions() called once from Link()'s prologue and from
  glProgramBinary's mandated failure - the link body never writes them, so
  a stage-4 worker cannot lose an invalidation against the draw path.
- ShaderObject: compile artifacts (TShader, preprocessed source, side-channel
  maps, status/log, consume-once flag) behind Compiled(); the P0b layer-1
  memo trio deliberately stays outside as the future non-joining
  COMPLETION_STATUS_KHR fast path.
- CompileEnv (new): a GL-thread snapshot of everything the compile pipeline
  used to read live from the backend mid-parse - compute limits (the
  GetIntegeri_v reach-back is gone from the worker path), advertised
  extensions, device quirks, TBuiltInResource inputs. Captured lazily per
  backend activation; the consume-once re-parse now runs against the same
  env as the original parse.
- The GL-thread prologue / worker-body boundary is marked in Link() where
  the stage sort ends; everything below is a pure function of the snapshot.

Public getter signatures unchanged - MG_Impl and both backends compile
untouched. Unit 476/476, Program suites 117/117, DirectGLES retrace 38/39 on
llvmpipe (the one failure is the known pre-existing non-CI iterationrp case;
the NVIDIA userspace driver was updated out from under the running kernel
module mid-session, so GLX there is down until a reboot).
2026-08-08 07:12:37 -04:00
BZLZHH 9152e88734 [Perf] (MG_State): dedupe shader compiles by source hash
Iris-style packs hand MobileGL the same source text repeatedly: probed across
three shaderpack traces, 28-32% of all glCompileShader work was redundant -
~9% same-object recompiles with byte-identical source, ~21% distinct shader
objects sharing identical source (the same common GLSL chunk glued into many
program stages). Two layers, both keyed by XXH64 + length with a full byte
compare on every hit (correctness never rides on the hash):

- Per-object: a successful (or failed) compile remembers its source hash;
  glShaderSource with byte-identical text keeps the compiled state and
  glCompileShader on unchanged source returns immediately. Deterministic
  (stage, source) pipeline makes the memo observationally identical to
  recompiling; the consume-once TakeShaderForLink re-parse path is untouched.

- Cross-object: a per-context bounded cache (ProgramState-owned, declared to
  outlive every shader object) shares the preprocessed source, both explicit
  side-channel maps, and the validation verdicts between objects with equal
  source; only the glslang parse stays per-object. Single-GL-thread today;
  flagged for a mutex when compiles go async (P1).

Interleaved A/B on the iterationrp trace (the recompile-heavy pack):
5.65s -> 5.46s median total replay, every round faster; BSL/complementary
stay flat (their duplicate sources are the small common shaders, so calls
drop but wall time is parse-bound on unique sources). Full DirectGLES
retrace, 445-test unit suite, and dedupe-semantics tests (no-op recompile,
invalidation on new source, failed-compile memo, cache bounds) all green.
2026-08-08 04:18:32 -04:00
BZLZHH 81bcbd6c14 [Fix] (MG_State): allocate program and shader names from one shared name space
GL 3.3 core 2.11 puts program and shader names in one name space: a shader
name passed where a program is expected must fail with INVALID_OPERATION,
and vice versa. Two independent IndexGenerators handed out colliding names
(shader 2 and program 2 could coexist), so CheckProgramNameValidity resolved
a shader handle to an unrelated linked program and the error checks in
KHR-GL30.get_uniform_tests.get_uniform were silently swallowed - the case
only ever passed because the collided program happened to reject the queried
location. One shared generator keeps the names disjoint; the per-kind object
tables are unchanged.
2026-08-08 01:25:37 -04:00
BZLZHH 07fa84fb8d [Fix] (MG_State, MG_Impl): defer deletion of the program in use
glDeleteProgram on the current program now only flags it: the name (and
every glGetProgram* query) stays valid until the program stops being
current, at which point UseProgram frees the slot and releases orphaned
attached shaders. Previously the name died immediately, so a second
glDeleteProgram - as issued by common CTS utility teardown - recorded
GL_INVALID_VALUE that poisoned the next test iteration's build
(KHR-GL33.clip_distance.functional now passes its build phase).

glIsProgram/glIsShader piggyback on the same rule: a flagged name is
still a program/shader while it stays GL-visible, which resolves the
long-standing FIXMEs there.
2026-07-31 18:21:14 -04:00
swung0x48 254cf1dc21 [Fix] (MG_Util/ShaderTranspiler, MG_State, MG_Impl/GLImpl, MG_Backend/DirectGLES): GL CTS uniform_block - coerce packed/shared block layouts to std140 at source preprocess (glslang rejects them when targeting SPIR-V; std140 is the only UBO layout the pipeline emits), GL-style block reflection (array "[0]" names, per-element struct-array expansion, unused members and declared-but-unread blocks stay active), vec4-padded GL_UNIFORM_BLOCK_DATA_SIZE, std140 array strides for struct-nested arrays (glslang reflects tight strides there), arrayed-block instances share the first instance member set, glDeleteShader-flagged names stay usable while attached, and backend ESSL emits against highp default precision so relaxed block members match across stages (KHR-GL33.shaders.uniform_block on llvmpipe: 659 Fail -> 828/828 Pass) 2026-07-16 12:09:11 -04:00
BZLZHH 15e24cda78 [Perf|Improvement] (All): Improve performance & optimize code. 2026-02-23 16:00:38 +08:00
BZLZHH 20fbfc2956 [Chore] (LICENSE): Switch project license to LGPLv3. 2026-01-18 09:33:33 +08:00
BZLZHH 3714e275ab [Chore] (All): Update copyright year in source code files. 2026-01-01 00:42:06 +08:00
BZLZHH e7d9da42dc [Chore] (All): Adjust copyright declaration. 2025-12-31 21:49:54 +08:00
BZLZHH 8ed8efe347 [Chore] (All): Add standardized source file header and update script. 2025-12-31 15:45:34 +08:00
BZLZHH ee971306ca [Misc] (All): Run format_code.sh 2025-10-02 15:11:38 +08:00
swung0x48 408b62a495 [Feat] (MG_State/Program): GetAttachedShaders 2025-08-11 23:07:15 +08:00
swung0x48 21a46ee151 [Feat] (MG_State/Program): glShaderSource, glUseProgram 2025-08-11 16:47:18 +08:00
swung0x48 19071c0217 [Fix] (MG_State/Program): typo in ProgramState::CreateProgram 2025-08-11 16:02:42 +08:00
swung0x48 2c40177f06 [Feat] (MG_Impl/Program, MG_State/Program): initial impl of shader related states 2025-08-11 15:19:35 +08:00
BZLZHH 93ca669929 [Refactor] (All): Restructure include system. 2025-08-09 23:10:34 +08:00
swung0x48 4ecd4e4a07 [Feat] (MG_State/Program, MG_Impl/Program): skeleton and initial impl of program/shader state machine 2025-08-09 13:08:26 +08:00