Commit Graph
47 Commits
Author SHA1 Message Date
swung0x48 500b63ea58 [Perf] (Magma): a renamed wire store reuses its retired buffers once their submissions complete (FCL inproc apply thread 1.74 -> 1.61 ms/frame at cpuhunt). 2026-10-08 15:26:24 -04:00
swung0x48 748ffc31c5 [Perf] (Magma): small busy wire buffers are renamed from a host shadow instead of a staged copy that ends the pass (FCL WriteWireBuffer 0.100 -> 0.077 ms/frame). 2026-10-08 06:59:40 -04:00
swung0x48 855b3b5009 [Perf] (Magma): converted wire vertex streams are cached across frames and default-framebuffer clears stay standalone (BSL Magma 19.5 -> 77.2 fps, MobileGlues 78.2). 2026-10-08 06:59:39 -04:00
swung0x48 785df78d77 [Perf] (P14): one UBO-ring lookup per draw, split residual copy/stamp loops, one context read for GPU-write marks, Magma slice stamps from the owner's counter (rd12 monolith Espryt 88.8 -> 90.3, Magma 69.8 -> 70.3 fps). 2026-10-07 16:06:02 -04:00
swung0x48 032dcd240e [Perf] (Magma): the wire vertex-input layout is kept on its vertex-elements record and buffer handles resolve through a slot-indexed front. 2026-10-07 14:58:22 -04:00
swung0x48 e6281b26de [Perf] (Magma): wire texture and buffer lookups get direct-mapped fronts and a continued pass stops re-marking its attachments (rd12 monolith 59.6 -> 60.5 fps). 2026-10-07 11:25:02 -04:00
swung0x48 de6ab16756 [Refactor] (P13): the legacy BufferBackendOps table, Magma's never-populated texture/render-pass/renderbuffer caches and VkClearManager go, and the buffer mock tests drive the resource op table instead (ratchet 36 -> 27). 2026-10-06 21:39:39 -04:00
swung0x48 7fb18840dc [Refactor] (P13 W5 S8): MOBILEGL_BUILD_RECORD_ARM is folded away - the record arm is unconditional - and retired_switches keeps it from coming back. 2026-10-06 15:03:11 -04:00
swung0x48 b13af729e4 [Refactor] (P13 W5 S5): the record arm's three knobs move from IpcTable to RecordArmTable, which every record-arm build parses. 2026-10-06 12:01:49 -04:00
swung0x48 22739a033c [Refactor] (P13 W5 S2): MG_Backend/DirectVulkan's record-arm guards name MOBILEGL_BUILD_RECORD_ARM. 2026-10-06 10:51:26 -04:00
swung0x48 f3a2fd9308 [Feat] (P13 W4): Magma monolith takes the record arm in one flip, with in-process persistent-map donation 2026-10-06 08:59:53 -04:00
swung0x48 58c3d9bde5 [Fix] (P14): latch a lost GPU device per session on Magma and Espryt instead of aborting the whole server 2026-10-02 13:37:58 -04:00
swung0x48 7e76a98d36 [Fix] (DirectVulkan): keep one renderer per session across its surfaces, follow bind_context to each context's surface, and keep the wire resource ops registered while any session lives 2026-10-02 06:24:13 -04:00
swung0x48 d639e54616 [Perf] (P8-D, Magma): the wire arm issues glDraw*Indirect / glMultiDraw*Indirect[Count] as vkCmdDraw*Indirect[Count] from the wire store instead of expanding them on the CPU behind a GPU-idle wait (create-indirect split: 321 waits / ~8 s -> 0), with an INDIRECT_COMMAND_READ barrier once per shader write; windr[] PipeStats counters, WireIndirectDrawScenario on both backends x every arm, notes/p8/D.md 2026-09-29 22:32:49 -04:00
swung0x48 7cde55b1c1 [Feat] (Magma, P11 B2): import a T0 store through VK_ANDROID_external_memory_android_hardware_buffer as a wire buffer (external VkBufferObject), serial-gated release, done-no-bytes readback, and the sustained-lock POST on the renderer's device 2026-09-29 17:50:50 -04:00
swung0x48 4d20746831 [MG_Backend] (P7): the descriptor memos fold in a wire-store destroy epoch, so a VkBuffer handle the mid-frame reclaim freed and the next mint handed back cannot revive a set baked to the dead store
- M2's defer path destroys a wire store MID-FRAME (the serial-0 park, the sweep's prefix walk, the watermark); before it a wire store only died at the boundary that also clears UniformManager's memos (BeginFrame from Present or a drain). Those memos are keyed on the VkBuffer HANDLE - m_descriptorReuseMemo hashes the VkDescriptorBufferInfo words, FastRebindMemo compares uboBuffer - and on the wire arm the direct-bind UBO and the SSBO descriptor name the store's own handle. A store destroyed at a park, its handle value re-minted by the next Create (a heap pointer under lavapipe, LIFO out of glibc's tcache), and a draw resolving the same (handle, range) reused a set whose descriptor still pointed at the dead store's memory: silent wrong bytes, no Fatal (ID-P7-43, the fable review's must-fix).
- VkBufferManager gains m_wireStoreDestroyEpoch, bumped on every path that destroys a wire store's VkBuffer (DeferWireRelease's immediate destroy, SweepDeferredWireReleases once per sweep that freed anything, DestroyAllDeferredWireReleases, Shutdown's record teardown) and never reset, with GetWireStoreDestroyEpoch(). UniformManager mixes it into the descriptor-reuse signature and records it in FastRebindMemo, refusing the fast rebind when it moved; all under MOBILEGL_BUILD_DISAGGREGATED. Deliberately not m_sliceEpochCounter, which every WriteWireBuffer bumps and would defeat the memo on every glBufferSubData. On the wire arm only the signature is reached (SetupWireDraw passes no sampler hint), the FastRebindMemo field is the defensive half.
- Two comments from the same review: SweepDeferredWireReleases no longer claims GetSyncPointSubmitIndex() never decreases - it steps back by one across an abandoned recording (RecreateSwapchain, the minimized Present), and the prefix walk is conservative across that, not wrong; and the header's "floor is not a proof" keeps its decision on its first reason alone, the second was a stale citation of a fix that is on the integration tree.
- Section 9 partition exception for UniformManager.{h,cpp} granted by the integrator for this fix.
2026-09-22 22:51:20 -04:00
swung0x48 f0356dbe61 [MG_Backend, MG_IntegrationTest] (P7): the wire watermark's sync point also fires above 1024 parked stores, so small orphans that never reach the byte budget are bounded by count
Measured on bsl-esc-menu-854 (spawn, lavapipe) with the byte watermark alone: the dead VkBuffers
at the end fell from 25,935 to 32, but one stretch with no submission still parked 12,498 stores
in 39.5 MB - under 64 MiB, so nothing forced a sync. A fixed ceiling
(VkBufferManager::kWireDeferredCountCeiling = 1024, not a knob) now rides the same
WaitForSubmitIndex call; MOBILEGL_IPC_WIRE_DEFERRED_MB=0 still turns both triggers off.
MagmaWireReclaimScenario gains the case: 3000 respecify-and-draw rounds of a 256-byte store in one
frame (750 KB, far below the lane's 8 MiB). Red before this commit on inproc and spawn (live peak
3001 vs a bound of 1027), green after (1025, two forced syncs).
2026-09-22 21:46:08 -04:00
swung0x48 9328bdc735 [MG_Backend, MG_Util, MG_Test, MG_IntegrationTest, scripts] (P7): integration-magma-split and -spawn carry MagmaWireReclaimScenario, which reads the server's wbuf[] gauges and reds on a wire arm that keeps orphaned buffer stores until the frame ends
Rule J's red-once for M2. The server role's VkBufferManager publishes four PipeStats gauges
(push builds only, so the pull image is untouched): wire buffer records alive, and the run
maxima of live wire VkBuffers and of parked bytes, plus the watermark's forced-sync count -
maxima because the defect lived inside one frame and a swap-time sample reads after the frame
boundary's own sweep. Two cases, each inside a single frame: 1024 undrawn respecifies of one
4 KiB buffer (live peak <= 4 x wire buffers), and 48 respecify-and-draw rounds of a 1 MiB store
under the lane's MOBILEGL_IPC_WIRE_DEFERRED_MB=8 (parked peak <= budget + one store, every strip
its own colour). Red at base (ec46a550's defer path, counted): 1025 vs 4 live, 49,287,168 vs
9,437,184 bytes and 49 vs 11 live, on both arms; green after: 1 live, 9,437,184 bytes / 9 live
with 5 forced syncs. The knob's negative control (=0) reds the second case with the same
49,287,168 bytes. Inproc and spawn only: the tcp server is the lane-wide fixture supervisor,
whose environment and log the entry cannot reach - named in spawn_lane_parity.py as
MAGMA_NOT_ON_TCP (the ID-P7-14 shape), and PipeStatsTest pins the four short names.
2026-09-22 21:46:08 -04:00
swung0x48 c569c916c0 [MG_Backend] (P7): when the orphaned wire buffer stores still parked exceed MOBILEGL_IPC_WIRE_DEFERRED_MB (64 by default), the server takes the host-access sync point mid-frame and reclaims them all
The defer-path sweep frees an orphan once its last submission has retired, but inside one long
frame a store that was drawn from is named by recorded, unsubmitted work, and nothing submits it
until the frame ends. The knob is read once through MG_Config::Ipc like its siblings; above it
the park calls WaitForSubmitIndex(GetSyncPointSubmitIndex(), flush) - the call
WaitForWireBufferHostAccess makes - and sweeps again, which empties the list because no entry can
be tagged past the sync point just waited out. A sync point that cannot complete dies through
the existing Fatal{ResourceUnavailable, "buffer-write-sync"} site, now one helper shared with the
host write (fatal_census: 79 abort sites, unchanged). 0 is the negative control.
2026-09-22 21:44:06 -04:00
swung0x48 b81a8827b5 [MG_Backend] (P7): a wire buffer store that glBufferData orphans is destroyed on the defer path as soon as no GPU command can still name it, instead of waiting for a frame boundary
RespecifyWireBuffer (every wire glBufferData) and DestroyWireBuffer parked the old VkBuffer in
the per-frame-slot buckets, which only BeginFrame and the idle drain empty. A snapshot-exiting
pbuffer replay delivers one present per run (ID-P7-32), so on bsl-esc-menu-854 the split server
held 25,923 dead stores. The wire arm now keeps its own flat list: a store whose last-use serial
is 0 (never named by a GPU command) is destroyed at the park; any other is tagged with the
renderer's sync-point submit index and destroyed by the sweep every park runs, once that index
has completed or the renderer is idle. The frame-serial floor is deliberately not a proof here
(it cannot move mid-frame, and on this base it is the floor B3 found unsound). The transient
arena's releases and the monolith OnRespecify path are unchanged.
2026-09-22 21:44:06 -04:00
swung0x48 cba0c3a110 [MG_Backend, MG_IntegrationTest] (P7): the busy predicate's comments name the floor as the guarantee and the submission term as a probe, and the term is consulted and re-stamped wherever the serial is
Review ID-P7-34, item 3. 02bb5789 presented the submission term as the fix
and still carried the every-8th-drain mechanism that magma-b3.md §2.4
refutes (the probe saw one successful drain and zero boundary works across
the OpenRA replay). The term is not sound alone: lastUseSubmitIndex is
stamped at AcquireWireSlice, but a mid-draw flush (BindProgramUniformBuffers
-> SyncWireTextureShape's preserve path -> FlushWirePendingCommandsFor-
TextureUpdate) can submit the stamped index without the draw, so the stamp
names S1 while the draw rides S2. The guarantee is the floor in
OnSubmitsCompletedUpTo; the term is a defence and a device probe.

- WriteWireBuffer's block comment, the WireBufferResource field comment,
  the AcquireWireSlice stamp comment, the StreamedBufferSubData CMake
  comment and the F1WireScenario case comment are rewritten on that
  footing; the drain story is gone. The drain probe's comment now says it
  refuted the drain as the mechanism.
- WaitForWireBufferHostAccess's early return also requires the submission
  term complete, so a caller with the serial term idle cannot skip a wait
  the term demands.
- ReadWireBuffer re-stamps lastUseSubmitIndex beside lastUseSerial after
  its wait, preserving the pending draw's reservation in both halves.

All disaggregated-only code. OpenRA DirectVulkan SPLIT / SPAWN retraces
ssim 1.000000, 0 px, zero MGWIRE-FLOOR lines; integration-magma-split 90/90,
-spawn 69/69, -tcp 70/70.
2026-09-22 20:25:11 -04:00
swung0x48 c9164f25d3 [MG_Backend] (P7): a streamed buffer write waits for the submission that reads it, not for a frame count
WriteWireBuffer answers a resource_subdata either with an ordered vkCmdCopyBuffer or with an
immediate host memcpy, and it chose on `lastUseSerial > GetCompletedSerial()` alone.
GetCompletedSerial is max(m_frameSerial - frameCount, m_completedSerialFloor) and its first term
is PURE COUNTING - it asserts that work from frameCount serials ago has finished with no fence
behind it. On the wire arm m_frameSerial is advanced not only by Present but by
TryDrainFrameTransients -> VkBufferManager::BeginFrame on every 8th drain, which happens
MID-FRAME, and package B established that mid-frame pooled-fence submissions are not covered by
frame.imageInFlightFence. So the predicate could answer "idle" about a buffer that a
recorded-or-in-flight draw still reads; the write was then applied unsynchronised and the draw
executed reading somebody else's vertices - drawing another batch's geometry, or nothing at all,
with no log line and no GL error.

That is OpenRA's frame: ~24 glBufferSubData -> glDrawArrays pairs over the SAME range of one
GL_DYNAMIC_DRAW buffer that is never orphaned, and a picture missing exactly one batch's quads.

The predicate now has a second, fence-backed term: WireBufferResource carries lastUseSubmitIndex,
set at every draw bind and staged copy to GetWireNextSubmitIndex() (m_submitCounter + 1 - what
RetireWireObjects already tags future objects with, and deliberately NOT GetSyncPointSubmitIndex,
which answers m_submitCounter when nothing is recorded yet, exactly the hole a draw acquiring its
vertex slices before BeginCommandRecording falls into), and IsSubmitIndexComplete reports an
unsubmitted index as incomplete. The counting term is kept because it only ever over-reports
busy; the two are OR'd, so the answer is monotonically safer. GetCompletedSerial itself is
untouched - it also serves the monolith VkBufferResource path and changing it would move the pull
build's .text.

MGITEST_MAGMA_FORCE_STALE_BUFFER_SERIAL forces the counting term to answer "idle" (all writes, or
the Nth one alone), which is the state a mid-frame idle drain puts the device into and which no
host lane reaches by itself. It forces only the lie, not a bypass of the ordered copy, so it stays
a valid red-once across this fix.
2026-09-22 18:44:07 -04:00
swung0x48 2207b6d9ee [docs, MG_Backend, MG_IntegrationTest] (P7): package A's eight refusals are on record, and only real sites still carry @P7
notes/p7/magma-a.md: the per-site table (retired / declined / kept-fatal with the
reason), every red-once with its captured output, the gate readings, and what is
handed on.

The three places package A's own prose still named `uniform-buffer-byte-tail@P7`
lose the marker. They are comments, not refusal sites, and leaving the marker in
would have made a tree-wide `grep @P7` count them against packages B and C. The
census now reads 15 -> 7, and all seven are real: WireFramebuffer.inc x4 and
VulkanRenderer.cpp x1 (package B), WireDraw.inc x2 (package C).

Three findings worth the integrator's attention, all in the note:

1. Most of this cluster was unreachable, and that is provable rather than
   assumed. The frontend validates buffer-range offsets against the very caps the
   backend then re-checks - GL_{UNIFORM,SHADER_STORAGE,TEXTURE}_BUFFER_OFFSET_ALIGNMENT
   and GL_MAX_UNIFORM_BLOCK_SIZE are all published from the same Vulkan limits -
   so five of the eight sites could only ever have fired on a malformed peer. The
   two that CAN fire are `uniform-buffer-byte-tail` (GL constrains a range's
   offset but not its size) and `unaligned-atomic-buffer-range` (GL gives
   GL_ATOMIC_COUNTER_BUFFER no queryable alignment, so the frontend only enforces
   offset % 4 while the backend needs minStorageBufferOffsetAlignment).

2. fatal_census does not move for any of this, and no re-baseline is due. Wave 0
   already routed these funnels through MGPipeSessionFailHook, so the abort lives
   in FatalFunnel.cpp and retiring a WireDescriptorFatal call site changes no
   census number: 79 abort sites / 43 family words, unchanged, RC=0. The brief
   expected the count to fall because the sites were bare aborts before wave 0.

3. DirectVulkan.Split.Full.IterationRPProgram203Scenario.FixedCompleteInputProducesFixedCompleteGoldenOutput
   is red on this worktree's build-split and is NOT ours: it reproduces 3/3 with
   package A's backend diff fully reverted, and the mismatch bits are identical
   to the S1 exposure texel in magma-two-process-first-run.md. The contract books
   that one against the tcp arm, and ~/w7/pipe's own build-split passes it, so
   whether it is a split-arm regression between 460a3a72 and 610fa281 or a
   configure-dependent S1 needs the integrator.

Gates on the finished branch: integration-magma-split 75/75, -spawn 54/54, -tcp
56/56, -buffers 21/21, full-split 512 entries 454 PASS / 57 SKIP / 1 red (the
pre-existing one above; base was 453/57), ctest -L unit 2408/2408,
spawn_lane_parity RC=0, link_ratchet unchanged at 186, fatal_census RC=0, and G1
pull .text = a52203 with nm --defined-only identical to p7-before/pull-syms.txt
(added=0 removed=0).
2026-09-22 16:22:40 -04:00
swung0x48 450d30759e [MG_Backend, MG_IntegrationTest] (P7): a uniform range that ends mid-word is copied, not refused
`uniform-buffer-byte-tail@P7` is retired. It was the live boundary the Redmi
Minecraft run reported (notes/p5f/magma-inproc-fix.md #5) and, since wave 0, a
named Fatal that took the session with it.

Only the SIZE of a uniform range can land off a word. glBindBufferRange already
forces the offset onto GL_UNIFORM_BUFFER_OFFSET_ALIGNMENT and records
GL_INVALID_VALUE otherwise (GL_Buffer.cpp ValidateBufferRangeOffsetAndSize), and
every device reports that as a multiple of four - but GL 4.6 core 6.1.1 puts no
such rule on size, so a 19-byte bind is legal and 19 % 4 is not 0. The sub-word
start is only reachable protocol-side; the new arm handles it anyway rather than
leaving a second shape to trip over later.

CopyWireBufferSubWordRangeToSlice rounds the read of the APPLICATION's store OUT
to whole words, clamped to the store's end so the widened window can never touch
a byte the application does not own, and lands it in a transient staging slice.
Only the staging -> padded-block shift is sub-word, and that happens inside our
own arena; vkCmdCopyBuffer places no alignment rule on a region's offsets or size
(unlike vkCmdUpdateBuffer / vkCmdFillBuffer), so the shift is a plain legal copy.
The block still starts at the padded slice's own offset, which is what lets the
descriptor keep a minUniformBufferOffsetAlignment-aligned dynamic offset.

red-once (R-16), all three Magma two-process arms, before the backend half:

    DirectVulkan.Split.Buffers.F1WireScenario.SubWordUniformBufferRangePadsMissingBytes
    DirectVulkan.Spawn.Buffers.F1WireScenario.SubWordUniformBufferRangePadsMissingBytes
    DirectVulkan.Tcp.Buffers.F1WireScenario.SubWordUniformBufferRangePadsMissingBytes
        Subprocess aborted - MGPipe: Fatal{UnmigratedVerb, "Magma:uniform-buffer-byte-tail@P7"}

and green after it. The case is the sibling of
ShortUniformBufferRangePadsMissingBytes with a 19-byte range; the byte just past
the range is poisoned, so a rounding that leaks it reads tail.x as 0.5f and the
red channel comes back 127 instead of 0.

Lanes: integration-magma-split 74 (was 73), -spawn 53 (52), -tcp 55 (54),
integration-magma-buffers 20 (19), all green. fatal_census unchanged at 79 abort
sites / 43 family words - these refusals stopped being bare aborts in wave 0, so
a retirement moves no census number and needs no re-baseline. link_ratchet
unchanged at 186.
2026-09-22 16:22:40 -04:00
swung0x48 a7d7e2bc44 [MG_Backend] (DirectGLES/DirectVulkan): the stencil-only scratch attachment and the aspect staging buffer's lifetime
Three follow-ups the first pass needed:

DirectGLES glGetTexImage has to attach a STENCIL_INDEX texture at
GL_STENCIL_ATTACHMENT. It went through the colour branch, which left the scratch
READ framebuffer incomplete for a stencil-only texture, so the readback bailed out
before the new stencil branch could run. ScratchFBOImpl::EnsureDepthAttachment2D
gains the stencil-only point; the shadow remembers which of the three points it
holds so a later depth attach is not deduped against it.

DirectVulkan's aspect staging buffer is recorded into the current frame's command
buffer and outlives the glBlitFramebuffer call that records it, so it retires with
the frame. VkBufferManager::DeferRelease is the manager's own deferred-release
entry point and moves to its public section for that.
2026-09-20 21:34:14 -04:00
swung0x48 fbb7873fbb [MG_Backend] (Disaggregated): consume Magma application buffers from server resources 2026-09-20 02:37:02 -04:00
swung0x48 fff9d639f0 [Fix] (DirectVulkan, tools/trace_replay): preserve resident buffer ordering and coherent trace copies - fix intermittent geometry and UI corruption in Minecraft 26.3-rc-3
Stage updates to adopted GPU-resident buffers on the command timeline instead of overwriting memory that earlier draws still read. Wait for submitted copies as well as pending commands before CPU readback.

Commit coherent mapped writes before Android trace buffer copies, and retire buffer shadows on deletion so copies see current vertices and name reuse cannot leave dangling dirty-shadow entries.

Validated draw ordering, submitted-copy readback, traced buffer copies, and mapped-buffer deletion/name reuse on Redmi Adreno 830. Minecraft 26.3-rc-3 Magma passed movement, HUD, and menu checks and exited normally; the new trace parsed all 64,410 frames without warnings.
2026-09-16 01:22:11 -04:00
swung0x48 59f7059bf4 [Fix, Test] (DirectVulkan): keep the draw when a program declares an image-backed resource the application left unbound 2026-08-24 05:31:15 -04:00
swung0x48 4446c861be [Fix, Test] (DirectVulkan): keep the draw when a program declares a storage block the application left unbound 2026-08-24 04:57:05 -04:00
swung0x48 f5b8a505ed [Fix] (MG_Backend/DirectVulkan): keep the transient arena's outgrown buffers alive until frame rewind - the transient drain freed them while frame-serial memos still bound them 2026-08-10 21:47:28 -04:00
BZLZHH f2d210b12d [Perf] (MG_Backend): memoise DirectVulkan's per-draw vertex binding resolution
Every draw re-resolved its whole vertex binding array: for each enabled binding,
look up the buffer, acquire a slice from the buffer manager, apply the binding's
base offset, fill the VkBuffer and offset arrays, bind. In the Minecraft-shaped
benchmark the same few hundred vertex array objects cycle for the whole run and
each one's answer is stable, so UploadAndBindVertexBuffers was the single largest
cost in the backend at 7.9% of the render thread, with AcquireResidentSlice
another 3.8% underneath it.

The resolved array is now kept per vertex array object and revalidated instead of
rebuilt. Validation is two-tier. The vertex array's own configuration version
already invalidates its backend vertex-input state, so a changed attribute,
format, buffer or base offset yields a different state object - the memo compares
both that object's address and its hash, which mixes the bound buffers and the
whole layout. What that does not cover is the slice moving underneath an
unchanged configuration, so the buffer manager now carries a monotonic epoch that
every writer of slice-deciding state bumps: resident storage creation, respecify,
sub-data, flush of a mapped range, the promotion and demotion between streamed
and resident storage, each fresh arena allocation, and bulk release. The counter
is manager-wide and never reset, so a resource created at a recycled address
cannot reproduce a value some memo still holds.

The miss path was the thing to get right, because the previous attempt in this
area regressed the texture-upload and sampler-churn cases by 60-85%: it added a
verification pass that re-ran the resolution work it was trying to skip, so every
miss paid for it twice. Here a miss is one pointer-keyed lookup and a few stores,
and nothing else runs that the full path would not have run anyway.

ns per draw, DriverBench on a GTX 1660 SUPER: mc_ubo_range 924 -> 767,
mc_vanilla_draw 1346 -> 1227, mc_sampler_churn 1397 -> 1279,
mc_sodium_multidraw 3365 -> 3266. Magma is now 4.1x the native driver on the
per-draw uniform-range case, from 5.4x when this round started. No case
regressed on either backend.

Unit tests 421/421.
2026-08-06 09:24:08 -04:00
BZLZHH 57aeeec053 [Perf] (MG_State, MG_Impl, MG_Backend): stop paying per draw and per upload for work already known
A per-draw CPU profile of a real Minecraft frame (perf on the render thread,
which sits at 100% of one core on both backends) said the deficit is translation
overhead, not the GPU, and named where it goes. This removes the largest items
it found, on both backends and in the shared frontend they both feed.

The single biggest one was not translation at all: IsBackendContextCurrentOnThisThread
called eglGetCurrentContext on every invocation, and glvnd answers that with a
getpid() fork check - a real syscall. The predicate sits two and three deep in
every draw (the deferred-release drain, the global-UBO ring availability check,
and the ring allocation), so it accounted for 16.3% of the render thread. EGL is
still the ground truth, but re-verifying it once per thread per frame catches an
external migration at the next frame boundary rather than the next call, which
recovers the same bookkeeping.

Texture uploads now carry a dirty region instead of a per-level flag. Minecraft
animates atlas sprites with 16x16 glTexSubImage2D calls into a 1024x512 atlas
and respecifies the lightmap every frame; a per-level flag turned each of those
into a full-level re-upload - about 3.6 MB a frame of texels nobody changed.
MipmapStorage accumulates the written box, Espryt uploads it with
UNPACK_ROW_LENGTH striding into the level shadow, and Magma stages just that box.
The box is a union, not a range list: repeated writes to one level widen it and
it degrades to exactly the old whole-level upload, which is the honest worst case.

glBufferData(NULL) is the orphaning idiom, and the backend was answering it by
uploading the stale CPU shadow - turning a rename the driver does for free into
a full synchronized upload. BufferObject now records that a NULL respecify leaves
the store undefined, and the upload is skipped until content is actually written.

The rest are smaller and of a kind: the deferred-release queue is probed without
taking its mutex, the UBO ring waits on the frame fence that frees the space it
needs instead of draining the whole pipeline with glFinish at the size cap, VAO
binds go through a shadow so a draw's second bind of the same object does not
reach the driver, the per-draw clean-texture probe short-circuits on the content
version before rebuilding shape info, glUniform drops byte-identical writes
(which otherwise dirty the whole UBO for the next draw), re-binding the texture
or VAO a slot already holds no longer bumps the generation counters a backend
fast path is keyed on, and the texture validators stopped taking shared_ptr by
value.

On Magma: descriptor-set reuse keeps four entries instead of one, because draws
alternating between two programs - the chunk/entity ping-pong - thrashed a single
slot into a full re-allocate and re-write every draw; a DynamicDraw buffer whose
contents survive two frame boundaries is promoted to resident storage instead of
being re-copied into the per-frame arena forever; and sampled-read barriers name
only the shader stages whose device feature is enabled, which also removes a
latent VUID violation (ALL_GRAPHICS names geometry and tessellation stages a
device need not have).

Measured with the Minecraft rig (render distance 32, p50 fps, same machine,
single sample each): vanilla 1.21.1 Espryt 10.8 -> 36.3 and Magma 31.3 -> 44.6;
26.2 snapshot Magma 114.5 -> 210.5. Fabric+Sodium moved inside noise on Magma
(854 -> 766) with the native baseline itself moving 838 -> 1031 between the two
sessions, so treat that cell as unresolved rather than a regression measured.
Unit tests 421/421. The CTS A/B was not run: these numbers and the test suite are
the whole of the evidence, and a conformance regression would not have been
caught here.
2026-08-06 06:24:37 -04:00
BZLZHH d39a706d57 [Perf] (MG_Backend): stop paying for descriptor slots and mip barriers nobody asked for
Five independent bits of per-draw and per-operation waste in the DirectVulkan
backend, all removing work whose answer was already known.

The per-draw descriptor walk iterated all 256 slots of bindingKinds to find the
one to eight bindings a real GL program declares, because that vector is sized to
the binding cap rather than to the program. Reflection now records the bindings it
actually assigned, and the draw path iterates that. It is built at the end of
ReflectLayout, not where bindingKinds is sized - at that point the vector is only
zero-initialised and the kinds are assigned further down, so a list built there
would be empty. It has to stay ascending: Vulkan consumes pDynamicOffsets in
binding order and the writer pushes them in iteration order, so an unordered list
would silently mis-pair dynamic offsets with their uniform blocks.

Descriptor pools were sized maxSets * the 256-binding cap, declaring 81,920
descriptors per pool and 245,760 across the frames in flight, for sets that hold
what shader reflection found. Sized from eight now; an outlier program is absorbed
by the VK_ERROR_OUT_OF_POOL_MEMORY path that already exists, which works because
pool sizes are aggregate budgets rather than per-set limits.

TrackLiveResource swept the whole live-buffer vector on every insert once it
passed 256 entries, and when the buffers are all live the sweep removes nothing
and the vector grows by one - so creating N live buffers cost about N^2/2
expired() checks. It sweeps on a doubling watermark now, with the same
reclamation semantics.

GenerateMipmap transitioned each destination level individually inside its loop,
but every generated level starts in the same layout and the loop only moves a
level out of TRANSFER_DST after writing it, so the whole range can be prepared in
one barrier - 3(N-1)+1 barrier commands become 2(N-1)+2. Each level is still
transitioned to TRANSFER_SRC before it is read, so the dependency between
consecutive levels is unchanged.

WaitForFrameSerial drained the entire graphics queue, as its own comment admitted.
Every submission records the frame serial it was made under, so it now waits on
the first fence at or past the requested serial. The narrow path deliberately does
not call NotifyDeviceIdle(): that claims every submission has retired, which is
only true after a real drain, so it stays on the fallback.

Verified with an 8213-case A/B (textures, buffers, queries, mipmaps, uniforms and
the whole direct_state_access suite): the Espryt failure list is identical, the
Magma failure list differs by one case, and both crash sets are unchanged on
Magma. That one case, buffer_storage.map_persistent_draw, does not reproduce in
isolation - running the buffer_storage group alone gives byte-identical results on
both builds (the same three failures, not including it), and it reports
NotSupported when run on its own. It is the same ordering-dependent behaviour this
suite shows elsewhere, and the three Espryt crash-set differences are the known
copy_image cluster moving chunk position. Flagging rather than hiding it.

direct_state_access stays at Espryt 370/371 and Magma 371/371; unit tests 421/421.
2026-08-05 15:19:37 -04:00
BZLZHH c069890ac7 [Feat] (DirectVulkan): GPU transform feedback capture via VK_EXT_transform_feedback
Second stage of GL 3.0 transform feedback: captured draws now write real
data.

- Device setup enables the VK_EXT_transform_feedback feature when present
  and loads the bind/begin/end entry points.
- Captured draws compile an XfbCapture program variant whose last
  vertex-processing stage gets XfbBuffer/XfbStride/Offset decorations from
  the program's resolved varyings (a new spirv-opt pass). A captured
  gl_Position is mirrored into a dedicated output written before every
  OpReturn - or before every OpEmitVertex in a geometry stage - ahead of
  the position fixup, so the captured value is the shader's own pre-remap
  position.
- DrawArrays/DrawElements wrap the draw in Begin/EndTransformFeedbackEXT;
  a small counter buffer resumes the append position across draws within
  one glBeginTransformFeedback (fresh Begin starts at the bound offsets).
- Capture targets are promoted to persistently-mapped host-coherent GPU
  storage (persistent-map storage now also carries the transform feedback
  usage), so MapBuffer/GetBufferSubData read the captured bytes after the
  fence wait glEndTransformFeedback now performs.
- Draw-mode/feedback-mode validation defers to the geometry shader's
  output primitive when one is present, and glGetBooleanv reports
  GL_TRANSFORM_FEEDBACK_ACTIVE/PAUSED so dEQP's per-case state reset can
  unwind an active capture.

KHR-GL33: transform_feedback capture_vertex_*/capture_geometry_*/
discard_*/draw_xfb and clip_distance.coverage now pass; queries
(PRIMITIVES_WRITTEN) and gl_ClipDistance capture remain.
2026-07-31 17:31:41 -04:00
swung0x48 c540fb88ee [Fix] (DirectVulkan): drain frame transients on present-less paths - readback waits, suspended presentation, blocking sync waits and flush completion polls now run Present's per-frame drains (deferred buffer/texture releases, transient arena rewind, descriptor cursors, retired command buffers, conversion caches) whenever every submission is provably complete, so offscreen/minimized workloads stay bounded; never blocks, frames-in-flight overlap untouched 2026-07-27 21:45:21 -04:00
swung0x48 9ed5dbf483 [Perf] (MG_Backend/DirectVulkan): return VkBufferResource by raw pointer from GetOrCreateResource to drop per-draw SharedPtr refcounting 2026-07-16 18:34:55 -04:00
swung0x48 c1743aa42d [Refactor] (MG_State, MG_Backend): PipeResource storage layer + zero-copy coherent persistent maps
Introduce a Mesa pipe_resource-style PipeResource that owns a GL buffer's bytes
and its backend GPU resource, abstracting WHERE the authoritative bytes live:
 - Shadow mode (non-persistent buffers): a CPU Vector; the backend keeps its own
   GPU copy in sync via BufferBackendOps, exactly as before.
 - Persistent mode (coherent GL_MAP_PERSISTENT maps): the backend's host-visible,
   COHERENT, persistently-mapped GPU memory is the single source of truth. The app
   writes into it directly, every reader resolves against it, and NO per-write
   backend transfer happens. The CPU shadow is released.

BufferObject no longer owns a raw shadow Vector; it holds a PipeResource and
exposes one accessor, MappedData(), that all readers go through. Every buffer-data
consumer (UBO payload, PBO texture upload, indirect draws, resident/streamed
uploads, both backends) was migrated from GetDataReadOnly()->data() to
MappedData(), so a persistent buffer's readers see GPU memory - not a stale
shadow. That stale-shadow inconsistency is what corrupted rendering (wrong UBOs ->
misplaced/"lost" vertices) in the first zero-copy attempt (625c8a6, reverted in
896cafc); routing every consumer through one accessor makes it structurally
impossible.

Backends provide the map via BufferBackendOps::AcquirePersistentMap:
 - DirectVulkan: a HOST_VISIBLE|HOST_COHERENT (required, not just requested),
   persistently mapped resident VkBuffer carrying every usage, seeded from the
   shadow, never recreated; AcquireResidentSlice binds it directly.
 - DirectGLES: EXT_buffer_storage immutable persistent+coherent glMapBufferRange,
   falling back to the shadow when the extension is absent.

Fixes the ~7GB GpuMemory OOM + 100%-CPU/ANR running modern Blaze3D Minecraft on
both Magma and Espryt (per-draw whole-buffer re-upload of the coherent persistent
ring buffer), without the coherency/stale-read hazards of the reverted attempt.

BufferTest: zero-copy stress guard (15,360 draws -> 0 per-draw transfers, and every
reader resolves to GPU memory) + a shadow-fallback test. Host suite: 203/203 pass.
Device verification pending.
2026-07-11 20:18:38 -04:00
swung0x48 eb090c6170 [Fix] (MG_Backend/DirectVulkan): fence-backed GL sync objects and per-frame descriptor rewind
- Track every graphics-queue submission with a real fence: pooled fences
  for mid-frame flushes, the frame slot's fence for Present and readback.
  Completion advances a submit counter via vkGetFenceStatus polls,
  slot-fence waits, and device-idle points, and raises the buffer-manager
  serial floor from the frame serial each submission carried.
- GL sync objects now capture the submission index that will carry the
  commands recorded so far; ClientWaitSync honors
  GL_SYNC_FLUSH_COMMANDS_BIT with a mid-frame submit (gated on the index
  still being unsubmitted so poll loops cannot split the render pass), and
  blocking waits flush then vkWaitForFences with the caller timeout.
- FlushPendingCommands retires the submitted command buffer and restarts
  recording on a fresh one; retired buffers are freed once the slot fence
  is next waited, so an executing buffer is never reset.
- Rewind descriptor-set cursors exactly once per frame in Present (after
  the slot-fence wait), plus after the synchronous readback drain,
  replacing the ten lazy per-draw-path rewinds.

Verified: host tests 168/168, trace-replay 70/70.
2026-07-10 10:55:10 +00:00
swung0x48andClaude Fable 5 139de76347 [Fix] (MG_State/MG_Impl/MG_Backend): render Flywheel instanced+indirect on both backends
Create 6 / Flywheel 1.0.6 now renders correctly with both flywheel:instancing
and flywheel:indirect on DirectGLES and DirectVulkan (verified in-game on
Adreno 830: waterwheels and cogwheels solid, animated, correct pairing, no
crashes across all four combinations).

- MG_State/MG_Impl: sync explicitly-ranged SSBO bindings of FLUSH_EXPLICIT
  persistent maps to the backend before compute dispatches. Flywheel writes
  its scatter-copy descriptors into the staging ring's persistent map and
  never flushes that span (UB per spec, works on drivers whose maps alias
  GPU-visible memory); our maps alias the CPU shadow, so the descriptors
  never reached the GPU: the scatter compute copied nothing (GLES: empty
  draw commands) or stale garbage (Vulkan: wild indirect commands ending in
  VK_ERROR_DEVICE_LOST).
- MG_Impl/MG_Backend: real glFenceSync objects backed by backend fences
  (GLES: native ES syncs guarded by context generation and owner thread;
  Vulkan: buffer-manager frame serials), replacing always-signaled stubs
  that let Flywheel reclaim staging memory the GPU still reads.
- MG_Backend/DirectGLES: compute dispatches now run the same per-program
  resource sync as draws (uniform-block bindings and sampler units must be
  re-established through the API because layout(binding) is stripped from
  transpiled ESSL) and rebind texture units afterwards; the cull shader
  used to read a stale _FlwFrameUniforms binding and the depth-pyramid
  downsample sampled a stale unit-0 texture, zeroing the Hi-Z pyramid and
  occlusion-culling all Flywheel geometry. Image uniforms are excluded from
  glUniform1i (ES bakes their unit via layout(binding)); image-unit sync is
  clamped to the device limit; eliminated/SSBO-classified uniform blocks
  are skipped.
- MG_Backend/DirectGLES: gl_BaseInstance in native indirect draws reads the
  GPU-written command buffer through an injected mg_IndirectParams SSBO
  view addressed per draw instead of the zero CPU shadow; layout(binding)
  is preserved for SSBO/image declarations (ES has no API rebinding for
  them); the ES context ownership claim moved to a global atomic owner
  thread with an EGL ground-truth check, and deferred buffer op state is
  mutex-guarded, so ops cannot silently no-op after context migration.
- MG_Backend/DirectVulkan: new RebaseInstanceIndexPass rewrites vertex
  InstanceIndex loads to (InstanceIndex - BaseInstance). glslang's relaxed
  Vulkan mode aliases gl_InstanceID to InstanceIndex, which includes
  firstInstance, but GL's gl_InstanceID is zero-based - draws with nonzero
  baseInstance paired meshes with wrong instance data (cogwheel drawn as a
  waterwheel, another wheel collapsed invisible). Gated on the
  shaderDrawParameters device feature. Sampled-read barriers additionally
  cover the compute stage (the Hi-Z downsample samples the depth
  attachment from compute), and short uniform-buffer ranges keep the
  existing zero-padding.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 06:30:10 +00:00
swung0x48andClaude Fable 5 b8ffd25148 [WIP] Mesa-style buffer overhaul: resource abstraction + immediate transfer ops
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 14:47:51 +08:00
swung0x48 cf165c0db5 [Fix] (MG_Backend/DirectVulkan): support Voxy rendering
Implemented:

- Advertise Voxy-required DirectVulkan extensions without raising the reported OpenGL version.

- Add DirectVulkan compute, indirect draw count, DSA, readback, and buffer state paths needed by Voxy.

Fixed:

- Enable Vulkan shaderInt64 and drawIndirectFirstInstance so Voxy baseInstance-driven LOD draws address the correct section data.

- Fix DirectVulkan synchronization, framebuffer, texture readback, and shader interface handling used by Voxy and Minecraft screenshots.

Tests:

- Add MG_Test coverage for DirectVulkan extension advertising, DSA buffer/texture/framebuffer/vertex-array behavior, persistent mapped readback, and shader/program paths.
2026-06-09 00:37:37 +08:00
swung0x48 5cfe9c8998 [Feat] (MG_Backend/DirectVulkan): support compute shaders 2026-06-05 11:52:10 +08:00
swung0x48 d7e768096e [Fix] (MG_Backend/DirectVulkan): fix Minecraft 26.2 startup crashes
- Support cube map face uploads with cube-compatible images and per-face array layers
- Add uniform texel buffer descriptor support for samplerBuffer bindings
- Cache transient vertex/index buffer uploads per frame to avoid VMA allocation failures
2026-05-10 08:50:55 +08:00
swung0x48 0209c5461f [Feat] (MG_Backend/DirectVulkan): implement naive transient/resident buffer, and heuristics to downgrade resident buffer to transient 2026-03-20 17:39:10 +08:00
swung0x48 7dfc2149d8 [Feat] (MG_Backend/DirectVulkan): unified transient buffer arena 2026-03-20 13:56:12 +08:00
swung0x48 9d1e8bd7cd [Feat] (MG_Backend/DirectVulkan): VkBufferManager transient upload now includes uniform 2026-03-20 13:31:51 +08:00
swung0x48 f2ab50b84e [Feat] (MG_Backend/DirectVulkan): VkBufferManager 2026-03-19 17:44:35 +08:00