Files
MobileGL/MobileGL/MG_State/GLState/BufferState/BufferObject.cpp
T
swung0x48 1f8de61be1 [MG_Remote, MG_Pipe, MG_Impl, MG_State, MG_Backend, MG_Test, MG_IntegrationTest] (Disaggregated): P5d round 3 - the profile-directed cuts on both roles
- The round-2 report guessed the apply thread's ~48% mutex time was the backend
  pendingMutex; the symbolized call graph at 56a77348 says it is the idle poll in
  ServerLoop::ApplyThreadMain taking m_controlMutex twice per iteration
  (ControlIsPending in the ready lambda, PumpControlRequest), with DrainRing at
  29% of the thread. Every package below is cut from that graph, not from self%.
- T (transport): the control mailbox gets an atomic shadow bit m_controlPosted so
  the idle poll takes no lock (handshake, C2 critical section and the
  NOT_INITIALIZED answer unchanged); Doorbell::Wait spins on a once-calibrated
  iteration budget and reads no clock in steady state (the park phase alone
  computes deadlines); client/server wait and park counts ride the stats line
  as wait[srv= srvpark= cli= clipark=] and the wire ledger as cliwait=/clipark=.
- D (role guards): OnApplyThread() is inline - one relaxed atomic key against the
  caller's thread pointer (aarch64 __builtin_thread_pointer, else
  std::this_thread::get_id) - and m_applyThreadId, the function-static guard and
  the pthread_equal call are gone; the three scope depth counters stop being
  thread_local (their only reader is the apply-thread guard, so they count only
  when OnApplyThread() is true; a GL-thread scope no longer exempts the apply
  thread, pinned by a new case); the guard chains are ordered cheapest-first.
- B (persistent maps): the shadow allocator rounds the SIZE to a page multiple, so
  TrackWriteMap aligns OUTWARD when the shadow is page-granular - a sub-page map
  is one protected page, edges and their per-draw XXH3 disappear; the inward arm
  stays as the fallback. Review found a hole the edge hashes had hidden since
  round 1: a map that faulted but was not consumed by the walk that advanced the
  epoch was never pushed. The walk now drains every marked member before the
  binding walk, registration bumps the epoch so a fresh map ships once on the
  draw path, and a non-4 KB kernel page refuses the mprotect arm by name.
- C (client per verb): an image-unit high-water mark (push build only, fed by
  glBindImageTexture) replaces the 192-unit sweep per draw; the residual fill's
  7-term supplied conjunction is a memo keyed on its four inputs; the server's
  63-field stamp per verb is a constexpr per-class mask table.
- generate_mipmap no longer skips its wait under MOBILEGL_IPC_BATCH_WAITS: its sink
  is the backend GenerateMipmap, which reads MGB_CTX->GetActiveTextureUnit(), and
  a batched publish let the next validate move the serial under it -
  Fatal{UnmigratedPipeInput, "GetActiveTextureUnit@ReadPixels"} on the gate tree.
  The rule is written at the site: a record may skip its wait only if its apply
  reads nothing the residual fill writes. The texture-death scenario gets the
  framebuffer case's fence before it reads the object_death tally.
- Redmi 2f7cbe2e, FCL + Minecraft 26.3-rc-3, VD12, CPU pinned: inproc 64.3 fps
  (client 15.0 ms/frame, apply 12.4) -> 103-114 fps (client 8.7-9.2 ms/frame on a
  mid core, apply 7.0-7.4); monolith 115 (120 Hz vsync-capped runs) / 206 (one
  uncapped run, 5.0 ms/frame on the big core). Gates: unit 2203/2203,
  integration-split 111/111 (the two formerly flaky cases 6/6 on repeat),
  generators, include closure, dirty surface and doc citations green.
2026-09-18 04:28:25 -04:00

1083 lines
58 KiB
C++

// MobileGL - MobileGL/MG_State/GLState/BufferState/BufferObject.cpp
// Copyright (c) 2025-2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
#include "BufferObject.h"
#include <Config.h>
#include <atomic>
#include <MG_Pipe/PipeMutation.h>
#if MOBILEGL_BUILD_DISAGGREGATED
// The client role's persistent-map tracker. MG_State reaching into MG_Remote/Client is the
// layering ARCHITECTURE.md:575 names - the CLIENT role IS MG_State plus MG_Impl - and the
// edge exists only in a build that has the transport at all.
#include <MG_Remote/Client/GpuWritePending.h>
#include <MG_Remote/Client/PersistentMapTracker.h>
#include <MG_Util/Debug/Log.h>
#include <cstdlib>
#include <MG_Util/Metrics/PipeStats.h>
#endif
namespace MobileGL::MG_State::GLState {
namespace {
const BufferBackendOps* g_bufferBackendOps = nullptr;
// Starts at 1 so a zero-initialized cache slot can never carry a live buffer's id.
std::atomic<Uint64> g_nextBufferLifetimeId{1};
#if MOBILEGL_BUILD_DISAGGREGATED
// P5c (hd, CONTRACT-P5C §3.8 / §6 layer 1): the frontend BufferObject's legacy
// accessors are a layer-1 surface. With an active transport, the pre-handle buffer
// arm that reads them (Managers.cpp's RespecifyStorageNow / UploadRangeNow /
// IsBufferDrawClean / the old EnsureBufferResource body) stays compiled but may not
// run: the apply thread calling one is Fatal{RoleViolation, "buffer-legacy-arm"} -
// the same shape as Fatal{PipeLegacyMemosDisabled} - so a cleared subsystem bit 7 no
// longer leaves the arm silently readable. Client-thread callers (the GL thread's
// own state) are unaffected.
// The refusal's BODY, split out from its two probes (P5d round 3, package D). A caller
// that has already asked "am I the server role" for its own reasons -
// SyncPersistentMappedRange does, one line later - calls this and pays for
// ServerLoop::OnApplyThread() once instead of twice. It is a Fatal, so it never
// returns; the probes live in RefuseLegacyBufferArmFromApplyThread below.
[[noreturn]] void FatalLegacyBufferArmFromApplyThread(const char* accessor) {
MGLOG_F("MGPipe: Fatal{RoleViolation, \"buffer-legacy-arm\"} - the apply thread "
"called BufferObject::%s on a frontend object. With an active transport "
"the server reads the applier's resource record and its own staged shadow; "
"a frontend object is client memory (rule E) and the pre-handle buffer arm "
"is monolith-only",
accessor);
std::abort();
}
// The accessors' own guard: two loads and two branches on the client thread, which is
// where MappedData/IsMapped/GetChangeSerial/HasDefinedContent are called from per draw
// (IsLivePersistentMap reads two of them for every live map). PushIsArmed() is a read
// of MG_Config::Transport and OnServerRole() is now one relaxed load of the apply
// thread's key (ServerLoop.h), so the monolith answer costs a load each.
void RefuseLegacyBufferArmFromApplyThread(const char* accessor) {
if (!MG_Remote::Client::PersistentMapTracker::PushIsArmed()) return;
if (!MG_Remote::Client::PersistentMapTracker::OnServerRole()) return;
FatalLegacyBufferArmFromApplyThread(accessor);
}
#endif
}
Uint64 BufferObject::AllocateLifetimeId() {
return g_nextBufferLifetimeId.fetch_add(1, std::memory_order_relaxed);
}
void SetBufferBackendOps(const BufferBackendOps* ops) {
g_bufferBackendOps = ops;
}
const BufferBackendOps* GetBufferBackendOps() {
return g_bufferBackendOps;
}
BufferObject::BufferObject(Uint externalIndex)
: m_externalIndex(externalIndex), m_size(0), m_usage(BufferUsage::StaticDraw), m_isMapped(false),
m_mappingAccess(BufferMappingAccessBit::Null), m_mappedRange({0, 0}), m_ownsStagingData{} {
#if MOBILEGL_PIPE_PUSH
// P3a D-A2: a resource EXISTS before anything can name it, so resource_create is
// emitted from the constructor and carries no storage - the store is defined lazily
// by the first respecify and every backend already tolerates a resource with none.
// The handle itself is minted whatever the subsystem bitmask says, because
// set_vertex_buffers names this buffer by handle out of a different subsystem.
MG_Pipe::MGPipeMintResourceHandle(*this);
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) MG_Pipe::MGPipeEmitResourceCreate(*this);
#endif
}
BufferObject::~BufferObject() {
#if MOBILEGL_BUILD_DISAGGREGATED
// Unconditional, not behind PushIsArmed(): the transport mode cannot change, but the
// tracker is a leaked singleton whose entries are raw pointers, and an entry that
// outlives its object is the one failure this set must not have. Forget is a no-op
// for a buffer that was never a member.
MG_Remote::Client::PersistentMapTracker::Instance().Forget(*this);
#endif
#if MOBILEGL_PIPE_PUSH
// P3a D-L: the buffer's death crosses as resource_destroy, which is the catalogue
// call for it - no seventh NotifyStateObjectDestroyed raiser is added, because that
// header exists for kinds that have no such call. The emit-then-free ORDER is fixed
// inside the helper and is not negotiable.
// The answer is the helper's LATCH - "was resource_create emitted for this buffer" -
// not a second reading of MGPipeResourceSubsystemEnabled(): a buffer constructed
// while a backend's table was registered and destroyed after it was unregistered has
// a pipe record to drop and no legacy backend object, and one constructed the other
// way round has the opposite, so the create's answer is the only one that pairs.
if (MG_Pipe::MGPipeEmitResourceDestroyAndFree(*this)) return;
#endif
if (m_resource.Backend() && g_bufferBackendOps && g_bufferBackendOps->OnDestroy) {
g_bufferBackendOps->OnDestroy(m_resource.ReleaseBackend());
}
}
void BufferObject::NotifyRespecify() {
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
#if MOBILEGL_PIPE_PUSH
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) {
MG_Pipe::MGPipeEmitResourceRespecify(*this);
return;
}
#endif
if (g_bufferBackendOps && g_bufferBackendOps->Respecify) {
g_bufferBackendOps->Respecify(*this);
}
}
void BufferObject::NotifySubData(SizeT offset, SizeT size) {
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
if (size == 0) return;
m_hasDefinedContent = true;
#if MOBILEGL_PIPE_PUSH
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) {
MG_Pipe::MGPipeEmitResourceSubData(*this, offset, size);
return;
}
#endif
if (g_bufferBackendOps && g_bufferBackendOps->SubData) {
g_bufferBackendOps->SubData(*this, offset, size);
}
}
void BufferObject::NotifyFlushMappedRange(Range1D range, Flags<BufferMappingAccessBit> appAccess) {
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
if (range.start >= range.end) return;
m_hasDefinedContent = true;
#if MOBILEGL_PIPE_PUSH
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) {
static_assert(sizeof(appAccess.GetRaw()) <= sizeof(Uint32),
"MGPFlushRange::AccessFlags is a Uint32 and carries the application's "
"real Flags<BufferMappingAccessBit>, unnormalised");
MG_Pipe::MGPipeEmitResourceFlushRange(*this, range.start, range.end - range.start,
static_cast<Uint32>(appAccess.GetRaw()));
return;
}
#endif
if (g_bufferBackendOps && g_bufferBackendOps->FlushMappedRange) {
g_bufferBackendOps->FlushMappedRange(*this, range, appAccess);
}
}
void BufferObject::NotifyContentWrite(SizeT offset, SizeT size) {
if (size == 0) {
// An empty write moves the serial and nothing else, exactly as NotifySubData
// and NotifyFlushMappedRange do: it wrote no byte, so it must not promote an
// undefined store to "has content" - that would cost the next orphaning
// respecification a full-size upload of bytes the application never wrote.
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
return;
}
m_hasDefinedContent = true;
if (m_resource.IsGpuResident()) {
// The write already landed in coherent GPU memory; the backend has no separate
// copy to sync. Only bump the serial so cached transient slices invalidate.
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
return;
}
NotifySubData(offset, size);
}
// A (re)definition of the store is about to write `size` bytes through Bytes().
// Sizing the shadow is all that takes for a shadow-backed buffer. A buffer whose
// bytes were adopted into backend GPU memory has to give the adoption back first,
// because the mapping it holds describes exactly the OLD store: writing the new
// contents through it runs past its end the moment the store grows, and a backend
// that replaces the storage for the new store - which is what an orphaning
// respecification asks for - would leave that mapping, and therefore every later
// read of this buffer, addressing storage nothing writes to any more. That was the
// transform feedback capture that wrote one buffer while the readback read another.
//
// Given back rather than renewed here, deliberately. Renewing in place would mean
// memcpying the new contents into storage that submitted-but-unretired draws may
// still be reading, which is precisely what the orphaning idiom exists to avoid;
// avoiding THAT would mean either stalling on a fence in the middle of a frame or
// teaching the persistent-map op to orphan, and the op must never orphan for the
// other kind of caller (an application-held GL_MAP_PERSISTENT_BIT mapping, whose
// pointer has to stay valid for the buffer's whole life). Handing the store back to
// the CPU shadow needs none of that: the backend's ordinary respecification path
// then does the busy-tracking and the conditional orphan it has always done, and the
// next binding that wants GPU residency takes a fresh mapping of the new store.
void BufferObject::RedefineStorage(SizeT size) {
if (m_resource.IsGpuResident()) {
m_resource.ReleasePersistentMap();
// Whatever a shader or a capture wrote is in the store being replaced, so
// there is nothing left to reconcile - and leaving the flag set would make
// the next read of this buffer wait for GPU work on behalf of bytes the
// application has just thrown away.
m_gpuWritePending = false;
}
m_size = size;
m_resource.ResizeShadow(size);
#if MOBILEGL_BUILD_DISAGGREGATED
// The store this buffer's membership was about no longer exists, and ResizeShadow is
// reserve+resize - a grow past the reserve reallocates - so a pushed block's source
// base has moved too. Both are the same event to the tracker: re-read the predicate.
NotePersistentMapStateChanged();
#endif
}
void BufferObject::Respecify(SizeT size, const void* data) {
// The store a live mapping wrote into is about to be replaced, so landing those
// bytes into it would copy a whole mapped range (an adopted arena's map is the
// arena) into storage the next line hands back.
ReleaseMemory(false);
RedefineStorage(size);
if (data && size > 0) {
Memcpy(m_resource.Bytes(), data, size);
}
// A NULL-data respecify (the orphaning idiom) leaves the store undefined;
// record that so backends skip uploading the stale shadow bytes.
m_hasDefinedContent = (data != nullptr) || size == 0;
m_isImmutableStorage = false;
// GL 4.6 core 6.2 defines glBufferData as glBufferStorage with
// DYNAMIC_STORAGE_BIT | MAP_READ_BIT | MAP_WRITE_BIT, so GL_BUFFER_STORAGE_FLAGS has to
// report those three afterwards. Reporting 0 - the value that belongs to a buffer whose
// store has never been specified - told an application that a perfectly writable
// glBufferData buffer accepted neither glBufferSubData nor a map. Only the IMMUTABLE flag
// distinguishes the two cases, and it is cleared just above.
m_storageFlags = GL_DYNAMIC_STORAGE_BIT | GL_MAP_READ_BIT | GL_MAP_WRITE_BIT;
NotifyRespecify();
TryAdoptLargeStorage();
}
void BufferObject::Resize(SizeT size) {
Respecify(size, nullptr);
}
void BufferObject::AllocateImmutableStorage(SizeT size, const void* data, GLbitfield storageFlags) {
// Same as Respecify: the bytes a live mapping staged have nowhere to land, the
// store they belong to is being replaced.
ReleaseMemory(false);
RedefineStorage(size);
if (data) {
Memcpy(m_resource.Bytes(), data, size);
} else if (size > 0) {
Memset(m_resource.Bytes(), 0, size);
}
m_hasDefinedContent = true;
m_isImmutableStorage = true;
m_storageFlags = storageFlags;
NotifyRespecify();
TryAdoptLargeStorage();
}
// Back a LARGE store with the backend's persistently+coherently mapped GPU
// storage the moment it is (re)defined, without waiting for the app to map it.
// Minecraft 26.3 streams chunk meshes into 128MB vertex arenas with plain
// glNamedBufferSubData - the one write API that carries no synchronization
// hint - and on Mali every route that hands the driver a write into a busy
// MUTABLE store either parks the calling thread (glBufferSubData, and
// glMapBufferRange even with GL_MAP_UNSYNCHRONIZED_BIT) or ghost-copies the
// whole destination on a driver worker (staged glCopyBufferSubData, and a
// range-invalidating map: ~167ms per touched arena, the recurring in-world
// hiccup). An adopted coherent map is the one shape with NO per-write driver
// call at all: every SubData lands as a plain memcpy into GPU-visible memory,
// and the shadow copy is dropped (a 128MB arena stops costing 128MB of RAM).
// Only attempted for stores the size of mesh arenas: small buffers keep the
// shadow model whose draw-time flush already prices them correctly.
void BufferObject::TryAdoptLargeStorage() {
constexpr SizeT kLargeBufferAdoptBytes = 16u * 1024u * 1024u;
if (MG_Config::Features.DisableLargeBufferAdoption) return;
if (m_size < kLargeBufferAdoptBytes) return;
if (m_resource.IsGpuResident()) return;
if (m_isMapped) return;
#if MOBILEGL_PIPE_PUSH
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) {
if (void* base = MG_Pipe::MGPipeEmitMapPersistent(*this)) m_resource.AdoptPersistentMap(base);
return;
}
#endif
#if MOBILEGL_BUILD_DISAGGREGATED
// R-6 IS "ALWAYS", AND THIS IS ITS SECOND DOOR. MGPipeApplyMapPersistent declines
// every acquisition under split - but a split build whose backend registered no
// MGPipe resource ops falls through to the LEGACY hook below, which would mint a real
// pointer and adopt it. A donated address is meaningless across a process, and an
// inproc lane that adopted would be green for a reason spawn cannot reproduce.
if (MG_Config::Transport != MG_Config::TransportMode::Monolith) return;
#endif
if (g_bufferBackendOps == nullptr || g_bufferBackendOps->AcquirePersistentMap == nullptr) return;
if (void* base = g_bufferBackendOps->AcquirePersistentMap(*this)) {
m_resource.AdoptPersistentMap(base);
}
}
void BufferObject::UploadData(DataPtr data, SizeT atOffset) {
MOBILEGL_ASSERT(atOffset + data.size <= m_size,
"UploadData out of bounds: atOffset (%zu) + data.size (%zu) > m_size (%zu)", atOffset,
data.size, m_size);
MOBILEGL_ASSERT(!m_isMapped || (m_mappingAccess & BufferMappingAccessBit::Persistent),
"Cannot upload data while buffer is non-persistently mapped.");
Memcpy(m_resource.Bytes() + atOffset, data.data, data.size);
NotifyContentWrite(atOffset, data.size);
}
void BufferObject::SetUsage(BufferUsage usage) {
m_usage = usage;
}
void BufferObject::ReleaseMemory(Bool landStagedWrites) {
if (!m_isMapped) return;
if (landStagedWrites &&
(m_mappingAccess & BufferMappingAccessBit::Write)) { // if we wrote to the buffer
if (!(m_mappingAccess & BufferMappingAccessBit::FlushExplicit)) { // if we didn't flush explicitly
const SizeT mappedLength = m_mappedRange.end - m_mappedRange.start;
if (m_resource.IsGpuResident()) {
// A persistent map of an adopted store wrote straight into coherent
// GPU memory: nothing to copy back, no range to push down. A
// NON-persistent write map is a different thing: the application
// wrote a staging copy (glMapBuffer and glMapBufferRange hand one out
// regardless of where the store lives), and GL requires those bytes
// to be visible to every later command the moment glUnmapBuffer
// returns. Residency used to come only from a coherent persistent
// map, which never has a staging copy, so the copy-back was simply
// skipped for a resident store; residency now also comes from a
// shader storage binding (EnsureGpuResidentStorage at draw time) and
// from large-store adoption (TryAdoptLargeStorage), both of which an
// application then re-initialises through an ordinary map/write/unmap.
// Skipping the copy-back dropped every one of those writes. Land the
// staged bytes through the same route glBufferSubData takes into an
// adopted store - the backend's flush op is for stores it keeps a
// separate copy of and must not run here.
if (!(m_mappingAccess & BufferMappingAccessBit::Persistent)) {
LandBytesIntoResidentStore(m_mappedRange.start,
{m_stagingData.data() + m_stagingBias, mappedLength});
}
} else {
if (!(m_mappingAccess & BufferMappingAccessBit::Persistent)) {
Memcpy(m_resource.Bytes() + m_mappedRange.start, m_stagingData.data() + m_stagingBias,
mappedLength);
}
NotifyFlushMappedRange(m_mappedRange, m_mappingAccess);
}
}
}
m_stagingData.clear();
m_isMapped = false;
m_mappingAccess = BufferMappingAccessBit::Null;
m_mappedRange = {0, 0};
m_stagingBias = 0;
m_ownsStagingData = false;
#if MOBILEGL_BUILD_DISAGGREGATED
// AFTER the reset, so the predicate reads the post-unmap state, and after the landing
// above, so the last bytes of a write map are already on the wire when the record
// that says "no live writer" goes out behind them.
NotePersistentMapStateChanged();
#endif
}
void BufferObject::FlushMemoryRange(SizeT offset, SizeT length) {
MOBILEGL_ASSERT(m_isMapped, "Buffer must be mapped to flush memory range.");
MOBILEGL_ASSERT((m_mappingAccess & BufferMappingAccessBit::FlushExplicit),
"Buffer must be mapped with FlushExplicit access to flush memory range.");
MOBILEGL_ASSERT((m_mappingAccess & BufferMappingAccessBit::Write),
"Buffer must be mapped with Write access to flush memory range.");
SizeT start = m_mappedRange.start + offset;
SizeT end = start + length;
MOBILEGL_ASSERT(end <= m_mappedRange.end, "Flush range out of bounds: mappedRange.end (%zu) < end (%zu)",
m_mappedRange.end, end);
// A FLUSH_EXPLICIT map can sit on an adopted store: the map itself never adopts
// (only a coherent persistent one does), but a shader storage binding or
// large-store adoption may have made the buffer resident before the map. The
// flushed bytes then take the same landing as any other CPU write into an
// adopted store - a persistent map already wrote them in place and only has
// to publish the change, a non-persistent map staged them and has to land
// them. The backend's flush op is for stores it keeps a separate copy of.
if (m_resource.IsGpuResident()) {
if (m_mappingAccess & BufferMappingAccessBit::Persistent) {
NotifyContentWrite(start, length);
} else {
LandBytesIntoResidentStore(start, {m_stagingData.data() + m_stagingBias + offset, length});
}
return;
}
if (!(m_mappingAccess & BufferMappingAccessBit::Persistent)) {
Memcpy(m_resource.Bytes() + start, m_stagingData.data() + m_stagingBias + offset, length);
}
NotifyFlushMappedRange({start, end}, m_mappingAccess);
}
void BufferObject::SyncPersistentMappedRange() {
#if MOBILEGL_BUILD_DISAGGREGATED
// P5c (hd): the named refusal comes FIRST - a silent return here used to let a
// server-side caller slip through with one MGLOG_D's worth of evidence (B3).
//
// ONE ROLE PROBE, NOT TWO (P5d round 3, package D). This used to call
// RefuseLegacyBufferArmFromApplyThread and then re-ask PushIsArmed()/OnServerRole()
// for the early-out below - the same two questions, answered twice on the same line of
// execution, on a function the draw path reaches per mapped buffer. The refusal's
// condition and the early-out's condition were already IDENTICAL, which is why the
// early-out's `return` was unreachable: with the push armed, the apply thread aborts
// in the refusal and never gets there. Folding them keeps the refusal first and drops
// the duplicate probe rather than the guard.
//
// Split's client pre-verb hook publishes these bytes. The retained backend sync
// sites must do nothing on the apply thread: re-entering this producer there would
// overwrite the server shadow through the monolith adapter without crossing the wire -
// and "do nothing" is spelled Fatal, not `return`, because a server-side caller here
// is a role violation and not a shape the design tolerates.
if (MG_Remote::Client::PersistentMapTracker::PushIsArmed()) {
if (MG_Remote::Client::PersistentMapTracker::OnServerRole()) {
FatalLegacyBufferArmFromApplyThread("SyncPersistentMappedRange");
}
MG_Remote::Client::PersistentMapTracker::Instance().PushBlocksFor(*this);
return;
}
#endif
if (!m_isMapped) return;
// GPU-resident: the app already wrote directly into coherent GPU memory. This is
// the whole point of the persistent-map path - the per-draw whole-buffer re-upload
// that used to run here is gone.
if (m_resource.IsGpuResident()) return;
if (!(m_mappingAccess & BufferMappingAccessBit::Persistent)) return;
if (!(m_mappingAccess & BufferMappingAccessBit::Write)) return;
if (m_mappingAccess & BufferMappingAccessBit::FlushExplicit) return;
if (m_mappedRange.start >= m_mappedRange.end) return;
NotifySubData(m_mappedRange.start, m_mappedRange.end - m_mappedRange.start);
}
#if MOBILEGL_BUILD_DISAGGREGATED
void BufferObject::PushMappedSpanBlock(SizeT offset, SizeT size) {
// NotifySubData and not MGPipeEmitResourceSubData directly: the serial bump, the
// defined-content promotion and the legacy-ops fallback are what the monolith span
// push does, and a second route to the same record is a second thing to keep in step.
if (size == 0) return;
// Assert ownership at the final producer entry too, before serial/aggregate updates.
if (MG_Remote::Client::PersistentMapTracker::OnServerRole()) {
MGLOG_F("MGPipe: Fatal{RoleViolation, \"PushMappedSpanBlock\"} - the server role (the apply "
"thread) reached the CLIENT persistent-map producer for buffer lifetime %llu "
"[%zu, +%zu). Under an active transport the server's staged copy is the draw's "
"only base (ID-52); a push from here would replace the transported bytes with "
"client memory through the monolith adapter",
static_cast<unsigned long long>(m_lifetimeId), offset, size);
std::abort();
}
NotifySubData(offset, size);
if (MG_Util::PipeStats::Enabled()) {
// THE persistent-map-push SITE: the bytes an application wrote through a map with
// no API call, which a split build therefore has to ship.
//
// THESE BYTES ARE ALSO COUNTED AS stage-buffer, and that overlap is stated in the
// inventory rather than avoided. At tier T2 a pushed block IS an ordinary
// resource_subdata, so it reaches Ops_H_SubData, is queued into pendingRanges and
// is staged like any other write. Subtracting it at the staging site would make
// stage-buffer under-report what the BACKEND actually moves, which is the question
// that class exists to answer; the two counters are different questions about the
// same bytes and PipeStats.cpp:33-35 now says so.
MG_Util::PipeStats::AddBytes(MG_Util::PipeStats::ByteClass::PersistentMapPush,
static_cast<Uint64>(size));
}
}
void BufferObject::NotePersistentMapStateChanged() {
if (!MG_Remote::Client::PersistentMapTracker::PushIsArmed()) return;
MG_Remote::Client::PersistentMapTracker::Instance().NoteMapStateChanged(*this);
// Teardown must retire membership, but only the client publishes live-host-write state.
if (MG_Remote::Client::PersistentMapTracker::OnServerRole()) return;
// THE LIVE-HOST-WRITES BIT (ARCHITECTURE.md 12, CONTRACT-P5.md section 3). A live
// WRITE map - persistent or not - mutates the shadow with no call, no serial and no
// epoch, which is exactly why IsBufferDrawCleanByHandle had to ask the frontend object
// whether it was mapped. Under a spawn there is no object on that side, so the fact
// has to cross; it rides MGPSubData's pad (see MGPipeTypes.h, and b1-v1.md 3.1 for
// why not MGPResourceDesc's).
//
// BOTH EDGES EMIT A RECORD, AND THE RISING ONE IS NOT OPTIONAL. An earlier cut let the
// rising edge emit nothing on the grounds that "the next content record carries the
// bit anyway" - and for the single idiom this feature exists to serve, a persistently
// mapped streaming arena behind a static VAO, there IS no next content record. The
// only one is the block push, the push for a vertex/uniform/SSBO map runs inside
// EnsureBufferResourceForHandle, and a draw-clean answer SKIPS that ensure
// (DirectGLES.cpp:691) and then LATCHES it (:697, vboCleanEpoch - and a coherent
// persistent map bumps no mutation epoch, which is its whole point). So: draw 1
// pushes, draw 2 probes clean, draw 3 onward never probes again, and the frame draws
// frame 1's bytes for ever with no diagnostic. That is Managers.cpp:2650-2655's
// regression one layer down, and the rising-edge record is what breaks the cycle: the
// server learns a host writer is live BEFORE the first probe, answers dirty while the
// map lives, and the ensure - and with it the push - runs every draw.
const Bool live = m_isMapped && (m_mappingAccess & BufferMappingAccessBit::Write) &&
!m_resource.IsGpuResident();
if (live == m_publishedLiveHostWrites) return;
m_publishedLiveHostWrites = live;
// ONE BLOCK, NOT THE SPAN. On the falling edge the span's bytes have already shipped
// on this same path (ReleaseMemory lands the staged writes and calls
// NotifyFlushMappedRange before this runs); on the rising edge the span has not been
// written yet. Either way the record exists to carry the STATE, and the bytes it
// carries are real, current and the cheapest honest ones there are. A zero-length
// record would have been the alternative and it is illegal by contract rule A.
if (m_size == 0) return;
const Uint64 blockBytes = MG_Remote::Client::PersistentMapTracker::BlockBytes();
// MOBILEGL_IPC_PERSISTENT_BLOCK_KB=0 TURNS THE WHOLE MECHANISM OFF, STATE RECORD
// INCLUDED (E3(a)). An earlier cut skipped the blocks and still shipped a whole buffer
// here, which meant an unmap delivered the bytes the control exists to withhold - and
// a negative control that still delivers them is not a control. With the knob at 0
// nothing is published and nothing is pushed, so the probe answers clean, the frame
// draws the last uploaded bytes, and PersistentCoherentMapScenario goes red.
if (blockBytes == 0) return;
const SizeT length = blockBytes >= static_cast<Uint64>(m_size) ? m_size
: static_cast<SizeT>(blockBytes);
NotifySubData(0, length);
}
Bool BufferObject::HasLiveHostWritesForWire() const {
return m_publishedLiveHostWrites;
}
#endif
void BufferObject::WritebackFromBackend(DataPtr data, SizeT atOffset) {
MOBILEGL_ASSERT(atOffset + data.size <= m_size,
"WritebackFromBackend out of bounds: atOffset (%zu) + data.size (%zu) > m_size (%zu)", atOffset,
data.size, m_size);
Memcpy(m_resource.Bytes() + atOffset, data.data, data.size);
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
#if MOBILEGL_BUILD_DISAGGREGATED
// THE THIRD STATE'S ONLY EXIT. In a split build SyncGpuWrites does NOT clear the
// flag before emitting, because between the emission and the answer the shadow is
// stale and the object has no way to say so; the answer landing here is what makes it
// current, so the answer is what clears. Gated: in monolith SyncGpuWrites has already
// cleared by the time the applier calls back, and a second clear here would also
// retire a GPU write announced by something other than a readback - a ReadPixels into
// a pack PBO writes back through this same function.
if (MG_Config::Transport != MG_Config::TransportMode::Monolith &&
atOffset == 0 && data.size >= m_size) {
m_gpuWritePending = false;
}
#endif
}
void BufferObject::MarkGpuWritten() {
m_hasDefinedContent = true;
m_gpuWritePending = true;
}
void BufferObject::SyncGpuWrites() {
if (!m_gpuWritePending) return;
#if MOBILEGL_BUILD_DISAGGREGATED
// THE THIRD STATE (CONTRACT-P5.md section 3, ARCHITECTURE.md:509). This function has
// only ever been able to say "pending" or "not pending", and clearing first is safe
// in monolith because the readback is synchronous INSIDE the applier: the caller sees
// the reconciled shadow on return. Under a transport there is a third state - the
// readback was emitted and the answer has not arrived - which the flag cannot express,
// and clearing optimistically leaves the shadow silently stale for the object's life.
//
// So: emit, then BLOCK until OnBufferWriteback lands, and let the writeback do the
// clearing. ARCHITECTURE.md:509 lists "the first CPU read of a GPU-write-pending
// buffer" among the UNAVOIDABLE blocking points, because monolith already glFinish()es
// here; under R-1's verb barrier the block is nearly free, since the client is already
// waiting for appliedSeq to reach the record it just emitted.
//
// NO NARROWING (ResourceTracker.h:587-592's rangeCount == 1 assertion stays): this
// phase only gives the client a conservative set, and a zero-range announcement stays
// illegal until P8/P9 make it mean "fully narrowed - nothing is dirty".
if (MG_Config::Transport != MG_Config::TransportMode::Monolith) {
#if MOBILEGL_PIPE_PUSH
if (m_size != 0 && MG_Pipe::MGPipeResourceSubsystemEnabled()) {
const SizeT sliceBytes = MG_Remote::Client::BufferWritebackSliceBytes();
if (sliceBytes != 0 && m_size > sliceBytes) {
// The writeback's bytes travel INLINE in a SEG_EVENT record (P5c ev),
// and one record must fit the ring (MaxRecordBytes == capacity/2) - a
// whole-buffer request for a buffer larger than that aborts the server
// on Fatal{EventRingOverflow}. Slice the request instead: every slice
// round-trips its own barrier + drain before the next is emitted, so
// the ring holds at most one slice's bytes at a time, and the in-order
// channel makes the last slice's landing imply every earlier one.
for (SizeT off = 0; off < m_size; off += sliceBytes) {
const SizeT left = m_size - off;
MG_Pipe::MGPipeEmitResourceReadbackRange(*this, off,
left < sliceBytes ? left : sliceBytes);
}
// No single writeback covered the whole buffer, so
// WritebackFromBackend's clear (above) never fired - but every slice
// has landed by here, which is exactly what the flag-clearing there
// says. Clearing by hand is what keeps AwaitBufferWriteback's
// third-state Fatal from misfiring on a sliced readback.
m_gpuWritePending = false;
} else {
MG_Pipe::MGPipeEmitResourceReadback(*this);
}
// The wait is the barrier's wait: the reply slot id IS the record's seq, so
// "my answer is back" and "appliedSeq reached me" are one condition. With no
// session (a build-split lane running monolith, and every unit case) the
// emission was synchronous and the writeback has already cleared the flag.
MG_Remote::Client::AwaitBufferWriteback(*this);
}
#endif
// A buffer with no readback route - no size, or a backend that registered no
// resource ops - can never catch up, and retrying on every subsequent read would
// only repeat the same no-op. That is the ONE case the monolith clear covers that
// the writeback cannot, so it is spelled out here rather than inherited.
if (m_gpuWritePending && !MG_Remote::Client::BufferWritebackIsReachable(*this)) {
m_gpuWritePending = false;
}
return;
}
#endif
// Cleared unconditionally: without a readback op the shadow can never catch up,
// and retrying on every subsequent read would only repeat the same no-op.
m_gpuWritePending = false;
#if MOBILEGL_PIPE_PUSH
if (m_size != 0 && MG_Pipe::MGPipeResourceSubsystemEnabled()) {
// The answer comes back through the reverse channel's OnBufferWriteback, which
// resolves this handle to this object and writes the shadow before the server
// bumps its mutation epoch. In monolith the whole sequence is synchronous inside
// the applier, so the caller sees the reconciled shadow on return exactly as it
// does today.
MG_Pipe::MGPipeEmitResourceReadback(*this);
return;
}
#endif
if (m_size == 0 || g_bufferBackendOps == nullptr || g_bufferBackendOps->ReadbackFromGpu == nullptr) {
return;
}
g_bufferBackendOps->ReadbackFromGpu(*this);
}
void BufferObject::UploadSubData(DataPtr data, SizeT atOffset) {
// GL 4.6 core 6.5 forbids only the OVERLAPPING write: a glBufferSubData that stays
// clear of a non-persistent mapping is legal, and the frontend lets it through.
MOBILEGL_ASSERT(!m_isMapped || (m_mappingAccess & BufferMappingAccessBit::Persistent) ||
atOffset >= m_mappedRange.end || atOffset + data.size <= m_mappedRange.start,
"Cannot upload sub data overlapping a non-persistent mapping.");
MOBILEGL_ASSERT(atOffset + data.size <= m_size,
"UploadSubData out of bounds: atOffset (%zu) + data.size (%zu) > m_size (%zu)", atOffset,
data.size, m_size);
// An adopted store's Bytes() IS the memory in-flight frames are reading, and
// GL orders a glBufferSubData after those already-submitted reads: the write
// has to take the resident landing, never a plain host write into the mapping.
// Shadow-backed stores need none of this: the Memcpy below touches only the
// shadow, and the backend's SubData op does its own ordering against in-flight
// work.
if (m_resource.IsGpuResident()) {
LandBytesIntoResidentStore(atOffset, data);
return;
}
Memcpy(m_resource.Bytes() + atOffset, data.data, data.size);
NotifyContentWrite(atOffset, data.size);
}
// A backend that can land the bytes on the GPU timeline takes them here, untouched
// by the mapping - an in-place host write into coherent memory tore the frames
// still reading the old bytes (Minecraft patches LIVE chunk sections this way).
// The bytes are then not current in the mapping until the backend's ordered copy
// executes, so reads reconcile through the same gate GPU-written buffers use.
//
// Without that op the write lands in place, after retiring the GPU writes this store
// is known to be waiting on: a backend that defers work (DirectVulkan's frame command
// buffer) may still be holding a recorded-but-unsubmitted dispatch that GL orders this
// write AFTER, and writing the mapping now would land the bytes underneath that
// dispatch - its increments then execute on top of the newer data and invert the call
// order. That gate only knows about work that WROTE the store (MarkGpuWritten); work
// that merely READS it - a draw sourcing an adopted vertex arena - is not tracked here,
// so a backend without the op still owes the ordering against its own recorded reads.
// NotifyContentWrite on a resident store only bumps the serial: the backend has no
// separate copy to sync, so no transfer op runs.
void BufferObject::LandBytesIntoResidentStore(SizeT offset, DataPtr bytes) {
#if MOBILEGL_PIPE_PUSH
// buffer_subdata_resident stays NULLABLE and stays asymmetric: one backend
// deliberately does not implement it, and the frontend checks the pipe table exactly
// as it checks the op table it replaces, so a backend without it keeps the legacy
// ordered in-place host write below.
if (bytes.size > 0 && MG_Pipe::MGPipeResourceSubsystemEnabled() &&
MG_Pipe::MGPipeResourceOpsHaveSubDataResident()) {
MG_Pipe::MGPipeEmitBufferSubDataResident(*this, offset, bytes.data, bytes.size);
m_hasDefinedContent = true;
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
m_gpuWritePending = true;
return;
}
// No resident op: the write lands in place below, after retiring the GPU writes this
// store is known to be waiting on - which is the same answer, and the same code, a
// backend with a null ResidentSubData gets today.
if (MG_Pipe::MGPipeResourceSubsystemEnabled()) {
SyncGpuWrites();
Memcpy(m_resource.Bytes() + offset, bytes.data, bytes.size);
NotifyContentWrite(offset, bytes.size);
return;
}
#endif
if (bytes.size > 0 && g_bufferBackendOps && g_bufferBackendOps->ResidentSubData) {
g_bufferBackendOps->ResidentSubData(*this, offset, bytes);
m_hasDefinedContent = true;
++m_changeSerial;
MGP_NOTE_AGGREGATE(BufferChange);
m_gpuWritePending = true;
return;
}
SyncGpuWrites();
Memcpy(m_resource.Bytes() + offset, bytes.data, bytes.size);
NotifyContentWrite(offset, bytes.size);
}
void BufferObject::FillSubData(DataPtr pattern, SizeT atOffset, SizeT size) {
MOBILEGL_ASSERT(pattern.data != nullptr && pattern.size > 0,
"FillSubData requires a non-empty pattern.");
MOBILEGL_ASSERT(size % pattern.size == 0,
"FillSubData size (%zu) must be a multiple of pattern size (%zu).", size, pattern.size);
MOBILEGL_ASSERT(atOffset <= m_size && size <= m_size - atOffset,
"FillSubData out of bounds: atOffset (%zu) + size (%zu) > m_size (%zu)", atOffset, size,
m_size);
MOBILEGL_ASSERT(!m_isMapped || (m_mappingAccess & BufferMappingAccessBit::Persistent),
"Cannot fill data while buffer is non-persistently mapped.");
if (size == 0) return;
// An adopted store takes the same landing as UploadSubData: the in-place write
// below would tear in-flight readers of the mapping. The pattern is expanded
// first because the landing takes the final bytes, not a repeat rule - which is
// why only a backend that actually takes them comes through here. Without that
// op the landing would memcpy the expansion into the mapping the loop below
// fills in place anyway, so a whole-arena clear would allocate a whole arena
// for nothing.
if (m_resource.IsGpuResident() &&
#if MOBILEGL_PIPE_PUSH
// The same question, asked of whichever table owns the family in this build.
(MG_Pipe::MGPipeResourceSubsystemEnabled()
? MG_Pipe::MGPipeResourceOpsHaveSubDataResident()
: (g_bufferBackendOps && g_bufferBackendOps->ResidentSubData))
#else
g_bufferBackendOps && g_bufferBackendOps->ResidentSubData
#endif
) {
Vector<Uint8> expanded(size);
if (pattern.size == 1) {
Memset(expanded.data(), *static_cast<const Uint8*>(pattern.data), size);
} else {
for (SizeT at = 0; at < size; at += pattern.size) {
Memcpy(expanded.data() + at, pattern.data, pattern.size);
}
}
LandBytesIntoResidentStore(atOffset, {expanded.data(), size});
return;
}
// A clear is ordered after all earlier GPU writes; partial clears additionally need
// the retained shadow bytes, and a resident store the backend cannot take the bytes
// for is written in place, which needs the same synchronization the landing does.
SyncGpuWrites();
Uint8* dst = m_resource.Bytes() + atOffset;
if (pattern.size == 1) {
Memset(dst, *static_cast<const Uint8*>(pattern.data), size);
} else {
for (SizeT at = 0; at < size; at += pattern.size) {
Memcpy(dst + at, pattern.data, pattern.size);
}
}
NotifyContentWrite(atOffset, size);
}
void BufferObject::DownloadSubData(void* dst, SizeT atOffset, SizeT size) const {
MOBILEGL_ASSERT(atOffset + size <= m_size,
"DownloadSubData out of bounds: atOffset (%zu) + size (%zu) > m_size (%zu)", atOffset, size,
m_size);
Memcpy(dst, m_resource.Bytes() + atOffset, size);
}
void BufferObject::CopyDataFrom(const SharedPtr<BufferObject>& src, SizeT srcOffset, SizeT dstOffset, SizeT size) {
MOBILEGL_ASSERT(!m_isMapped || (m_mappingAccess & BufferMappingAccessBit::Persistent),
"Cannot copy data while destination buffer is non-persistently mapped.");
MOBILEGL_ASSERT(!src->IsMapped() || (src->GetMappingAccess() & BufferMappingAccessBit::Persistent),
"Cannot copy data from a buffer that is non-persistently mapped.");
MOBILEGL_ASSERT(srcOffset + size <= src->GetSize(),
"Source buffer copy out of bounds: srcOffset (%zu) + size (%zu) > src->GetSize() (%zu)",
srcOffset, size, src->GetSize());
MOBILEGL_ASSERT(dstOffset + size <= m_size,
"Destination buffer copy out of bounds: dstOffset (%zu) + size (%zu) > m_size (%zu)", dstOffset,
size, m_size);
src->SyncGpuWrites();
// An adopted DESTINATION takes the same landing as UploadSubData: the in-place
// write below would tear in-flight readers of the mapping, and pending recorded
// GPU writes to it must retire before the copy lands or they would execute on
// top of it.
if (m_resource.IsGpuResident()) {
LandBytesIntoResidentStore(dstOffset, {src->m_resource.Bytes() + srcOffset, size});
return;
}
Memcpy(m_resource.Bytes() + dstOffset, src->m_resource.Bytes() + srcOffset, size);
NotifyContentWrite(dstOffset, size);
}
void* BufferObject::AcquireMemory(Bool markMapped, Bool read, Bool write) {
SyncGpuWrites();
if (markMapped) {
m_isMapped = true;
m_mappingAccess = (read ? BufferMappingAccessBit::Read : BufferMappingAccessBit::Null) |
(write ? BufferMappingAccessBit::Write : BufferMappingAccessBit::Null);
m_mappedRange = {0, m_size};
#if MOBILEGL_BUILD_DISAGGREGATED
// glMapBuffer never takes the Persistent bit, so this buffer can never join the
// push set - but a WRITE map still mutates the shadow with no call, which is the
// half of the live-host-writes bit that is not about the push at all.
NotePersistentMapStateChanged();
#endif
if (m_mappingAccess & BufferMappingAccessBit::Write) {
// glMapBuffer maps from offset 0, so no bias: the allocation's own
// GL_MIN_MAP_BUFFER_ALIGNMENT-aligned base is what the application must get.
m_stagingBias = 0;
m_stagingData.resize(m_size);
m_ownsStagingData = true;
if (!(m_mappingAccess &
(BufferMappingAccessBit::InvalidateRange | BufferMappingAccessBit::InvalidateBuffer))) {
Memcpy(m_stagingData.data(), m_resource.Bytes(), m_size);
}
return m_stagingData.data();
}
}
return m_resource.Bytes();
}
Bool BufferObject::EnsureGpuResidentStorage() {
if (m_resource.IsGpuResident()) {
return true;
}
// Adoption releases the CPU shadow, and a live mapping may BE that shadow: a
// persistent map that did not itself adopt (a FLUSH_EXPLICIT one, or a read map)
// handed the application shadow + offset, and GL keeps that pointer valid while
// the buffer is drawn with - which is exactly when this runs, on the storage
// binding walk. Freeing it under the application is a use-after-free, so a mapped
// buffer keeps the shadow model until it is unmapped; the binding that follows
// adopts then. Same rule as TryAdoptLargeStorage.
if (m_isMapped) {
return false;
}
#if MOBILEGL_PIPE_PUSH
if (m_size != 0 && MG_Pipe::MGPipeResourceSubsystemEnabled()) {
void* pushedBase = MG_Pipe::MGPipeEmitMapPersistent(*this);
if (pushedBase == nullptr) return false;
m_resource.AdoptPersistentMap(pushedBase);
return true;
}
#endif
#if MOBILEGL_BUILD_DISAGGREGATED
// R-6's second door, as in TryAdoptLargeStorage above.
if (MG_Config::Transport != MG_Config::TransportMode::Monolith) return false;
#endif
if (m_size == 0 || g_bufferBackendOps == nullptr || g_bufferBackendOps->AcquirePersistentMap == nullptr) {
return false;
}
void* base = g_bufferBackendOps->AcquirePersistentMap(*this);
if (base == nullptr) {
return false;
}
m_resource.AdoptPersistentMap(base);
return true;
}
void* BufferObject::AcquireMemoryRange(Range1D range, Flags<BufferMappingAccessBit> access) {
MOBILEGL_ASSERT(range.end <= m_size && range.start <= range.end,
"AcquireMemoryRange out of bounds: range (%zu, %zu) exceeds m_size (%zu)", range.start,
range.end, m_size);
// The app is about to look at the bytes; a shader may have rewritten them since
// the shadow was last authoritative. Also needed for a write map without an
// invalidate bit, whose staging copy is seeded from the shadow.
//
// One map shape looks at nothing: a non-persistent write map that discards the
// range it maps gets a staging copy the seeding below skips, so no reader of the
// store exists between here and the unmap. Reconciling an ADOPTED store would
// still cost the backend's full drain-and-wait (its queued landings are made
// visible to the CPU by finishing the pipeline), once per map, on exactly the
// streaming arena the adoption exists to keep cheap. The outstanding-write flag
// stays set, so the first read that DOES look at the bytes still pays for it.
const Bool discardsWhatItMaps =
(access & BufferMappingAccessBit::Write) && !(access & BufferMappingAccessBit::Persistent) &&
(access & (BufferMappingAccessBit::InvalidateRange | BufferMappingAccessBit::InvalidateBuffer));
if (!(m_resource.IsGpuResident() && discardsWhatItMaps)) {
SyncGpuWrites();
}
m_isMapped = true;
m_mappingAccess = access;
m_mappedRange = range;
#if MOBILEGL_BUILD_DISAGGREGATED
// BEFORE the adoption attempt below, deliberately. Under R-6 the acquisition always
// declines, so the predicate this publishes is already final; if a later phase ever
// mints one, the adoption path notes the change itself (it does now - the call is
// beside AdoptPersistentMap) and the entry is withdrawn there rather than never
// having been made, which is the order that keeps the set conservative under both
// answers.
NotePersistentMapStateChanged();
#endif
if (access & BufferMappingAccessBit::Persistent) {
m_ownsStagingData = false;
// Zero-copy: for a coherent (non-FLUSH_EXPLICIT) persistent write map, ask the
// active backend for host-visible, coherent GPU storage and adopt it as the
// single source of truth. The backend seeds it from the current shadow before
// returning; AdoptPersistentMap then releases the shadow. Falls back to the
// shadow when the backend declines (returns null). Only attempted once - the
// storage is immutable and outlives unmap/remap.
#if MOBILEGL_PIPE_PUSH
if (!m_resource.IsGpuResident() && (access & BufferMappingAccessBit::Write) &&
!(access & BufferMappingAccessBit::FlushExplicit) &&
MG_Pipe::MGPipeResourceSubsystemEnabled()) {
if (void* pushedBase = MG_Pipe::MGPipeEmitMapPersistent(*this)) {
m_resource.AdoptPersistentMap(pushedBase);
#if MOBILEGL_BUILD_DISAGGREGATED
// An adoption takes the buffer OUT of the push set and out of the
// live-host-writes state: the application now writes coherent GPU memory
// and there is nothing to ship. Under R-6 this is unreachable; it is here
// because the comment below used to claim the adoption path withdrew the
// entry and nothing did, which would have left the published bit true
// across an adoption until unmap (latent for P11).
NotePersistentMapStateChanged();
#endif
}
return m_resource.Bytes() + range.start;
}
#endif
if (!m_resource.IsGpuResident() && (access & BufferMappingAccessBit::Write) &&
!(access & BufferMappingAccessBit::FlushExplicit) &&
#if MOBILEGL_BUILD_DISAGGREGATED
// R-6's second door, as in TryAdoptLargeStorage: no legacy mint under a
// transport, whatever the backend registered.
MG_Config::Transport == MG_Config::TransportMode::Monolith &&
#endif
g_bufferBackendOps && g_bufferBackendOps->AcquirePersistentMap) {
if (void* base = g_bufferBackendOps->AcquirePersistentMap(*this)) {
m_resource.AdoptPersistentMap(base);
}
}
return m_resource.Bytes() + range.start;
}
if (access & BufferMappingAccessBit::Write) {
// ARB_map_buffer_alignment constrains (returned pointer - offset), not the pointer:
// a map at offset 63 must hand back a pointer 63 bytes past the alignment grid, which
// is exactly what the read path below gets for free from shadowBase + offset. The
// staging store has to be biased by the same phase to match, so it over-allocates by
// it and the mapped bytes start at data() + m_stagingBias.
m_stagingBias = range.start % MIN_MAP_BUFFER_ALIGNMENT;
const SizeT mappedLength = range.end - range.start;
m_stagingData.resize(m_stagingBias + mappedLength);
m_ownsStagingData = true;
if (!(access & (BufferMappingAccessBit::InvalidateRange | BufferMappingAccessBit::InvalidateBuffer))) {
Memcpy(m_stagingData.data() + m_stagingBias, m_resource.Bytes() + range.start, mappedLength);
}
return m_stagingData.data() + m_stagingBias;
} else {
m_ownsStagingData = false;
return m_resource.Bytes() + range.start;
}
}
const Uint8* BufferObject::MappedData() const {
#if MOBILEGL_BUILD_DISAGGREGATED
RefuseLegacyBufferArmFromApplyThread("MappedData");
#endif
return m_resource.Bytes();
}
#if MOBILEGL_BUILD_DISAGGREGATED
SizeT BufferObject::ShadowAllocationBytes() const {
// The extent of the very pointer MappedData() hands out, read by the same caller
// (the tracker's registration) on the same thread: the same rule-E surface, the
// same refusal.
RefuseLegacyBufferArmFromApplyThread("ShadowAllocationBytes");
return m_resource.ShadowAllocationBytes();
}
#endif
Bool BufferObject::IsBackendPersistentMapped() const {
return m_resource.IsGpuResident();
}
SizeT BufferObject::GetSize() const {
return m_size;
}
Bool BufferObject::IsImmutableStorage() const {
return m_isImmutableStorage;
}
BufferUsage BufferObject::GetUsage() const {
return m_usage;
}
Uint64 BufferObject::GetChangeSerial() const {
#if MOBILEGL_BUILD_DISAGGREGATED
RefuseLegacyBufferArmFromApplyThread("GetChangeSerial");
#endif
return m_changeSerial;
}
Bool BufferObject::HasDefinedContent() const {
#if MOBILEGL_BUILD_DISAGGREGATED
// §3.6's surface as well as §3.8's: the descriptor's bit and the staged coverage
// answer on this side; the frontend flag is client memory.
RefuseLegacyBufferArmFromApplyThread("HasDefinedContent");
#endif
return m_hasDefinedContent;
}
const SharedPtr<BackendBufferResource>& BufferObject::GetBackendResource() const {
return m_resource.Backend();
}
void BufferObject::SetBackendResource(SharedPtr<BackendBufferResource> resource) {
m_resource.SetBackend(std::move(resource));
}
Bool BufferObject::IsMapped() const {
#if MOBILEGL_BUILD_DISAGGREGATED
RefuseLegacyBufferArmFromApplyThread("IsMapped");
#endif
return m_isMapped;
}
Range1D BufferObject::GetMappedRange() const {
return m_isMapped ? m_mappedRange : Range1D{0, 0};
}
void* BufferObject::GetMappedPointer() const {
if (!m_isMapped) return nullptr;
if (m_mappingAccess & BufferMappingAccessBit::Persistent) {
// GPU-resident maps return the coherent GPU pointer; shadow-backed persistent
// maps return the shadow. m_resource.Bytes() resolves both.
return const_cast<Uint8*>(m_resource.Bytes()) + m_mappedRange.start;
}
if (m_ownsStagingData) {
return const_cast<Uint8*>(m_stagingData.data()) + m_stagingBias;
}
return const_cast<Uint8*>(m_resource.Bytes()) + m_mappedRange.start;
}
Flags<BufferMappingAccessBit> BufferObject::GetMappingAccess() const {
return m_isMapped ? m_mappingAccess : BufferMappingAccessBit::Null;
}
GLbitfield BufferObject::GetStorageFlags() const {
return m_storageFlags;
}
Uint BufferObject::GetExternalIndex() const {
return m_externalIndex;
}
} // namespace MobileGL::MG_State::GLState