[MG_Remote] (P6/cp): a real trace retraces across two processes, pixel-for-pixel

OpenRA.DirectGLES.SPAWN is green: ssim=1.000000, mismatchPixels=0, zero
Fatal{ lines, against a server in a process the client started by name. The
inproc SPLIT arm stays beside it as the control, so "did the transport break
it" and "did the phase break it" remain separately answerable.

Five defects stood between the handshake and that picture. Each one was
silent in its own way, and none of them looked like what it was.

1. THE DOORBELL COULD NOT WAKE ITSELF. SocketDoorbell's one-fd form sends on
   the end it polls, so a write is delivered to the PEER - correct for waking
   the other process, useless for waking a thread in this one. The server's
   control pump posts into the apply thread's mailbox and must ring the bell
   that thread parks on. Symptom: every eglInitialize crossed, landed in the
   mailbox, and timed out at 5 s. The two-fd form (park on [0], notify on [1])
   is restored here, and the bell handover goes from two descriptors to THREE
   - the client needs the same split for the same reason. Seven offers now.

2. InitCapabilities HAD NO WIRE KIND. CONTRACT-P6 §2 predicted this exactly
   ("the request has a live client caller and the codec refuses to encode
   it"). SurfaceOpKind gains 11, append-only; the decode range's upper bound
   moves with it, or the new tag reads as out-of-range and surfaces as
   Fatal{ProtocolCorruption} rather than the missing row it is. The op's name
   loses its InprocOnly suffix, which stopped being true here.

3. THE CHILD LOADED NO CONFIG AT ALL. mobilegl_server_main calls
   InitServerRoleForSpawn directly, and MG_ConfigLoader::Init is a step of
   MobileGL::Initialize - so every MG_Config value in the server was its
   static default. MOBILEGL_BACKEND_TYPE survives the launcher's scrub on
   purpose and was read by nobody: a DirectVulkan session would have brought
   up a DirectGLES server and then disagreed with its client about which
   backend the capability mask describes.

4. AND THEREFORE Transport READ AS Monolith, which CONTRACT-P6 §3.1 (D1)
   forbids in as many words. a6 measured the cost: 226 live lines test
   `Transport != Monolith` to select the SERVER arm, 148 in MG_Backend.
   Monolith flips all of them to frontend glue in the one process with no
   frontend. Observed precisely as predicted - UnitTexturesByHandle() answered
   false, Clear took the legacy arm that dereferences borrowed FRONTEND slot
   pointers, and the apply thread took a SIGSEGV inside SyncNeccessaryTextures
   on the first Clear record of the trace. It is pinned in-process rather than
   through the environment because the environment is where it must NOT be:
   LooksLikeAClientEnvironment exits 65 on a surviving MOBILEGL_TRANSPORT, and
   that check is anti-recursion catch (b). Dial, not Transport, is that axis.

5. THE LOG WAS CORRUPT, NOT MERELY INTERLEAVED. Two processes opened the same
   MOBILEGL_LOG_FILE_PATH with "w" and each carried its OWN file offset, so
   they overwrote each other's bytes - half-lines like `ote client: CapsMirror
   generation 1 adopted`, and the server's whole startup block vanishing under
   the client's later writes. This cost real diagnosis time twice, because a
   missing line read as a code path not taken. O_APPEND now; the client still
   truncates once so a fresh run is not a pile of old ones, and the server
   never does - it would delete the very startup lines a failed session needs.
   run_trace_case.cmake's Fatal{ census reads exactly this file, so it was
   also reading a lie.

The retrace runner's split assertions become transport-aware: one row per
transport it admits, and a transport with no row is a FATAL rather than a
fall-through, because the fall-through is the accident the block exists to
prevent. The spawn row carries a SECOND sentence and it is the stronger half
- ConfigLoader's marker proves the value RESOLVED, while `spawn ARMED - the
server role runs in pid N` is written only after the launch, the connect and
the handshake all succeeded, and carries a pid that is not this process's.
inproc resolves its own marker while running the server role on a thread
here; nothing on that path can write this one.

G14: one test name added (TheEleventhFramedOpIsInitCapabilitiesAndItCrosses-
TheWire), none removed. The ten-op table keeps its name and its scope rather
than being renamed to eleven.

Two unit failures predate this change and are environmental in this tree:
PipeCatalogue.FrontendNeverTakesAnApplierAddress (the frontend source root
resolves empty) and SessionHandshakeTest.TheAbiFingerprintChangesWhenAnyOf-
ItsInputsDoes (GIT_COMMIT_HASH_SHORT is empty here, which a6 already recorded
as red in this configuration).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tools/trace_replay/CMakeLists.txt

@@ -465,6 +465,22 @@ function(add_trace_replay_split_test CASE_NAME)
     endif()
     add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPLIT TRANSPORT inproc
             IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
+
+    # --- P6: the SPAWN arm, beside the inproc one rather than instead of it ------------------
+    #
+    # TWO ARMS, NOT ONE RENAMED. The inproc arm is the CONTROL: it is the same
+    # library, the same codec, the same applier and the same picture, with the
+    # only difference being which process the server role runs in. A spawn arm
+    # that had replaced it would have left "did the transport break it?" and
+    # "did the phase break it?" answerable only together.
+    #
+    # G14 is satisfied by construction - a name is added, none is removed - and
+    # the VARIANT gives it its own output and artifact directories, which is the
+    # half the comment above says is easy to forget: both arms write
+    # output/mobilegl.log, and that log is the ONLY valid refusal census because
+    # the console sink is compiled out of these configurations.
+    add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPAWN TRANSPORT spawn
+            IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
 endfunction()
 
 set(MOBILEGL_TRACE_CASES_CMAKE "${CMAKE_CURRENT_BINARY_DIR}/trace_cases.cmake")

tools/trace_replay/run_trace_case.cmake

@@ -262,7 +262,34 @@ if(DEFINED ENV{MOBILEGL_TRANSPORT} AND NOT "$ENV{MOBILEGL_TRANSPORT}" STREQUAL "
         # DEBUG to debug a split failure. The sentence below exists only in
         # ConfigLoader::InitTransport's InProcess arm, which exists only under
         # MOBILEGL_BUILD_DISAGGREGATED.
-        set(split_expected_marker "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream")
+        #
+        # ONE ROW PER TRANSPORT THIS RUNNER ADMITS. A transport with no row is a
+        # FATAL below rather than a fall-through, because the fall-through is the
+        # exact accident this whole block exists to prevent: an arm that named a
+        # transport nobody asserted on and went green having proved nothing.
+        #
+        # THE SPAWN ROW CARRIES A SECOND SENTENCE, and it is the stronger half.
+        # The ConfigLoader marker proves the VALUE RESOLVED - it is a statement
+        # about a parser. `spawn ARMED - the server role runs in pid N` is
+        # written by ClientSession::StartSpawned only after the connect and the
+        # handshake succeeded, and it carries a pid that is not this process's.
+        # inproc resolves its own marker while running the server role on a
+        # thread HERE; nothing on that path can write this one.
+        set(split_expected_second "")
+        if("$ENV{MOBILEGL_TRANSPORT}" STREQUAL "inproc")
+            set(split_expected_marker "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream")
+        elseif("$ENV{MOBILEGL_TRANSPORT}" STREQUAL "spawn")
+            set(split_expected_marker "MOBILEGL_TRANSPORT=spawn - the MGPipe record stream")
+            set(split_expected_second "spawn ARMED - the server role runs in pid ")
+        else()
+            message(FATAL_ERROR
+                    "MOBILEGL_TRANSPORT=$ENV{MOBILEGL_TRANSPORT} is set for ${split_case} and this "
+                    "runner has no marker for it, so the run could not be checked. ConfigLoader "
+                    "recognises monolith|inproc|spawn|unix:<path>|pipe:<name> and REFUSES the last "
+                    "two BY NAME, staying on monolith - a run that asked for one of those would "
+                    "have replayed monolith. Add a row above when the transport lands; do not let "
+                    "an unasserted transport reach a green.")
+        endif()
         string(FIND "${split_log}" "${split_expected_marker}" split_armed_at)
         if(split_armed_at EQUAL -1)
             # Say which transport was actually asked for. ConfigLoader REFUSES spawn / unix: /
@@ -270,11 +297,14 @@ if(DEFINED ENV{MOBILEGL_TRANSPORT} AND NOT "$ENV{MOBILEGL_TRANSPORT}" STREQUAL "
             # has a different diagnosis from one that set inproc against a monolith library, and
             # the old message named `inproc` either way.
             set(split_refusal "")
-            if(NOT "$ENV{MOBILEGL_TRANSPORT}" STREQUAL "inproc")
+            if("$ENV{MOBILEGL_TRANSPORT}" STREQUAL "spawn")
                 set(split_refusal
-                        " NOTE: this run asked for '$ENV{MOBILEGL_TRANSPORT}', which P5 does not "
-                        "implement - ConfigLoader recognises spawn / unix: / pipe: and REFUSES them by "
-                        "name, staying on monolith. Only 'inproc' can resolve in P5.")
+                        " NOTE: this run asked for 'spawn'. The server is a SEPARATE PROCESS here, so "
+                        "the two ways to get this far with no marker are a library that has no spawn "
+                        "arm (pre-P6) and a launcher that never resolved the transport at all. "
+                        "MOBILEGL_IPC_SERVER_PATH must point at a libMobileGLServer.so built from this "
+                        "same tree; LaunchServer REFUSES an unresolvable image BY NAME and never falls "
+                        "back to running the server role locally.")
             endif()
             message(FATAL_ERROR
                     "MOBILEGL_TRANSPORT=$ENV{MOBILEGL_TRANSPORT} is set for ${split_case} and the library "
@@ -288,6 +318,23 @@ if(DEFINED ENV{MOBILEGL_TRANSPORT} AND NOT "$ENV{MOBILEGL_TRANSPORT}" STREQUAL "
                     "admits INFO (at WARN or above the line is compiled out and this reds for no "
                     "defect).${split_refusal}")
         endif()
+        # ... and the half that a same-process run cannot satisfy. Checked
+        # SEPARATELY from the marker above so the two failures read differently:
+        # no marker means the transport never resolved, while marker-but-no-pid
+        # means it resolved and the session never came up across the boundary.
+        if(NOT "${split_expected_second}" STREQUAL "")
+            string(FIND "${split_log}" "${split_expected_second}" split_second_at)
+            if(split_second_at EQUAL -1)
+                message(FATAL_ERROR
+                        "${split_case}: ${mobilegl_log} reports MOBILEGL_TRANSPORT="
+                        "$ENV{MOBILEGL_TRANSPORT} resolved, but carries no "
+                        "\"${split_expected_second}\". That sentence is written by "
+                        "ClientSession::StartSpawned AFTER the launch, the connect and the handshake "
+                        "all succeeded, and the pid in it is the SERVER's - it is the only evidence "
+                        "in this log that the server role left this process. Without it the run "
+                        "resolved a transport and then replayed something else.")
+            endif()
+        endif()
         # The refusal census. Recorded on every split run, pass or fail.
         file(STRINGS "${mobilegl_log}" split_fatals REGEX "Fatal\\{")
         list(LENGTH split_fatals split_fatal_count)