MobileGL/docs/Disaggregated/notes/p12/map-server-surface.md

# Server-side surface path and both backends' window surfaces (commit 9f669e52)

Everything below was read in code unless it is marked **[INFER]**. Nothing was edited or built.

## 1. How the control/surface ops travel today [VERIFIED]

**Wire schema**
- `SurfaceOpKind` is 0..11 (`protocol.fbs:218-237`). `WindowKind` is `None=0, AndroidNativeWindow=1, X11=2, Win32Hwnd=3, Surfaceless=4, Pbuffer=5, MetalLayer=6` (`:239-250`). Both are append-only.
- `SurfaceOp` carries `seq, kind, display, surface, windowKind, nativeToken, width, height, swapInterval, readSurface, context` (`:252-268`). The comment says "Android transfers the window out of band" (`:258`).
- `SurfaceReply.defaultFb` still exists in the schema (`:275`), but the encoder writes 0 (`SurfaceOpCodec.cpp:192-193`) and CONTRACT-P6.md:75 calls the slot burned.
- `SurfaceProgress` is at `:351-354`.

**Client emission**
- Every `Server*` forwarder builds a `SurfaceControlFrame` (`ServerLoop.cpp:1289-1404`) and calls `RunSurfaceControlFrame`.
- Under spawn or tcp, `m_remoteSink` routes the frame to `ClientSession::RunRemoteSurfaceControlFrame` (`ServerLoop.cpp:793-795`, `ClientSession.cpp:1205-1249`). An encode error there comes back as `MOBILEGL_ERR_UNSUPPORTED` (`:1217-1226`).
- `BackendObject_Remote::CreateEGLWindowSurface` always sends `ServerSetWindowHandle(handle)` first and then `ServerCreateEGLWindowSurface` (`BackendObject_Remote.cpp:215-216`). There is no split check.
- SwapBuffers is never forwarded (`:279-288`). Present travels as a class-B record.
- The frame is values only, and a compile-time binding enforces it (`SurfaceControlFrame.h:75-100`, `:113-114`). The four inproc-only kinds are 12..14 plus `InitCapabilities=11` on the wire (`:46-73`, `SurfaceControlFrame.cpp:34-54`).

**Server intake**
- Path: `ServerMain.cpp:594-612` → `ServerApplyWireSurfaceOp` (`SurfaceOpCodec.cpp:207-246`) → `DecodeWireSurfaceOp` (`:150-188`) → `ServerLoop::RunSurfaceControlFrame` (`:785-807`), which posts to the apply thread or runs inline if already on it (`:803`) → `ApplySurfaceControlFrame` (`:1097-1283`).
- `OpNeedsAWindowBackend` is true only for CreateWindowSurface and SetWindowHandle (`SurfaceOpCodec.cpp:60-62`).
- For a pbuffer, the encoder fills in `WindowKind::Pbuffer` itself (`:133-135`).

**Dispatch in `ApplySurfaceControlFrame`**
- A null backend answers `NOT_INITIALIZED` for every op (`:1110-1111`).
- **CreateWindowSurface**: `UnpackWindowHandle` (`:1050-1058`) turns the token back into a `void*`, then calls `backend->CreateEGLWindowSurface` and `ForgetCurrentTuple` (`:1126-1138`).
- **ResizeWindowSurface** (`:1139-1143`) and **CreatePbufferSurface** (`:1144-1151`) call the matching backend method.
- **MakeCurrent**: `ApplyMakeCurrent`, then a CapsSnapshot republish on a native bind (`:1152-1188`).
- **ReleaseCurrent**: recorded, not forwarded (`:1189-1200`).
- **SetSwapInterval**: `:1201-1204`.
- **ReleaseSurface**: `:1205-1216`.
- **ReleaseResources**: `ReleaseEGLResources` plus `SetContextLive(false)` (`:1217-1231`).
- **SetWindowHandle**: `backend->SetWindowHandle` (`:1232-1241`).
- **InitCapabilities**: `:1242-1254`.
- **Present**: `ServerVerbSink::OnPresent` → `table->Present()` (`PipeApplier.cpp:313-331`), then the present credit is returned (`:357-359`).

**Refusals, by name**
- `Fatal{UnmigratedSurface,"AndroidNativeWindow@P12"}` for AndroidNativeWindow on a window op (`SurfaceOpCodec.cpp:166-171` and `:217-223`).
- `Fatal{UnmigratedSurface,"MetalLayer@P12"}` (`:172-175`, `:224-228`).
- `WindowKindNamesNoBackend` for Surfaceless/Pbuffer on a window op, and `UnknownWindowKind`; both become `Fatal{ProtocolCorruption,"SurfaceOp"}` (`:176-182`, `:229-235`).
- `SurfaceOp.windowBackend` for an out-of-range tag (`ServerLoop.cpp:1039-1048`).
- `SurfaceOp.kind` for an unknown kind (`:1279-1282`).
- `ApplyThreadNotRunning` (`:815-831`).
- In an armed session child all of these latch the session instead of aborting (`SurfaceOpCodec.cpp:211`, `:215-216`).
- **The X11 refusal is not in the code.** X11, Win32Hwnd and None are accepted, and their token is copied through (`:183-184`). CONTRACT-P6.md:438-446 specifies `X11@P12` and `Win32Hwnd@P12`; they never landed.
- The client-side mirror of the Android refusal (CONTRACT-P6.md:451-456) is also missing: there are no `@P12` hits under `MG_Remote/Client`.

## 2. Backend window surfaces [VERIFIED]

**Base class, `BackendObject`**
- `WindowBackend {Android, X11, MetalLayer, Win32, Count, Unknown=-1}` (`BackendObject.h:569-577`); `WindowHandle` (`:579-584`); `SurfaceKind {None, Window, Pbuffer}` (`:616-620`).
- `CreateEGLWindowSurface` = `RegisterEGLWindowSurface` + `ActivateEGLSurface` (`BackendObject.cpp:191-194`).
- Register refuses Unknown/null (`:227-230`) and stores `surfaceState.Window = handle` (`:232-238`).
- Activate calls `SetWindowHandle(surfaceState->Window)` (`:283`), then `InitWindowSurface()` (`:284`). For a pbuffer it calls `InitPbufferSurface(w,h)` (`:288-292`).
- MakeCurrent activates lazily (`:318-323`, `:332`).
- **Stores to `m_windowHandle`:** `:475` (SetWindowHandle), `:365` (ResetEGLRuntimeState), and `:206-207` (Resize, width/height only; nothing reads those).
- `SwapEGLBuffers` → `Present()` (`:396`).

**DirectGLES (Espryt)**
- `InitWindowSurface` casts `m_windowHandle.Handle` → `DirectGLES::InitWindowSurface` (`BackendObject_DirectGLES.cpp:899-907`).
- `CreateEGLWindowSurface` checks the window backend (`:955-960`). The **sameHandle** dedup key (`:962-967`) requires the same surface, kind Window, and the same Backend and Handle. Otherwise it runs `DestroyEGLContext` + `ResetEGLRuntimeState` (`:969-972`).
- Process globals `g_Display/g_Context/g_Surface/g_Config` (`DirectGLES.cpp:15814-15817`).
- `InitDisplayAndContext` destroys the old context first (`:16159`), then `eglGetDisplay(EGL_DEFAULT_DISPLAY)` (`:16177`). The config has a single surface bit (`:16085`).
- `InitWindowSurface` → `eglCreateWindowSurface` (`:16230`) → MakeCurrent → publish (`:16225-16241`). The pbuffer twin is `:16243-16258`.
- `Present` → `eglSwapBuffers(g_Display, g_Surface)`, **return value ignored** (`:16842`).
- `DestroyEGLContext` destroys the surface and context and calls `eglTerminate` (`:16907-16919`).
- Resize is not overridden (`BackendObject_DirectGLES.h:39-44`), so it only updates bookkeeping.
- **Width/Height gap:** `PublishDefaultFramebufferDepthStencilFormat` posts `MGPSurfaceInfo` with the format only, W/H = 0 (`DirectGLES.cpp:15850-15852`, `:15943-15950`). `QueryCurrentSurfaceSize` exists (`:15819-15837`) but is not used for publishing.

**DirectVulkan (Magma)**
- `CreateEGLWindowSurface` only registers (`BackendObject_DirectVulkan.cpp:418-431`). The real creation happens at the client's MakeCurrent: `Activate` → `InitWindowSurface` → `VulkanRenderer(nativeWindow)` (`:337-352`).
- `InitPbufferSurface` uses a config with `SurfaceWidth/Height` and a null window (`:354-365`).
- Resize → `RequestSwapchainResize` (`:433-446`, `VulkanRenderer.cpp:16533-16542`).
- Window: `vkCreateAndroidSurfaceKHR` (`VulkanRenderer.cpp:16196-16202`).
- Offscreen on Android: headless surface (`:16112-16121`), otherwise **an AImageReader ANativeWindow** (`:16129-16158`).
- Swapchain extent = `currentExtent`, or the clamped desired extent when currentExtent is UINT32_MAX (`SwapchainObject.cpp:194-202`). Quarter-turn swap (`:204-207`). `preTransform = currentTransform` (`:234`).
- `OnSurfaceChanged` carries **Width/Height** plus format (`:323-339`).
- On present: OUT_OF_DATE, or an extent/transform change (`VulkanRenderer.cpp:16501-16531`), → `RecreateSwapchain` (`:14381-14418`) → re-publish. Zero area sets `m_presentSuspended` (`:14386`). `VK_VERIFY` aborts on any other present or acquire result (`:14394`, `:14456`).

**How the size and format reach the client**
- `ServerOnSurfaceChanged` → `kEventSurfaceChanged` (`ServerSession.cpp:483-499`, installed at `:827-828`).
- `ApplySurfaceChangedToClient` (`ClientSession.cpp:306-347`): if W/H ≠ 0 it reallocates all three attachments (`:316-338`); otherwise it changes the format only (`:339-346`).
- The client drains after create, resize and make-current (`BackendObject_Remote.cpp:224`, `:233`, `:272`).

## 3. What the device split path renders into today [VERIFIED]

- The trace app forces a pbuffer for spawn (`trace_replay_jni.cpp:173-188`). apitrace creates a pbuffer sized `ResolveWidth/Height` (`apitrace_glws_android.cpp:198-219`).
- `EGLImpl.cpp:458-469` passes `EGL_WIDTH/EGL_HEIGHT` (default 1) → frame width/height → server `InitPbufferSurface(w,h)`.
  - Espryt: `eglCreatePbufferSurface`.
  - Magma: a headless surface or an AImageReader swapchain at w×h.
- Present on Espryt is `eglSwapBuffers` on a pbuffer (a no-op). On Magma it is `vkQueuePresent` to a surface nobody sees.
- The service exec's `libMobileGLServer.so --serve` (`MobileGLServerService.java:60-61`). The supervisor forks one child per session (`ServerMain.cpp:985`, `:1123`). The child runs `RunSession` (`:268`, `[[noreturn]]`), sets `Transport=Spawn` (`:356-358`) and calls `InitServerRoleForSpawn` (`:377`).
- This whole pbuffer chain is what must stay.

## 4. What `WindowKind::ServerOwned` needs on the server

- **Injection point [VERIFIED as the natural seam].** Substitute the server's window token in `ServerApplyWireSurfaceOp`, after decode and before `RunSurfaceControlFrame` (`SurfaceOpCodec.cpp:243`). Downstream nothing changes: `UnpackWindowHandle` → Register (`:232`) → Activate → SetWindowHandle (`:283`), which is exactly the inproc path today.
- **sameHandle** (`BackendObject_DirectGLES.cpp:962-964`) then compares the server's pointer with the server's pointer and dedups correctly. The concern a6 §6(d) raised disappears.
- **Present:** unchanged. The Present record → `DirectGLES::Present` `eglSwapBuffers` on the window surface, or Magma `vkQueuePresentKHR` on the real swapchain.
- **Size and rotation**
  - Magma already reports window resize and rotation (`SwapchainObject.cpp:323-339`); rotation is compensated inside the server via preTransform.
  - Espryt reports nothing (§2 gap).
  - Resize requests from the client do not move the extent on either backend. Espryt only updates bookkeeping; Magma uses currentExtent. A client-requested size therefore has to be applied to the window itself: `ANativeWindow_setBuffersGeometry` or Java `SurfaceHolder.setFixedSize`, which is the lever the code comment names (`VulkanRenderer.cpp:14402-14405`).
- **Window absent:** an absent window must be checked both when CreateWindowSurface is decoded and when the surface is activated, because on Magma activation is deferred to MakeCurrent (`BackendObject.cpp:319`, `:332`).
- **Existing precedent:** the codebase refuses by name and never falls back silently (`ServerLoop.cpp:180-186`, `:824-831`).

## OPEN QUESTIONS / RISKS

1. **Process topology is the main blocker [INFER].** An ANativeWindow from a SurfaceView is a Binder proxy living in the APK's JVM process. The server is an exec'd binary that forks one child per session, and neither exec nor fork carries a usable window (Binder does not survive fork). The options:
   - (a) Run the session in-process via JNI, in a dedicated `android:process` that is restarted per session. `RunSession` exits the process when it ends, and `ServerLoopInstance` is a leaked singleton (`ServerLoop.cpp:980-984`).
   - (b) Pass a Parcel to the exec'd child. This needs API 34 and a binder route to a native process; impractical.
   - (c) Keep the server offscreen and send `AHardwareBuffer`s over the existing AF_UNIX socket (API 26) to an APK-side presenter.
2. **Selection knob.** Should the mode be client-driven (the client sends ServerOwned) or a server environment policy? Should a client pbuffer be allowed to go on-screen?
3. **Espryt Width/Height publishing.** Publishing W/H through the transport arm switches the client to the reallocating arm (`ClientSession.cpp:316-338`), which is a divergence from monolith. It should be gated to ServerOwned only.
4. **Initial viewport and client default-FB extent.** Does the frontend's initial viewport or the default-FB extent depend on the 512×512 placeholder? That is another slice.
5. **Losing the window mid-session.**
   - Espryt: destroying `g_Surface` destroys the context as well (`DirectGLES.cpp:16907-16919`). Parking on a pbuffer would need `g_*` de-globalisation and a config with both WINDOW and PBUFFER bits (`:16085`).
   - Magma: it aborts on `SURFACE_LOST` (`:14394`), and `vkCreateAndroidSurfaceKHR` failure is `VK_VERIFY` rather than a `false` return.
   - Espryt ignores the `eglSwapBuffers` error (`:16842`).
6. **`surfaceDestroyed` threading.** Teardown must be serialized onto the apply thread before the Java callback returns.
7. **Unrelated gap.** The X11/Win32/None hole in CONTRACT-P6 §7.2 is still open.

## MINIMAL CHANGE LIST

- **`protocol.fbs:249`**: append `ServerOwned = 7` and regenerate. **`SurfaceOpCodec.cpp`**:
  - Add a static_assert for the new value; widen the `IsOutRange` bound (`:178-179`).
  - In the decoder, add a ServerOwned arm before the Android refusal. It requires `nativeToken==0` (otherwise `Fatal{ProtocolCorruption,"SurfaceOp.nativeToken"}`) and sets `windowBackend=Android`.
  - In `ServerApplyWireSurfaceOp`, fill the token from the provider. Add a new `SurfaceWireError::ServerWindowAbsent` that answers `ok=false`, is logged by name, and does not latch; the client then gets `EGL_BAD_NATIVE_WINDOW` (`EGLImpl.cpp:153-156`).
  - Encoder (client side): under spawn/tcp, map `WindowBackend::Android` → ServerOwned with token 0, and send the client's desired w/h.
- **New `Server/ServerWindowProvider.{h,cpp}`**:
  - A process-wide holder, `SetServerOwnedWindow(ANativeWindow*)` with acquire/release, a generation counter, and `ApplyGeometry(w,h)`.
  - It is set by the APK's JNI before a session starts; no window means the ServerOwned path is refused.
  - This is the "both paths" switch: pbuffer ops are untouched.
- **`SurfaceControlFrame.h/.cpp`, `ServerLoop.cpp`**:
  - Append an inproc-only `ServerWindowLost` kind (15), posted by the provider on `surfaceDestroyed`.
  - Its dispatch releases the surface and latches through a new `FatalFamilies.def` row (e.g. `ServerWindowLost`).
  - Recheck the provider at activation (`BackendObject.cpp:283-284`) or in `UnpackWindowHandle` (`ServerLoop.cpp:1050-1058`).
- **`DirectGLES.cpp`**:
  - `:15947-15950`: fill W/H from `QueryCurrentSurfaceSize` when the surface is a server-owned window.
  - `:16842`: check the `eglSwapBuffers` result, re-query the size after the swap, and call `OnSurfaceChanged` when it changed.
  - `g_*` de-globalisation is not needed while there is one backend per session process.
- **`VulkanRenderer.cpp:14394`, `:14456`, `:16202`**: turn `SURFACE_LOST` and a failed surface creation into named latches instead of aborts. The on-screen path itself needs nothing new, because `InitWindowSurface` already takes a real window.
- **`BackendObject_DirectGLES.cpp:962-967`**: no change once the token is the server's.
- **`ServerMain.cpp`, `MobileGLServerService.java`, JNI**: decide the topology in risk 1. Option (a) needs an in-process entry point without fork; the service is gated by environment or intent extras.
- **Tests**:
  - `SurfaceControlFrameTest.cpp:125-163`, `:236-249`, `:323`, `:376` (enum bounds).
  - `PeerLatchTest.cpp:1250-1257`: add ServerOwned rows (absent, nonzero token).
  - `ServerLoopTest`: a lost-window control.
- **Client (another slice)**: `BackendObject_Remote.cpp:215` currently sends the client pointer unconditionally. With the encoder mapping above it becomes harmless.