mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-19 16:01:57 +09:00
[MG_Remote, MG_Test, MG_IntegrationTest, scripts] (P7): a spawned server reports SurfaceProgress while its apply thread runs a surface op, so the client's reply budget bounds silence and a cold bring-up past 20 s is waited for
This commit is contained in:
@@ -394,8 +394,8 @@ is a pull-build symbol and G1 admits **no resize**, which is the same reason the
|
||||
| `MOBILEGL_IPC_STRICT_ERRORS` | 0 | promotes BARRIER-PULLED reads — and, in a split build, the seven sticky forwards — to `Fatal`. |
|
||||
| `MOBILEGL_IPC_AUDIT` | 0 | `0xDD` over retired staging bytes (rule C's mechanical control). |
|
||||
| `MOBILEGL_IPC_SERVER_AFFINITY` | `auto` | kept as the raw string; whoever starts the apply thread logs the **resolved mask**, because an affinity that silently did nothing looks exactly like one that worked. |
|
||||
| `MOBILEGL_IPC_CONTROL_TIMEOUT_MS` | 5000 | (P6 D5b, parsed from P7) a spawn/tcp client's wait for one surface-control `SurfaceReply` once the server's backend is up. Expiry is **not** fatal: dead latches device-lost, alive-but-silent is a named diagnostic. Range 100..600000. |
|
||||
| `MOBILEGL_IPC_COLD_START_MS` | 20000 | (P7) the same wait for `CreatePbufferSurface` / `CreateWindowSurface` / `MakeCurrent` until the session's first `MakeCurrent` is answered ok - the ops a server brings its native backend up inside, lazily. Never shorter than `CONTROL_TIMEOUT_MS`. Measured need: retrace-split spawn legs outlasted 5 s on loaded CI runners. The test lever is the server's `MOBILEGL_TEST_DELAY_FIRST_BRINGUP_MS`. |
|
||||
| `MOBILEGL_IPC_CONTROL_TIMEOUT_MS` | 5000 | (P6 D5b, parsed from P7) a spawn/tcp client's wait for one surface-control `SurfaceReply` once the server's backend is up. Expiry is **not** fatal: dead latches device-lost, alive-but-silent is a named diagnostic. Range 100..600000. Since p7/spawnhang it bounds **silence**, not the op: each `SurfaceProgress` the server sends while its apply thread runs the op (every 250 ms, `ServerLoop::kControlProgressIntervalMs`) restarts it, up to `kBarrierTimeoutMs` (120 s) from the send. |
|
||||
| `MOBILEGL_IPC_COLD_START_MS` | 20000 | (P7) the same wait for `CreatePbufferSurface` / `CreateWindowSurface` / `MakeCurrent` until the session's first `MakeCurrent` is answered ok - the ops a server brings its native backend up inside, lazily. Never shorter than `CONTROL_TIMEOUT_MS`. Measured need: retrace-split spawn legs outlasted 5 s on loaded CI runners, and one outlasted 20 s (run 35912252677, `CreatePbufferSurface` seq 2) - which is why, since p7/spawnhang, this too is a silence budget that the server's `SurfaceProgress` reports restart (row above). The test lever is the server's `MOBILEGL_TEST_DELAY_FIRST_BRINGUP_MS`. |
|
||||
|
||||
**One consequence, stated so it is not rediscovered.** In a build *without*
|
||||
`MOBILEGL_BUILD_DISAGGREGATED`, `MOBILEGL_TRANSPORT=inproc` is accepted by the environment and
|
||||
|
||||
@@ -335,7 +335,9 @@ into the pull build too, and is now guarded for the same reason.
|
||||
its expiry is `Fatal{ControlReplyTimeout, "<op>@<seq>"}`. (b) The client's *cross-process* reply wait
|
||||
is bounded by `MOBILEGL_IPC_CONTROL_TIMEOUT_MS` (default 5000) and its expiry is **not** fatal: the
|
||||
client consults the doorbell's death latch — dead latches device-lost, alive-but-silent is a named
|
||||
diagnostic. Packages: `cp` (a), `dl` (b).
|
||||
diagnostic. Packages: `cp` (a), `dl` (b). (P7, p7/spawnhang: "silent" is now literal — a server whose
|
||||
apply thread is running the op says so every 250 ms with `SurfaceProgress`, and each report restarts
|
||||
the client's bound, up to 120 s; a server that never took the op, or is frozen, sends none.)
|
||||
|
||||
> **D5c — the client's self-bell CANNOT witness the server's death, and `sm` is why.** Measured, not
|
||||
> predicted: `minecraft-1.21.4-fabric-iris-iterationrp-in-world` kills the *server* process outright
|
||||
|
||||
@@ -1251,9 +1251,32 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// The reply. A BOUNDED wait whose expiry is NOT fatal (CONTRACT-P6 §5.4
|
||||
// D5b): a peer that is gone is a different fact from a peer that is slow,
|
||||
// and only the doorbell's death latch may say which.
|
||||
//
|
||||
// P7 (p7/spawnhang): THE BUDGET BOUNDS SILENCE, NOT THE OP. It used to be one deadline
|
||||
// for the whole reply, so a server that was still running the op - a cold native
|
||||
// bring-up, eglInitialize loading a software rasteriser off a runner's cold disk for
|
||||
// ~20 s (retrace-split run 35912252677) - was reported ALIVE BUT SILENT and its surface
|
||||
// creation failed, although it was about to answer. The server now says so while it runs
|
||||
// an op it has taken (Wire::SurfaceProgress, every ServerLoop::kControlProgressIntervalMs),
|
||||
// and each report restarts this budget. What still times out, named as before: a server
|
||||
// that never took the op or is frozen (no report arrives), and one that reports progress
|
||||
// past kBarrierTimeoutMs - the bound the verb barrier gives one record's apply, which is
|
||||
// what a surface op that is being run IS.
|
||||
const auto sentAt = std::chrono::steady_clock::now();
|
||||
const Uint32 ceilingMs = std::max<Uint32>(budgetMs, kBarrierTimeoutMs);
|
||||
Uint32 progressReports = 0;
|
||||
Uint32 lastProgressMs = 0;
|
||||
for (;;) {
|
||||
std::vector<Uint8> reply;
|
||||
const MobileGLResult received = ReceiveControlFrame(reply, budgetMs);
|
||||
Uint32 waitMs = budgetMs;
|
||||
if (progressReports != 0) {
|
||||
const auto spentMs = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now() - sentAt).count();
|
||||
waitMs = spentMs >= ceilingMs ? 0u : std::min<Uint32>(budgetMs, ceilingMs - static_cast<Uint32>(spentMs));
|
||||
}
|
||||
// A zero wait is the ceiling, reached: answered exactly as an expired budget is.
|
||||
const MobileGLResult received =
|
||||
waitMs == 0 ? MOBILEGL_ERR_TIMEOUT : ReceiveControlFrame(reply, waitMs);
|
||||
if (received != MOBILEGL_OK) {
|
||||
// `dl` (D5b): THE EXPIRY IS NOT FATAL AND IT IS NOT ONE ANSWER. The transport
|
||||
// already distinguishes the two states this wait can end in, and the contract
|
||||
@@ -1272,6 +1295,14 @@ namespace MobileGL::MG_Remote::Client {
|
||||
} else if (const Transport::Doorbell* bell = m_producer.SelfDoorbell();
|
||||
bell != nullptr && bell->PeerHungUp()) {
|
||||
LatchDeviceLost("a control reply timed out and the peer had hung up");
|
||||
} else if (progressReports != 0 && waitMs == 0) {
|
||||
MGLOG_E("MG_Remote client: the server is ALIVE AND BUSY past the ceiling - it "
|
||||
"reported %u time(s) that it was still running %s seq %llu (the last "
|
||||
"at %u ms) and has not answered within %u ms, the bound the verb "
|
||||
"barrier gives one apply. This is NOT device loss (CONTRACT-P6 5.4); "
|
||||
"the op itself is not returning on the server's apply thread.",
|
||||
progressReports, Server::SurfaceControlOpName(frame.kind),
|
||||
static_cast<unsigned long long>(frame.seq), lastProgressMs, ceilingMs);
|
||||
} else {
|
||||
MGLOG_E("MG_Remote client: the server is ALIVE BUT SILENT - no SurfaceReply "
|
||||
"for %s seq %llu within %u ms (%s) and the control stream is still "
|
||||
@@ -1283,9 +1314,23 @@ namespace MobileGL::MG_Remote::Client {
|
||||
coldStart ? "the cold-start budget, MOBILEGL_IPC_COLD_START_MS: the "
|
||||
"server may still have been bringing its backend up"
|
||||
: "the steady bound, MOBILEGL_IPC_CONTROL_TIMEOUT_MS");
|
||||
// WHICH SILENCE (p7/spawnhang). A server running the op reports progress, so
|
||||
// none at all means it never started it (or froze before it could say so);
|
||||
// reports that stopped mean it froze mid-op.
|
||||
if (progressReports == 0)
|
||||
MGLOG_E("MG_Remote client: no SurfaceProgress arrived for seq %llu either (a "
|
||||
"server reports every %u ms while it runs an op): it did not start "
|
||||
"the op, or it is frozen",
|
||||
static_cast<unsigned long long>(frame.seq),
|
||||
Server::ServerLoop::kControlProgressIntervalMs);
|
||||
else
|
||||
MGLOG_E("MG_Remote client: the server had reported progress on seq %llu "
|
||||
"%u time(s), the last at %u ms, and then stopped",
|
||||
static_cast<unsigned long long>(frame.seq), progressReports,
|
||||
lastProgressMs);
|
||||
}
|
||||
MGLOG_E("MG_Remote client: no SurfaceReply for seq %llu within %u ms (rc=%d)",
|
||||
static_cast<unsigned long long>(frame.seq), budgetMs,
|
||||
static_cast<unsigned long long>(frame.seq), waitMs == 0 ? ceilingMs : budgetMs,
|
||||
static_cast<int>(received));
|
||||
return received;
|
||||
}
|
||||
@@ -1304,6 +1349,30 @@ namespace MobileGL::MG_Remote::Client {
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (envelope->msg_type() == ::MobileGL::Wire::CtrlMsg::SurfaceProgress) {
|
||||
// THE SERVER IS STILL RUNNING THIS OP (p7/spawnhang, above): the budget restarts.
|
||||
// One op is outstanding at a time (m_remoteControlMutex), so a report naming any
|
||||
// other seq is a server that has lost track of which op it is running.
|
||||
const auto* progress = envelope->msg_as_SurfaceProgress();
|
||||
if (progress == nullptr || progress->seq() != frame.seq) {
|
||||
MGLOG_E("MG_Remote client: a SurfaceProgress names seq %llu while seq %llu is "
|
||||
"the op outstanding",
|
||||
static_cast<unsigned long long>(progress == nullptr ? 0 : progress->seq()),
|
||||
static_cast<unsigned long long>(frame.seq));
|
||||
return MOBILEGL_ERR_PROTOCOL_MISMATCH;
|
||||
}
|
||||
if (progressReports == 0) {
|
||||
MGLOG_I("MG_Remote client: the server is still running %s seq %llu (%u ms after "
|
||||
"it was posted to its apply thread); the reply budget of %u ms restarts "
|
||||
"on each progress report, up to %u ms",
|
||||
Server::SurfaceControlOpName(frame.kind),
|
||||
static_cast<unsigned long long>(frame.seq), progress->elapsedMs(), budgetMs,
|
||||
ceilingMs);
|
||||
}
|
||||
++progressReports;
|
||||
lastProgressMs = progress->elapsedMs();
|
||||
continue;
|
||||
}
|
||||
if (envelope->msg_type() == ::MobileGL::Wire::CtrlMsg::Fatal) {
|
||||
// `dl` step three: the server published a SessionFault before it aborted, so this
|
||||
// is a death that NAMES ITSELF instead of arriving as a bare EOF. Log the family
|
||||
|
||||
@@ -458,7 +458,8 @@ namespace MobileGL::MG_Remote::Client {
|
||||
Uint64 m_remoteControlSeq = 0;
|
||||
// P7 CI: false until this session's first MakeCurrent is answered ok. Until then the
|
||||
// ops the server may bring its native backend up inside wait MOBILEGL_IPC_COLD_START_MS
|
||||
// for their reply instead of MOBILEGL_IPC_CONTROL_TIMEOUT_MS (Config.h has both).
|
||||
// for their reply instead of MOBILEGL_IPC_CONTROL_TIMEOUT_MS (Config.h has both). Either
|
||||
// budget bounds SILENCE (p7/spawnhang): a Wire::SurfaceProgress for the op restarts it.
|
||||
// Guarded by m_remoteControlMutex, like the seq above; reset by Stop().
|
||||
Bool m_serverBackendWarm = false;
|
||||
|
||||
|
||||
@@ -77,6 +77,9 @@ struct FatalBuilder;
|
||||
struct LogLine;
|
||||
struct LogLineBuilder;
|
||||
|
||||
struct SurfaceProgress;
|
||||
struct SurfaceProgressBuilder;
|
||||
|
||||
struct CtrlEnvelope;
|
||||
struct CtrlEnvelopeBuilder;
|
||||
|
||||
@@ -506,11 +509,12 @@ enum class CtrlMsg : uint8_t {
|
||||
Refuse = 11,
|
||||
LogFlush = 12,
|
||||
DataBind = 13,
|
||||
SurfaceProgress = 14,
|
||||
MIN = NONE,
|
||||
MAX = DataBind
|
||||
MAX = SurfaceProgress
|
||||
};
|
||||
|
||||
inline const CtrlMsg (&EnumValuesCtrlMsg())[14] {
|
||||
inline const CtrlMsg (&EnumValuesCtrlMsg())[15] {
|
||||
static const CtrlMsg values[] = {
|
||||
CtrlMsg::NONE,
|
||||
CtrlMsg::Hello,
|
||||
@@ -525,13 +529,14 @@ inline const CtrlMsg (&EnumValuesCtrlMsg())[14] {
|
||||
CtrlMsg::LogLine,
|
||||
CtrlMsg::Refuse,
|
||||
CtrlMsg::LogFlush,
|
||||
CtrlMsg::DataBind
|
||||
CtrlMsg::DataBind,
|
||||
CtrlMsg::SurfaceProgress
|
||||
};
|
||||
return values;
|
||||
}
|
||||
|
||||
inline const char * const *EnumNamesCtrlMsg() {
|
||||
static const char * const names[15] = {
|
||||
static const char * const names[16] = {
|
||||
"NONE",
|
||||
"Hello",
|
||||
"Welcome",
|
||||
@@ -546,13 +551,14 @@ inline const char * const *EnumNamesCtrlMsg() {
|
||||
"Refuse",
|
||||
"LogFlush",
|
||||
"DataBind",
|
||||
"SurfaceProgress",
|
||||
nullptr
|
||||
};
|
||||
return names;
|
||||
}
|
||||
|
||||
inline const char *EnumNameCtrlMsg(CtrlMsg e) {
|
||||
if (::flatbuffers::IsOutRange(e, CtrlMsg::NONE, CtrlMsg::DataBind)) return "";
|
||||
if (::flatbuffers::IsOutRange(e, CtrlMsg::NONE, CtrlMsg::SurfaceProgress)) return "";
|
||||
const size_t index = static_cast<size_t>(e);
|
||||
return EnumNamesCtrlMsg()[index];
|
||||
}
|
||||
@@ -613,6 +619,10 @@ template<> struct CtrlMsgTraits<MobileGL::Wire::DataBind> {
|
||||
static const CtrlMsg enum_value = CtrlMsg::DataBind;
|
||||
};
|
||||
|
||||
template<> struct CtrlMsgTraits<MobileGL::Wire::SurfaceProgress> {
|
||||
static const CtrlMsg enum_value = CtrlMsg::SurfaceProgress;
|
||||
};
|
||||
|
||||
template <bool B = false>
|
||||
bool VerifyCtrlMsg(::flatbuffers::VerifierTemplate<B> &verifier, const void *obj, CtrlMsg type);
|
||||
template <bool B = false>
|
||||
@@ -2220,6 +2230,64 @@ inline ::flatbuffers::Offset<LogLine> CreateLogLineDirect(
|
||||
text__);
|
||||
}
|
||||
|
||||
struct SurfaceProgress FLATBUFFERS_FINAL_CLASS : private ::flatbuffers::Table {
|
||||
typedef SurfaceProgressBuilder Builder;
|
||||
struct Traits;
|
||||
enum FlatBuffersVTableOffset FLATBUFFERS_VTABLE_UNDERLYING_TYPE {
|
||||
VT_SEQ = 4,
|
||||
VT_ELAPSEDMS = 6
|
||||
};
|
||||
uint64_t seq() const {
|
||||
return GetField<uint64_t>(VT_SEQ, 0);
|
||||
}
|
||||
uint32_t elapsedMs() const {
|
||||
return GetField<uint32_t>(VT_ELAPSEDMS, 0);
|
||||
}
|
||||
template <bool B = false>
|
||||
bool Verify(::flatbuffers::VerifierTemplate<B> &verifier) const {
|
||||
return VerifyTableStart(verifier) &&
|
||||
VerifyField<uint64_t>(verifier, VT_SEQ, 8) &&
|
||||
VerifyField<uint32_t>(verifier, VT_ELAPSEDMS, 4) &&
|
||||
verifier.EndTable();
|
||||
}
|
||||
};
|
||||
|
||||
struct SurfaceProgressBuilder {
|
||||
typedef SurfaceProgress Table;
|
||||
::flatbuffers::FlatBufferBuilder &fbb_;
|
||||
::flatbuffers::uoffset_t start_;
|
||||
void add_seq(uint64_t seq) {
|
||||
fbb_.AddElement<uint64_t>(SurfaceProgress::VT_SEQ, seq, 0);
|
||||
}
|
||||
void add_elapsedMs(uint32_t elapsedMs) {
|
||||
fbb_.AddElement<uint32_t>(SurfaceProgress::VT_ELAPSEDMS, elapsedMs, 0);
|
||||
}
|
||||
explicit SurfaceProgressBuilder(::flatbuffers::FlatBufferBuilder &_fbb)
|
||||
: fbb_(_fbb) {
|
||||
start_ = fbb_.StartTable();
|
||||
}
|
||||
::flatbuffers::Offset<SurfaceProgress> Finish() {
|
||||
const auto end = fbb_.EndTable(start_);
|
||||
auto o = ::flatbuffers::Offset<SurfaceProgress>(end);
|
||||
return o;
|
||||
}
|
||||
};
|
||||
|
||||
inline ::flatbuffers::Offset<SurfaceProgress> CreateSurfaceProgress(
|
||||
::flatbuffers::FlatBufferBuilder &_fbb,
|
||||
uint64_t seq = 0,
|
||||
uint32_t elapsedMs = 0) {
|
||||
SurfaceProgressBuilder builder_(_fbb);
|
||||
builder_.add_seq(seq);
|
||||
builder_.add_elapsedMs(elapsedMs);
|
||||
return builder_.Finish();
|
||||
}
|
||||
|
||||
struct SurfaceProgress::Traits {
|
||||
using type = SurfaceProgress;
|
||||
static auto constexpr Create = CreateSurfaceProgress;
|
||||
};
|
||||
|
||||
struct CtrlEnvelope FLATBUFFERS_FINAL_CLASS : private ::flatbuffers::Table {
|
||||
typedef CtrlEnvelopeBuilder Builder;
|
||||
struct Traits;
|
||||
@@ -2273,6 +2341,9 @@ struct CtrlEnvelope FLATBUFFERS_FINAL_CLASS : private ::flatbuffers::Table {
|
||||
const MobileGL::Wire::DataBind *msg_as_DataBind() const {
|
||||
return msg_type() == MobileGL::Wire::CtrlMsg::DataBind ? static_cast<const MobileGL::Wire::DataBind *>(msg()) : nullptr;
|
||||
}
|
||||
const MobileGL::Wire::SurfaceProgress *msg_as_SurfaceProgress() const {
|
||||
return msg_type() == MobileGL::Wire::CtrlMsg::SurfaceProgress ? static_cast<const MobileGL::Wire::SurfaceProgress *>(msg()) : nullptr;
|
||||
}
|
||||
template <bool B = false>
|
||||
bool Verify(::flatbuffers::VerifierTemplate<B> &verifier) const {
|
||||
return VerifyTableStart(verifier) &&
|
||||
@@ -2335,6 +2406,10 @@ template<> inline const MobileGL::Wire::DataBind *CtrlEnvelope::msg_as<MobileGL:
|
||||
return msg_as_DataBind();
|
||||
}
|
||||
|
||||
template<> inline const MobileGL::Wire::SurfaceProgress *CtrlEnvelope::msg_as<MobileGL::Wire::SurfaceProgress>() const {
|
||||
return msg_as_SurfaceProgress();
|
||||
}
|
||||
|
||||
struct CtrlEnvelopeBuilder {
|
||||
typedef CtrlEnvelope Table;
|
||||
::flatbuffers::FlatBufferBuilder &fbb_;
|
||||
@@ -2429,6 +2504,10 @@ inline bool VerifyCtrlMsg(::flatbuffers::VerifierTemplate<B> &verifier, const vo
|
||||
auto ptr = reinterpret_cast<const MobileGL::Wire::DataBind *>(obj);
|
||||
return verifier.VerifyTable(ptr);
|
||||
}
|
||||
case CtrlMsg::SurfaceProgress: {
|
||||
auto ptr = reinterpret_cast<const MobileGL::Wire::SurfaceProgress *>(obj);
|
||||
return verifier.VerifyTable(ptr);
|
||||
}
|
||||
default: return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,7 +64,8 @@ extern "C" {
|
||||
// it runs as ProtocolSchema.RevisionPinsDigest (label unit) and in CI's flatc-check job. After a
|
||||
// bump: `python3 scripts/ci/protocol_revision_pin.py --write` records the new row.
|
||||
// 1 P7 wave 2-F, PH-7 (4): Welcome.dataNonce and the DataBind message.
|
||||
#define MOBILEGL_PROTOCOL_CONTROL_REVISION 1
|
||||
// 2 P7 (p7/spawnhang): the SurfaceProgress message.
|
||||
#define MOBILEGL_PROTOCOL_CONTROL_REVISION 2
|
||||
|
||||
#define MOBILEGL_ABI_VERSION(major, minor) (((uint32_t)(major) << 16) | (uint32_t)(minor))
|
||||
#define MOBILEGL_ABI_MAJOR_OF(version) ((uint32_t)(version) >> 16)
|
||||
|
||||
@@ -340,6 +340,19 @@ table LogLine {
|
||||
text: string;
|
||||
}
|
||||
|
||||
// P7 (p7/spawnhang). THE SERVER IS STILL RUNNING THE CLIENT'S SURFACE OP. Sent only by the
|
||||
// server's control pump, only while its apply thread is running a SurfaceOp it has TAKEN, and
|
||||
// only before that op's SurfaceReply - once per progress interval after the op has run that long
|
||||
// (ServerLoop.cpp, kControlProgressIntervalMs). It is how a client tells a server that is BUSY
|
||||
// with its op (a cold native bring-up inside the first surface creation or MakeCurrent: eglInitialize
|
||||
// loading a software rasteriser off a cold disk) from one that is SILENT: the client's reply budget
|
||||
// restarts on each report, up to a ceiling. A server that never took the op, or is frozen, sends
|
||||
// none and still times out by name (CONTRACT-P6 5.4 D5b).
|
||||
table SurfaceProgress {
|
||||
seq: ulong; // the SurfaceOp.seq being run
|
||||
elapsedMs: uint; // since the op was posted to the apply thread
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Envelope
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -363,6 +376,7 @@ union CtrlMsg {
|
||||
Refuse,
|
||||
LogFlush,
|
||||
DataBind,
|
||||
SurfaceProgress,
|
||||
}
|
||||
|
||||
table CtrlEnvelope {
|
||||
|
||||
@@ -633,9 +633,21 @@ namespace MobileGL::MG_Remote::Server {
|
||||
// The dispatch runs OUTSIDE the lock, exactly as the old work(user) did, and fills the
|
||||
// frame's reply half; the reply is then published back into the slot under the lock so
|
||||
// the poster's copy-out after m_controlDone sees it.
|
||||
const auto dispatchStart = std::chrono::steady_clock::now();
|
||||
const MobileGLResult result = ApplySurfaceControlFrame(
|
||||
frame, m_controlProbeHook.load(std::memory_order_acquire),
|
||||
m_controlProbeUser.load(std::memory_order_acquire));
|
||||
// p7/spawnhang: A SLOW DISPATCH NAMES ITSELF. The first surface creation / MakeCurrent is
|
||||
// where a lazy native bring-up runs, and on a cold CI runner it outran the client's whole
|
||||
// reply budget with nothing in this log to say so - the investigation had to infer it.
|
||||
const auto dispatchMs = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now() - dispatchStart).count();
|
||||
if (dispatchMs >= static_cast<long long>(kSlowControlDispatchMs)) {
|
||||
MGLOG_W("MG_Remote server: %s seq %llu ran %lld ms on mgl-srv-apply (a lazy native "
|
||||
"bring-up runs inside the first surface creation or MakeCurrent)",
|
||||
SurfaceControlOpName(frame.kind), static_cast<unsigned long long>(frame.seq),
|
||||
static_cast<long long>(dispatchMs));
|
||||
}
|
||||
{
|
||||
const std::lock_guard<std::mutex> lock(m_controlMutex);
|
||||
m_controlFrame = frame;
|
||||
@@ -844,11 +856,42 @@ namespace MobileGL::MG_Remote::Server {
|
||||
if (m_session != nullptr) {
|
||||
m_session->DataLink()->ConsumerBell().Notify();
|
||||
}
|
||||
m_controlDone.wait(lock, [this] { return m_controlFinished; });
|
||||
// p7/spawnhang: THE WAIT IS SLICED, AND A SLICE THE APPLY THREAD SPENT RUNNING THIS FRAME IS
|
||||
// REPORTED (SetControlProgressSink). It used to be one untimed wait, so the poster - under
|
||||
// spawn, the control pump in ServerMain, the only thread that talks to the client - was mute
|
||||
// for as long as the dispatch ran, and a client could not tell a cold bring-up from a wedge.
|
||||
// "Running" is read under this lock from the mailbox's own fields: the pump clears the
|
||||
// shadow when it TAKES the frame (PumpControlRequest), and m_controlPending stays true until
|
||||
// the reply is published under this same lock. Posted-but-not-taken says nothing. The sink
|
||||
// runs without the lock, so a dispatch finishing meanwhile is not held up by the send; the
|
||||
// predicate is re-asked on the way back in. Only the timing of the wakeups changed: the
|
||||
// handshake, the C2 exit block's answer and the copy-out below are exactly as before.
|
||||
const auto postedAt = std::chrono::steady_clock::now();
|
||||
while (!m_controlDone.wait_for(lock, std::chrono::milliseconds(kControlProgressIntervalMs),
|
||||
[this] { return m_controlFinished; })) {
|
||||
const Bool running = m_controlPending && !m_controlPosted.load(std::memory_order_acquire);
|
||||
const ControlProgressSink sink = m_progressSink;
|
||||
void* const user = m_progressUser;
|
||||
if (!running || sink == nullptr) continue;
|
||||
const auto elapsedMs = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now() - postedAt).count();
|
||||
lock.unlock();
|
||||
sink(user, frame.kind, frame.seq,
|
||||
static_cast<Uint32>(std::min<long long>(elapsedMs, 0xFFFFFFFFll)));
|
||||
lock.lock();
|
||||
}
|
||||
frame = m_controlFrame; // the reply half, written by the dispatch
|
||||
return m_controlResult;
|
||||
}
|
||||
|
||||
void ServerLoop::SetControlProgressSink(ControlProgressSink sink, void* user) {
|
||||
// Under the mailbox's lock, which the poster holds when it reads the pair, so a sink is
|
||||
// never called with another sink's user.
|
||||
const std::lock_guard<std::mutex> lock(m_controlMutex);
|
||||
m_progressSink = sink;
|
||||
m_progressUser = user;
|
||||
}
|
||||
|
||||
MobileGLResult ServerLoop::RunProbeOnApplyThreadForTesting(ControlProbeHook hook, void* user) {
|
||||
SurfaceControlFrame frame;
|
||||
frame.kind = SurfaceControlOp::ProbeForTesting;
|
||||
|
||||
@@ -249,6 +249,28 @@ namespace MobileGL::MG_Remote::Server {
|
||||
using ControlProbeHook = MobileGLResult (*)(void* user);
|
||||
MobileGLResult RunProbeOnApplyThreadForTesting(ControlProbeHook hook, void* user);
|
||||
|
||||
// P7 (p7/spawnhang). WHAT A POSTER SAYS WHILE THE APPLY THREAD RUNS ITS FRAME.
|
||||
//
|
||||
// A posted frame is waited for in slices of kControlProgressIntervalMs, and after every
|
||||
// slice in which the apply thread is RUNNING it - taken, not yet answered - the poster calls
|
||||
// this sink with the frame's kind, its seq and how long ago it was posted, WITHOUT
|
||||
// m_controlMutex held. ServerMain installs one that sends Wire::SurfaceProgress on the
|
||||
// control connection: that is how a spawn / TCP client tells a server BUSY with its op (a
|
||||
// cold native bring-up - eglInitialize loading a software rasteriser off a cold disk ran
|
||||
// ~20 s on a CI runner) from a SILENT one, and restarts its reply budget. A frame posted but
|
||||
// NOT TAKEN reports nothing, because that silence is exactly what the client's budget
|
||||
// exists to name. Null (inproc, and every case that does not set one): the slices are
|
||||
// waited out and nothing is said. Read on the posting thread, set from any (both under
|
||||
// m_controlMutex).
|
||||
using ControlProgressSink = void (*)(void* user, SurfaceControlOp kind, Uint64 seq, Uint32 elapsedMs);
|
||||
void SetControlProgressSink(ControlProgressSink sink, void* user);
|
||||
// Well inside the smallest default reply budget (MOBILEGL_IPC_CONTROL_TIMEOUT_MS, 5000),
|
||||
// and a frame's worth of bytes per interval only while an op runs that long.
|
||||
static constexpr Uint32 kControlProgressIntervalMs = 250;
|
||||
// A dispatch that runs at least this long is logged, with its op, seq and duration, on the
|
||||
// apply thread when it returns - the line the retrace-split investigation did not have.
|
||||
static constexpr Uint32 kSlowControlDispatchMs = 1000;
|
||||
|
||||
// How many frames this loop has dispatched (every kind, probe included). Reset by
|
||||
// Start(). The frame channel's own red-once handle: a forwarder that stopped posting
|
||||
// frames leaves this unmoved.
|
||||
@@ -451,6 +473,10 @@ namespace MobileGL::MG_Remote::Server {
|
||||
// published; held until its reply returns. Re-entrant probes use call-local arguments.
|
||||
std::atomic<ControlProbeHook> m_controlProbeHook{nullptr};
|
||||
std::atomic<void*> m_controlProbeUser{nullptr};
|
||||
// SetControlProgressSink's pair (p7/spawnhang). Guarded by m_controlMutex: the poster reads
|
||||
// both while it holds that lock between slices, so the pair is always read whole.
|
||||
ControlProgressSink m_progressSink = nullptr;
|
||||
void* m_progressUser = nullptr;
|
||||
|
||||
// The BOUNDED join's other half. std::thread::join has no deadline, so a lost wakeup
|
||||
// would wedge CI rather than fail it; the thread signals here last and Stop() waits
|
||||
|
||||
@@ -57,6 +57,22 @@ void ForwardLog(void* user, const char* text) {
|
||||
(void)transport.SendFrame({builder.GetBufferPointer(), builder.GetSize()});
|
||||
}
|
||||
|
||||
// p7/spawnhang. THE PROGRESS REPORT: Wire::SurfaceProgress, sent by the thread that posted the op
|
||||
// - RunSession's control pump, waiting in ServerApplyWireSurfaceOp - after each
|
||||
// ServerLoop::kControlProgressIntervalMs that mgl-srv-apply has spent running it
|
||||
// (ServerLoop::SetControlProgressSink). The client's reply budget restarts on each one, so a cold
|
||||
// native bring-up that outruns MOBILEGL_IPC_COLD_START_MS is waited for instead of being reported
|
||||
// ALIVE BUT SILENT (retrace-split run 35912252677). A send failure is left to the reply's send,
|
||||
// which already ends the session on it.
|
||||
void SendSurfaceProgress(void* user, Server::SurfaceControlOp, std::uint64_t seq, std::uint32_t elapsedMs) {
|
||||
auto& transport = *static_cast<SocketTransport*>(user);
|
||||
flatbuffers::FlatBufferBuilder builder(64);
|
||||
auto progress = Protocol::CreateSurfaceProgress(builder, seq, elapsedMs);
|
||||
auto envelope = Protocol::CreateCtrlEnvelope(builder, Protocol::CtrlMsg::SurfaceProgress, progress.Union());
|
||||
Protocol::FinishCtrlEnvelopeBuffer(builder, envelope);
|
||||
(void)transport.SendFrame({builder.GetBufferPointer(), builder.GetSize()});
|
||||
}
|
||||
|
||||
void RefuseBusy(SocketTransport& transport, const char* detail) {
|
||||
// Unread Hello bytes can make close send RST and discard our Refuse.
|
||||
std::uint64_t bytes = 0;
|
||||
@@ -459,6 +475,10 @@ void DelaySessionHandshakeForTest() {
|
||||
if (tcp && (!forwarding || std::strcmp(forwarding, "0") != 0))
|
||||
MobileGL::MG_Util::Debug::SetLogForwarder(&ForwardLog, control.get());
|
||||
auto& loop = Server::ServerLoopInstance();
|
||||
// p7/spawnhang: this thread posts every surface op the client sends (ServerApplyWireSurfaceOp,
|
||||
// below) and is the one that says "still running" while the apply thread runs it. Cleared
|
||||
// after the loop, on this same thread, so no post can be in flight when it goes.
|
||||
loop.SetControlProgressSink(&SendSurfaceProgress, control.get());
|
||||
if (loop.Start(session) != MOBILEGL_OK) ::_exit(70);
|
||||
WireLogError("MG_Remote server: pid=%d transport=spawn role=server ready control=%s data=%s",
|
||||
selfPid, tcp ? "tcp" : "unix", tcp ? "stream" : "shm");
|
||||
@@ -595,6 +615,7 @@ void DelaySessionHandshakeForTest() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
loop.SetControlProgressSink(nullptr, nullptr);
|
||||
loop.Stop();
|
||||
// Publish the final server window while log forwarding is still attached.
|
||||
MobileGL::MG_Util::PipeStats::Shutdown();
|
||||
|
||||
Reference in New Issue
Block a user