mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-07 01:53:24 +09:00
[Perf] (MG_Remote): the residency budget does not make the window usable either - the answer is gone before the client next looks, so a deferral needs RETENTION, not a budget
Steps 1 and 2 of the plan the design review set out, landed and measured on the
device. Step 1 answers the question step 2 depended on; step 2 answers the one the
whole line depended on, and the answer is that the window's shape is wrong.
STEP 1 - CAN AN ANSWER BE LOST IN DELIVERY? NO. ReplyPool now counts what the link
took (PostedReplies) and what it refused (FailedReplies), and a non-OK result is
NAMED instead of swallowed: it checked only BUFFER_TOO_SMALL and let
TRANSPORT_CLOSED vanish with no line and no counter, so "the server never sent it"
and "it was sent and the buffer overwrote it" were the same symptom with opposite
fixes. Measured: posted=4501 failed=0. The twenty-answer gap between the server's
tally and the client's was a sampling difference, not a drop - and the server's
m_answered counts ATTEMPTS (incremented before PostReply), which is why it could
never have settled it.
STEP 2 - DOES A RESIDENCY BUDGET FIX THE WINDOW? NO.
Wire_ResourceCreate now measures how many reply-owning records the client has
emitted since the oldest window entry was pushed (ClientSession::ReplyPostings, a
new clock) and takes the oldest answer back as soon as that approaches the reply
buffer's depth, instead of deferring past the point where the answer can still
exist. On the device:
P12win call=4501 cfg=2 eff=2 pending=2 taken=63 pushed=65 blocked=4435
budget=4451 readfail=8872 posts=54377
kind=frame-op frame=14 wait_replies=9078 by_op=2=9071 wall 27.87 s
budget=4451: it fires on essentially EVERY create, the window stops deferring
(pushed=65), and the drain still reads nothing back. THE REASON IS STRUCTURAL, not
a constant:
* the window only gets control WHEN A CREATE ARRIVES. Between two creates this
client emits on the order of a thousand records - at the FIRST failed read the
reply ring already spanned [1175..1504] while the head was seq 310 - and nothing
looks at the window in that gap;
* what displaces a deferred answer is not what this client emits, it is WHAT THE
SERVER ANSWERS. The server is the one behind (one answer per ~5 ms wakeup while
the client runs ahead), so the replies arriving during the gap are its LATE
ANSWERS TO THE EARLIER CREATES - they fill the 8 slots first;
* so the check fires too late by construction. Tightening it does not fix that; it
only removes the deferral entirely, which is what pushed=65 is.
WHAT THAT LEAVES. A deferral is readable only if the client KEEPS the answers it
deferred. The buffer is bounded by COUNT and the client cannot bound how many
answers the server will post between two of its creates, so hoping the buffer still
holds one is a race the client loses - which is also why the earlier two diagnoses
(reply-pool churn, then FIFO-with-no-way-out) were both describing symptoms of the
same thing. The shape that can work is retention: the reader keeps the answers for
the seqs still outstanding (a handful), drops them when read, and reports a LOSS
when a read asks for one it no longer has. That is a transport change - StreamLink's
ring becomes a retention map with a cap and a lost counter - and it is the next step.
BOTH STEPS STAY: step 1 is a net gain on its own, and step 2 is the safe direction
(it can only degrade the window to no-deferral, i.e. the pre-window shape) and its
clock is the one the retention design needs. MOBILEGL_IPC_CREATE_WINDOW stays at 1.
172/172 and 13/13 after the probes were removed. The measurements and the revised
conclusion are in notes/p12/CREATE-WINDOW-MEASURED.md section 9.
This commit is contained in:
BZLZHH
@@ -1898,6 +1898,10 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// creates or parameter sets, and those three have different fixes.
|
||||
const auto replyMetricStart =
|
||||
Transport::LinkMetricsBeginReply(ownsReplySlot, static_cast<std::uint32_t>(op));
|
||||
// P12: the residency budget's clock. Placed HERE, where the record is certainly
|
||||
// published, rather than beside rowCarriesReplySlot: a call that declined before
|
||||
// encoding emits nothing and must not advance it.
|
||||
if (rowCarriesReplySlot) ++m_replyPostings;
|
||||
m_producer.PublishAndNotify(seq);
|
||||
if (op == MG_Pipe::MGPWireOp::Present) {
|
||||
// A credit is permission to run ahead, not permission to retain the
|
||||
@@ -2458,6 +2462,8 @@ namespace MobileGL::MG_Remote::Client {
|
||||
return m_producer.WaitForApplied(seq, timeoutMs);
|
||||
}
|
||||
|
||||
Uint64 ClientSession::ReplyPostings() const { return m_replyPostings; }
|
||||
|
||||
Bool ClientSession::ReadReply(Uint64 seq, void* outBytes, Uint64 outCapacity, Int32* outStatus,
|
||||
Uint64* outSize) {
|
||||
const void* bytes = nullptr; Uint64 size = 0; Int32 status = 0;
|
||||
|
||||
@@ -380,6 +380,18 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// only way a client blocked in the barrier survives a server that went away.
|
||||
Transport::SessionWait WaitForApplied(Uint64 seq, Uint32 timeoutMs);
|
||||
|
||||
// P12: HOW MANY REPLY-OWNING RECORDS THIS CLIENT HAS EMITTED, the clock a deferred
|
||||
// answer's RESIDENCY is measured against. An answer is readable only until the bounded
|
||||
// reply buffer reuses its slot, so a caller that defers an answer (the create window)
|
||||
// has to know how many other answers have gone by since it was posted.
|
||||
//
|
||||
// EVERY reply-slot row is counted, not only the rows this client waits for, and that is
|
||||
// the conservative direction on purpose: a server that ignores the no-reply bit answers
|
||||
// records the client does not read, and those answers displace a deferred one just the
|
||||
// same. A budget that counted only "answers I will read" would be wrong in exactly the
|
||||
// deployment where it matters most.
|
||||
Uint64 ReplyPostings() const;
|
||||
|
||||
// The reply slot for `seq`, addressed seq % slots with the seq stamped back into the
|
||||
// header for self-check (R-3). `outStatus` is 0 OK / 1 DECLINED / 2 ERROR, and
|
||||
// DECLINED IS A REAL ANSWER - MapPersistent's nullptr and the four Bool acceptances.
|
||||
@@ -513,6 +525,7 @@ namespace MobileGL::MG_Remote::Client {
|
||||
Transport::EventRingConsumer* m_events = nullptr;
|
||||
Transport::ITransport* m_transport = nullptr;
|
||||
Bool m_started = false;
|
||||
Uint64 m_replyPostings = 0; // P12: the residency budget's clock (ReplyPostings())
|
||||
// Written by whichever thread first notices the hangup - the GL thread in the barrier,
|
||||
// or the event pump - and read by glGetGraphicsResetStatus on the GL thread.
|
||||
std::atomic<bool> m_deviceLost{false};
|
||||
|
||||
@@ -416,6 +416,10 @@ namespace MobileGL::MG_Remote::Client {
|
||||
Uint64 Seq = 0;
|
||||
MG_Pipe::MGPipeKind Kind = MG_Pipe::MGPipeKind::None;
|
||||
MG_Pipe::MGPipeHandle Handle{};
|
||||
// P12: ClientSession::ReplyPostings() at the moment this entry was pushed. The
|
||||
// difference between it and the live count is how many answers could have displaced
|
||||
// this one - see the residency budget below.
|
||||
Uint64 ReplyStamp = 0;
|
||||
};
|
||||
PendingCreate g_createWindow[kCreateWindowMax];
|
||||
Uint32 g_createWindowHead = 0;
|
||||
@@ -428,6 +432,7 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// answers from existing at all - the second is what a no-reply bit keyed on wantReply
|
||||
// would produce, and it would leave every other reading green.
|
||||
Uint64 g_createWindowTaken = 0;
|
||||
Uint64 g_createWindowBudgetStops = 0;
|
||||
|
||||
// A HANDLE WHOSE CREATE CAME BACK DECLINED IS NEVER WINDOWED AGAIN. Without this the window
|
||||
// would UNDO the repair it exists to keep: every later use of the object sends another create,
|
||||
@@ -548,6 +553,57 @@ namespace MobileGL::MG_Remote::Client {
|
||||
if (status == 1) ++g_declined;
|
||||
if (status == 0) ClearCreateSuspect(payload->Resource);
|
||||
}
|
||||
// ---- P12: THE RESIDENCY BUDGET --------------------------------------------------------
|
||||
//
|
||||
// A deferred answer is readable only while the reply buffer has not reused its slot, and
|
||||
// that buffer is bounded by construction: kDefaultReplySlotCount slots in the shared pool,
|
||||
// and the stream link's ring is that same depth (StreamLink.cpp's kReplyRing, whose
|
||||
// comment says why). So the window may defer only an answer it can still read back, and
|
||||
// what it measures that against is HOW MANY REPLY-OWNING RECORDS IT HAS EMITTED since the
|
||||
// oldest entry was pushed - an upper bound on the answers that could have displaced it.
|
||||
//
|
||||
// ONE LESS THAN THE BUFFER, because the eighth newer answer is the one that displaces it.
|
||||
//
|
||||
// THIS IS PREVENTION, NOT REPAIR, AND THAT IS THE WHOLE POINT. The alternative - read the
|
||||
// head, and when the read fails call the object unknown - UN-LATCHES a create that very
|
||||
// probably SUCCEEDED, and the server's create starts that object's record over
|
||||
// (PipeApply.cpp:2033: `*record = MGPipeResourceRecord{}`), PendingUploads included
|
||||
// (PipeApply.h:332); the client cleared its dirty flags at EMISSION (PipeApply.h:298-304),
|
||||
// so nothing re-sends those texels. Silent content loss for a healthy object. An answer
|
||||
// read inside its residency never has to be given up on at all.
|
||||
// ---- MEASURED: NECESSARY BUT NOT SUFFICIENT, AND THE CLOCK IS WHY ---------------------
|
||||
//
|
||||
// On the device this fires on essentially every create (budget=4,451 of 4,501 calls) and
|
||||
// the drain still reads nothing back (taken=63, readfail=8,872, by_op 2 = 9,071, wall
|
||||
// 27.9 s). The clock is wrong, and the reason is structural rather than a bad constant:
|
||||
//
|
||||
// * THE WINDOW ONLY GETS CONTROL WHEN A CREATE ARRIVES. Between two creates this client
|
||||
// emits on the order of a thousand records (measured: at the FIRST failed read the
|
||||
// reply ring already spanned [1175..1504] while the head was seq 310), and nothing
|
||||
// looks at the window during that gap.
|
||||
// * WHAT DISPLACES A DEFERRED ANSWER IS NOT WHAT THIS CLIENT EMITS, IT IS WHAT THE
|
||||
// SERVER ANSWERS. The server is behind - one answer per ~5 ms wakeup while the client
|
||||
// runs ahead - so the replies arriving during the gap are its LATE ANSWERS TO THE
|
||||
// EARLIER CREATES, and they fill the 8-slot buffer before the client comes back.
|
||||
// * SO THIS CHECK FIRES TOO LATE BY CONSTRUCTION: it is consulted at the next create,
|
||||
// and by then the answer is already gone. Tightening the constant does not fix that;
|
||||
// it only removes the deferral entirely, which is what `pushed=65` is.
|
||||
//
|
||||
// A DEFERRAL IS THEREFORE READABLE ONLY IF THE CLIENT KEEPS THE ANSWERS IT DEFERRED. The
|
||||
// reply buffer is bounded by COUNT (8) and the client cannot bound how many answers the
|
||||
// server will post between two of its creates, so hoping the buffer still holds one is not
|
||||
// a policy - it is a race the client loses. The shape that can work is RETENTION: the
|
||||
// reader keeps the answers for the seqs still outstanding (a handful), drops them when
|
||||
// they are read, and reports a LOSS when a read asks for one it no longer has. That is a
|
||||
// transport change (StreamLink's ring becomes a retention map with a cap and a lost
|
||||
// counter). Kept as-is because it is the safe direction - it degrades to the pre-window
|
||||
// shape - and because the clock it needs is the one ReplyPostings() already provides.
|
||||
constexpr Uint64 kCreateReplyResidency = Transport::kDefaultReplySlotCount - 1;
|
||||
Bool CreateWindowBudgetSpent(ClientSession& session) {
|
||||
if (g_createWindowCount == 0) return false;
|
||||
return session.ReplyPostings() - g_createWindow[g_createWindowHead].ReplyStamp >=
|
||||
kCreateReplyResidency;
|
||||
}
|
||||
void Wire_ResourceCreate(const MG_Pipe::MGPResourceDesc* payload, MG_Pipe::MGPReplySlot* reply) {
|
||||
if (RunsAsTheServerRole()) { MG_Pipe::MGPipeMonolithScreen().ResourceCreate(payload, reply); return; }
|
||||
ClientSession& session = RequireSession("ResourceCreate");
|
||||
@@ -566,7 +622,11 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// The poll is not politeness: it is what keeps a burst shorter than the window from
|
||||
// leaving its answers unread until the next burst.
|
||||
DrainCreateWindow(session, /*blockForOldest=*/false);
|
||||
if (g_createWindowCount >= window) {
|
||||
// A SPENT BUDGET IS TREATED EXACTLY LIKE A FULL WINDOW: take the oldest answer back
|
||||
// NOW, while it still exists, instead of deferring another one past its residency.
|
||||
const Bool budgetSpent = CreateWindowBudgetSpent(session);
|
||||
if (g_createWindowCount >= window || budgetSpent) {
|
||||
if (budgetSpent) ++g_createWindowBudgetStops;
|
||||
DrainCreateWindow(session, /*blockForOldest=*/true);
|
||||
// IF IT IS STILL FULL the oldest answer is not readable yet, and the window is a
|
||||
// fixed array: pushing would silently OVERWRITE a confirmation. This record takes
|
||||
@@ -597,7 +657,8 @@ namespace MobileGL::MG_Remote::Client {
|
||||
return;
|
||||
}
|
||||
g_createWindow[(g_createWindowHead + g_createWindowCount) % kCreateWindowMax] =
|
||||
PendingCreate{seq, CreateRecordKind(*payload), payload->Resource};
|
||||
PendingCreate{seq, CreateRecordKind(*payload), payload->Resource,
|
||||
session.ReplyPostings()};
|
||||
++g_createWindowCount;
|
||||
reply->Id = seq;
|
||||
MG_Pipe::MGPipePostReply(*reply, Wire::ReplySink::kStatusOk, 1u);
|
||||
@@ -1216,6 +1277,11 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// the ones still outstanding. A create the window accepted and then LOST would break that
|
||||
// sum, which is the failure a provisional accept makes possible and a count cannot show.
|
||||
Uint64 ClientCreateWindowTaken() { return g_createWindowTaken; }
|
||||
// How many times the residency budget, and not a full window, forced the blocking read. A
|
||||
// non-zero value is the budget DOING its job (an answer was about to outlive its slot); a
|
||||
// value near the record count means the window is deferring almost nothing, which is the
|
||||
// honest way to say "the budget is too tight for this workload".
|
||||
Uint64 ClientCreateWindowBudgetStops() { return g_createWindowBudgetStops; }
|
||||
Uint32 ClientCreateWindowEffective() { return CreateWindowEffective(); }
|
||||
Uint32 ClientCreateWindowArraySize() { return kCreateWindowMax; }
|
||||
// Objects whose create came back DECLINED: out of the window, next create blocking, and back
|
||||
|
||||
@@ -91,6 +91,11 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// drain that never collects leaves Pending at the ceiling and every other reading unchanged,
|
||||
// which is exactly what the device measured before the stream link learned to answer by seq.
|
||||
Uint64 ClientCreateWindowTaken();
|
||||
// How many times the RESIDENCY BUDGET - not a full window - forced the blocking take: the
|
||||
// oldest answer was about to outlive its reply slot. Non-zero is the budget doing its job;
|
||||
// a value near the record count says the budget is too tight for this workload and the
|
||||
// window is deferring almost nothing, which is the honest way to read that.
|
||||
Uint64 ClientCreateWindowBudgetStops();
|
||||
Uint32 ClientCreateWindowEffective();
|
||||
Uint32 ClientCreateWindowArraySize();
|
||||
Uint32 ClientCreateWindowSuspects();
|
||||
|
||||
@@ -66,14 +66,30 @@ namespace MobileGL::MG_Remote::Server {
|
||||
void ReplyPool::PostReply(Uint64 seq, Int32 status, const void* bytes, Uint64 size) {
|
||||
if (m_link) {
|
||||
const auto result = m_link->PostReply(seq, status, bytes, size);
|
||||
if (result == MOBILEGL_OK) {
|
||||
++m_posted;
|
||||
|
||||
return;
|
||||
}
|
||||
++m_failed;
|
||||
if (result == MOBILEGL_ERR_BUFFER_TOO_SMALL)
|
||||
SessionFail(MGFatalFamily::ReplyTooLarge, "MGPipe: Fatal{ReplyTooLarge, stream reply}");
|
||||
// EVERY OTHER FAILURE IS NAMED AND COUNTED RATHER THAN SWALLOWED (P12). This used to
|
||||
// `return` on anything that was not BUFFER_TOO_SMALL, so a TRANSPORT_CLOSED reply
|
||||
// vanished with no line, no counter and no Fatal - and the client's only symptom is a
|
||||
// deferred answer that never arrives, which is indistinguishable from a buffer that
|
||||
// overwrote it. Once per call site rather than once per reply: a broken link fails
|
||||
// every time, and a log per answer would drown the run it is there to explain.
|
||||
MGLOG_E_ONCE("MGPipe: reply seq %llu was NOT delivered (%d); the client will not read"
|
||||
" this answer. FailedReplies() counts them",
|
||||
static_cast<unsigned long long>(seq), static_cast<int>(result));
|
||||
return;
|
||||
}
|
||||
Transport::ReplySlotPool pool(m_base, m_size, m_slots);
|
||||
// Fatal inside Post when the answer does not fit a slot: P5 does not chunk replies,
|
||||
// and the client knows an answer's size before it emits the record.
|
||||
pool.Post(seq, status, bytes, size);
|
||||
++m_posted;
|
||||
}
|
||||
|
||||
Uint32 ReplyPool::SlotBytes() const { return m_slotBytes; }
|
||||
|
||||
@@ -97,12 +97,25 @@ namespace MobileGL::MG_Remote::Server {
|
||||
// chosen from that and an overflow means the two sides disagree about the frame.
|
||||
Uint32 SlotBytes() const;
|
||||
|
||||
// P12: DID THE ANSWER ACTUALLY GO OUT? PostedReplies counts the ones the link took,
|
||||
// FailedReplies the ones it refused. THE PAIR EXISTS BECAUSE THE DECODER'S TALLY CANNOT
|
||||
// ANSWER IT: m_answered is incremented BEFORE PostReply, so it counts ATTEMPTS. Without
|
||||
// these two, a client that cannot read an answer has no way to tell "the server never
|
||||
// sent it" from "it was sent and the bounded buffer dropped it" - and those two need
|
||||
// opposite fixes, one in the transport and one in the deferral. (Found the hard way: a
|
||||
// device run had the server report 4,521 answers and the client receive 4,501, and
|
||||
// nothing in the tree could say which twenty were which.)
|
||||
Uint64 PostedReplies() const { return m_posted; }
|
||||
Uint64 FailedReplies() const { return m_failed; }
|
||||
|
||||
private:
|
||||
Transport::ILink* m_link = nullptr;
|
||||
Uint8* m_base = nullptr;
|
||||
Uint64 m_size = 0;
|
||||
Uint32 m_slots = 0;
|
||||
Uint32 m_slotBytes = 0;
|
||||
Uint64 m_posted = 0;
|
||||
Uint64 m_failed = 0;
|
||||
};
|
||||
|
||||
// ---- MGPClear's two discriminants ---------------------------------------------------
|
||||
|
||||
@@ -190,7 +190,43 @@ if (x.replySeq != seq) return ... MOBILEGL_ERR_PROTOCOL_MISMATCH; // 等的是
|
||||
2. **两端必须都升才生效**：客户端先落、服务端不升时，行为与改前逐字节相同（旧端忽略 bit 5 照旧回包）——
|
||||
这正是 B6 说的那一档，本轮在真机上先撞到了一次。
|
||||
|
||||
## 9. 复算
|
||||
## 9. 第 1、2 步落地与实测：预算也不够，缺的是「保留」
|
||||
|
||||
按 §8 的次序做了两步并上真机。
|
||||
|
||||
**第 1 步（让回包失败可归因）**：`ReplyPool` 现在有 `PostedReplies()`/`FailedReplies()`，非 OK 的返回不再被吞
|
||||
（原来只检查 `BUFFER_TOO_SMALL`，`TRANSPORT_CLOSED` 静默消失）。实测：
|
||||
|
||||
服务端 P12post posted=4501 failed=0
|
||||
|
||||
**⇒「答案没送到」被排除**：服务端 `answered` 与客户端 `seen` 的那 20 条差额是两端采样时刻不同，不是丢包。
|
||||
|
||||
**第 2 步（residency budget）**：客户端数「自最老窗口条目 push 以来发出了多少条带 `kReplySlot` 的记录」
|
||||
（`ClientSession::ReplyPostings()`），逼近 8 就强制阻塞取回。实测：
|
||||
|
||||
P12win call=4501 cfg=2 eff=2 pending=2 taken=63 pushed=65 blocked=4435 budget=4451 readfail=8872 posts=54377
|
||||
kind=frame-op frame=14 wait_replies=9078 by_op=2=9071 墙钟 27.87 s
|
||||
|
||||
`budget=4451`——预算在**几乎每一条 create** 上都判超支，于是窗口几乎不再延后（`pushed=65`），而 drain 依然
|
||||
读不回来（`taken=63`、`readfail=8872`）。**预算无效，且原因是结构性的：**
|
||||
|
||||
1. **窗口只在 create 到达时才有机会动作。** 两条 create 之间客户端可以发出上千条记录——实测第一次读失败时
|
||||
回复环已经是 `[1175..1504]`，而队头是 **seq 310**；中间那段没有任何人看窗口。
|
||||
2. **挤掉延后答案的不是客户端发了什么，是服务端回了什么。** 服务端是落后的一侧（每次唤醒 ~5 ms 回一条，
|
||||
客户端一路跑到前面），所以那段时间到达的回复是它**对更早那批 create 的迟到答案**，它们先把 8 个槽占满。
|
||||
3. **所以这个判据在构造上就太晚**：它在下一条 create 才被问到，而那时答案早就没了。收紧常数只会让延后彻底消失
|
||||
（`pushed=65` 就是那个样子），不会有别的结果。
|
||||
|
||||
**结论（本轮最终认识）：延后要成立，客户端必须「保留」它延后的答案。** 回复缓冲按**条数**有界（8），而客户端
|
||||
**无法**约束服务端在它两条 create 之间会回多少条答案——所以「指望缓冲里还有」不是策略，是一场客户端必输的竞争。
|
||||
能工作的形状是**保留**：读线程把「还没被取走」的那几个 seq 的答案留着（同时只有个位数），取走即丢，
|
||||
读不到就**报一次丢失**而不是无限重试。那是传输层改动（`StreamLink` 的环 → 带上限与丢失计数的保留表），
|
||||
也是下一步该做的。
|
||||
|
||||
**两步都留在树里**：第 1 步是净收益；第 2 步是安全方向（只会让窗口退化成不延后＝pre-window 形状），
|
||||
而且它用的时钟 `ReplyPostings()` 正是保留方案要用的。
|
||||
|
||||
## 10. 复算
|
||||
|
||||
~~~sh
|
||||
# 两臂的帧读数（handoff-2 §1 的读法）
|
||||
@@ -199,6 +235,10 @@ grep -a "kind=frame-op frame=" mgl-device/logs/mcx-fix4.client | tail -1
|
||||
grep -a "kind=frame-sent frame=" mgl-device/logs/mcx-fix4.client | tail -1
|
||||
# 一轮设备 run（自带 force-stop、就绪判据、负载帧检测、截屏、SIGTERM 收尾）
|
||||
cd mgl-device && python3 run-once.py <tag> [MOBILEGL_IPC_CREATE_WINDOW=4]
|
||||
# 窗口自己的读数（临时探针）：taken / pushed / budget / readfail / posts
|
||||
grep -a "P12win" mgl-device/logs/<tag>.client | tail -1
|
||||
# 回包送达（服务端）：posted / failed
|
||||
grep -a "P12post" mgl-device/logs/<tag>.server | tail -1
|
||||
# 门
|
||||
build-split/MobileGL/MG_Test/Wire/RemoteClientTest --gtest_filter='*CreateWindow*:*RefusedCreate*'
|
||||
build-split/MobileGL/MG_Test/Wire/StreamLinkTest --gtest_filter='*OlderReply*'
|
||||
|
||||
Reference in New Issue
Block a user