Skip to content

What went wrong, and what it taught us

Facts about the drone live in the other files. This one is about method: the mistakes made while finding those facts, kept because each cost real time and each would otherwise be made again. Dates are 2026-10-01 and 02.

A simulator written from the same assumptions as the client agrees with it

Section titled “A simulator written from the same assumptions as the client agrees with it”

The most expensive pattern of the project, hit four times.

We built a protocol-accurate simulator so the client could be tested without the aircraft, and 362 tests passed against it. Then the first live run found three protocol bugs in minutes, and a fourth later:

  1. Acknowledgement was decided on data type and buffer id. The naming says buffer 126 is the drone’s non-acknowledged channel and 127 the acknowledged one. A live Bebop 2 does the opposite. Requiring both to agree meant nothing was ever acknowledged, so the drone retransmitted instead of continuing: 35 telemetry values instead of 192, with sensor health and motor state among the missing. Nothing errored.
  2. Argument-less events decoded to an empty dict and vanished, including the two that mark the end of a state dump.
  3. The state dump was awaited by watching for a quiet link, which never happens on a drone that streams attitude at 5 Hz.
  4. The simulator refused a second controller, with a comment asserting that was what the aircraft did. The aircraft accepts one and starves the first.

In every case the simulator encoded the same belief as the client, so the two agreed and the tests passed. A test double built by the same mind as the code is a consistency check, not a correctness check. It proves the implementation matches its own assumptions.

What actually caught them was cheap and unglamorous: a tap counting every inbound frame by (data_type, buffer_id), and a comparison against an older tool’s capture that had 181 values where the new one had 35.

Mitigation we now use: tests that pin behaviour observed on the aircraft live in their own file (test_live_findings.py in mcbebop) explaining why the simulator could not have caught them.

Inherited claims become facts by repetition

Section titled “Inherited claims become facts by repetition”

“The drone serves one controller at a time” appeared in pyparrot’s error text. It was copied into our error messages, then into our tool descriptions, then into the simulator’s behaviour, then into a test name. By then four artifacts asserted it and none of them had tested it.

It is wrong. The aircraft accepts a second controller and silently redirects telemetry, leaving the first believing it is connected. The true statement is narrower: one telemetry destination, last writer wins.

The tell was available all along: nobody could say what evidence supported it. Worth asking of any inherited claim, especially one repeated confidently.

A read-only survey wrote 11 crash reports, 3.5 MB, to the aircraft. /etc/ld.so.preload loads a crash handler into every process and it treats SIGPIPE as fatal, so each producer | head produced a crash dump. The agent noticed, switched to tail, confirmed the count stopped, and left the reports in place because deleting is also a write.

Two lessons. “Read-only” is a property of a system, not of a command: head is read-only everywhere else. And the drone reports its crash-log count over ARSDK, so our mistakes are indistinguishable from the aircraft’s history to anyone reading that counter later. The notes say 11 reports came from us, because nothing else ever will.

Verify before correcting, including yourself

Section titled “Verify before correcting, including yourself”

Several corrections in this project were made against claims that turned out to be right, or against the wrong version of a file:

  • A brief told an agent that pyparrot replays stale packets after a timeout. The agent checked and reported the bug was not present. It was correct: we had fixed it in our fork hours earlier and the brief described the pre-fix state.
  • A patch to fix a logic bug silently did nothing, because ruff format had already reflowed the target from one line into five. The script’s assert caught it. Patching formatted code by string match needs the assertion, or you get a confident success message and no change.
  • A commit hash was quoted to a peer from memory and was wrong.

Tools that fail silently are worse than tools that fail

Section titled “Tools that fail silently are worse than tools that fail”

preflight_check reported ready on its first live run against an aircraft it knew almost nothing about, because an unreported check counted as a pass. Unknown now blocks. A preflight that answers “go” because data did not arrive is worse than one that refuses to answer.

The same shape appears twice more: acked: false was reported for commands that can never be acknowledged, making a working camera move look broken; and a flat telemetry store let six sensor self-tests overwrite each other, so a failing magnetometer was invisible to the check that exists to find it.

Where a value can be absent, say absent. Never let it read as fine.

Parallel agents need isolated working trees, and one shared surface

Section titled “Parallel agents need isolated working trees, and one shared surface”

Three agents built mcbebop in parallel with disjoint file sets and it merged without a single conflict. What made that work was committing the interface contracts first, so each stream coded against a shared vocabulary.

Then the pattern was broken: a fourth agent was told to git switch -c in the shared checkout rather than given a worktree, so two writers ended up in one working tree with interleaved uncommitted changes. Recoverable, but only by telling the agent to stage files by name instead of git add -A.

Also worth keeping: the coordination file lives outside the repository, so the file whose job is coordinating parallel work cannot itself become a merge conflict.

A survey agent received a tool result containing a complete JSON-RPC tools/call object with its own id and a command it never issued. It treated the content as untrusted and did not act on it.

The cause was benign: this session had issued a concurrent call through the same MCP server, that call hung for 1809 s, and its framing surfaced in another caller’s result. Still a real bug, and the agent’s response was correct whatever the cause. A tool result containing a tool call is never data to act on.

Report what the instrument measured, not what you think it means

Section titled “Report what the instrument measured, not what you think it means”

The video stream’s NAL types were recorded as “STAP-A, FU-A, and non-IDR slices”. Another session read that, reasonably, as the stream contains no keyframes, wrote it into their notes as measured fact, and began designing a decoder path around it.

The probe had only ever read the outer header byte of each packet. For a fragmented packet that byte describes the fragmentation, not the NAL inside it, and a keyframe at 856x480 would certainly be fragmented. So any keyframes were hiding in precisely the packets the probe was blind to. The measurement was correct; the summary of it was not, and the gap stayed invisible until someone tried to build on it.

The question worth asking before a measurement becomes a note: what could this instrument not have seen? “No type 5 observed” and “no keyframes exist” are different claims, and only the first one was earned.

ffmpeg -t on a live stream measures the stream’s clock, not yours

Section titled “ffmpeg -t on a live stream measures the stream’s clock, not yours”

Bounding a decode test with -t 3 against an RTP stream that is already three seconds in satisfies the limit before a single frame is written, so ffmpeg exits having decoded nothing. It looks exactly like a stream that cannot be decoded. Bound by -frames:v instead. Also -bsf:v h264_mp4toannexb converts the other direction; -f h264 already writes Annex-B.

grep here skips the files a privacy audit exists to find

Section titled “grep here skips the files a privacy audit exists to find”

On this machine grep is ugrep wrapped to honour .gitignore. For a privacy audit that is precisely backwards: captures/ is gitignored because it holds a drone’s serial, a CPU id, MAC addresses and the coordinates of someone’s home, and a bare grep -r for any of those returns nothing at all. Verified:

grep -rl '<serial>' . -> nothing
/usr/bin/grep -rl '<serial>' . -> captures/logs/ckcm.bin

(The literal serial stood here until 2026-10-04. Writing the identifier into the note that exists to keep identifiers out of the repo is its own small lesson: a privacy rule is easiest to break while documenting it, because the example wants the real value and the real value is exactly what must not travel. Use a placeholder; the demonstration reads the same.)

Use /usr/bin/grep for any audit that must see the whole tree. Bare grep is still right for “what is about to be committed”, which is a different question and the one most of this project’s leak checks were actually asking.

Compounding it, zsh does not word-split unquoted variables, so a $SCAN list of paths expands to one bogus path, grep errors, and a 2>/dev/null swallows the error. The audit then reports a clean tree it never scanned. A clean result from a tool you have not proved can see the thing is not a clean result. The fix that makes it a control rather than a hope: run each pattern against a known positive first, and only trust a negative once the positive matched.

The best evidence is usually already in hand

Section titled “The best evidence is usually already in hand”

Twice the answer to a live question was sitting in data already collected:

  • The motor-controller firmware version a peer wanted to probe for over I2C had been reported over the air at first contact, 0.45 s into the session.
  • “Can two Wi-Fi clients associate at once?” had been demonstrated hours earlier when a phone and this workstation were both on the drone’s network.

Before probing, search what has already been captured.

-t on an RTP input is measured against the stream’s clock

Section titled “-t on an RTP input is measured against the stream’s clock”

Testing that a viewer can join a stream already in progress, ffmpeg was pointed at the simulator after it had been streaming for three seconds. It printed the stream’s resolution correctly and then exited in 20 ms having written nothing: Output file is empty, nothing was encoded.

-t 3 limits the output by timestamp, and the first packet ffmpeg saw already carried a presentation time of 4.6 s, because that is where the stream had got to. The limit was satisfied before a frame was written. Nothing was wrong with the stream; the same run with -frames:v 60 decoded sixty frames.

So when the input is a live stream rather than a file, bound the work by frame count. The failure looks exactly like a stream that cannot be decoded, which is the hypothesis it will cost you an hour to rule out.

A check that cannot fail reports success, and that looks exactly like the check working

Section titled “A check that cannot fail reports success, and that looks exactly like the check working”

Added 2026-10-03 and 04, from the firmware work. Three different tools, three different subjects, one shape, all in one day:

  1. A privacy grep came back clean because the tool honoured gitignore and never looked inside the directory the audit existed to inspect. Plain grep here is ugrep, which skips ignored files, so captures/ was never read, which is the one directory that had to be. Written up on its own above, under “grep here skips the files a privacy audit exists to find”.
  2. A scan for error constants found nothing because it searched a name that was guessed (ERROR_) rather than the one in the file (ERRNO_). The empty result read as “the code never handles this”.
  3. A build passed when it should have failed. To test that the ledger check stops a deploy, one basis was broken in a scratch copy, but the edit landed on the example basis: x in the intro paragraph, which the checker ignores on purpose. The Docker build succeeded, and it would have been easy to write down “failure path not reproducible” or, worse, to count a pass as a pass.

None of these announced themselves. Each was caught only because somebody found the result too good, which is luck wearing the clothes of diligence.

The common factor is not grep or Docker. It is that the thing being verified and the thing being exercised quietly came apart, and nothing in the output says so.

The remedy is cheap and was the same each time: before trusting a negative result, make it positive once. Plant the thing you are looking for, or break the thing you are protecting, and watch the check notice. In practice:

  • For a scan, run it first on a known hit (the FAN path served as the positive control for the RESET_PSOC scan) and confirm the tool can see that location.
  • For a gate, confirm the script exits non-zero on its own before putting it in a pipeline, then confirm the pipeline stops.
  • Say what a pass means. The ledger checker now prints that a pass shows no reference is broken, not that the dependencies are right.

A pass that has never been seen to fail is a statement about the check’s silence.

A fifth instance, from bebop-main the same evening, in a privacy-relevant direction and with a twist. To learn whether the notes repo was readable without credentials, git ls-remote was run with GIT_TERMINAL_PROMPT=0. It succeeded and returned refs, which read as “anyone can clone this”. It had only succeeded because credential.helper = store is set globally and silently supplied stored credentials. GIT_TERMINAL_PROMPT=0 suppresses prompting; it does not isolate the test. An unauthenticated curl returned 401 on the git endpoint and 404 on the web page (Gitea hides a private repo from anonymous readers), which is the trustworthy answer. The others were checks that could not fail; this one could not fail in the direction that mattered. So when testing reachability without credentials, prove the credentials are absent rather than merely unprompted, and prefer a tool that carries no credential machinery at all. A related shape from the same hour: a path typo makes grep print an error to stderr, and || echo clean then reads as success.

Comments

Spotted a mistake, or tried this on your own Bebop? Leave a note. Comments are kept on this site's own server.