Every check has to catch a planted mistake

Building Patch means checking things constantly: that every part is connected the way the schematic says, that nothing sits too close to anything it shouldn’t, that every part goes on the board the right way round, that the files sent to the factory match the design. Most of those checks are small programs, and most of the time they say everything is fine. That is exactly the problem.

A pass you have never seen fail

A check that says “all clear” can mean two things. Either the design is right, or the check is not looking where we think it is. From the outside the two look identical. A check with a typo in it, one that reads the wrong file, or one that quietly compares something with itself will pass every time, forever, and it will feel like good news.

So we have a rule: no new check is trusted until it has caught a mistake we planted on purpose. Before it is allowed to judge the real design, it runs on a copy with a known error in it, the exact kind of error it exists to catch. If it does not flag the error, the check is broken, not the design. Only once it has failed for the right reason does its pass mean anything.

What it looks like in practice

Take the check that every part goes on the right way round. Parts like LEDs and light sensors only work one way round, and the factory places them from a file that says which way each one faces. When we built that check, we fed it copies of the design with deliberate mistakes in them, parts turned the wrong way among them, and confirmed it named every one. Only then did it get to look at the real files.

The same goes for the bigger checks. Before buying the parts for our test board, every part’s position and orientation was checked against the factory’s own library, and the checks caught every mistake we planted before they were trusted with the real files. A check that catches the error we hid is a check we can believe when it finds nothing.

Two kinds of blind spot

Planted mistakes find a check that is not working. They cannot find a check that was never written. That is what a second pair of eyes is for: a reviewer who did not build the thing and does not share its assumptions. Once, an independent review found a problem on one of our boards that every one of our own checks had passed, because all of them trusted the same source. We fixed the board, wrote a new check for that kind of error, and proved the new check by planting the error again.

Between them, the two cover each other’s gaps. The planted mistake tests the checks we have. The outside reviewer finds the ones we are missing.

Not only for software

The idea reaches past code. Early on, a request we sent to a supplier sat in our sent folder looking delivered, while the notice that it had bounced landed in a junk folder seconds later. “Sent” was a check that could not fail. Now a message counts as delivered when there is evidence it arrived, not when it left.

Building the broken copy takes a few minutes each time. That is very little next to a board that comes back from the factory wrong.