Checking the check: two mistakes in my own work

Four days ago I published an audit of an automated report. My engine, FORGE, had written a brief about an April 2026 arrest. I checked it claim by claim and found real events joined by links no source supported.

This post is about what happened next. I rebuilt the report with stricter rules, and while doing it I found two mistakes: one in my audit and one in the rebuild.

What I changed in the rebuild

I left the old brief online as a case study, with a banner saying it contains errors, and froze a copy of it in the repository’s history (the frozen copy). Then I rebuilt the report on a fresh database with four rules:

• Hand-picked signals, each labelled. A signal is a subject (the Limpopo arrests), a control (the arrest the old brief tied to Limpopo) or a decoy (another Limpopo corruption story, link not examined).
• A claim ledger on the page. Every claim from the old brief gets a verdict and a source.
• An open list. What still needs checking is published with the report.
• Names only as far as a record goes. A person appears only where a public record names them, and only with their legal status.

The result is the rebuilt report. It supports the real arrests, keeps them apart and says what it cannot show.

Mistake one: my audit said a source did not exist

In the first post’s table I wrote that I found no source for a Polokwane tender probe in September 2025, and I marked the brief’s claim Unsupported. The category was right. The sentence beside it was wrong.

On 17 September 2025 the Hawks visited Polokwane municipality’s offices with a court order, to obtain documents for an investigation into alleged corruption involving municipal principals (Sowetan, 18 September 2025). The municipality disputes the word “raid”. It says the Hawks asked for specific documents and were given them (the municipality’s statement).

I had searched, got nothing, and written down “no source found”. When I later ran FORGE’s own news query for the event, a narrow phrasing returned no articles and a looser one returned 28. An empty search showed that my words did not match. It did not show that the event had not happened.

The brief’s link is still unsupported. Nothing I found ties the Polokwane matter to the April arrests. The Molemole case traces, according to the Hawks spokesperson as reported by Eyewitness News, to incidents reported in September 2024, and it concerns a different municipality. A third event shares a month with it: in September 2024 the Hawks searched Polokwane municipality in the VBS matter (Daily Maverick). Three September events across two years, and no demonstrated link. The only things they share are a province and a police service.

I have added a dated correction to the first post.

Mistake two: my rebuild said it used a scoring system it never ran

The first version of the rebuilt report said, in its provenance note, that its scoring used the changes I had just made to the scoring code. Then I looked at the numbers. Of the 30 signals pinned to the case, 28 had the identical gravity score, 0.3, and two had none.

The news collector writes a fixed 0.30 on every signal when it saves it. The pass that computes real scores had never run on this data. The case page’s “evidence confidence” is the average of those scores, so it was the same constant.

The claim that the new scoring had been used was false. I corrected the note the same day. The report now says scoring was not exercised, states the 28-of-30 figure as a number computed from the database, and ends with a dated correction.

What I can now say about the old brief

In the first post I wrote that I could not tell from outside how FORGE built the link between the arrests. From inside, I can say more. The brief’s article text is prose that an AI coding agent wrote, and the pipeline stored it in a script that writes it into the database. I searched the code and found nothing that produces its linking sentences or its claim that FORGE “cross-references” an earlier probe. The numbers in the text, three pinned signals and an evidence weight of 0.42, were typed in as well. They probably matched the case when they were typed. The case then changed and the page did not.

So the machine’s vocabulary, gravity score, evidence weight and “cross-references”, gave authority to sentences that its scoring and correlation code had not produced. That is a finding about this report. It is not a measure of how the engine behaves in general.

How to correct your own work

1. Date the correction and put it where readers will see it. A line at the top of the page, not only a note in a changelog.
2. Say what was wrong, what is right and what changes. Here, one claim stays Unsupported even though the event exists.
3. Keep the original reachable. Dated copies and a frozen tag let a reader see what I said before.
4. Fix every place the error lives. The post, the project’s records and the pages that link to them.
5. Recompute numbers instead of typing them. Both mistakes involved a statement I had written and not re-checked against the data.
6. Check your own claims as hard as the machine’s. I audited the brief carefully and then trusted my own search result.

What this does not show

• Two mistakes found by chance and by looking at the numbers are not an error rate for my audits.
• The rebuilt report still rests on press reports that quote the Hawks and the NPA. I have not read the Hawks’ own statements or the court records, and the report lists them as open.
• It does not show that the changes to the scoring code work, because the corpus was never scored.
• An arrest is not a finding of guilt. Everything here is an allegation, and no suspect is named in this post.

To practise the checks, Exercise 005: Same month, different case uses an invented scenario built from the same traps.

This post was drafted with AI assistance. I checked each claim against the pages and sources linked in it.