A 200 Response Is Not a Record
A crawler can be correct about HTTP and still wrong about the result.
Abrade started from a small useful premise: generate patterned URLs, probe them, and record the candidates that appear to exist.
That premise is still useful. It is also too small if the crawler treats every kind of “success” as the same claim.
The 2026 refresh of Abrade, a C++23 command-line crawler, is mostly about separating those claims. The pressure point is simple: 2xx is transport evidence, not content truth. A server returning HTTP 200 may only prove that the request reached an HTTP application and received a successful response class. It does not prove that the response is the record being sought.
It can be an empty result, a placeholder, a login wall, a shell page, a default route, or a route that exists without containing the domain object.
That difference matters because “the request completed,” “the body contains evidence,” and “the run had operational problems” are not interchangeable. Abrade v0.3.0 makes that distinction explicit. It still generates candidate resource paths and performs concurrent HTTP requests. It still defaults to HEAD, and it can fetch bodies with GET. But the refreshed tool adds clearer boundaries around what is being measured: TLS with SNI for DNS host names, optional platform CA and DNS host verification with --verify, conservative redirects, dry-run generation, body-only output, body filters, and run summaries that account for filtered bodies, runtime errors, bytes written, and throughput.
One diagnostic sample in the canonical post exposed two common crawler mistakes. Without SNI, HTTPS failed before there was any meaningful application response. After SNI, many 200 responses still were not useful records. TLS success was one claim. Verified identity was another. Status-code success was another. Content evidence had to be checked separately.
The content-oriented pass is where the crawler stops pretending status success is enough. --contents writes body-only files. --require and --require-regex keep bodies containing required evidence. --reject and --reject-regex remove bodies containing absence, shell, or other non-record evidence.
Those strings are not universal predicates. They have to come from authorized knowledge of the application being tested. That is also the use boundary: authorized resource discovery, migration checks, and repeatable diagnostics against systems you own or are allowed to test.
Read the full essay on lospino.so:
Abrade After the Happy Path
Original essay published on lospino.so on 2026-07-09. This Substack dispatch is an adapted pointer to the canonical version, not a mirrored copy.