How Accurate Are AI Tool Directories? We Audited Our Own 366 Listings
A full audit of our own directory: 75% of launch dates were placeholders, 21% of related-coverage links were false, and six listings pointed at products that no longer exist.
In this story
Turning the ruler on ourselves
A directory is only as credible as its details — a launch date, a price, a parent-company name — and readers quote them as fact. Almost nobody re-checks those fields: they are typed in once, indexed, copied onward, and the error sets. So we audited our own tool directory against public sources. This is what we found. Every figure below is a historic snapshot taken when this piece first ran on 2026-09-23: the sample is the tool records on disk that day, and the method is at the end. The directory and the audit list have kept moving since; these numbers do not.
Finding one: we did not know most of our launch dates
Of 366 listings at the start (the 2026-09-23 snapshot), 273 (75%) ended in "the first of some month", and 152 were exactly 1 January. Those were not birthdays but placeholders for "we only know the year". The site rendered them as full dates and passed the same invented precision to search engines through structured data.
The first fix was not to find dates but to stop claiming them: show a year when only a year is known, a month when a month is evidenced, a day only when a source states one. A product that genuinely launched on the 1st now reads one step less precisely, which is the harmless direction for an error. After that came the evidence hunt: as of the snapshot (2026-09-23), 266 records had been checked against a source (including ones we checked and deliberately left alone), with well over a hundred corrections. Some still only support a year, and we label them as such rather than guessing.
Finding two: the products move, the directory does not
Six listings pointed at products that no longer exist and were removed: Phind (its domain answers 404 DEPLOYMENT_NOT_FOUND), Supermaven (a dated vendor sunset notice), Clockwise ("no longer be available starting on March 27, 2026", team to Salesforce), Socratic (the domain now renders a Google Lens page), GitHub Models (documentation states it was fully retired on 2026-07-30) and Youper (a vendor notice ending service on 2026-09-30 and deleting data). Two more were duplicate listings of a single product — Codeium, now Devin Desktop, and Bing Image Creator, now served inside Microsoft Designer — and were merged.
Renamed-but-alive products keep their old name in the blurb, so searching it still finds them. Three suspected renames were left untouched because a redirect proves relocation, not a rename date: DreamStudio, Galileo and INK.
Finding three: our related-coverage links were lying
Tool pages matched article text by substring, which turns short names into skeleton keys: "Pi" matched every Anthropic story, "Exa" matched Alexa+, "INK" matched Thinking. Measured across the corpus (the same 2026-09-23 snapshot), 28 of 131 article-tool mention pairs (21%) were false.
Switching to word-boundary matching removed the junk and exposed the real gap: only 40 of the 364 listings in that snapshot have any coverage on this site at all. Those pages used to go silent after the description, or pad the shelf with false links; they now state plainly that the desk has not covered the tool and route the reader to the category index.
Method, and what remains
Every change required a dated first-party source or a registry timestamp (an App Store releaseDate, a first PyPI or npm publish, a first GitHub release). An archive capture only bounds a date from below, so it can justify a year, never a month. Press coverage is enough to raise a question, never enough to rename a product — two agent-supplied conclusions were rejected on that basis. The audit state now lives in the repo as a checked-records list (the file scripts/launch-audit-checked.json) and a selector for what remains, so the work can be resumed by someone else instead of living in one person's memory.
Traceable summary: what this audit changed
- Snapshot date: 2026-09-23, the day this piece first ran. Every count on this page is that day's historic snapshot, not a live figure; the directory and the audit list have kept moving since.
- Sample: the 366 tool records in the repo when the audit started; 2 duplicate listings were merged and 6 were removed after their products were confirmed gone, so the related-coverage comparison uses a 364-record basis.
- Method: every record was checked against a dated first-party source or a registry timestamp (previous section). The checked list and the "take the next unchecked record" script are maintained in the repo, and later waves keep appending to them; the numbers here stay frozen at the snapshot date rather than growing with the list.
- What kind of change: this round's directory fixes (delistings, merges, date and link corrections) are genuine revisions, each backed by a dated source. They are not the same thing as our daily updates — daily updates publish new stories, revisions change what was already published. Corrections to a published story update its revision date and appear on the Revisions page; changes to a directory record are recorded in the checked list itself.
- This page: to state the snapshot scope and method, this piece was revised on 2026-10-14 and its revision date updated.
The honest remainder: 324 tools we have never written about in that snapshot, and prices that drift — Zoom's assistant moved from "included" to a paid licence mid-audit. Any one-off verification expires; the claim "we checked" is only good until the next rename.
This is original content first published by AI Tools Daily. Please credit us when republishing.
AI Tools Daily is a bilingual newsroom covering AI tool launches, product updates and industry trends. Editorial standards · Report a correction
