How web directories organize and categorize the web

The received wisdom goes like this: web directories are a quaint relic, killed off somewhere around 2011 when Google’s algorithms got good enough to make human-curated lists redundant. Yahoo’s directory shut. DMOZ closed in 2017. The case looked closed.

I do not buy it. Eight years working alongside search and directory companies left me with a different read on the evidence, and the more I look at how people actually find specialist information, the more I think the obituary was written too early. This article makes the contrarian case. Not that directories beat search engines at general queries (they obviously do not), but that the classification problem they were built to solve is still unsolved, and pretending otherwise produces worse outcomes for everyone hunting for niche, vetted, or context-dependent information.

The myth of algorithmic supremacy

Search engines do something extraordinary: they rank a near-infinite pool of pages in milliseconds, using signals that approximate relevance well enough that most users never bother with page two. That is genuinely impressive. It is also frequently confused with a stronger claim, which is that ranking solves classification. It does not.

Why search engines aren’t the whole story

Ranking and categorising are different jobs. A search engine asks: given this query, which pages best answer it? A directory asks: given this category, which entities belong inside it, and how do they relate to neighbouring categories? You can rank without categorising (Google does, mostly) and you can categorise without ranking (a library catalogue does). Confuse the two and you assume that because Google answers questions well, classification itself is solved.

It isn’t. Try a query like “independent ceramic restorers in the East Midlands who handle 18th century porcelain”. Google will hand you a mix of national chains, Etsy sellers, a Reddit thread, and three pages that mention the words but offer nothing useful. A purpose-built directory, if one exists, gives you the seven relevant businesses in alphabetical order. The directory loses on coverage and gains on precision, which is exactly the tradeoff classification is supposed to make.

What directories still do better than crawlers

Three things, in my experience. First, they enforce membership criteria a crawler cannot: this business is real, is reachable at this number, and operates in this county. Second, they preserve relationships between entities that a query-based interface hides, so a user can browse adjacent categories and discover options they would never have thought to search for. Third, they survive vocabulary mismatch. If you do not know whether to call something a “luthier” or a “guitar repair shop”, a hierarchy lets you arrive at the right node from either direction. Search engines fudge this with synonyms, and most of the time they get it right; the interesting cases are the ones where they do not.

The classification problem Google never solved

Google has tried and largely abandoned several taxonomy projects. The Knowledge Graph is the most ambitious survivor, and it is excellent for famous people and chain restaurants but thin in long-tail verticals. The reason is structural. Building a usable taxonomy means committing to a particular view of how the world divides up, and any committed view is wrong somewhere. Algorithms hate committing. They prefer to defer the decision to query time, which works beautifully when the user can phrase their need precisely and not so beautifully when they cannot.

Evidence against the “directories are dead” narrative

The strong form of the death narrative says directory traffic has collapsed to irrelevance. The weak form says general-purpose directories have collapsed but vertical ones thrive. The weak form is correct; the strong form is not, and conflating the two does damage to anyone trying to make a sensible discovery strategy.

Traffic data from niche directory verticals

Look at what survived. Legal directories (Chambers, Legal 500, Avvo). Medical directories (Doctify, Healthgrades). Trade directories (Checkatrade, TrustATrader). Academic indexes (PubMed, which is a directory in every meaningful sense). Restaurant directories (OpenTable, which most people forget is structured as a directory with a booking layer on top). The list goes on. These are not zombies; they are healthy businesses with paying members and substantial referral traffic. The Yellow Pages model died. The vertical directory model adapted and, in several cases, became the dominant lead source for its sector.

I have watched conversion data from law firm clients where their directory listings outperform organic search by a factor of three to four on cost per acquired client. That is not because the directory has more traffic. It is because the traffic that arrives via a directory has already self-selected as serious. A user who browses to “commercial property solicitors in Bristol” inside a curated index is much closer to instructing a firm than someone who Googled “do I need a solicitor for a lease”.

How specialized directories outperform general search

The mechanism is intent compression. A directory’s category page is, in effect, a pre-filtered results page that the user has agreed to by clicking into the category. By the time someone lands on it, they have done two filtering passes (the directory itself, and the category within it) and the listings can be presented in a way no general engine would dare, because the user has accepted the editorial frame.

The librarian’s case against pure algorithms

Information scientists have been making this argument for thirty years and being ignored for most of them. The core point: classification systems encode disagreements about how the world works, and surfacing those disagreements is part of the service. A query-driven interface hides the disagreement and substitutes its own implicit ontology, which is worse, not better, because the user cannot inspect it. When I worked on directory taxonomies, the most painful meetings were always about where to put borderline categories. Is “naturopathy” health, or wellness, or alternative medicine? The answer matters because it determines what a user expects to find. Google’s answer is “we will figure it out from the query”, which works until it does not.

Human curation versus machine classification

The honest version of this debate is not “humans good, machines bad”. It is: which problems benefit from human judgement, which from algorithmic scale, and where do the two combine usefully. Pretending it is a winner-takes-all contest is the failure mode I keep seeing in product strategy meetings.

graph TD
  A[Web Directory] --> B[Top-Level Category]
  B --> C[Sub-Category]
  C --> D[Leaf Category]
  D --> E[Vetted Listing]
  B --> F[Sub-Category 2]
  F --> G[Leaf Category 2]
  G --> H[Vetted Listing 2]
  E --> I{Editorial Check}
  H --> I
  I -->|Pass| J[Published]
  I -->|Fail| K[Rejected]

Where editorial judgment beats automation

Editorial judgement wins whenever the cost of a false positive is high. A directory of registered electricians cannot afford to list an unregistered one; the reputational cost of a single bad listing outweighs the marginal value of a hundred good ones. Algorithms are improving at fraud detection but they still struggle with the kinds of subtle signals a human editor catches: a website that technically claims membership of a professional body but links to a defunct registration page, an address that resolves to a residential serviced office, a phone number recycled from a previously delisted business.

The same logic applies to categorisation accuracy. Machine classifiers are quite good at obvious cases and quite bad at edge cases, which is fine for a search engine because edge cases are diluted by ranking. In a directory, edge cases are the whole game; they are exactly the borderline entries a user is trying to disambiguate.

The cost and accuracy tradeoff

Human review is expensive. A trained editor can vet, perhaps, twenty new listings an hour for a moderately complex vertical, including verification calls. Automated submission processing handles thousands. The arithmetic looks brutal until you account for downstream costs: every bad listing produces complaints, refund requests, and brand damage that scales with directory size. The break-even point varies by vertical, but in regulated sectors (legal, medical, financial) human review pays for itself within a year on quality-driven retention alone. In low-stakes verticals (recipe blogs, hobbyist sites) it does not, and you see exactly the curation patterns the economics predict.

Why hybrid models keep emerging

Every successful modern directory I have studied uses a hybrid: algorithms for ingestion, deduplication, and freshness checks; humans for vetting, category placement, and dispute resolution. The pure-algorithmic directories drown in spam within eighteen months. The pure-human directories cannot scale past a few thousand listings without their editorial team becoming the bottleneck. The hybrid model is not a compromise; it is the only configuration that survives contact with real submission volume.

Honest counterpoints worth taking seriously

If I only listed the arguments for directories, I would be doing exactly what I criticised at the start. The case against has real substance and ignoring it produces lazy strategy.

The scalability ceiling of manual review

Manual curation does not scale linearly with the web. The web added something like a billion new pages while I was writing this paragraph, slightly hyperbolically. A directory that depends on humans reviewing every submission will, by definition, cover a vanishingly small fraction of relevant entities. This is fine if the directory’s value proposition is curation itself (“we list the 200 best X”), and disastrous if the user expects comprehensiveness. Be honest about which one you are building.

The corollary: directories that promise comprehensiveness and deliver curated quality eventually get caught in the gap. Users notice their favourite local business is missing, lose trust, and stop checking. I have watched this slow death three times in different verticals. The pattern is always the same: editorial team cannot keep up, submission backlog grows, quality slips as shortcuts get taken, and the brand promise quietly stops being true.

Spam, pay-to-play, and quality erosion

The second honest objection is that many directories, including most of the ones that crowded the SEO landscape in the 2000s, became pay-to-play link farms. The business model rewarded inclusion volume over quality, and the result was a generation of “directories” that were really just lists of paid placements with no editorial logic. Google’s penalisation of low-quality directory backlinks (Penguin, 2012, and subsequent updates) was a rational response to this, and it killed a lot of weak operators. Good. The survivors had to actually justify their existence.

The lesson for anyone evaluating a directory today is to look at the editorial process before you look at the traffic figures. A directory with strict inclusion criteria, like Web Directory or the surviving legal indexes, behaves very differently from one that accepts anyone with 29 pounds and a URL. The former adds value to its members; the latter mostly does not, although it may still rank for branded queries through sheer link equity.

When algorithmic discovery genuinely wins

For ad-hoc factual queries (capital of Bolivia, conversion between fluid ounces and millilitres, opening hours of the nearest pharmacy), search engines are not just better than directories; they make directories pointless. For long-tail conversational questions (“why is my sourdough not rising”), language models now handle most of the work and a directory would be actively unhelpful. For very high-coverage discovery (every business in a city), the comprehensiveness gap is too large for curation to close.

So the right framing is not directory versus search. It is: in which discovery contexts does each approach create value, and where does each fail? Anyone selling you a universal answer is selling, not analysing.

A framework for choosing your approach

If you are building a discovery experience, or deciding which directories to engage with as a business owner, the question is not which side of the debate to take. It is which combination of signals matches your situation. Here is the decision framework I use with clients.

Audience intent as the deciding factor

Start with intent, not technology. If your audience arrives with a precise query they can articulate, search wins. If they arrive with a need they cannot yet phrase, browse-based discovery wins. Most real audiences are a mix, weighted differently by vertical. Wedding photographers get browse-heavy users (couples who do not know what to ask). Plumbers get query-heavy users (boiler is leaking, fix now). The same business in both categories needs different discovery strategies.

A useful diagnostic question: can your typical customer name three competitors before they start looking? If yes, they have vocabulary and search is their tool. If no, they need a map, and a directory or category-based interface is doing them a favour they may not consciously appreciate.

Vertical depth versus horizontal reach

The second axis is depth versus reach. A vertical directory that covers one sector deeply (every certified arborist in the UK, say) beats a horizontal directory that covers many sectors shallowly. The horizontal model worked when the web was small enough that one team could reasonably curate “everything good”; it does not work now. If you are building, build narrow. If you are evaluating directories to submit to, prefer vertical ones in your sector over horizontal ones that include your sector as a tab.

When to build, submit, or ignore directories entirely

Three situations, three different answers. Build a directory when there is a vertical with no good incumbent, where buyers struggle to find sellers, and where you can credibly impose editorial standards. This is rarer than it looks; most “no good incumbent” assessments turn out to be wrong on closer inspection. Submit to directories when your sector has working ones and your competitors are listed; not being there is a positioning problem more than a traffic problem. Ignore directories when your acquisition channels are fundamentally different (B2B enterprise sales, for instance, where buyers do not browse anyone’s index) or when the directories in your space are obviously low-quality and listing alongside their existing members would damage your brand.

The case studies I find most instructive are the ones where businesses discovered directory value retroactively. A mid-sized accountancy I worked with treated their professional body directory as a bureaucratic obligation for years; when they finally added call tracking to that listing, it turned out to be their third-largest source of qualified inquiries, behind referrals and Google but ahead of LinkedIn and content marketing combined. They had been undervaluing it because the traffic was invisible in their analytics, which only counted clicks, not the listing’s role in pre-decision research.

My recommendation, after all of this: stop treating the directory question as a binary. Build a working theory of where your audience sits on the intent axis, run the small experiment of being listed in two or three carefully chosen vertical directories for a quarter, and measure not just direct traffic but assisted conversions and inbound calls that mention seeing you somewhere. The data you collect in three months will settle the argument for your specific case better than any general article can, including this one. The directories that deserve your attention will reveal themselves quickly. The ones that do not will reveal themselves faster.