In an age of LLMs, is it time to reconsider human-edited web directories?

Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.

These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.

Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.

Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.

Lycos, Excite, and of course Yahoo all offered web directories of this sort.

(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)

By the late '90s, the standard narrative goes, the web got too big to index websites manually.

Google promised the world its algorithms would weed out the spam automatically.

And for a time, it worked.

But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.

And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.

My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?

Do we really want to search every single website on the web?

Or just those that aren’t filled with LLM-generated SEO spam?

Or just those that don’t feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your “free trial” subscription?

At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?

And is it time to begin considering what a modern version of those early web directories might look like?

@degoogle #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM

  • Fardels Bear
    1•1 year ago

    @ajsadauskas @degoogle

    It would be sad to go back to walled gardens like AOL, particularly since they were corporate-owned. But a sort of Kite Mark, certifying a site is free of LLMs, would be useful. Then users could choose for themselves.

  • Brad Enslen
    6•1 year ago

    @ajsadauskas @degoogle Since I run a small directory this is a fascinating conversation to me.

    There is a place for small human edited directories along with search engines like Wiby and Searchmysite which have human review before websites are entered. Also of note: Marginalia search.

    I don’t see a need for huge directories like the old Yahoo, Looksmart and ODP directories. But directories that serve a niche ignored by Google are useful.

    • Bernard Sheppard
      3•1 year ago

      @bradenslen @ajsadauskas @degoogle looksmart! There’s a blast from the past.

      As a very early internet user (suburbia.org.au- look it up, and who ran it) and a database guy, what I learnt very early is that any search engine needed users who knew how to write highly selective queries to get highly specific results.

      Google - despite everything - can still be used as a useful tool - if you are a skilled user.

      I am still surprised that you are not taught how to perform critical internet searching in primary school. It is as important as the three Rs

    • ᴇᴍᴘᴇʀᴏʀ 帝English
      2•1 year ago

      But directories that serve a niche ignored by Google are useful.

      This is a good point - as search is increasingly enshittified too (from top down, with corporate interests, and bottom up, from SEO manipulation and dodgy sites) it makes sense for topics or communities often drowned out by the noise.

      I also see you are using webrings - another blast from the past that has it’s uses.

    • ᴇᴍᴘᴇʀᴏʀ 帝English
      2•1 year ago

      Indeed. As I mentioned below, something like a webring (a FedRing) might be the solution to something I was pondering.

      It is increasingly clear to me that a lot of directions Web 1.0 was evolving in were diverted or just killed off by Big Tech’s landgrab which built walled gardens. I see the Fediverse as a return to the idea of blogs (micro and macro), forums, etc but in a more natural progression to interoperability. This still isn’t perfect and there may be other early web ideas, like webrings, that improve discoverablity.

  • The Octonaut
    9•1 year ago

    Reddit and Lemmy are supposed to be what you want: link aggregators.

    We’re supposed to link to sites and pages and people vote on how good they are in the context of the sub community topic.

    Of course, then Ron Paul happened, and now it’s just memes and Yank politics so… maybe deploy Lemmy and turn off comments.

      • ᴇᴍᴘᴇʀᴏʀ 帝English
        1•1 year ago

        Yeah, it’s the lack of organisation that is the issue and if we are thinking about web directories, there is the missing element of deliberate creation.

    • ᴇᴍᴘᴇʀᴏʀ 帝English
      2•1 year ago

      Yeah, I was just looking at a webring and thinking “these still have a use”. They could definitely help with discoverablity on a broad front. I help Admin feddit.uk and had pondered reaching out to other British Fediverse services to make a Britiverse. However, how to hold it all together and navigate between them was proving tricky or clunky until I was looking at the webring and thought “FedRing”. Now that could work.

  • @[email protected]English
    3•1 year ago

    What’s to say we won’t have AI-curated lists and directories? That way we don’t have to deal with link rot and the like. I think the issue is the algorithms used for search. We need better ones, better AI, not more frivolous human labor.

  • Atemu
    8•1 year ago

    I’d argue that link aggregators like Lemmy (from which I’m posting o/) are the new world version of that. Link aggregators are human-edited web directories; humans post links and other humans vote whether those links are relevant to the “category” (community) they’re in. The main difference is that it’s an open communal effort with implicit trust rather than closed groups of permitted editors.