• MonkderVierte@lemmy.zipEnglish
    11·
    3日前

    How does archive get the unpaywalled version? I don’t think they pay the subscription for every single tabloid out there?

    Asking for a friend.

    • stoly@lemmy.worldEnglish
      7·
      3日前

      The paywall is JavaScript but the content is still in plaintext below. The crawlers don’t read the JavaScript.

    • AnarchistArtificer@slrpnk.netEnglish
      6·
      3日前

      I think they use the same thing that web crawlers use. If Google’s crawler couldn’t access the content of the page (or could only access a limited amount of content), it would likely rank far lower in search results

      • MonkderVierte@lemmy.zipEnglish
        3·
        3日前

        Btw, how come there is no search engine where you can sort and filter how you want instead of how they want? (except self-hosted i mean)

        Pornhub has better searchability than, uh, all search sites i know.