Comment

Comment on Running web sites makes me not want to - Lowkey Party

Dave@piefed.nz ⁨4⁩ ⁨hours⁩ ago

Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.

Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.

source

Sort:hotnew top

frongt@lemmy.zip ⁨2⁩ ⁨hours⁩ ago
It doesn’t need to. Like any similar site, it can cache a lot of that in databases like redis, or cache the rendered page in part or in whole.

source
Zak@lemmy.world ⁨4⁩ ⁨hours⁩ ago
I believe you, but the linked article is not about hosting Lemmy.

source
- Dave@piefed.nz ⁨4⁩ ⁨hours⁩ ago
  No it’s not, it was more of a tangential comment.
  
  I think this is the photography site’s talking about.
  
  He mentioned “so many scrapers hit my site that it saturated my upload with generated content feeding the them.” which seemed unusual to me. I’m not sure why such a site would have a public upload. It doesn’t feel like a site that would have a technical challenge with the scrapers, but I also get the impression the main issue is the thought of hosting all his photos just for AI scrapers to train their image generation models.
  
  source
ms_lane@lemmy.world ⁨4⁩ ⁨hours⁩ ago
Why not just use more static pages?

4chan has been ‘static’ forever, the backend builds each page every ~15s, pages served to users are always static. It’s ‘justwerked’ for 20 years now on a shoestring.

source
- rollin@piefed.social ⁨2⁩ ⁨hours⁩ ago
  Sounds like 4chan rolled their own server-side page cache. It probably made more sense for 4chan to do that then, back when the internet was still pretty new.
  
  Nowadays there are a few packages that perform a similar function, like Varnish. Varnish picks up requests from the internet and where possible, it returns the response as static HTML from its own cache.
  
  There’s also other types of caching at different stages of the request lifecycle that already reduce the gap between dynamic and static website performance.
  
  source