Downtime of several hours on two mornings this past week

[matthew1]

We suffered another few weeks of attacks, ending approximately 2nd week of April. Now I don't think it was a competitor or DDoS at all. I think it's 2nd/3rd-tier AI company scraper bots requesting page after page, even forum searches/search results, up to dozens of page requests per second.

This data is very valuable. It's used to train LLMs... Large Language Models, the things that AI uses as its base of knowledge.

I blocked a few dozen AS numbers, which are essentially groups of many IP addresses. After the attacks (bot scrapes) waned, I unblocked them. It was very distributed, not coming from one country or source.

[matthew1]

In case anyone is interested:

So three months later and a year later, after lots of turning over ideas about MVS server performance in my head, I've concluded that the traffic spikes that have been happening are from AI scraping. Not scrapers that are AI (you don't need much intelligence there), but scrapers collecting info to feed AI. That data is used to build what's called LLMs. https://www.cloudflare.com/learning/ai/ ... age-model/

So some of this yearlong off/on analysis I admit to some paranoia about DDoS attacks. Scrapers and DDoS look the same (to me) until I had months of pondering under my belt.

[abscate]

Isn’t there a simple way to limit request to human timescales?

[matthew1]

Yes and I do. The problem is that the requests are distributed in source, i.e.: they hit from 150 or 500 or 4000 IPs concurrently. Any one of those will satisfy any sane rate limiting I apply, but together even at normal page request intervals... it's a locust swarm.