It's so long since I did this that I've forgotten the names of the faciliies that I used to use, years and years ago. But whatever they were, they will have been superseded, hence the question:
I'd like to display a "mystery object" for the group, and get pointers to what it might be. What's the simplest and least risky way of doing this these days?
Cheers AJ
Upload a photo file to something like
I find Dropbox useful for that
Or use the diy wiki:
If you use a Mac or iPhone the Photos app makes sharing quite straightforward.
Just tried that and it failed. Said that the site was experiencing technical difficulties.
Now working again.
Yup we were experiencing some rather unexpected loading issues...
The site would be working fine and then for no apparent reason would go non responsive, with the loading on the server maxing out CPU and RAM. The real time monitoring would then go ape, posting us warnings the site was not responding. That would repeat hourly. In the first couple of days the sever was up, it had something like 1.5 million total requests, was getting 250K visitors/day and had served over 35GB of traffic.
Turns out it was being very aggressively crawled by bots and spiders, many from AI scrapers - the worst being some tracing back to IP addresses owned by Anthropic. They would get into the history page on an article and the request every single revision of the page. Since that causes mediawiki to run diff between the versions - it swallows loads of CPU time. There are other "expensive" pages to serve that also have lots of content for a crawler to "digest".
We installed a bad bot blocker on the server (which to give you sense of the scale of the problem blocks 7000+ bad referrers, nearly 700 bad clients / user agents, and 200+ fake google crawlers). That helped but still did not fix it. We messed about with rate limiting, again a bit better but not enough.
In the end we decided to take advantage of the fact that our wiki is "private" (i.e. you can't post unless logged it) so we could use mediawiki extensions that allow you to restrict pages that can be served to different classes of user. We looked at the pages that were being targetted, and moved those into the "users" group - so basically you can't get to them if you are not signed in.
That did the trick - the server loading is *way* down, and the response times for real users are now exceptionally quick. There is a slight downside however in that some pages are no longer visible to human visitors if not signed in; things like page history, and access to some of the "special" pages like those that can show a user list, or a gallery of images will now just say you need to login to access.
Redirect the AI scrapers to a feed of bollox training data?
Thanks for the background info John, it's good (for me at least) to get some insights into this kind of thing.
Regards, J^n
Or a certain uk.d-i-y scraping website? ;-)
Chris
And make AI even stupider than it already is?
+1. Sounds a nightmare.
It makes me wonder about the 'intelligence' these AI models arrive at, and what it's used for.
I've been using Claude every so often and usually ask for sources. So far so sort of OK.
Have something to add? Share your thoughts — no account required.
Ask the community — no account required