Donmai

Getting 403 error while trying to download images

Posted under Bugs & Features

flatio

I have a bot that can download images from danbooru, but today it started getting 403 error while trying to download images. All requests are made with auth+user-agent headers set up (I use "Danbooru user flatio" as user-agent). Was there any changes regarding this error and can you help me resolve this issue?

Reply
kittey

Something seems to be up at the moment. I’m getting the same issues while using the site normally with a browser. Only images, though, which is quite annoying because I don’t get the Cloudflare user verification page; images just fail to load. I can only fix it by opening a direct image link in a new tab so that I can see the Cloudflare page, which fixes it for a while.

Reply
Mexiguy

RedsAnimeGirlDungeon said:

Danbooru doesn't want us to use the API for image bots anymore?

Well, someone was scraping so hard that RAM, CPU and disk usage dropped down by 40-ish percent the moment the API was disabled, so I guess it will remain down until admins figure it out.

Reply
kittey

dai1313 said:

On Firefox on android it seems like I can no longer download images.
Just normal web browser, not through the API. kittey I'm curious if you are also Firefox on Android?

No, Firefox on desktop Linux.

Reply
evazion

Some of you may have noticed that images have been slow to load or sometimes failing to load completely over the last few weeks. That's because over 30% of our bandwidth is taken up by bots and scrapers now and the image servers can no longer take it. So I'm blocking all bots to see how well the site performs with only real humans. It turns out that made things significantly better, reducing average response times from 2 to 4 seconds to 0.5 seconds.

I will gradually loosen the rules to try to block fewer real users and benign bots. But in general, traffic from real people takes priority over bots. There are too many AI bros trying to scrape the entirety of Danbooru these days and it degrades the site's performance too much for real people. I've tried blocking individual bots and IPs but it's a never-ending game of whack-a-mole. So at this point I'm left with wide-ranging bans to try to reduce the number of bots overall.

Reply
meatwagon

evazion said:
I'm blocking all bots to see how well the site performs with only real humans.

Thanks for doing this work.

I'm a bot-owner who tries hard to not be a nuisance, but I agree with you that traffic from real people must take priority, and I appreciate the time and effort you put in to keep the site working.

Reply
kittey

evazion said:

I will gradually loosen the rules to try to block fewer real users and benign bots.

Hopefully, you can post some info on the forum about what would be considered “benign”, especially if delays or rate limits are required to stay below some otherwise invisible threshold.

Reply
Shimakaze010

dai1313 said:

On Firefox on android it seems like I can no longer download images.

Just normal web browser, not through the API. kittey I'm curious if you are also Firefox on Android?

I'm having the same issue. Firefox on Android, no special add-ons or vpn. Tried both desktop mode and mobile, cannot download images. Laptop Firefox browser works fine though.

Just tried the generic "Internet" app that comes stock from Samsung and that works. Must be an issue with Firefox itself.

Updated

Reply
Claverhouse

sealplease said:

Does using something like imgbrd-grabber once a day count as a bot? Really hope not, this place is a big part of my daily image downloads.

Same for me both.

About midday here [ Britain ] I fired up B.I.D --- downloader, and through it had worked earlier that day, and though the images show up fine in a browser [ Basilisk on Linux ], in B.I.D. the frames turn up with blanks for the images.

I hope this will be fixed whn possible: it's fairly distressing, particularly as I am accumulating a backlog in the browser...

Thank you very much though. I love this site.

Reply
DarknessEternal

Running into this myself, with both tooling I've written and Imgbrd-Grabber, but with user agent strings that make it clear its my tooling and that i'm not masquerading as a bot. This was working yesterday, so something in how you're implementing restrictions, etc. is causing issues.

Firefox and Chrome seem to be able to access things, but non-web-browsers and such are getting hard-fails. And I seem to remember from past discussion that you want to *avoid* our tooling masquerading as web browsers or such as well...

Reply
1 2 3 4