Regular search timeouts

I've avoided making a forum post about it because I'm assuming it has something to do with the scraper situation from last week (mentioned in topic #22822), but I'm seeing regular database query timeouts in queries that I used to be able to reliably go through thousands of pages on without encountering a single timeout, sometimes even on page 1 of results (but most often & reliably on page 200+ of a given search).

Are there still a lot of scrapers slowing down the site, or is there something else that may be contributing to the performance degradation?

Ah, so it's a common issue right now?

Was trying to use the artist list to try and find someone who I KNOW changed their name by using their old alias but it keeps giving me timeouts.
I'll just repeat the search every now and again and hope it'll work, I guess. None of the other sites I use have an artist list that's as complete or easy to use.

Yeah, it's breaking saved searches (especially search:all) too.

It seems like the database timeout issue is becoming more common now

It's happening for me with almost every tag I search for today.

Been happening to me lately too.

Okay, a few weeks later and I'm still noticing timeouts happening a lot more frequently than they used to. A lot of searches I used to be able to do, no problem, are still timing out almost all the time now. What's going on?

At this point I've separated most of my searches out into smaller ranges or more specific queries, which for the most part has eliminated timeout issues except towards the end of a given range, but also has the unfortunate side effect of breaking out what used to be 7-8 searches I had to stay up to date on into something like 25ish.

Would be nice if we could get a comment from someone with more insight as to what's going on.

I believe the site admins have activated a rate limiter due to this. I think this recent issue I have is related; whenever I upload, it takes a very long time to load the image. And downloads are not going above 100 Kb/s. Not a local issue as my Internet speed is about a gigabit per second. Maybe implementing a captcha when uploading would help, hopefully not being permanent.

Unfortunately the Internet is plagued with bots, they make over 1/3rd (maybe 1/2 right now) of traffic for sure. I do not believe there's a possibility of reversing this. (They are profitable.) I would have hoped the Internet would stay a place for communities and self-expression as it always had been, but it seems over the past decade to have become the garden for unnatural websites, algorithms are everywhere.

I don't know why you think adding a captcha to the upload page of all things would help at all. Do you really think bots are just hammering the upload page?

What searches in particular are timing out? I can't really help you without knowing what you're doing.

Talulah said:

I don't know why you think adding a captcha to the upload page of all things would help at all. Do you really think bots are just hammering the upload page?

You're right, that escaped me entirely... since my primary use of this site is uploading I applied it to robots as well, but it's not what they do currently.

evazion said:

What searches in particular are timing out? I can't really help you without knowing what you're doing.

I can't really pin it down to particular searches because it seems to affect all of them, but at 100 results/page, if I try to go past page ~75-100 it starts to become flaky, and anywhere past ~250, search timeouts become more common than not. Trying to navigate beyond page ~350 on just about any search is basically impossible.

I wish I could pin when this started down to a particular date, but I took a break from visiting from the middle of November through the end of December, so that's the most narrow range I can give. Before the break I was able to search through basically the entire archive for any given tag and rarely encounter any search timeouts.

evazion said:

What searches in particular are timing out? I can't really help you without knowing what you're doing.

Welp I am also encountering timeouts very frequent in the last weeks / days.

And for me i can even pin point it, if i try to search with "order:score" i always get a timeout (since yesterday), even if i clear all my tags in the blacklist, and limit to 5 images per page.
If i dont have a "order" in there it works most the time, sometimes i still get a timeout tho, but after i reload the site (essentially search again) it works.

H_A_Change said:

Welp I am also encountering timeouts very frequent in the last weeks / days.

And for me i can even pin point it, if i try to search with "order:score" i always get a timeout (since yesterday), even if i clear all my tags in the blacklist, and limit to 5 images per page.
If i dont have a "order" in there it works most the time, sometimes i still get a timeout tho, but after i reload the site (essentially search again) it works.

Clearing blacklisted tags wouldn't do anything to help query performance as blacklisted images are hidden client-side via JavaScript

knight_artorias said:
I can't really pin it down to particular searches because it seems to affect all of them, but at 100 results/page, if I try to go past page ~75-100 it starts to become flaky, and anywhere past ~250, search timeouts become more common than not. Trying to navigate beyond page ~350 on just about any search is basically impossible.

And I've seen it as early as the low double-digits. In extreme cases sometimes in the single-digits too. At any rate, the fact it's seemingly every search after a certain point suggests that there's something causing the database itself to be slowed down.

TARDISES said:

And I've seen it as early as the low double-digits. In extreme cases sometimes in the single-digits too. At any rate, the fact it's seemingly every search after a certain point suggests that there's something causing the database itself to be slowed down.

I've had it happen a couple of times on page 1 of a given search, but I didn't feel it was necessary to report it that way because it wasn't consistent.

The thing I think is the problem is that no matter what tag you search for, if the database can't find anything after "3 SECONDS" it timeouts, canceling the search. But, if you're a gold member, you get "6 SECONDS" of search time. I think this is part a marketing angle, trying to get more people to pay to use the site, and a complete screwup in the time search function. Here's a perfect example: I've just recently tried searching for the "chemise" tag, about ten times, and got a search timeout for each one.

Cvlsoldier said:

The thing I think is the problem is that no matter what tag you search for, if the database can't find anything after "3 SECONDS" it timeouts, canceling the search. But, if you're a gold member, you get "6 SECONDS" of search time. I think this is part a marketing angle, trying to get more people to pay to use the site, and a complete screwup in the time search function. Here's a perfect example: I've just recently tried searching for the "chemise" tag, about ten times, and got a search timeout for each one.

the search time limit has been a thing for a very long time, no, as suspicious as i tend to get, i really don't think it is a ploy to get more people to buy gold membership, especially now that the gold purchases arent even available. No one would purposefully slow down their whole website just so a bit more people get tempted to buy a membership anyway

My big reason for upgrading past a basic membership was the increased tag limit, not the search time limit.