Replacing tag subscriptions with saved searches

Over the next few months I will be phasing out the tag subscriptions feature. The intent is to replace it with saved searches, which should duplicate their functionality.

As a reminder, this is what a tag subscription is currently:

• Each user can have up to 5 tag subscriptions.
• A single tag subscription represents multiple searches.
• A single tag subscription can have at most 20 searches.
• If you have a gold account, you can view all the posts for a single tag subscription, or across all tag subcriptions.
• These searches involve asynchronous searches that return the X most recent posts for that tag query, where X is inversely proportional to the number of searches in the subscription. This means that the more searches a tag subscription has, the fewer posts will be returned.
• These asynchronous searches refresh (in general) once a day at a random time.

In contrast, this is what a saved search is currently:

• A single saved search represents a single tag query.
• Platinum users can have at most 1,000 saved searches. Everyone else can have at most 250 saved searches.
• A saved search can belong to a category, which is just a free form string you can choose.
• If you have a gold account, you can view all posts for a single category, or across all saved searches.
• Saved searches are updated asynchronously, on demand.
• Each saved search will return the most recent 1,000 posts matching that tag query.

I want to make whatever changes are necessary to make sure saved searches reach feature parity with tag subscriptions so there are no issues with migrating people over.

Why am I doing this? The UI for tag subscriptions is not intuitive. You have to go into your user settings and manually enter in tags, on separate lines, and to maximize the utility you have to learn special meta tags. The way they are architected also means they are updated irregularly, and as tag subscriptions become more popular with more users the problem will only worsen.

A saved search, in contrast, is something that is accessible from any post search listing. Most of the processing is handled off-site so it should be much simpler to scale out. The way it is architected also means much less redundant work is being done.

The major difference in terms of usability is that you will have to open a saved search to trigger an update, after which the posts will get updated within a few minutes. If your general usage pattern is to check your tag subscription every day then the end result won't change much. I am thinking of ways to preempt updates. If this is a major concern I can investigate it further.

The migration plan is to simply offeer a link to convert your tag subscription into equivalent saved searches. At some point tag subscriptions will stop updating. I have no intent on deleting existing subscriptions but you will not be able to create new ones.

If there are any questions or concerns I'd like to hear your opinions.

I like how the tag subscriptions are right on the front page of my profile because I the way I use it reminds me of stuff. Saved searches don't do that. Also, isn't it true that saved searches are still limited to the amount of tags a certain user can search at a time (e.g. two for regular users, nine for platinum)?

albert said:

A saved search can belong to a category, which is just a free form string you can choose.

tapnek said:

I like how the tag subscriptions are right on the front page of my profile because I the way I use it reminds me of stuff. Saved searches don't do that.

Would it be possible/feasible to show some recent uploads for each saved search category on our profile pages?

I use tag subscriptions for easy access to low-count artists I'm monitoring and frequently check up on. I don't know what I can suggest in terms of functionality but this is how I use that service.

Would be nice perhaps if you could search the arttags from all your uploads and search them by how many posts they have. Subscriptions and saved searches both feel limited and confusing and aren't that useful to me.

chodorov said:

Would be nice perhaps if you could search the arttags from all your uploads and search them by how many posts they have.

Oh yes. That would be a very useful feature. This way, I can go back and check back on artists I barely encounter on a Pixiv or Tumblr search.

I migrated my subs to SS's, which seemed to work well enough. Problem is I can't seem to get the Gallery to bring up anything besides a Bad Gateway, same with the categories. Clicking an individual search works, but that loses a lot of functionality from what I was able to do before by just using my subs as a tag for all of it.

Is this a case of needing to give it more time, or needing to redo the format of my searches?

I use subscriptions to follow interesting things that are uploaded irregularly, be it copyrights or artists. The way it's displayed in user profile is beneficial for this kind of usage, since any changes are noticeable right away.

So, would saved searches still display posts right away in user profile? If they would not, then unfortunately it's not a valid replacement to subscriptions - even RSS reader would be more useful. Transparent migration would be creating a category for each current subscription, migrating every subscription line to the separate search query, then aggregating first posts for a category.

The major difference in terms of usability is that you will have to open a saved search to trigger an update, after which the posts will get updated within a few minutes. If your general usage pattern is to check your tag subscription every day then the end result won't change much.

Well, if I'll have to open every saved search, wait a few minutes and then open it again to see if anything new was added, then it actually would change much, since it involves quite a lot of manual work as opposed to a quick glance on profile.

I am thinking of ways to preempt updates. If this is a major concern I can investigate it further.

The simplest way I can think of is updating upon logon, if certain time period has passed since last update. For Gold+, that is. Regular members would have to update manually. If done like this, it would pretty much mirror the existing paradigm with added benefit of not processing anything for inactive users. Also, I'm pretty sure that manually updating every search manually is going to be a major pain in the back, so updating the entire category at once should be there, too. Maybe there's no point in updating separate searches, even. If you're worried about performance hit, limit regulars to N searches per category, single category update per hour, and allow category update only if 24 hours have passed since last update.

Also, can you show how those searches would look in the profile, UI-wise? Since every search is now separate, and is processed on another server, there is a few possible improvements. Namely, every search could report the date of the newest post it contains, and could be ordered within category like that. Every category/search could report last update date and time. If search is currently updating, it could be indicated as well.

Right now I am using tag subscriptions to follow my favorite artists. To do that as efficiently as possible I create subscriptions looking like this:

Tag Subscription 1 - "Artists_1"

~orange_bullet ~popobobo ~racer_(magnet) ~redcomet ~rustle ~ryunnu ~saber_01 ~sadamoto_yoshiyuki ~saitou_masatsugu ~sakura_kotetsu ~sakuya_tsuitachi ~sawasawa
~sayori ~shimano_natsume ~shinama ~shokuyou_mogura ~siraha ~sody ~sugina_miki ~suzuhira_hiro ~suzuki24 ~takamichi ~takeuchi_takashi ~tamon_maru
~tanabe_kyou ~tanaka_takayuki ~tenako_(mugu77) ~tomiyama_akiji ~toraishi_666 ~tsubasa_tsubasa ~ttomm ~unasaka_ryou ~watanabe_akio ~wokada ~xil ~yamada_(gotyui)
[...]

Tag Subscription 2 - "Artists_2"

~yang-do ~yokaze_japan ~yoshida_seiji ~yotsuyu ~40010prototype ~774_(nanashi) ~92m ~aaaa ~abe_ranzu ~abec ~abubu ~akahoshi_kenji
~akaza ~akesaka_iku ~alexa_mahone ~an2a ~anbe_yoshirou ~aner_(qqan00) ~anyannko ~aoi_nagisa_(artist) ~arakure ~arisaka_ako ~baariya ~bitibiti
~blackblades ~boko_(maniacpurple) ~bosshi ~bridow ~bubuzuke ~butakoma_300g ~cait ~chakabo ~chenbo ~chikado ~chikusan_nurseman ~choco
[...]

Tag Subscription 3 - "Artists_3"

[...]

As a platinum user I can create queries with up to 12 tags. Tag Subscription allow me to create up to 5 subscriptions with up to 20 queries each. A single tag subscription provides me a listing of the 200 most recent posts matching my queries.

Let's do the math: 5 subscriptions * 20 queries * 12 tags = 1200 tags
That means that I could subscribe to 1200 different artists maximum.

Okay, now I'll try the same with saved searches.

albert said:

• A single saved search represents a single tag query.
• Platinum users can have at most 1,000 saved searches. Everyone else can have at most 250 saved searches.
• A saved search can belong to a category, which is just a free form string you can choose.
• If you have a gold account, you can view all posts for a single category, or across all saved searches.
• Saved searches are updated asynchronously, on demand.
• Each saved search will return the most recent 1,000 posts matching that tag query.

Looks like I can create up to 1000 saved searches.
Math: 1000 queries * 12 tags = 12000 tags
That means that I could "subscribe" to a maximum of 12000 different artists and each saved search would return the most recent 1,000 posts matching my tag query.

Altogether, this looks like a huge update for me on paper. However, I also share pretty much the same questions and worries with @Type-kun. The usability should not suffer from these changes.

The major difference in terms of usability is that you will have to open a saved search to trigger an update, after which the posts will get updated within a few minutes. If your general usage pattern is to check your tag subscription every day then the end result won't change much.

I am thinking of ways to preempt updates. If this is a major concern I can investigate it further.

Being able to cover 12000 different artists with my saved searches list doesn't mean anything if I have to manually click on every single search to trigger an update.

The UI is not very intuitive, but I do use subscriptions as a way to keep am eye on artists who I'm interested in bit who don't update daily. Unless saved searches can be automated to run and also show results without me doing more than opening one page as I do with subscriptions (going to my profile), I'd say I'm not that interested in the change.

Especially not if I have to trigger it. Unless the saved searches were automatically run once a day so the results showed up on the saved searches page under each search like how subscriptions show in my profile.

Type-kun said:

The simplest way I can think of is updating upon logon...

I can't speak for everyone else, but I always remain "logged on" (because of browser cookies). Would it be possible to trigger an update event instead when a user's authentication and session info is passed to Danbooru, if a certain amount of time has passed since the last update like you said?

Well, I don't know if the above is possible or not...? If it is, then that'd be ideal. It would also be nice if any of the "saved searches" could also be updated on demand, in case someone wanted to check it multiple times throughout the day.

BrokenEagle98 said:

I can't speak for everyone else, but I always remain "logged on" (because of browser cookies). Would it be possible to trigger an update event instead when a user's authentication and session info is passed to Danbooru, if a certain amount of time has passed since the last update like you said?

That's what I meant, I just wasn't clear enough. I'm pretty sure the session is destroyed when browser is closed, but you log in automatically via saved cookie information.

Type-kun said:

That's what I meant, I just wasn't clear enough. I'm pretty sure the session is destroyed when browser is closed, but you log in automatically via saved cookie information.

Yes, the sessions_controller destroys the session on sign out. The _danbooru_session cookie itself expires at the end of the browsing session (browser close).

If a user decides to check the "remember" option on sign in, permanent user_name and password_hash cookies will be created. These cookies are used to generate a temporary _danbooru_session cookie each time Danbooru is visited.

reiyasona said:

Right now I am using tag subscriptions to follow my favorite artists. To do that as efficiently as possible I create subscriptions looking like this:

Tag Subscription 1 - "Artists_1"

I do exactly the same thing, down to Artists 01 etc. etc..

.

Very useful to have all the Grand Chibi Masters ready to hand, and checkable with a click.

Mirroring what was said by several individuals here. I use subscriptions to follow specific things that aren't uploaded very regularly, and with them being displayed on my profile and updated automatically, it's very easy to check for new things. Unless saved searches can do both of these things, then they won't really be a replacement to me.

Agreeing with most of the things other people have said too. I use it to check up on artists, copyrights and characters I like and I like how easy it is to look when they are displayed on my profile and updated automatically.

And I don't know if it would change anything but I never close my danbooru page except when restarting my computer, I always have at least 4 different tabs open 24/7.

Are the subscriptions taxing because "~1 ~2 .. ~50" --> "first 6 posts search" are intensive, or because there are so many subscriptions?

Update on login or on (own) profile view (cache 1~24h) could smooth it out.

Btw, i can't seem to view reiyasona's subscriptions?

itsonlyaname said:

Btw, i can't seem to view reiyasona's subscriptions?

Didn't make them public. Fixed!

Type-kun said:

That's what I meant, I just wasn't clear enough. I'm pretty sure the session is destroyed when browser is closed, but you log in automatically via saved cookie information.

Ah good, I thought that was what you meant. I also wasn't sure.

On a related issue, if Saved Searches gets the ability to be automatically/manually updated, it would be nice if there was an option to turn that off for specific categories. This is because the primary use case of Saved Searches for me is to keep track of my multiple tag gardening efforts.

For instance, I've been going through all of the tags with the angry tag to add the fume tag as appropriate. Currently, my saved search has the category name "Fume" with just a single search angry id:1300000..1400000. This lets me know that I have already gone through post ids 0..1400000, and then I've been doing them in 100000 increments.

In cases like the one above, I wouldn't want Danbooru to waste resources updating them either automatically or manually. I would just want it to open up a search as if I had entered those tags manually (like it currently does). If that would be too difficult, I could always go back to using bookmarks, but that unfortunately isn't as portable between computers.

I didn't realize profiles were that often used. If people are interested in a kind of portal page that has more relevant information than the profile page maybe that's worth investigating. At the very least I'll look into exposing saved searches in the profile page. That'd be a good way to test the server load at least.

In practice, my statement that you have to open a saved search before triggering an update isn't true. It will aggregate the searches on demand, synchronously, although the data you see may be a few hours stale. The only time this isn't true is when you create a saved search for a query that has never been searched for by anyone, but your act of creating the saved search ensures that this is only a one-time event.

albert said:

I didn't realize profiles were that often used. If people are interested in a kind of portal page that has more relevant information than the profile page maybe that's worth investigating. At the very least I'll look into exposing saved searches in the profile page. That'd be a good way to test the server load at least.

In practice, my statement that you have to open a saved search before triggering an update isn't true. It will aggregate the searches on demand, synchronously, although the data you see may be a few hours stale. The only time this isn't true is when you create a saved search for a query that has never been searched for by anyone, but your act of creating the saved search ensures that this is only a one-time event.

I use my profile page to keep track of my stats, as well as my subscriptions - it's a quick and easy way to get them, and to see if any new updates have popped up. Plus, it's an easy way to get to my uploads page without typing in the specific search.

A portal page to my saved searches to show the first 10 newest results for each tag would help take the sting out of losing subscriptions, if that's how it worked.