[New Feature] Post Replacements

A new feature has been added that allows approvers to replace a post's image. This allows image samples to be upgraded to the full version without having to delete the sample post and upload the full version separately.

How it works

• Replacing a post replaces the image only. Everything else in the post is left the same.
• Replacement is approvers only.
• Click the "Replace Image" link in the sidebar to replace an image. This opens a dialog where you paste in the new source.
• The source works the same way it does on the upload page: you can put in a html page and it will grab the full image. Be careful about Pixiv galleries; it will grab the first post if the link isn't for a specific page.
• DanbooruBot will leave a comment recording the details of what changed (unless the MD5 hash is the same).
• The replacement will be logged in /post_replacements too.
• The old file will be deleted from the server after 30 days. You can download it using the old md5 up until then.
• Admins can stop a deletion by cancelling the job on the /delayed_jobs page.

Issues

• issue #3785: ugoira broken
• issue #3569: id:<=2000000 (images hosted on raikou) broken
• issue #3815: Notes must be manually readjusted if the replacement is a different size

API

If you scrape images from Danbooru, you have to be aware that existing posts can be replaced with new images. You can use these API endpoints to track replacements:

• /post_replacements.json
• /posts/2724013/post_replacements.json

Params:

• search[id]=<id1,id2,...>
• search[creator_id]=<id1,id2,...>
• search[creator_name]=<name>
• search[post_id]=<id1,id2,...>

Past discussions:

• issue #2949
• topic #13646
• topic #13586

It's finally here~

One thing to note that this must only be used for image samples or confirmed md5 mismatches of indubitably lower quality -- but it's good to know that this was finally taken care of.

Just that now, well, we might still have people interested in replacing image samples the old fashioned way, so the quicker we get through everything the better. @Qpax @Randeel @chinatsu Let's get to work maybe?

Oh, and by the way, try to avoid anything that's an md5 mismatch from tumblr since there's still funky stuff going on. Some images have the same dimensions and same exact image data but a higher filesize -- it's kind of strange.

Excellent, and just a few days after I became an approver, too.

Mikaeri said:

One thing to note that this must only be used for image samples or confirmed md5 mismatches of indubitably lower quality -- but it's good to know that this was finally taken care of.

To expand on this:

• Don't replace posts that were revised. Upload the revision as the parent instead.
• Don't replace Twitter posts with Pixiv posts. Or at least don't do that unless we decide to (I don't think we should).

There are some other cases where this could arguably be used, but I'd like to bring them up for discussion first:

• Upgrading scans, say if someone has a cleaner version.
• Replacing webm conversions of ugoiras with the actual ugoira. You can find these under ugoira filetype:webm + ugoira filetype:mp4. These came from people uploading manual conversions before Danbooru had official support for ugoira files.
• Replacing hard translations?

Mikaeri said:

One thing to note that this must only be used for image samples or confirmed md5 mismatches of indubitably lower quality

Just to be overly pedantic and remove any ambiguity, this should be used for replacing in image of the same source. It may be clear to you in particular, but just so everyone is clear on this, don't go around replacing Twitter posts with Pixiv ones, Tumblr with Pixiv, and so on. It should be one for one (i.e. replace Pixiv sample with Pixiv original, Tumblr sample with Tumblr original).

And for the love of God's green earth please don't replace in image just because its been revised! As an archival site, we want to keep all iterations of a post, hence the revision should be made the parent post, not a replacement image.

How do you submit replacements via the API? From inspecting a manual submission, it looked like POST /posts/2725223/replacements.json with post_replacement[replacement_url] as the parameter, but that got me 500 Internal Service Error...

Since this feature is for approvers only, I think I can trust you guys enough not to screw things up and replace images with revisions and images from sources like Twitter. One more question, can approvers themselves undo these replacements in case they do screw up or can only admins do that?

RaisingK said:

How do you submit replacements via the API? From inspecting a manual submission, it looked like POST /posts/2725223/replacements.json with post_replacement[replacement_url] as the parameter, but that got me 500 Internal Service Error...

That should work, not sure why it didn't. You can also try POST /post_replacements.json with params post_id=<id> and post_replacement[replacement_url]=<url>.

You're welcome to test on my http://devbooru.ml if you want to test in a safe environment.

tapnek said:

Since this feature is for approvers only, I think I can trust you guys enough not to screw things up and replace images with revisions and images from sources like Twitter. One more question, can approvers themselves undo these replacements in case they do screw up or can only admins do that?

If someone screws up they can replace the post back to the original source. The original file is kept on the server for 30 days, so in a worst case scenario you can download /data/<old_md5>.jpg to get it back.

I would give a tentative no to replacing "cleaned" scans, as cleaning the scan often results in loss of fine detail, so it really depends on a case-by-case basis. Personally, I occasionally redraw details after cleaning but that isn't always the case.

@RaisingK

Just curious, but are you tracking reuploads for your Pixiv scanner...? As for myself, I'm automatically saving every reupload that I find to a file for later reattacks at the 1 week and 1 month point...

@evazion

Nevermind, I just had a typo in my code.

@BrokenEagle98

My scanner works off of the post tag history, so it notices replacements same as ordinary source changes and re-evaluates accordingly.

Ah, got it... we're working at it a bit differently then...

I check /posts and /uploads instead. For the latter, if (upload['updated_at'] - post['created_at']) > 600, then I consider it a reupload.

This is not much of an issue, but at this rate delayed job page would consist solely of "deleting old files", like it was flooded with updating subscriptions before. Maybe it's better to move this to daily maintenance with same grace period of 30 days since replacement?

I guess that's more doable now that there's a model for tracking replacements. Although currently the filenames of the to-be-deleted files aren't stored in the model. That would have to be changed to be able to do it in daily maintenance.

The other approach would be to fix issue #2577 (add search options to filter out these jobs).

My main concern is this:

chinatsu said:

Just to be overly pedantic and remove any ambiguity, this should be used for replacing in image of the same source. It may be clear to you in particular, but just so everyone is clear on this, don't go around replacing Twitter posts with Pixiv ones, Tumblr with Pixiv, and so on. It should be one for one (i.e. replace Pixiv sample with Pixiv original, Tumblr sample with Tumblr original).

And for the love of God's green earth please don't replace in image just because its been revised! As an archival site, we want to keep all iterations of a post, hence the revision should be made the parent post, not a replacement image.

What kind of automated checks, if any, are there to stop people going around purging version 1.0 posts? I still remember rantuyetmai deleting everything he deemed a duplicate the moment he became a janitor (and made them permanent deletions once he became a mod).

I have that concern as well about revisions but I can trust the approvers we have enough about that and hopefully, we won't have another rantuyetmai on our hands.

Hillside_Moose said:

What kind of automated checks, if any, are there to stop people going around purging version 1.0 posts? I still remember rantuyetmai deleting everything he deemed a duplicate the moment he became a janitor (and made them permanent deletions once he became a mod).

There are none, beyond not deleting the file for 30 days. Don't make untrustworthy people approvers? I would've made this mod+ if we had enough active mods to do this work, but we don't.

evazion said:

There are none, beyond not deleting the file for 30 days. Don't make untrustworthy people approvers? I would've made this mod+ if we had enough active mods to do this work, but we don't.

^ This is admittedly the biggest danger, but I think we can generally trust the approvers. I'd prefer stricter merit-based promotion on choosing approvers than having test approvers seemingly chosen at random, but for now we can assume the damage is mitigable within the 30 days allotted to a replacement. Hopefully we won't have to use this feature much after the bulk of samples are replaced.

But yes, evazion is right, we don't have enough Moderator+ users to do this, and perhaps none of them are interested in doing it -- it's an arduous, time-consuming job that sucks to do alone. The most active users as of recent are all Builder, typically with both unrestricted uploads and approval permissions (myself, Qpax, Randeel, chinatsu) and some with just approval perms (NWSiaCB, Flandre5carlet, reiyasona). I've talked to most of them, and we're all sorta planning to walk through these at a reasonable pace without burning out.

tapnek said:

I have that concern as well about revisions but I can trust the approvers we have enough about that and hopefully, we won't have another rantuyetmai on our hands.

The problem with trust is we're getting new approvers all the time, and we can't expect everyone, new or old, to keep up with all the rules and nuances of the site. That's why I want to nip this in the bud instead of waiting it to blow up.

evazion said:

There are none, beyond not deleting the file for 30 days. Don't make untrustworthy people approvers? I would've made this mod+ if we had enough active mods to do this work, but we don't.

Everyone thought rantuyetmai was trustworthy, which is why he had seven consecutive positive records. The moment he came into power he immediately abused it, and everyone handled it with kid gloves until he was promoted to moderator and started expunging posts instead.

Perhaps make a separate "can replace posts" permission?