Donmai

All right, I'm out on uploads - the auto-approval is crap

Posted under General

This topic has been locked.

reiyasona

albert said:

It's basically a system of escalating upload timeouts that restrict how much you can upload. You could subject it to any contributor and (after an initial warning period) they'd be restricted to 1 upload an hour or per day.

There are lots of problems and dangers with a system like that (like who should have the right to issue probations, and is a system of negative reinforcement a good thing) which is why I'm still tweaking the draft. It's one possible solution to this complaint, however.

BrokenEagle98 said:

Whatever we choose, it will need to be robust and transparent enough to assure all users that the system of privileges on Danbooru is fair and that those who have those privileges continue to deserve them.

fossilnix said:

I'm not in favor of a vote of no confidence system either, seeing as how we had an entire thread of no confidence against a certain user last year that led to changes in the moderation process, but no probation actions taken against that user. The no-confidence approach seems like more drama than it's worth.

I share your concerns regarding a system of negative reinforcement, especially if it is a system that is based on non anonymous users accusing contributors (users with unrestricted upload permission) of having terrible taste. This will cause unnecessary QQ, drama and shitstorms.

I'd prefer a transparent automated system that is based on raw stats. Stats that inform unrestricted uploaders about the quality of their uploads. As far as I can see, we are already able to collect the basic data points for such a feedback system (forum #13112). It would be possible to create tables for "Unrestricted uploaders with worst active to deleted ratio (per week/month/total)" and "Unrestricted uploaders with most undertagged uploads (per week/month/total)". An automated QC system could then use this data to regulate the amount of uploads that are randomly pushed to the mod queue (spot-check inspection). It could also send out warnings to contributors who cross certain borders, e.g.:

  • 10% of uploads are deleted after either being flagged or not passing the mod queue
  • 10% of uploads are undertagged

This approach has the major advantage that users will be discouraged to personally attack each other. This is due to the fact that the relevant data is transparent and can be examine by everyone. It is also important to note that privileged uploaders won't be able to claim that sanctions took them totally by surprise if warnings and quality indicators visible on their profile page speak a clear language. Users who fails to improve their upload and/or tagging habits before the statistic based quality rating hits rock bottom simply loose their privileges. True to the motto: "No QQ, no drama. You brought it upon yourself and you should have seen it coming."

Conclusion of what I said here and earlier (forum #118742 & forum #118746):

  • The mod queue should continue to remain the main QC mechanism here on Danbooru.
  • The flagging system should be regarded as the second most important QC mechanism.
  • The uploads of all user who didn't demonstrate their uploading capabilities yet (users without unrestricted upload permission) should be pushed to the mod queue in their entirety.
  • Unrestricted uploaders should undergo spot checking quality control. The amount of posts that are randomly pushed to the mod queue should depend on a quality rating (backed up by transparent statistic data).

Provence said:

And yeah, scorerboards are good, but it wasn't only BrokenEagle who did work :<.......

Well ... props to everyone who is participating! Been quite busy, so haven't had the time to look into it yet.

Updated

Sacriven

Provence said:

This sounds like you know why Schrobby doesn't contribute for nearly three months and maybe longer anymore..?

Let's just say that Schrobby has reached his final milestone on his Danbooru life. I won't say anything more.

Jarlath

You know what might work, on top of the flagging system?

Automated QC - posts that meet certain criteria are automatically flagged for review,with none of the random move to mod queue requirements. That, and have a mod go into the upload pages of ALL Contributor users to spot check on a regular basis, say two to three times a year.

For the automatic QC flagging, I'd go with a few criteria - undertagged posts (5 tags or less) on upload or after they've sat that way for a few minutes, post mistakes (either needing to correct the rating or removing 3-5 tags from a post which were added by the Uploader), users with multiple flags in their posts in a month (if I'm getting two posts flagged a month, that might be an issue), and excessive use of certain tags (tagme is a big trigger for me,and when it's used is usually in situations where it shouldn't be). That should apply to ALL users, not just Contributors.

As for spot checks, if approvers only need to spend 20 minutes a day (as Provence claims) to finish the queue, then they should have plenty of time to review one or two Contributors' recent uploads to confirm they've been uploading and tagging appropriately. By applying it to all Contributors and making sure those Contributors know they could be spot checked at any time, and will be reviewed at least once a year, that may improve things with the optics. Plus, it's more fair to those Contributors if all of their uploads in a short period age checked, versus a random percentage.

And I'd like to confirm if Approvers really fans as small a load as Provence suggests in forum #118811 - because if that's the case, I don't see why I've had uploads sit in the approval queue for the full three days unless there's only a few approvers acrive or they're choosing not to review anything but their favourites. I don't see why decent art has often sat in the queue for days when a nude drawing often gets insta-approved.

Between auto-QC flagging, spot checks that can happen at any time (versus the random number of uploads, which also alerts the Uploader they're being watched), and faster processing of the upload queue, it mount work better. Especially if there are others reviewing reports on users who might also be good Contributors based on their upload to delete rate as well as their tagging. Those same reports could be used to flag problem Contributors too.

Updated

Rastamepas

Jarlath said:

post mistakes (either needing to correct the rating or removing 3-5 tags from a post which were added by the Uploader),

Are both bad criteria. I've had people change ratings because they think that the bikini was to "sexy" People shouldn't be held accountable because other people are prudes. Also people make mistakes tagging at times doesn't mean they should get demoted or "marked" if they misunderstood a tag or mistook the details of picture. I find shit all the time and I'm not here raising a fit about it.

Jarlath

Rastamepas said:

Are both bad criteria. I've had people change ratings because they think that the bikini was to "sexy" People shouldn't be held accountable because other people are prudes. Also people make mistakes tagging at times doesn't mean they should get demoted or "marked" if they misunderstood a tag or mistook the details of picture. I find shit all the time and I'm not here raising a fit about it.

That why it only needs to go into auto-QC. If there's really no problem, then they can investigate whoever changed te rating instead. And it applies to ALL users - not just Contributors.

richie

Sacriven said:

I'm gonna straight here. Yes. I don't like my uploads getting limited for third time. If you're talking about quality, then you got the wrong guy. I've learned my lesson back then, exactly one year ago. There is no way I'd let anyone disturb our privileges that we got through dedication and hard work.

I appreciate your sincerity. Especially because I suspect you're not the only one who's sharing such feelings.

Besides, all contributors were already judged by several moderators for their worthiness, but that doesn't mean they must be PERFECT all the time.

No disagreement here.
The problem is... it still doesn't work properly.
And it's not that I'm targeting you. Nothing personal.

I admit that the system that you implemented above is good, but shouldn't it be applied to only select few? To only "rogue" contributors? Like, probation period or something? That way, I can still accept.

In order to do that, someone would have to pick these rogue ones. And not once but on regular basis. Who's going to do it?
We already have feedback system online.
How many negs have you seen being given to contributors for their bad uploads lately? Bah, to janitors? Or maybe mods?
If the answer is (almost) none, then do you really believe it's because everything was fine or maybe it's only because nobody cared?
Truth is, pointing fingers at long term users (and often respected ones - hey, we're talking about contributors+ here!) is an abslolutely shitty job. Even admins often don't have enough authority, guts, conviction and commitment to do this.
And yet you're admitting that

even contributors make mistake and sometimes several contributors abuse the bypass system.

Did you give someone neg for such abuse? Why not?

So we're back to square one.

Also, aren't you a bit forceful here? It seems that you're trying to force anything that you think is "good", behind the words "for the site's sake".

Well I think it's good, so perhaps I am. But I also believe it's for site sake.

Elfaleon said:

As someone with unlimited uploads, there are times when I mean to kick something to the mod queue, but forget to check the box to do so. Reading through this, I think I'm going to do a search and flag some of my own subpar stuff, because I know there's quite a bit of chaff.

Hats off to you, good sir.
BTW there was a time I tried to track down contributors+ who are using this option.
I found one. One single contributor who was using it more or less regularly.
Perhaps there were some of them I missed, but scarecity of them was for me undeniable.

In other words - most of contributors+ truly believe that everything they're posting is of an absolutely unquestionable quality, huh?
Yeah, right.

reiyasona said:

  • 10% of uploads are deleted after either being flagged or not passing the mod queue

1. Putting these two kinds into one bag is fundamental mistake.
2. Not to mention banned artist removals.
3. Regardless, do you understand that implementing automatic demotion after losing 10% of uploads implies automatic (re)promotion when geting over 90%?
As long you want to be true to the "No QQ, no drama" motto, of course.

  • 10% of uploads are undertagged

Undertagging is far from being objective so good luck to make it automated.

  • The mod queue should continue to remain the main QC mechanism here on Danbooru.
  • The flagging system should be regarded as the second most important QC mechanism.

No problems here...

  • The uploads of all user who didn't demonstrate their uploading capabilities yet (users without unrestricted upload permission) should be pushed to the mod queue in their entirety.

Why?

  • Unrestricted uploaders should undergo spot checking quality control. The amount of posts that are randomly pushed to the mod queue should depend on a quality rating (backed up by transparent statistic data).

You'll add more work to mods then (without relief from previous point which you're apparently against).

reiyasona

richie said:

reiyasona said:

  • 10% of uploads are deleted after either being flagged or not passing the mod queue

1. Putting these two kinds into one bag is fundamental mistake.
2. Not to mention banned artist removals.
3. Regardless, do you understand that implementing automatic demotion after losing 10% of uploads implies automatic (re)promotion when geting over 90%?
As long you want to be true to the "No QQ, no drama" motto, of course.

1. ???
2. status:banned ≠ status:deleted
3. Why 90%? I think you meant to ask: "What will happen if a user improves his/her upload quality again after crossing the 10% (example value) border? Is there a chance for repromotion?" No, I don't think we should automatically repromote any contributor or unrestricted uploader after he/she messed up.

richie said:

reiyasona said:

  • 10% of uploads are undertagged

Undertagging is far from being objective so good luck to make it automated.

Yeah, that's true.

richie said:

reiyasona said:

  • The uploads of all user who didn't demonstrate their uploading capabilities yet (users without unrestricted upload permission) should be pushed to the mod queue in their entirety.

Why?

Because I don't like the idea of allowing any percentage of posts uploaded by regular users to bypass the mod queue.

richie said:

reiyasona said:

  • Unrestricted uploaders should undergo spot checking quality control. The amount of posts that are randomly pushed to the mod queue should depend on a quality rating (backed up by transparent statistic data).

You'll add more work to mods then (without relief from previous point which you're apparently against).

Provence said:

I don't reallx feel overwhelmed by the mod queue. There are 2 to 3 pages at total with 200 posts each. Looking through them does not even require a huge amount of time and I look through a lot pics manually. So this reason flies out of the window pretty fast IMO. Seriously...people who think that we have too many post...I don't really understand that. Doesn't even take 10 minutes^^.

Jarlath said:

And I'd like to confirm if Approvers really fans as small a load as Provence suggests in forum #118811 - because if that's the case, I don't see why I've had uploads sit in the approval queue for the full three days unless there's only a few approvers acrive or they're choosing not to review anything but their favourites. I don't see why decent art has often sat in the queue for days when a nude drawing often gets insta-approved.

Workload too high? Add more approvers!

Oh, and don't forget that approvers can mess up, too. INB4 let's create an automated system to count posts that get flagged+deleted after having been approved.

Updated

lkjh098

fossilnix said:

If it can't score at least 10 within 24 hours, should it be automatically deleted?

That would be a very fast way to turn this into even more of a porn-only site than it is already.

Elfaleon said:

I"m not opposed to having a certain percent of uploads just being hard-randomed into the queue at all; especially after seeing some incredibly low-quality comic submissions that squeak by from Touhou, Kancolle and LoveLive.

Personally, I'm in favor of sending a fraction of contributor uploads (maybe even just 5%) to the mod queue randomly. I don't think any sort of fancy system with sliding scales is needed. Even without any other changes, that would help contributors see if their quality level is staying high. With a bit of logging (what was sent to the mod queue, why [not a contributor, checkbox, random %, flagged], and the result), that would provide some interesting data to help guide future decision-making. If the overall deletion chance for randomly-selected contributor posts is reasonably low, then nothing really needs to be done; if it's high, we can start digging for patterns. I don't like jumping straight to assuming that it's the fault of a few contributors.

(Note: I was demoted from Builder during a previous wave of bad-contributor-hunting. After two weeks of everything going through the queue my approval rate was around 95% and I successfully petitioned to be reinstated. This makes me extra suspicious of "solutions" proposed without supporting data.)

user 460797

@richie said
How many negs have you seen being given to contributors for their bad uploads lately? Bah, to janitors? Or maybe mods?
If the answer is (almost) none, then do you really believe it's because everything was fine or maybe it's only because nobody cared?
Truth is, pointing fingers at long term users (and often respected ones - hey, we're talking about contributors+ here!) is an abslolutely shitty job. Even admins often don't have enough authority, guts, conviction and commitment to do this.
And yet you're admitting that
Did you give someone neg for such abuse? Why not?

About that: One Janitor has received multiple bad feedbacks but nothing has happened since. He also received multiple positive ones. But the bad feedbacks were given by moderators or other janitors. I don't know what poaition they had when created these feedbacks, but it was definitely higher than the Builder level or the Contributor status.
But I can't say that I don't care. The thing is that it is difficult to write a bad feedback to a person who is higher up than you are. That works easier for "Platinum creates neg feedback for Contributor" but since Mods and Admins are staff then only other mods or admins should give them feedback because they are higher up...And sometimes I really would like to give a bad feedback to Janitors, but since I'm still in trial, I think my hands are tied there.
So it's not laziness, but a normal barriere to create bad/neutral feedback for higher ups. At least for me. But I don't rest there, but I send DMs to these Mods, Janitors or Admins.

And @reiyasona
Worload is not too high. It's ok. But then I'm only one of few Janitors who goes through the queue next to @Ephyon and I also think @Not_One_Of_Us so I'd be nice to hear from them if the workload is really too high and works as an argument against sending some posts from Contributors randomly to the queue. But still, 5% (-> 5 posts out of hundred don't increase the work that much, if at all)

But another thing is: If we agree to take back randomly some posts to the queue: When should a line be drawn? By two deleted uploads or really by five deleted uploads?

Also I didn't see any moderator or admin in this topic by now. Would be nice if they'd also say something, because if only users are discussing, then this will lead to nothing. Well, except Type-Kun and OOZ662, but there I've to say, at least to Type-Kun: The issue is not: "We have to many Contribs", but "We have Contributors who might upload too many crap because of no/few monitoring"

And if we have uploads randomly send back to the queue, Contributors might think twice before uploading something iffy. Because the "Upload for approval" function is nice, but do you really think that a Contributor is using that function because they upload something in question? Well, I did that with post #2468668 and post #2468666 right now since I feel that there is a imbalance between those thighs and the slim torso. But I think that Contributors just bypass that and hope that it won't get flagged.

Updated

Jarlath

Maybe the janitor/approver group needs the real review, versus the Contributor list. If they weren't mostly inactive, then we wouldn't have this many gripes about bad Contributors bypassing the queue if everyone was seeing that their uploads were being processed in under a day. I've complained about it often enough, especially as I've had uploads time out with 1-2 reviewers.

Especially if we're going to add randon Contributor uploads to the queue. Instead of frustrated regular users having to appeal timed out posts and complaining about Contributors, it'll be Contributors AND regular users bitching about how Janitors aren't working and forcing everyone (Uploader and Approver) to waste time with appeals and re-reviewing posts that should have been approved or rejected with a solid reason (bad art, not high enough quality, etc).

We need to fix that queue wait - I uploaded one of my usual "this art is good, and I've clicked the approval checkbox because I'm tired" pictures, and I'm keeping an eye on how long it's waiting.

Needless to say, between that and another few favourites I'm tracking, I can understand why Richie and the OP are bitching about abuse of Contributor privilege. And this is a worse problem than bad Contributors, IMO, as it affects a lot more people when you've got only 2-3 active janitors, all of whom may pass the buck.

I really hate seeing posts with good art deleted because "3 did not like it enough to approve" rather than providing concrete feedback as to why it was deleted. Like 'did not like the copyright' or 'bad art'. I remember asking (forum #110394) about changing the approval system so that, should a user with Approval powers fdig into the queue, they couldn't just pass by a post with 'did not like it enough to approve' and move on.

At the time, I was told that nobody wanted changes to the Approval process and the checkboxes to avoid making Approvers dread the process. And yet, with. The current system, we're not seeing short approval times and are getting a lot of user resentment towards people who don't have to go through having ALL of their posts waiting three days.

I really think someone needs to review the Approver list. And do so regularly, to boot. Active Approvers and a short wait for rejection of approval plus clear feedback if rejected (other than "did not like enough to approve") would go a long way to stopping this from coming up every few months.

Updated

Type-kun

Sorry to ruin the fun, everybody, but I still fail to see the problem you are trying to fix here. If the contributor uploads crap, flag it. If this persists, or crap quantity is enormous, inform mods or admins, anonymously via dmail if you want, and they will be demoted.

Unrestricted uploads permission essentially means "we trust your art sense". If your sense suddenly stops to align with the approval staff (or userbase), the permission is revoked. It should be that simple.

I do honestly believe that current system works well enough as-is. If you think it is broken, summarize what exactly fails, in two or three sentences. Sorry to be rude, but it's hard to follow your walls of text, with ideas and thoughts jumping around in every direction. Properly formulating the problem is half of the solution, so start with that.

user 460797

Did you even read the posts above?
Well, I think that flagging posts is not the best way to tell if an Contributor is uploading bad pictures. First because we have a very high amount of uploader and then to write an explanation costs a lot amount of time. Flagging should be a last resort.
The idea here is to have an automated process and sorry, Contributors have not entirely proven their innocence. iF they'd have, your reason that flagging is there is contadictory. Contributors have proven that they might show some understanding of what is good and what not. But that is not reason to not control them anymore. Flagging would be good, if the amount of Contributors would be low. That system is outdated. It might have worked earlier but the last years had a huge increase in total uploads.
So to have send some posts randomly back to the queue by default solves in any case the problem that Contributor uploads are not unsupervised. It might not be the best solution, but if Danbooru calls itself a high-quality place for mostly anime-art, it should only be in its interests to also keep good track of what its Contributors are uploading.
It's not like you're screwing out people with that. If these posts are getting approved then there is nothing wrong with the Contributor. But if some posts are getting deleted, then the alarm should go on and the Contributor should get a closer look by a moderator or admin. Didn't @reiyasone post a little link about that, too?

To sum things up: It is like an automated flagging, simply because there are so many posts floating around. To keep track of them all and write a text takes very much time. In older times, with less traffic, manual flagging was good. But today?
Good uploads will be approved and bad ones not. If there are too many deleted posts (aka. bad images) then they should get a manual check.

Updated

Jarlath

Provence said:

Did you even read the posts above?
Well, I think that flagging posts is not the best way to tell if an Contributor is uploading bad pictures. First because we have a very high amount of uploader and then to write an explanation costs a lot amount of time. Flagging should be a last resort.
The idea here is to have an automated process and sorry, Contributors have not entirely proven their innocence. iF they'd have, your reason that flagging is there is contadictory. Contributors have proven that they might show some understanding of what is good and what not. But that is not reason to not control them anymore. Flagging would be good, if the amount of Contributors would be low. That system is outdated. It might have worked earlier but the last years had a huge increase in total uploads.
So to have send some posts randomly back to the queue by default solves in any case the problem that Contributor uploads are not unsupervised. It might not be the best solution, but if Danbooru calls itself a high-quality place for mostly anime-art, it should only be in its interests to also keep good track of what its Contributors are uploading.

That also assumes we've got an active Approval staff that can process them quickly and efficiently. Without them, we're just getting more deleted posts which may end up being appealed and waste everyone's time. With more active approvers and shorter queue waits, we might not be seeing this issue as then you may have more Contributors using the upload for Approval function voluntarily - I have no incentive to use it most of the time if idea reasonable expectation that a possibly good enough post will time out because it's not kantai_collection or a naked girl getting her rocks off. Forcing Contributor posts back into the mod queue fixes nothing without moderators who are active in monitoring quality.

user 460797

Jarlath said:

That also assumes we've got an active Approval staff that can process them quickly and efficiently. Without them, we're just getting more deleted posts which may end up being appealed and waste everyone's time. With more active approvers and shorter queue waits, we might not be seeing this issue as then you may have more Contributors using the upload for Approval function voluntarily - I have no incentive to use it most of tn the time if idea reasonable expectation that a possibly good enough post will time out because it's not kantai_collection or a naked girl getting her rocks off.

Like I said: I have absolutely no problems with the approval process. Without my permission, it would be very hard to say this, but I only need 10 minutes at most to go through all posts in the queue. So I think that the reason "it increases work for approvers" does not work.

But I've to ask: How are Janitors supervised? Are they supervised at all? What is about Janitors that are going around here for years like zigzag, Ephyon, FeKa, Not One Of Us, etc.? Because if they are not, then I see indeed no reason to control Contributors as well, because they once got their approval from one of them and were promoted to Contributor. And I don't say that we should compare these two ranks, but maybe we could use such a tool for Contributors, too.

Updated

BrokenEagle98

Type-kun said:

If you think it is broken, summarize what exactly fails, in two or three sentences.

flagqualitycutoff = bad
approvequalitycutoff = good

if (iscontributor) and (postquality > flagqualitycutoff):
then pass

if (not iscontributor) and (postquality < approvequalitycutoff):
then delete

Summarized...

Jarlath

Provence said:

Like I said: I have absolutely no problems with the approval process. Without my permission, it would be very hard to say this, but I only need 10 minutes at most to go through all posts in the queue. So I think that the reason "it increases work for approvers" does not work.

But I've to ask: How are Janitors supervised? Are they supervised at all? What is about Janitors that are going around here for years like zigzag, Ephyon, FeKa, Not One Of Us, etc.? Because if they are not, then I see indeed no reason to control Contributors as well, because they once got their approval from one of them and were promoted to Contributor. And I don't say that we should compare these two ranks, but maybe we could use such a tool for Contributors, too.

So you're saying the reason most (80%) posts sit in the queue for days of they're not from a favorite artist or copyright is either a) most uploads are crap which occasionally get pity approved since they're not approved in hours or b) approvers are ignoring them? That's the impression I'm getting here.

Or do you think that the complaints that the OP and about the moderation process / contributors which appear every few months are baseless? There wouldn't be so much pushback about the Upload for Moderation requirements for regular users or the proposed one for Contributors if there wasn't the expectation based on experience that posts would wait forever (up to the point that they show how many approvers ignored them) in there.

1 2 3 4 5 6 7