morning with delicious spam

yup

there was even a fan spammer on MSE

CrimsonCrisp

→ 12 messages moved to Chimney

nice how nature simply takes over once we leave the place

Maybe they consider it their pet-frog

@Shog9 I think the approach of replacing the icon to increase review participation doesn't work

I think reviewing has been suffering from the "nothing new in a year almost" syndrome in that its systemic issues have been made clear to people and there is no longer any expectation of change coming

It's very discouraging if you see robo reviewing everywhere from people either not caring about review bans or using scripts to bypass the audits while at the same time people who try legitimately get shafted by the occasional mishandled audit. I think right now the audit system is one of the big points of increasing (especially new reviewer) burnout

evil plan of the day:

Recently, on the Science Fiction & Fantasy chat board a new trend of posting images of fluffy animals has taken hold.

should we send @Telkitty there?

no one here

no two here

I'll triple that

zzzZZZZzzzzZZz

At least I got your attention, so it did wake you up ... coffee served their goal

I think this was the first day ever I used all my suggested edit reviews for a day on MSE.

Thanks to @Glorfindel ...

@rene you weren't around when this happened?

→ 3 messages moved to Chimney

@Glorfindel Obviously I missed that opportunity for easy rep reviews ...

The last message will be posted in 1 hour.

Everybody panic!

:o

@Magisch This is where @DiminutiveColossus is coming from too, I think.

But... There are two problems.

Burnout is an issue, and one we've been trying to address for some time - not very successfully, but still

Lack of discoverability is a new issue, and one that is affecting participation a LOT more severely than burnout

Taking Triage as an example... There are maybe 100-200 users who together handle a hair less than 50% of the queue in any given month

There are probably 10x that many users who handle the remainder

So failing to address discoverability means we'd need to completely solve burnout and cut the number of tasks entering the queue by half

That's... Not an appealing option.

@Shog9 Or you could do a combination of reducing tasks by more smartly prioritizing them and lowering reviews per task

Less dime a dozen reviewers also means less robo reviewing so you can afford that

@Shog9 One way to adress burnout would be to award a very small amount of rep, maybe 1 per review or 1 per 2 reviews, for reviewing

Not enough to be appealing to rep hunters but a bit so it adds up if you review every day

People already only sporadically really clamor for rep by doing edits, and these take less effort then a review and award 2 per accepted one

I disagree with the rep. Right now we provide up to 1000 rep for edits. Allowing that for reviews gets you 2/3 of the way through the privileges list and allows you to edit without review. Suddenly adding rep for reviews is going to do exactly what happened with documentation: People will rush in and click a button to get their points, regardless of quality. Queues will drop temporarily but I will bet you my rep that the quality of reviews will drop.

Throwing rep at the problem isn't a great way to solve the problem.

@Andy Hence "too low to be worth it for rep hunters"

You really think uninterested people would review for 1 rep per review?

Yes.

There are much easier ways to get rep then that

If you have no scruples

20 free rep a day?

Why wouldn't they go click "No Action" or "I'm done"

2 minutes of work

Step 1: Go to javascript / java / php tag

Step 2: Answer a super easy "gimmeh the codez" question

Step 3: More rep then you would have gotten in an entire hour of reviewing

effort: less then 5 minutes

That's a decent way to gather a down vote too. Reviews don't give you those.

It averages out to a lot of upvotes and not a lot of downvotes

People gamed docs for rep because it was spend 5 minutes editing something trivial -> rep for months

Not a "one and done" kind of deal

Hardly anyone gets the full 1000 rep from edits I'd wager

At least I only ever saw one guy

@Magisch we've already done this several times. Bluefeet spent a fair bit of time on that last year. It helped a lot... Until in the span of a couple months we were back at pre-change backlogs.

@Shog9 If you want to adress burnout, you need to find a carrot to put before people besides the one gold badge per queue

A tangible carrot. Something that people want

The desire to make the site better often isn't enough to overcome the necessary systemic barriers to reviewing

If I want to make the site better I won't need or use your review queues for that. That's what people make stuff like charcoal or SOBOTICS or SOCVR for, a place where we control the rules of engagement to a small extent. If you want people to use the official tools, there needs to be an incentive

SE of all places should know this, its success is built on the perpetual carrot

I think a big carrot would be per-emptively removing low quality posts from the queue. It gets frustrating to go through 20-40 reviews and only seeing crap. I'd much rather see good posts that need a few things to make them great.

Maybe another gold badge every 500 reviews after the first thousand

ch-ching!

just something that people want. Relying on the goodwill of people works only to a point

How about just some more postive feedback when you reviewed? Not just the Thank you for reviewing 20 suggested edits today; come back in 6 hours to continue reviewing. but something like: With your effort 7 posts were bumped to the front page, 5 question were closed and 10 low quality posts were removed. You're quality rank for today is 234 on the total of 596 reviewers.

We all love stats, right?

I'd actually like that

that's a good idea

It is an idea, maybe low hanging fruit that is ripe in 6 to 8 weeks

It is a bit in the spirit of what Closey does: github.com/SO-Close-Vote-Reviewers/SOCVR-Chatbot/wiki for SOCVR

Watching Dune now on TV... it's like playing a CGA game... lol

The review system needs something like this

@DiminutiveColossus literally. There's a bunch of stuff we can't do simply because the schema + volume makes it too heavy.

Like... I'd kill to be able to visit a tag & see on the sidebar, "3 suggested edits, 10 questions awaiting triage, 30 questions pending closure, 2 answers flagged" - and have them link directly to applicable filters.

That'd be worlds better than a link in the top bar.

And it's completely infeasible.

due to performance impact?

yeah

How about In the last 15 minutes 20 reviews in tag c# were completed, click to see if you can help

The test run for that feature was actually running targeted ads... Guess we could go back to that.

That's ... creative ...

If memory serves, they had an insanely high click-through rate

I think showing what has been done can have the same effect

@rene I like this idea, btw. Wanted something like that for help/improvement review, but again too expensive.

Pondering building a review-tracking system outside of SE. Display call-to-action via ads, post stats in a meta post.

@Shog9 That is weird. I would think just keeping track of has been done already would be cheap.

go figure, eh?

:(

that's the problem: we designed the review system at a time when there was essentially no one reviewing; the notion of having a hundred active reviewers per day sounded pretty good. We've scaled waaay beyond that.

Yeah, if you never designed stuff for high concurreny, lots of users, it is frigging hard to refactor that into something that performance well

And it's worked really well - I mean, back in 2012, there were mods who pretty much just spent all day handling close flags. Now review handles close flags, bulk of VLQ/NAA flags, automatic system flags... But it's overloaded and creaky.

@Shog9 can't you make that a community run project with a public github repo, a bit like SEDE?

maybe

I mean, except that the necessary data isn't public

would have to add APIs

Well, SEDE has also parts of its data private once it is in production

Is that a query you run?

That is, 49 items that are "locked" to a given reviewer

and that is across all queues?

yup

Is that query cheap?

yup

(inaccurate, but cheap)

bunch just expired - now 25

You could run that every minute: 49 items being reviewed and show that in the topbar

(there's a 5-minute "lock" when you open an item where it can't be assigned to others)

run it every 5 minutes then

@rene this sorta thing is kinda what /review/<queue>/stats was intended for

almost too slow to load now though

Yeah, I just realized that the close vote graph I run takes 5 minutes snapshots of that page

Is there a dev or group of devs allocated / assigned to review?

no

that's what makes most substantive improvements infeasible; any change is stealing time from some other group

So what kinds of things get devs working on the review system. Is it just bug-fixing at this point?

@Shog9 yes, but it is so hidden away, so many clicks, so many scary looking screens. I do think the ads with a live feedback count would be really useful.

I was beyond happy when Oded got the time to make suggested edit overrides happen earlier this year - that took quite a bit of time

to implement or before he was able to find time?

How feasible would it be to open source the majority of the review system and start accepting pull requests?

Some of our best reviewers are also damn good web programmers

@rene both

@Shog9 ahh

IIRC, that was originally suggested on meta in late 2012; I'd originally sent it over in... 2013-2014, but it got rejected. I sent it over again in late 2016, and it finally got done in March, tested and deployed in April.

Easily the most work that's been done on review (outside of Docs) in a couple of years; I was pretty psyched

Dev perspective here: the review queue codebase needs to be rewritten, from scratch and with what we know now that it needs to handle. The codebase and system design are gnarly at best.

design an api, let us do the work.

do crazy things

Ask Spolsky, I'm pretty sure he did things back in the days that sounded crazy at first

An API needs to be built on stable/good foundations... Need to get to those first.

Sure, but that design has a different timescale and resource demand then building the frigging stuff. Beyond that you can offload the work.

And yet - this needs to work with all sites, not just SO. And be discoverable and usable by everyone (with the privileges), not just those with the API keys.

Sure, it is not easy, I never suggested that. And there are numerous issues to be addressed and designed for. And it could maybe be like SEDE. Basically you run the community build and maintained review system.

If it so big and complex that you're no longer able to change it, just launch weird idea's.

You have a community with people than can generate those idea's at an instant

Dev gives up due to the sheer underestimating of the complexity by the oblivious community

where to ask "What is the clamping force of human hands?"

@x0a biology.se maybe? Or Physics.se, check their help and meta first.

heh, its a mix of both. ill try the metas, thanks

Maybe sports.se if you give it sports context, but that would definitely first need a meta post

that's a good idea, a lot more appropriate than biology or physics

I'm full of good idea's. They come for free ...

→ 6 messages moved to Chimney

I'm out for the night ...

@Oded There is a precedent to building things on SO first then pushing them to the rest of the network. It isn't like we would have to immediately replace the review system either, we could develop and test the new system while the old system is still in play. I just figure that if there aren't enough devs to make this stuff happen, why not outsource to your massive free labour base?

@DiminutiveColossus still needs to have a UI for users who are not Meta/Chat frequenters and those who may not be comfortable with just an API (or with installing user scripts). Not saying we can't have two systems side-by-side if designed that way, but that the idea of an API only system is a no go.

@Oded Oh for sure, I'm proposing that you make an API, so that the community can work on a separate review system that could eventually be merged back into SO / SE.

"Sharp drop at 10-50 likely indicates that many users find it difficult to work in review queue. Drop after 250 (silver badge) suggests that even after substantial amount of reviews, many users still fail to discover a way to work productively..." Folks dropping off after 50 reviews because they (wrongly) feel it's too hard are unlikely to be attracted to do more by badges and stats

@DiminutiveColossus Making an API isn't free. Reviewing and integrating a completely outsourced system isn't free either. It's not as simple as "make someone else do it"

@AdamLear I never said it would be easy. I said it would be better than nothing, which is what it seems like is going to be happening at this rate.

Here's the thing. If we had resources to throw at it, we'd just do the work instead of doing the work to enable someone else to do the work. If that makes sense.

But making an API would be a lot less work than making a review system.

Hell, coming up with a design and workflow that everyone agrees on is probably the most time consuming part of the whole thing. Outsourcing even just that part to the community is bound to save a bunch of work.

@DiminutiveColossus Maybe, maybe not. Spending time on a system that would either be thrown out eventually or that we then have to maintain indefinitely doesn't sound awesome. And we're still starting with zero available resources.

Sure, but as far as I can tell right now, the consensus is that the current review system is largely broken, can't handle the scale, and there isn't going to be any reasonable amount of dev time devoted to fixing or rewriting it, so.... basically the current review system is a somewhat failed experiment at best.

Nah, it just means the current review system hasn't aged well. Not everything that eventually breaks down is failed or an experiment.

There's probably a bunch that could be done with the existing public API. I'm hazy on which voting bits exist in it, if any.

Looks like there's some flagging, but no close/reopen. Edit is there.

One of these days we should release v2.3. I think that has more auth-gated features in it. But, see above, re: time and resources.

On that note, errands/dinner time. back later.

@AdamLear that's in the API too (even works with custom reasons). Most of what's needed is informational (what's been flagged, what's been reviewed), which is also where the weak points for review plumbing lie.

@AdamLear Yeah, I'm not worried at all about actions... whatsoever. I can handle that. What I want is the review feeding and stuff like that. Basically, you have a list of posts that need to be reviewed for a given queue and a way of selecting which review goes to which person, and I'll handle the user interface part.

Yeah, that's the bit that's gonna take some creative re-engineering, regardless of what interface gets put above it.

@Shog9 For sure, at least y'all know what I want. I don't expect anything to happen, but if it does I would be ecstatic. I am (and I can think of a few others who are) willing to work for free to make whatever needs to happen happen.

that, at least, is public