related discussion in SO election chat

(Just me, not the entire council): Doing a dry-run without someone actually appealing to the council may be a problem, as part of the council will get insight into their history without them actually saying/agreeing the council should have that insight. At least for the reinstatement processes, it's probably going to put too much of a strain on an already stretched thin employee team to create fake data to do these dry runs with the council.

@Tinkeringbell we had this discussed in chat referred above and I eventually decided that introducing appeal for cases like that would be troublesome even in real evaluation - and for the purpose of a dry run it would be simply unacceptable. "Hey, hello, we made 5th amendment to reinstatement process and we are now smoke testing how it would work like without any real consequences, would you mind to appeal again to help in our test?"

I've seen those messages. I'm confused if we're agreeing or disagreeing: your post says the council should use real cases for dry runs, while your comment suggests that you see the trouble with that.To be clear, what I'm trying to say is that I think the council can't just go investigating cases like they're escalated when they're not escalated, especially not for the purpose of a 'dry run', as the whole process includes SE sharing data with the council about e.g. suspensions or reasons for denial that isn't otherwise available to council members, and a dry run without is basically useless.

nope @Tinkeringbell I only have trouble involving demand for repeated appeal(s) in that. As for handling real use cases, I want just that. Worth stressing here that all council members are already bound by mod agreement. Which should cover getting insight already - because, if it is not, then I guess we've got in some serious trouble handling cross-site spammers and trolls

Okay, cross-site spammers and trolls are a lot different than reviewing reinstatement process details, since cross-site spammers or trolls generally don't make it to moderator first, then resign/get removed, then apply for reinstatement. As for what you want, I don't really think that's all that relevant, it's about what the people that were denied reinstatement want: Do they want the detailed reasons for the denial of their reinstatement to be presented to the people on the mod council? Using real people for "dry runs" like that without their consent seems rather rude to me.

Especially as it'd be just a dry run, so it won't be real and the results won't matter anyways.

what I want @Tinkeringbell is to avoid things like "oh council made mistake approving / rejecting this request because we had no prior experience and everyone was in such a hurry and stress that we had no time to properly learn all involved complicated things". If this is not relevant then I don't know what is. Refusing to learn before judging people is what really looks rude to me

...as for results that won't matter - yes I insist on than (now) and I firmly believe that this is the only reasonable way (now). Because without trying there is no way to tell if this whole process will work at all, and it makes no sense to additionally load is with promises that may turn out empty. If (if) trial run proves that everything worked out perfectly then (only then) this can be reconsidered, maybe even somehow "upgraded" to get a real result somehow - but at this point there is no sense discussing this - because we can't tell the future

@gnat Still, I don't think dry runs using real people's data are the way to go for those.

Let's say SE denied a request because they have evidence a moderator has a mental illness.

Without that moderator escalating the issue to the mod council, that mental illness is not something the council should know about.

That falls under private information.

And I think the CMs and CLT are just stretched too thin to focus on whipping up some fake data right now for a whole fake reinstatement process to actually take place.

@Tinkeringbell Another mod council person here: fully agree. We should not be getting people's private data without them kicking off the process and expressly granting us that permission.

@gnat If someone offers up their willingness to be a test case, then maybe we'd have a starting point.

great thing about dry run is you don't have to push it to completion at any price and you are free to stop, take a break and have a patient discussion for as long as needed. You are even free to cancel the whole thing.

Using example with mental illness (or more precisely with "data impossible to share for legal reasons" because I doubt that company would explicitly communicate it), council would stop dry run at that point, discuss and decide how to proceed in future cases like that and document their findings the future. Things like this is much better to do now than wait until it falls on you in the middle of the "live" evaluation

@gnat The sharing of information is an integral part of the process at many stages. Also without information what kind of discussion would we have? What would we learn? I'm just confused about the whole issue. I'd be ok with fake data and fake users, or a willing test subject. But so far I'm not sure we have the ability to have either.

That information occurs at many points in the process as I understand it so it isn't like once we were beyond a certain point we'd be good.

the real question that was revealed by the mere proposal of dry run (which by the way is philosophically already a start of dry run) is, whether mod agreement even suffices to get insight or it needs to be amended somehow to make it possible. And second real question would be (in case if first part is cleared), whether it is required / desired / unnecessary to get some additional confirmation to dig from those who appeal

as for fake data I don't have very strong opinion on that. I certainly would prefer using it but on the other hand if it's so prohibitively difficult to make as you say then I would recommend using real data. Because if you don't, then you will eventually end up doing evaluation live - using real data - and with much higher risk of making high cost mistakes

@gnat What's the differences you see in making a mistake in a "dry run" with real data vs a "real run" with real data? It seems to me that many failure modes are shared and that real harm could be done still.

(but naturally I could be missing something here)

what you are missing is at dry run you are guaranteed to have enough time to stop, think, and make a thoroughly evaluated decision on how to proceed further. At live evaluation you can only hope for something like that (but better be prepared to stressed timing and emotional pressure which won't give you a luxury of dry run pace). This is all already stated in my answer

@gnat Meh. You don't learn to handle the stress of timing and emotional pressure by being in situations where there is none (which a dry run would be). I'm also not sure there even is a time limit on the mod council evaluation that's as tight as you're suggesting.

Also, I do think the people on the council, and moderators in general, have learned to handle a bit of stress and emotional pressure, but if you don't trust them to have that... there's not much a dry run is going to do to fix that. I can tell you next week we did one and it went charmingly well?

dry run is generally not about learning to act in stress. It is about learning and clarifying little simple routine things that would otherwise take so much needed time and effort when doing live, real and possibly stressful evaluation. As for the timing matters,you better drop the assumption of fully controlling it. For example, last year SO moderators have chosen to resign once in few weeks. That's a fairly smooth rate per se - but it was totally out of your control.

...had they chosen a faster pace, you'd have to adjust your timing

quoting self, "refusing to learn before judging people is what really looks rude to me"

It's not refusing to learn. Like two people pointed out right now we simply can't do dry runs with real people data unless those real people kickstart an appeal to the mod council.

@Tinkeringbell Or I suppose do some weird thing where they allow us to use their real data in some kind of a dry run.

Which would be...strange but not entirely impossible I guess.

@Rubiksmoose I'm still not sure how much use that'd be then, as right now I don't know of anybody that went through a denied request for reinstatement + a first appeal handled by the CLT.

Oh I fully agree on it seeming to be less than useful, just stating that as a possible other option for data.

I don't mind dry runs for stuff, but if the costs outweigh any possible benefits in my mind, it's not something to spend much time on. Which... I'm leaning towards is the case here.

Especially since the messages in the MSO election chat seem to suggest this isn't entirely about the dry run, but about getting particular people reinstated.

All the hypotheticals being floated don't help the original point of the discussion: We can't and shouldn't want to access data from people that haven't lodged an appeal to the mod council for a dry-run.

^

I object against meisreading my election chat message like that. My intent in there was to clearly separate these two matters and specifically, keep SO part totally out of MSE and I tried my best to express that - "it is going to be of more general nature, not specifically about SO election". If there was something in my message over there that could mis/read differently, I would like to see it quoted and make all needed corrections / clarificatioons to resolve any possible ambiguity about that

I find it particularly sad to see misinterpretation like that when my post at MSE and whole discussion in this chat have nothing to do with it. My past experience at MSE made me used to things like that but somehow, it still hurts a bit every time it happens again

@gnat FWIW the SO thing doesn't play at all into my core objection about the use of data

@gnat Hmm. My apologies then, this message you linked me to before really made it seem you were talking about Robert's reinstatement specifically, and this one you linked under your question does express your wish to see him reinstated. Misinterpretation can happen, I hope this explains a little how I even ended up with that conclusion.