Current status: hammering API to get all of a beta site's comments into a db

19800/25166

@Undo Really?

:P

@hichris123 Yup! Almost done

Done!

Heh.

Space?

@hichris123 GD

JohnB was complaining about lack of comment flags.

Okay. :P

So now to MySQL?

Once I get the JSON into a CSV

Then CSV -> MySQL & Charcoal

(not sure about the Charcoal part)

Well first you have to figure out if there's any comments worth flagging.

Oh, sure there are.

Obsolete ones, certainly.

New lesson learned: Don't use a random guy's webservice to convert JSON to CSV without checking the size of the file first.

danmandle.com/blog/json-to-csv-conversion-utility

Why...?

^ Just tried to use that to convert a 8 MB file, I think I crashed his server for about 20 seconds :P

I think I'll run it locally :P

At least it wasn't SO's comments :P

Heh. Well, we can do that for Animuson's website: load that tool and have it endlessly convert a 20GB file. :P

@hichris123 A botnet that gives it 20BiG files to convert!

(BiG == GiB, but bigger)

Hmm, well how do we add it to his website?

Not sure

animuson.me/contact

Hehe

We should send him mail asking for a circus clown.

Oh gosh: animuson.org

AAaack!

And animuson.com

And animuson.me/terms and animuson.me/privacy

Why does @animuson always make it sound like there are multiple of him?

He takes himself too seriously? I don't know.

> Developers Only - Developers directly work for the network and may require any variety of information in order to debug and solve issues in the construct of the network. Any information they encounter is only discussed between other developers and all developers follow the same non-disclosure policy as moderators.

> Moderators Only - Moderators don't necessarily work for us, but are trusted members of the community given special privileges to help moderate the sites and keep the network clean. We only provide essential information for moderators to function properly. As well, moderators also must agree to follow and physically sign a non-disclosure policy for any sensitive user information they encounter that isn't visible to the public.

:P

I will laugh if I receive something in the mail addressed to the Circus Clown Department.

And Animuson Network.

Then I will send something to you, @animuson

I will print out some of Pekka's rep-dollars.

@Undo animuson.me/credits

It's the stuff he uses for the webpage.

I should update the SSL. I don't use StartCom anymore.

@animuson So how do we add something to one of your webpages? :P

Get me to fix the site so it accepts submissions?

I don't even have any way of adding stuff, aside from manually editing files.

@animuson Make a JSON to CSV converter, I have something for you.

exports SO comment DB to JSON

@animuson Please?

Oh right. I don't have Dreamweaver anymore, and I never set up my file structure in Expression Web. Meh. Doesn't sound interesting right now.

@animuson Just do it. Don't ask questions, do it!

I wasn't asking questions. I was just stating I don't want to do it. Big difference.

Whatever. Do it?

@animuson Just do it. Don't state things, do it!

What am I doing?

Adding in this.

So that I can convert this big JSON file to a CSV

So that I can flag a buncha stuff

Someone needs to host it.

Why not you, @animuson?

It's okay, guys. I made it.

Because if I hosted something like that it would be hidden away in my dev panel, which is also non-functional.

@jadarnel27 Huh wha? Really?

Yeah. Totally. I made it...back from my Craigslist sale.

Oh

Yay!

\o/

When I'm here, I'm "not working." When I'm not here, I need to get in here and start "working." @hichris123 and @Undo, time for you two to have an epic battle to determine whether the Tavern is "work" for me, once and for all.

@Pops One question: what's the point of this battle?

"to determine whether the Tavern is 'work' for me, once and for all."

But why would Undo and I battle about that?

You are of opposing beliefs on the topic.

We are?

Perhaps they are just mentally deranged, @Pops.

They dont even know they disagree with one another.

I can't even remember that we are. Maybe I didn't read the relevant parts of the transcript...

Should the Java tag be removed?

I got back from the gym to this:

"The program can also be write in java language" Obviously, every code snippet can. But then again: her code is as good as java code

@Pops Oh, well that was a real CM'y question. It was worthy of your appearence.

@JeroenVannevel I think it should; otherwise it's like I haz this code that doesn't work, but if you write it for me in Java it will work! (over exaggerated, of course).

Person removed the tag.

Yeah, I was leaning towards it as well. She already removed it, all good

@animuson Why do you always refer to yourself as We and Animuson network on your website? :P

Because I am all.

All what?

All.

So are you above us humans?

@hichris123 It's obvious, animuson is actually a network.

Someone really messed up the formatting here:stackoverflow.com/questions/21126806/…

Who puts in <br/>'s in the code (and puts it as inline code)?

Lots of people.

Like people that don't know what they're doing :P

stackoverflow.com/revisions/21126806/3

heh, I became a 10K+ user. :)

@LaszloPapp Yay!

@LaszloPapp Congrats!

10k Party!

:)

the orange indicator became 186 from 5.

@LaszloPapp Yup. Your job is to make it 0

@LaszloPapp Congrats! That's my cue to flag a bunch of stuff. :P

Don't let that @Undo fellow pawn off his work on you.

Hey!

Sssshhhhhh!!!

@animuson Well, do you have any NAA recomendations right now?

has found work for @Pops to do

Yay!

@TimStone Mmm?

I suppose really any employee would do, but I've found people respond better to feeling special.

@Pops Figure out how in the world this user's meta account there links to a different (invalid) Stack Exchange profile ID than their other accounts.

At first I thought there was a problem with the election site, but I was relieved to find out I could blame someone else.

Well, now I feel un-special and used.

what is the daily reputation record?

Can I filter the delete votes reviews per tag?

@Pops (It's okay, I just said that so the other employees wouldn't feel bad)

Hmm, this seems to have gotten 5 flags, but it looks OK now: stackoverflow.com/questions/18629474/…

is it OK to mark it as invalid?

@LaszloPapp Looks like an NAA to me, just IMO

Meh, it seems like something better for a comment, but I don't know.

oh, I just read the last line, too, and I am confused about its meaning.

@TimStone Shog solved it.

What, did Shog want to feel special?

it is auto-generated as excessively long: stackoverflow.com/questions/21120961/…

I voted for closing it, but should I mark the flag as invalid? (The other reasons look out of place to me)

@Pops Hooray! I'll give you the credit anyway, thanks!

What happens if I do [chat]?

Nothing, apparently.

@qwertynl So what are you going to do with this account?

@Shog9 199 people committed to the Beer proposal, who's going to be the 200th? :P

in general, should I commit even if I'm not sure I can satisfy the 10 post requirement even though I feel active?

@Jan What do you mean?

should I commit even though I don't like asking and I'm not sure I can answer 10 questions?

@Jan I would say ask yourself this: Can I help a growing beta community out, and will I consider myself active enough to help them?. I'd say if that's yes, then commit.

Thanks. I'll consider it this way

Disclamer: All advice may or may not be correct. :P

^ Emphasis on the may not.

@animuson Is that part of the mod agreement? All actions may or may not be correct, don't sue us? :P

And are you saying that I give bad advice? :P

return rand(0,1);

1 = yes, 0 = no?

@hichris123 1

Gah! PHP.net got a re-design!

@Jan Heh.

@animuson What, did they get compromised again?

I kid, I kid.

Are there stats on all Area 51 proposals ever? Which ones made it to which phase with X many supporters or whatnot?

there are so many flagged posts as NAA. Do I as a 10K+ user really need to mark them again the same way? Is it really helpful for mods?

LOLZ

@JanDvorak You're asking if you should commit to something you don't intend to commit to?

need to get that "sign in blood over the 'Net" feature working...

@Shog9 The way you word it sounds like I shouldn't :-/

point taken

@JanDvorak I don't know if you should or not. That's a personal decision.

But if you don't think you're gonna try to be an active member of the site, I would encourage you not to sign up.

I would try to. I'm worried about the "success" part.

Rather launch a site with dedicated enthusiasts and then open it up to the curious later on, than launch with a bunch of idle curious and... never open it up. Y'know?

...timely

I still think the beta process is broken because I don't think the area 51 process is designed to attract enthusiasts over the curious

@JanDvorak do you know anything about beer?

@Shog9 I had a different proposal in mind

"software recommendation"

@JanDvorak oh

I'm... so sorry.

unless you really love that idea, I would avoid it like the dread lurgy

@jmac fair point, but... it's what we have at present

so if I now flag 60-70 NAA posts with downvotes, I will lose my 10K+ privilege. :)

Understood. I'm the "curious" one in soft-rec :-/

@Shog9 I keep thinking I should propose an alternative system, but I'm not that creative.

@jmac folks here have been tossing around ideas for a few years now. We'll do something better eventually.

@Shog9 6-8 years?

heh...

let's hope not

As long as I have you here, are you guys working on a response to the hullabaloo about close reasons and the elimination of 'minimal understanding' and 'share your code' reasons?

I do not understand this in the privelege description because I could even vote for deletion when a question had 209 upvotes weight:

read on

Debating whether to write more than that; keeping an eye on how folks are behaving

"read on"?

@Shog9 I think that we're on a seesaw with the close reasons. The new close reasons (that eliminated too localized, and not a real question, and not constructive) were designed to help out the user to understand what they were doing wrong. The new change is to prevent the closers from using an ambiguous reason to close questions for the wrong reason, despite confusing users more.

@jmac yeah... See, there's where I might need to write something. "Minimal understanding" wasn't all that helpful to askers, as it turned out.

What if the closers saw a different description than the closees?

Yeah, it wasn't. But I don't think 'unclear' will help either (and perhaps be more confusing) since it's prancing around what closers really want to say.

@JanDvorak Now that is a very interesting idea.

(or rather, the description + usage notes)

@JanDvorak that happens in one case (duplicates). We talked about doing this for all of them, but decided it was too much unnecessary complexity

I think that you're spot on when you say that the 'master' close reasons are more important than worrying about the sub close reasons (camels, not gnats), but I also think that the problem is that the 'gnats' are actually often camels just better described as gnats.

maybe it isn't so unneccesary?

I think the community is good at closing bad questions. Very good at it. I think the frustration is from the poor connect from the user to the close reason, and again between the closer and the close reason

The bad part is that sometimes the community is too eager to close based on literal rules

I think that's a small part of the community (though a vocal one), and that most of the close voters are more focused on being allowed to close clearly bad questions than rule lawyer to close perfectly reasonable ones.

I am one of the part you describe as bigger

I'm a bit of both :-/

I wonder if something as simple as a flow chart would help....

FWIW: author editing stats for all close reasons since June...

Close Reason               Pct author edited
-------------------------- -----------------
off topic                  3
duplicate                  5
exact duplicate            6
not constructive           8
not a real question        9
too localized              9
off-topic                  9
primarily opinion-based    10
too broad                  14
unclear what you're asking 17

I really want to borrow Shog-stat-powers for a day.

I suggest measuring reopen after edit as the success rate

Close Reason               Pct author edited then reopened
-------------------------- -------------------------------
exact duplicate            0.61
duplicate                  0.63
primarily opinion-based    0.88
too broad                  1.19
not a real question        1.42
off-topic                  1.75
unclear what you're asking 2.67
too localized              4

thanks. Still a nice number

Throwing localized in there is kinda unfair, given it hasn't been an option for the entire time period I'm measuring.

@JanDvorak I think that both are effective stats. Editing shows that the user realizes it can be removed (that the status is temporary), reopening shows that the community agreed somehow and is measuring something different

so, a 20% chance of reopen if edited?

ideally we want both percentages to be higher (showing that the users know they can improve it, and do so, and that the community acknowledges that and helps them reopen it)

It seems that a way higher percentage of edited unclear questions is reopened than it is for offtopic.

The reopen/edit rate for too broad is pretty low, however

FYI: here's the breakout of author edits by off-topic reason:

Close Reason                             Pct Closed then edited
---------------------------------------- ----------------------
This question belongs on another site in 0
Questions on **professional server, netw 3
Other (add a comment explaining what is  4
Questions about **general computing hard 4
This question was caused by **a problem  4
Questions asking us to **recommend or fi 6
Questions asking for code must **demonst 9
Questions must **demonstrate a minimal u 10
Questions concerning problems with code  11

Simple attempt at a flow chart:

so, don't close regex questions? :'-(

That would be "edit and/or downvote"

Then I predict their numbers will grow.

it would be interesting to compile a list of regex questions you think are bad, and get a dev (@Shog) to compile stats about how many people come across them from google, and how long the people stay on the page (or whatever they do) to get an indication of usefulness

@jmac you should post that here

@JanDvorak seriously though, what is the beef with regex questions?

I get the whole "folks trying to use regex for stuff it's bad at" frustration

The point is that if they are useful, no matter how distasteful, we shouldn't close them.

@Shog9 They're almost universally too localised, and typically zero-effort

@Shog, because of the nature of regex, most questions 'feel' like one-off, or somehow that they won't be useful to future visitors because each pattern you use regex for is different

@Shog9 Done and done

@JanDvorak localized is... really hard to judge accurately. Effort is a red herring.

I think that the concept of keeping SO clean probably went out the window in 2008. If there are long-tail questions that are sucking a few bytes somewhere, so be it. They may not be useful to anyone, but so what? Storage is cheap. Share the bandwidth love.

just grabbed that off the list

ignoring the horrible usability involved, seems like a rather common problem

in fact, it's probably a duplicate

nevermind, he has a bug somewhere else

From a meta answer on a separate SE site, I wrote:

@Shog9 "huge amounts of alike questions that technically aren't duplicates nevertheless" is objective and, I believe, accurate. Not phone number in specific, but is "one letter, then three digits, then X" a duplicate of "X or Y, then four or five digits, then a letter"?

@JanDvorak No, they aren't duplicates. And I'll bet you there are a lot of repeat questions of the same format, and I'll bet that they get trafficked a lot from google to boot if the title is good.

@jmac How can they be useful if they're near duplicates?

@JanDvorak no, I mean that question is a bad example because his regex is fine - there's something else going wrong with the library he's using. I've edited.

I mean, a user posts his homework here, gets a slew of downvotes and (duplicate and non-explaining) answers, rewards each of them with an upvote (if he has the rep). User is helped, rep-whores are richer, SO has one more near-duplicate. Duplicate enough for googlers, but not duplicate enough for closure (according to @jmac's criteria)

@JanDvorak Because if I am living in Japan, I want to know how to search for a phone number of format (xxx)yyyy-zzzz and don't care about answers of format (xxx)yyy-zzzz

Even if I can figure out how to go from the regex for the latter to the former, I would much prefer having the latter pop up first so I don't have to make the effort. Because that's what makes the internet grand.

And if you or the other regex pros feel that answering the former if you already have the latter is a waste, then feel free to downvote it (preferably pointing them in a comment to the other answer so I can find it when I do my search for the pattern I want)

there is no need to close it -- if I answer the question based on the old answer for the slightly different pattern, and feel like posting that answer in the question that got me here then tremendous, the internet has become a better place and you have lost nothing

@jmac but then you have not one "how to write a phone number regex" question, but hundreds of them. One for Alaskan phones, one for Alabama phones, one for Zulu phones... Some of them are going to be unanswered or poorly answered.

Why is that a problem?

Some of them are going to be unanswered or poorly answered.

Why is that a problem? There are 650k questions with no answer on SO.

SO, your suggestion is "slap them for laziness, then spoonfeed without hesitation"?

Don't spoon feed them if you don't feel the need to.

Nobody is forcing you.

Ain't SO grand?

but then someone else will spoonfeed them with worse food

and be rewarded

And they will have to eat poor food, or figure out how to make it tasty on their own. No harm no foul. Closing it will mean they starve. That better?

(my impression was that we were here to provide good resources for future visitors, not determine whether we should answer questions based on our impression of the person asking. Personally I don't see the point in volunteering my time to help people if I am going to put energy in to deciding who is worth volunteering for, but if you want to discriminate, downvote, ignore, and move on)

If the hunger forces you to learn to cook, it's a win.

"good resources for future visitors" - now that's the problem with "three letters, a digit, three more letters". The sole winner is the asker and some repwhores.

and those who answer for the benefit of the asker

Inb4 SO becomes a regex-this-for-me service

Then all questions could just be edited in to that (you could make it community wiki!) and close everything that ever gets asked as a "regex this for me" as duplicates of those canonical questions

(repeat as necessary for other slightly-different but conceptually-the-same patterns)

The long and the short of it from my perspective is that as volunteers trying to make a resource, I don't think our job is to punish people, I think our job is to help. If we don't want to help for whatever reason, as volunteers we can refrain from helping. But we shouldn't get in the business of punishing folks who haven't even visited just because we don't like someone with a similar question.

We can only help so many people. People who don't want to search don't deserve us to search for them.

(as volunteers in a community-moderated process, you are more than welcome to disagree with me and go in a different direction -- that's healthy, what isn't healthy is if we hurt the quality of the resource by giving conflicting messages and driving away potentially productive users who can help us improve stuff)

So just leave the question unanswered? What is the merit by closing it?

to prevent further alike questions

So you think, "How do I match (xxx)yyy-zzzz in regex" is a bad question?

What would a good regex question be?

heavy duplication is bad.

The first time, it might become a nice canonical reference. The hundredth time, it's just a lazy dupe.

I don't see how "How do I match (xxx)yyy-zzzz" can be a good question when "How do I match (xxx)yyyy-zzzz" is a bad question. They are equivalent quality in the sense of what they are asking, the issue you take is that the second person had the bad luck of answering after you had already answered the first.

Do you suggest we have a canonical phone number regex question for each country?

Nope, I suggest that you have a single canonical answer for the phone number patterns for each country (or group of countries with a shared pattern)

"How do I parse phone numbers with regex?" as the canonical question, "US/Canada - (xxx)yyy-zzzz - s/foo/bar", etc.

You mean, every country in the same question? This smells of too broad.

Canonical question. Solves your problem. One-stop shop for phone number regexes. Community wiki. No rep whoring. Easy access to closing as duplicates. Far easier to find.

Far too big if it has to include every possible format, I'm afraid

Wikipedia did the hard work, why not use that as a template?

Better: teach them how to write a regex in that canonical answer, then let them apply that knowledge to their particular pattern.

The upside is then that you don't have to update the answer anytime some country's format changes due to e.g. exhaustion.

So you want people to go to regexlib instead of SO?

Given the size of the wiki page, I'm afraid your answer wouldn't even fit into the 30k character limit.

I am saying you take the patterns from the wikipedia page (not the explanations), and list each pattern and its regex below.

I want to teach them how to fish

Yes, but most people who find SO are not trying to learn how to fish, they want to eat fish.

I prefer general knowledge over specific and less applicable knowledge.

If only because general knowledge is easier to navigate

And so the canonical answer isn't for you. But if others enjoy giving specific knowledge for commonly asked questions (which is a perfectly fine application for SO if I understand it correctly), then why is it bad to let them do so?

@jmac If I had a penny for every time I closed something as a dupe of "How do I compare strings in Java?", "Scanner issue when using nextLine() after nextXXX" or "Is Java "pass-by-reference"?"...

How would you organise your huge answer? Alphabetically? By the continent? Without a list of hyperlinks in the question, the question will become a mess.

I am not seeing the problem. I mean. I'm seeing the frustration, but I don't see why this necessitates closing all these questions.

@DennisMeng that was already suggested ;-)

@Jan, I think you're focusing too much on the specific example. I don't care how it is organized. You can make one for each country if it so suits you. The point is that if these questions keep popping up it's because they don't have an answer on SO. So long as that remains the case, the questions will continue to get asked

@JanDvorak Sorry, didn't see it in the scrollback. Also sort of for describing my first thought after I saw discussion on canonical posts

Even if you punish everyone who has the bad luck to post such a question over the next year, there are millions of people who are coming to SO for the first time, who won't find an answer, and will continue to ask it until the ends of time.

@jmac Let's do an experiment and keep them open. I predict the growth of their count will increase. You don't even have to downvote - they are clear enough, they might be useful, it's not an exact duplicate and thinking shouldn't be needed.

@DennisMeng I mean, reputation rewards for dupe closure

@JanDvorak Oh, lol

I don't understand what the issue is with keeping them open or them multiplying, and I don't understand what the benefit is by closing them by extension. What exactly would your experiment prove?

Speaking of which, I want to say 2/3rds of my "actions" are reviews >.>

Then there are mods complaining about "too liberal duplicate closures"

@jmac positive feedback. It will become impossible to find a question worth answering among those countless duplicates and writemearegexes

@JanDvorak So what is a question worth answering in regex? Something tells me that many members would be exceedingly happy if their was somehow an "SO.elite" that was a migration path for "questions which actually ask interesting non-trivial questions for the serious expert programmers"

@jmac You know what? "questions which actually ask interesting non-trivial questions for the serious expert programmers" is exactly what SO is supposed to hold.

not countless duplicates of basic questions

part of the reason why i haven't posted a single question on SO

not regex-for-me questions where the specs are so specific the sole benefactor is the asker

I think the issue is that what began as a question that would have qualified as a interesting and non-trivial rapidly become uninteresting and trivial as the knowledge of the user base increases by answering what are now trivial and uninteresting questions

@jmac the problem isn't "boring for experts". The problem is "didn't bother to search/think/debug"

And anyway, if we start closing questions as being "too basic" then there won't be any users left (Jon Skeet would cast us all out of the kingdom).

lolol

If the person did not search/think/debug, then downvote the question, add a comment if you so please, and move on.

Hover over the downvote button: "This question does not show any research effort; it is unclear or not useful"

@jmac so, you don't mind if these questions SO wasn't meant for multiply?

As someone who spends all of his time on vanilla-SO answering questions in a tag with only basic questions, yes, I don't mind personally.

But then again I couldn't code my way out of a black box.

I am not at the level of the average amateur programmer, let alone developer, and I don't understand half of the stuff you guys take for granted. What I want isn't really relevant here.

But, what about those hard questions that people might actually benefit from if only they were answered?

Why don't you answer them?

@jmac I can't find any

Why don't you ask one?

I have found no such challenge yet

Most of the time I resolve weird bugs when preparing an SSCCE

the last time it was a multiplication instead of division. Not worth posting on Stack Overflow.

90% time my question is a duplicate. Instead of reasking, I upvote.

meh, I mispressed a button at a review.