Sandbox

12:37 PM

One space . Two spaces . Three spaces . Some special chars: ¬¦!"£$%^&*()_+-=[]{};'#:@~,./<>?

Ugh, forgot to add filters. Take #2. One space . Two spaces . Three spaces . Some special chars: ¬¦!"£$%^&*()_+-=[]{};'#:@~,./<>?

Line 1

Line 2

Line 3

Sam

12:53 PM

`¬¦!£$%^&*()_+-={}[]:@~;'#<>?,./|\

RAWR

lol heya

hi :)

Just trying to figure out how chat encodes messages ;)

Those last two messages were encoded to: %60%C2%AC%C2%A6!%C2%A3%24%25%5E%26*()_%2B-%3D%7B%7D%5B%5D%3A%40~%3B'%23%3C%3E%3‌F%2C.%2F%7C%5C&

Which just makes things more difficult since the .Net framework doesn't have any methods (that I'm aware of) that encode in the same way :/

Unihedron

:/

Well, the problem we get over here is that we recieve the HTML of the message in the json connection receiver end, while Markdown would be more helpful.

So your message:

4 mins ago, by Sam

`¬¦!£$%^&*()_+-={}[]:@~;'#<>?,./|\

Would turn into \`¬¦!£$%^&*()_+-={}[]:@~;'#<>?,./|\ like this

without the backslash in front

After asking fox how chatexchange works and he'd link me to the relevant code, I found out that they converts the message ID to a history link chat.meta.stackexchange.com/messages/2645635/history and then scraps the HTML for the markdown.

Sam

1:00 PM

Hmm

That seems quite intensive.

@Unihedron .Net has a method exactly for decoding those sorts of strings.

Unihedron

Yes, so I'm trying to work around that as well. However from watching fiddler and chrome network instances, there doesn't seem to be a specific way to get the message as markdown.

@Sam Ah, lucky. You can get the message ID and convert it to like this, which includes raw text for the HTML message content, otherwise received through a json connection: http://chat.meta.stackexchange.com/message/{0}

A simple GET request will do.

That's garbage in Java though.

heresome inline code <<<< >>>>> !#%#@*

<pre class='full'>heresome inline code <<<< >>>>> !#%#@*</pre>

Sam

Actually, I was just gonna parse whatever the socket throws at me.

Wait, are you "watching" the room via sockets or html polling?

Unihedron

The old method in the chatbot itself used an ugly hack. We would create a Web Agent instance thru HTMLUnit library, then inject a custom connection handler:

It acts like the web connection the Chrome browser does.

Sam

Ok...

Unihedron

webClient = new WebClient(BrowserVersion.CHROME);
webClient.setWebConnection(jsonChatConnection = new JSONChatConection(webClient, this));

So I'm looking for a way to change that and properly use sockets, too.

But it won't give proper messages.

Sam

1:07 PM

@Unihedron How come?

Unihedron

It returns HTML instead of the actual message, so for example when we try to eval code, the accidental print character escaping screws us over.

I'll show you!

hastebin.com/lawafabexi.tex

Foreign locale, since I couldn't fix the environment.

Sam

I guess that's another thing on your todo list? ;)

Unihedron

source: chat.stackoverflow.com/transcript/message/19845211

Sam

checks out links

Unihedron

@Sam Yes, you'd think using websockets would make everything easier, especially since the library came with it. But then it just had to return html instead of the message. So I guess it's a necessity to parse the history page for us.

By us I mean myself who's going to pull an all nighter on the weekends and wrench everything together.

Sam

1:13 PM

> @Unihedron, \u003ccode\u003eMap\u003c/code\u003e has \u003ccode\u003e.forEach()\u003c/code\u003e too

^ That looks pretty easy to decode.

Unihedron

@Sam That's the unicode literal version because it's printed to the logger. When we deserialize the json, we get this:

> @Unihedron, <code>Map</code> has <code>.forEach()</code> too

Sam

Oh

Unihedron

We could scrap the HTML tags, which just came to mind.

Maybe we don't care about the formatting after all. Hmm.

Then there's this, which is the bot catching his own message

>

{"event_type":1,"time_stamp":1415383650,"content":"\u003cpre class=\u0027full\u0027\u003estartup failed:\r\nUserScript: 1: unexpected char: 0xFFFF @ line 1, column 87.\r\n   stem.out::println??) will work\r\n                                 ^\r\n\r\n1 error\u003c/pre\u003e","user_id":4070640,"user_name":"JavaBot","room_id":139,"me‌ssage_id":19845294},

Sam

@Unihedron But that'd be difficult if the message is like: You should totally drop that and try html tags: <some><tags></to-screw></up-deh-parser>

Unihedron

@Sam yep...

oh boy, this is a tricky puzzle

Sam

1:18 PM

@Unihedron Simply ignore all messages with the bot's ID.

Then again, if you use a proper html dom parser, then you shouldn't run into those sorts of issues.

Unihedron

Possibilities:
- X Scrapping HTML tags - If we don't care about the formatting
Ruled out: broken html tags will screw up the parsing
- Scrapping the HTML of the history page
resource intensive, but done by ChatExchange, not preferred but viable
- Use a proper html dom parser - very difficult in Java, dependencies require extra attention
may be viable

wait...

@Sam

@Unihedron But that&#39;d be difficult if the message is like: <code>You should totally drop that and try html tags: &lt;some&gt;&lt;tags&gt;&lt;&#47;to-screw&gt;&lt;&#47;up-deh-parser&gt;</code>

Yeah, just maybe it would work if the HTML tags are scrapped, and then we decode the printed characters.

Sam

Ah, yeah. That should work.

Unihedron

Hmm... I'll just keep the original string as metadata if we ever need it again.

Ha, I just realized I've been rubber ducking myself.

Sam

(completely forgot everything was encoded)

lol

Unihedron

Rubber duck debugging

Rubber duck debugging is an informal term used in software engineering for a method of debugging code. The name is a reference to a story in the book The Pragmatic Programmer in which a programmer would carry around a rubber duck and debug his code by forcing himself to explain it, line-by-line, to the duck. Many other terms exist for this technique, often involving different inanimate objects. Many programmers have had the experience of explaining a programming problem to someone else, possibly even to someone who knows nothing about programming, and then hitting upon the solution in the process...

Sam

1:22 PM

At least you're not shotgun debugging ;)

Unihedron

:)

Sam

Don't forget to add multi-room support ;)

Unihedron

It's as simple as telling the Web Agent to open new tabs. ;)

... And logging in if not already logged in to the domain, of course. :P

Sam

Right. Quick side Q: have you run into any cookie issues?

Unihedron

Nope, because the web agent takes care of that for us. It stores cookies, which gets lost every time the application restarts, and we would have to programmatically login again.

Sam

1:27 PM

Well at least you don't have to manually handle cookies.

brb

Unihedron

Sounds like it's harder to do in C#.

in Low Quality Posts HQ, 25 secs ago, by Andrew T.

@Unihedron No problem at all. I understand Sam has limitation on his bandwidth, and I really don't want to push him for it. On the other hand, I really appreciate for all these bots :)

Sam

1:56 PM

Annnd back.

Sam

2:41 PM

test

test again

test

<b><i>test</b></i>

¬`¦!"£$%^&*()_+-={}[]:@~;'#<>?,./|\

a b

> Test

google.com

test

Test test test test test test test test test test test test test test test test test test test test test test test test test.

Sam

3:10 PM

`¬¦!"£$%^&*()_+-={}[]:@~;'#<>?,./

&

Sam

3:25 PM

Test test test.

@Uni Well I'm just going to strip out all the tags (via regex <a href=".*?" rel="nofollow">|</a>|</?b>|</?i>|</?code>) then decode the html.

Atm, it looks like the best approach.

Unihedron

@Sam Hmm, so you're switching to websockets and hence the change, or you're using the /message/{0} approach?

Sam

@Unihedron I was using sockets from the start ;)

Unihedron

Oh! :D

Yay! Problem is solved!

'd you still have to handle the cookies though?

Sam

Yeah :/

And I'm still struggling with generating the wss url.

But besides that, it's all fine.

Unihedron

Hmm, they show up on chrome dev consoles.

message

^ that did this:

Sam

3:38 PM

Yeah, I've got fiddler running too. ;)

Although, on SO the usr cookie is passed instead.

Unihedron

Along with a referrer token.

Sam

Yeah, thankfully the sockets have built-in support for passing the referrer.

Btw, messing with this js may be of use to you (it's helped me out, so)...

$.post('/chats/651/events', {
            since: 0,
            mode: 'Messages',
            msgCount: 1,
            fkey: fkey().fkey
        }).success(function (eve) {
            console.log(eve.time);
            // call ws-auth to get the websocket url
            $.post('/ws-auth', { roomid: 651, fkey: fkey().fkey }).success(function (au) {
                console.log(au);
                // start the webscoket
                var ws = new WebSocket(au.url + '?l=' + eve.time.toString());

Unihedron

Thanks, looks very handy, I'll play with it!

Sam

(Obviously change 651 to the room number)

Unihedron

rawr

Yay spammed with dom updates!

Sam

3:42 PM

lol :D

I've got our entire convo in the console.

Unihedron

O-o

Sam

Yeah, I've kinda left it running.

Unihedron

Aww, printed characters.

<

Is there a way to inverse the colors of the chrome dev console or something?

It's near midnight, it's hurting my eyes

Sam

Erm, I guess you'd have to install an app.

Unihedron

Do I also have to sign up for spam and bloatware?

Best deal ever! googles for such app

Sam

3:45 PM

(for global inversion)

@Unihedron Don't know, I've never looked.

Unihedron

Wikihow advises to use the magnifying glass. wikihow.com/Invert-Colors-on-Windows-7

AAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHH

Sam

:O

Unihedron

... I can deal with the white dev console.

Sam

falls off chair

Unihedron

LOL

Sam

3:52 PM

Ugh, my inv avatar looks ugly.

Unihedron

thing that's even more scary than a picture of vampire appearing on screen suddenly (for Uni): large white windows

Sam

lol :D

Sam

4:08 PM

Here goes another ce.net test...

Oh wait. I never got round to encoding the messages facepalm

Unihedron

Oops lol

I gotta work out a way to encode the messages.

Don't forget that <a tags may include the title attributes. Markdown looks like this:

[text](http://url "title")

Sam

Huh? I'm parsing outgoing messages, not incoming.

Unihedron

41 mins ago, by Sam

@Uni Well I'm just going to strip out all the tags (via regex <a href=".*?" rel="nofollow">|</a>|</?b>|</?i>|</?code>) then decode the html.

I thought you might want to add optional "title" after "href".

'Cuz I noticed that while working on my parser.

Sam

Oh... that. Right, thx.

Unihedron

no worries :)

there's also that tag which applies to ~~these~~, so watch out as well. One day it might bite you in the back.

Sam

4:17 PM

kk, I'll rethink the regex after I've finished figuring out how SE's weird chat parser works ;)

Unihedron

!

That sparks a reminder! I know I've seen Jeff's SOMarkDown on Googlecode somewhere.

well, it's for the main site, but it's partially useful, just mess around with it.

Sam

Oh?

Unihedron

code.google.com/p/pagedown/source/browse

It's called Pagedown.

> PageDown is the version of Attacklab's Showdown and WMD as used on Stack Overflow and the other Stack Exchange sites.

It includes a converter that turns Markdown into HTML, a Markdown editor with realtime preview of the generated HTML, and a few useful plugins, e.g. for sanitizing the generated HTML according to a whitelist of allowed tags.

The Markdown converter can be used both in the browser (usually in conjunction with the editor, to display a real time preview), and on the server using Node.JS.

Sam

Great! Thanks! :D

Another test. ¬¦!£$%^&*()_+-={}[]:@~;'#<>?,./