@Tim: What is this data like? String data? Arrays? Objects?

The data is tabular, and consists of numerical and string data only.

Could you generate checksums out of it?

That way you could store the checksum in a table, and easily search for a given row as to avoid duplication...

Or, you could store the data elsewhere, outside of the database, and just store the checksum, and use that to look it up later

Yeah, that'd be possible.

The first part anyway, the current storage format is not friendly to random access, heh.

Just brainstorming here a bit, since I don't know the specifics of your situation

Yeah, I might be able to illustrate the problem a little better, one second. :)

(Also, thanks a lot for the help!)

Sure thing, glad to be of assistance

brb

Alright, now that I didn't break Google Docs

spreadsheets.google.com/…

LOL. Let me have a looksee

(and don't worry, the data isn't "real")

Basically, the data comes in the form on the Details tab

And I need tables that will be flexible enough to handle that and produce the first two tabs as well.

OK... so...

I think the message column from the details tab should be it's own table

That will save a lot of space

Yeah

As for the data itself, is that the Value column of the Details sheet?

@TheUnhandledException Yep. The values correspond to the column names in the Variable column

The value comes in key-value pairs, it's just flattened for the sake of display there

Hmmmm... I understand now

Yeah, there's a ton of repetition, but only in the context of a single source (identified by the Name column)

Yeah, I'm thinking a data table, with columns for variable and value

and then an error_data table, containing error_id and data_id

so a one-to-many relationship between error messages and data rows

So error_data would have a record for each record that has an error of a given type then right?

@Tim yeah

so that first row in the details column would have one error row and two error_data rows

@TheUnhandledException Alright, great. That was close to where I had gotten, but for some reason my brain wouldn't make that extra leap. :)

the two error_datas would be (Label,BRTHDTC),(Variable,BRTHDTC)

so for example, data id#1 = LABEL,BRTHDTC

data id#2 = variable,BRTHDTC

Then you can have any number of errors refer to those

Rows 57-66 are a better example

You'd only store "PC" and "PCSEQ" once

Right

maybe data ID#22 is RDOMAIN,PC. Then all the errors can have error_data id= the error row ID, data_id=22

it will still be a ton of data, but at least it won't be duplicated

@AidenBell if you're around, do you agree?

@TheUnhandledException Yeah, that definitely works. Awesome!

cool

Feel free to ask others for help, like I said, I've designed quite a few databases but am not an expert! :-)

I've learned what I needed to, I'm sure others could critique my suggestions :-)

Yeah, I was too unsure of myself here to feel confident about trying anything due to the issues that might arise from a terrible design, heh. So your insight should help me at least get something workable, which we're then going to pump full of data to see how it performs :)

@Tim Awesome! I hope it works out for you

@TheUnhandledException Thanks, and thanks again!

This room isn't very popular yet, I hope moving it to SO chat will gat more attention/people in here

@Tim You're very welcome

@TheUnhandledException Yeah, admittedly I wasn't aware that it existed until now, heh.

@Tim I should drive more people in here

@TheUnhandledException do you know when SO Chat will go live?

Problem is, if nobody's here to help, it doesn't do much good. catch-22 :-)

@PiersMyers I've been asking for weeks

Looks like any day now

chat.stackoverflow.com/beta-access

Yeah, I'm going to leave myself parked in here now. ;)

@TheUnhandledException what's the password?

@PiersMyers Not this again! :-D

(We were trying to guess it for like 3 hours this morning in The Tavern)

hehe, I missed that