Big data is a misnomer. Complex data is a better description. Having a terabyte of simple data with 2 columns is really not that difficult to analyze and won't give you much information. Whereas having a few hundred mbs data with complex relationships and many dimensions can yield tons of information and is far more difficult to analyze.

Difficulty in "big data" should be about its horizontal breadth (covering many aspects of a system) rather than its vertical depth (covering one aspect of a system in great resolution).

Not to go all senselessly pedantic, but doesn't Excel have a limit of like 55,000 rows?

The devil is in the details. Big Data is really a massive cluster of VMs running maxed out Excel spreadsheets, and instrumented to restart automatically and restore from redundant backup, a la RAID, when the Excel process crashes one of the Windows VMs.

If you're being pedantic, it's 1,048,576 rows from Excel 2007, 65,536 rows before that.

2^20 rows! :)

Current limit is 1,000,000 pr worksheet. However there is a tool called PowerPivot which lets you get around that limit and do analysis on larger data sets.

I didn't see this response when I replied to the parent. Power Pivot is pretty great when you can use it.

Not since 2007 where it got bumped to over 1 million. http://office.microsoft.com/en-us/excel-help/excel-specifica...

He's saying his file has way more rows than that, so maybe they upped the limit in the more recent versions of excel? (I think he also wrote a bunch of VBA and hooked into some external systems too)

Not when you use it with Power Pivot.

16 384 columns and 1 048 576 rows actually

2^14 columns * 2^20 rows * 8 bytes per cell = 128GB. Bang on.