Tell me more ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.
up vote 4 down vote favorite

I'm looking to produce plots from some very large data sets. To start I've got a data set with 40 different series in it, each with 5000 samples that I'd like to produce a line chart out of, with each series represented by a line in the plot.

Numbers chokes after about 450 samples. Excel fares a little better and gets to 1000 samples. But neither of them can handle the full set of samples across all the series.

Is there better software I can use to generate the plot? Would OmniGraphSketcher handle this size of data set? What about R? Or gnuplot?

Something with an easier learning curve is preferred (i.e. I'm avoiding gnuplot like the plague until it's clear it's my only hope...) and it doesn't bother me if the generation of the plot can't be done in real time. I don't mind setting things up and then waiting for an image of the plot to be produced in the background. Bonus points if it can produce plot output that's as easy on the eyes as Numbers will produce for smaller data sets.

share|improve this question
Won't the graph be a little spaghetti-like? What format is the data currently in? (Doesn't make a huge difference, but it would help to know.) I use R a lot and I think it could definitely handle the graph you're talking about. I just tried generating a 40 column by 5000 row matrix and plotting it using matplot and it took 23 seconds on my Mac on battery power. – Wayne Apr 5 '11 at 0:01
@Wayne: the look is okay for my purposes. The data is a lot of zeros in the series with periods of non-zero values that don't really overlap from series to series. It's in CSV with each series a column and the first column is the timestamp for the sample interval. – Ian C.♦ Apr 5 '11 at 0:09
There are three different graphics packages in R, and each probably has several ways of doing what you want. In the base graphics package, I think matplot would do what you want, and read.csv can read in your data. – Wayne Apr 5 '11 at 0:43

2 Answers

up vote 5 down vote accepted

I assume that you mean 5000 observations. R will work, and RStudio will get you to use R with a bit more GUI than the native application. Commercial solutions like Stata -- which has a soft learning curve, in comparison especially to R -- will also work that kind of sample without any difficulty (by statistical standards, 5000 observations is a relatively small dataset).

The learning curve will matter if you want to reach beyond a mere plot; otherwise, a quick look at the documentation for line graphs should suffice (for R, check there and there, for instance). HTH!

share|improve this answer
That first R tutorial link looks really promising. – Ian C.♦ Apr 5 '11 at 0:05
That first tutorial plus RStudio got me up and plotting this data. Thanks! – Ian C.♦ Apr 5 '11 at 4:01
up vote 0 down vote

R will definitely handle that size of data for you. If you routinely manage data sets, I would definitely recommend learning it.

Shortcut to giving you the OpenOffice.org suggestion (also known as OOo).

I'm surprised, however, that Excel crashes on you. I used Excel (on Windows) recently to manage 8 series of 27 000 samples with no issues, producing complex charts in fractions of a second.

At what point do these two applications bug? Is it, by any chance, at the time they import your data set, or do they bug when you attempt to compile them into a chart? I don't know about Numbers, but Excel should be able to handle hundreds of thousands of rows without missing a step.

share|improve this answer
Numbers spins indefinitely trying to draw the plot after I select the 480th row. Excel: the 1000th row. I'm using the latest Numbers but my MS Office is several years old. Both display the data in tables just fine, they only fall apart when plotting it. – Ian C.♦ Apr 5 '11 at 0:03
Microsoft Excel for Mac is notoriously unreliable, even more than the PC version. I also faced crashes in the rare cases where I used it for more than data cleaning. – Fr. Apr 5 '11 at 0:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.