4

I'm looking to produce plots from some very large data sets. To start I've got a data set with 40 different series in it, each with 5000 samples that I'd like to produce a line chart out of, with each series represented by a line in the plot.

Numbers chokes after about 450 samples. Excel fares a little better and gets to 1000 samples. But neither of them can handle the full set of samples across all the series.

Is there better software I can use to generate the plot? Would OmniGraphSketcher handle this size of data set? What about R? Or gnuplot?

Something with an easier learning curve is preferred (i.e. I'm avoiding gnuplot like the plague until it's clear it's my only hope...) and it doesn't bother me if the generation of the plot can't be done in real time. I don't mind setting things up and then waiting for an image of the plot to be produced in the background. Bonus points if it can produce plot output that's as easy on the eyes as Numbers will produce for smaller data sets.

  • Won't the graph be a little spaghetti-like? What format is the data currently in? (Doesn't make a huge difference, but it would help to know.) I use R a lot and I think it could definitely handle the graph you're talking about. I just tried generating a 40 column by 5000 row matrix and plotting it using matplot and it took 23 seconds on my Mac on battery power. – Wayne Apr 5 '11 at 0:01
  • @Wayne: the look is okay for my purposes. The data is a lot of zeros in the series with periods of non-zero values that don't really overlap from series to series. It's in CSV with each series a column and the first column is the timestamp for the sample interval. – Ian C. Apr 5 '11 at 0:09
  • There are three different graphics packages in R, and each probably has several ways of doing what you want. In the base graphics package, I think matplot would do what you want, and read.csv can read in your data. – Wayne Apr 5 '11 at 0:43
5

I assume that you mean 5000 observations. R will work, and RStudio will get you to use R with a bit more GUI than the native application. Commercial solutions like Stata -- which has a soft learning curve, in comparison especially to R -- will also work that kind of sample without any difficulty (by statistical standards, 5000 observations is a relatively small dataset).

The learning curve will matter if you want to reach beyond a mere plot; otherwise, a quick look at the documentation for line graphs should suffice (for R, check there and there, for instance). HTH!

  • That first R tutorial link looks really promising. – Ian C. Apr 5 '11 at 0:05
  • That first tutorial plus RStudio got me up and plotting this data. Thanks! – Ian C. Apr 5 '11 at 4:01
0

R will definitely handle that size of data for you. If you routinely manage data sets, I would definitely recommend learning it.

Shortcut to giving you the OpenOffice.org suggestion (also known as OOo).

I'm surprised, however, that Excel crashes on you. I used Excel (on Windows) recently to manage 8 series of 27 000 samples with no issues, producing complex charts in fractions of a second.

At what point do these two applications bug? Is it, by any chance, at the time they import your data set, or do they bug when you attempt to compile them into a chart? I don't know about Numbers, but Excel should be able to handle hundreds of thousands of rows without missing a step.

  • Numbers spins indefinitely trying to draw the plot after I select the 480th row. Excel: the 1000th row. I'm using the latest Numbers but my MS Office is several years old. Both display the data in tables just fine, they only fall apart when plotting it. – Ian C. Apr 5 '11 at 0:03
  • 1
    Microsoft Excel for Mac is notoriously unreliable, even more than the PC version. I also faced crashes in the rare cases where I used it for more than data cleaning. – Fr. Apr 5 '11 at 0:15
  • @Fr. I will back this statement up, I would strongly suggest using Windows when using Microsoft Office for large/more complex data. – Skeleton Bow Jun 7 '18 at 19:21
  • 1
    @SkeletonBow Hasn't gotten better in the last 7 years, eh? That's a shame. :( – msanford Jun 7 '18 at 19:53
  • As of today, I would advise against using MS Excel in any circumstance, and use Google Sheets instead for basic data collection, management and visualization. – Fr. Jun 8 '18 at 20:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .