data visualizationr
I find R can take a long time to generate plots when millions of points are present – unsurprising given that points are plotted individually. Furthermore, such plots are often too cluttered and dense to be useful. Many of the points overlap and form a black mass and a lot of time is spent plotting more points into that mass.
Are there any statistical alternatives to representing large $n$ data in a standard scatterplot? I have considered a density plot, but what other alternatives are there?
I suggest adding an example or two of what you are presently doing so we can better see what you are dealing with.
What you are concerned with is an important issue: how do you convey the "overall" pattern in the time series data while also not misleading viewers by showing just average values? One way I have dealt with this situation is plotting an average or median line along with surrounding quantile bands. For example,
Here, the time series data are from a bootstrap-based simulation so there are hundreds of values associated with each time point. The actual data are plotted in the black line with colored bands showing the variability of values from the simulation. This particular plot is maybe not the best example to show, but you can see that some points have much more variability than others, and you can also assess how the variability is skewed above/below the actual values depending on the position in the series.
UPDATE: Given your update here are some additional questions and thoughts... What decisions, if any, are made from this visualization? For example, are you looking for specific points in time where there is very slow response time, perhaps above a specific threshold? If so, it may be better to simply plot all of the points as a scatter plot, and then also plot a time series line showing the average value, as well as some lines delineating the bounds you are concerned about. This recommendation is not appropriate if you have numerous observations at some time points (too much clutter), or if your time measurement is not sufficiently coarse (in which case you can bin response data into minute-wide time of day intervals). But the visualization recommendation will certainly be affected by what decision(s) will be supported with it. In my example, I was looking at such plots side by side, one from one simulation and the other from another simulation (each simulation using different parameters) so I could assess the variability of the underlying model due to sampling error.
Python with MatplotLib. Python is pretty good at manipulating csv files.
but haven't seen any simple examples for plotting multiple columns of data onto the same X and Y axes as a spreadsheet scatterplot will do.
Have you explored ggplot2? You can keep adding series to a plot using ggplot2. It also has a very good facet plotting feature.
Best Answer
This is a hard task with no ready solutions (this is of course because density plot is so a tempting fallback than no one really cares). So, what can you do?
If they really overlap (i.e. have exactly the same X & Y coordinates) and you are not using alpha, the best idea would be just to reduce the overlap using
unique(with alpha, it may be summed over such groups).If not, you may manually round the coordinates to the nearest pixels and use the previous method (yet this is a dirty solution).
Finally, you can make a density plot only to use it to subsample the points in the most dense areas. This on the other hand will not make the exactly same plot and may introduce artifacts if not precisely tuned.