Solved – Automated approach to find patterns and correlations between multiple sets of data

correlationpattern recognition

I'm trying to find a correlation or repeated patterns between several sets of data, at this stage 15, but I plan to expand this to several hundred data sets.

At the moment I am plotting the data in sets of 3s on a line chart similar to the image below and trying to see if there are any patterns or correlation. This has very obvious downsides, mostly because a) it's manual; I could easily miss something especially if it's subtle b) doing this 3 (or even 4 or 5) at a time and trying to cover several hundred sets of data and all the combinations will take forever.

Is there an automated approach I can take to this, namely I'm looking for historical patterns or correlations between 2 or more of the data sets? I was looking into the correl function in Excel, but that won't show me patterns: it just tells me the correlation coefficient. (I'm not limited to Excel; I'm open to any program / language.)

Ive attached an example of the sort of data im working with in a Google Spreadsheet HERE

multi line chart

Best Answer

first of all when you calculate correlations, don't just use the default correlation functions in any package, they all refer to pearson correlations. also calculate the kendall's tau and spearman's rho. (Use R).

Next for the case of stock prices, only correlation may be one property, but a more useful property is called cointegration which is used in a better manner if you consider things like pairs trading.

Next these properties in case of the financial data do not stay constant, so what you should do is divide that data into slices which could be monthly or yearly or quarterly and then calculate these values for each of these slices and then come to a conclusion as to how do the two stocks behave together.

Next calculate the cross correlation function for any two stocks but once again after dividing the prices into different windows.

Next, do not use wavelets as mentioned in one of the other answers if you do not understand them, first go through the theory before even thinking about it.

Next define the local maximum and local minimum of each of the stocks and then check out the behavior of the other stock around the point when one stock has formed the local maximum or minimum. this can be done by calculating the maximum value for a group of prices for the max and the minimum for the min. this will resemble your support and resistance analysis.

Next calculate the beta between the returns of the stocks versus one common index and then group the ones that are of similar beta.

All of this can be done very easily using R.

Related Question