Cross-correlation of average arrays
I have 16 1D arrays with approximately 10-11 million double precision elements. I need to cross-correlate between them, i.e. 1s 2, 1s 3, ..., 1s 16, 2s 3, 2s 4, ..., 2s 16, etc. This cannot be effectively implemented on my 2.4GHz Intel Core 2 Duo MacBook with 4GB of RAM. My question is what is the typical approach if not the brute force (faster processor, more RAM) that people use to overcome this problem, or a problem like this? Thank!
source to share
If you are calculating the Fourier transform of each of your arrays, you should be able to use the transformed arrays to efficiently compute the cross-correlation between each pair of original input arrays. See the Properties section of the Wikipedia article I linked to for using the ID.
source to share
The cross-correlation function in numpy is ridiculously slow. The openCV library has numerous friendly cross-correlation functions. Even if you try to implement a frequency domain aproach, you won't beat the openCV library, as there is more room to speed up the cross-correlation calculation. I've written about this before:
Calculate the cross-correlation function?
I believe the code is based on the tricks described in the following article:
J. P. Lewis, “Rapid Pattern Matching,” in Vision Interface, 1995, vol. 95, pp. 120-123.
source to share