Very slow function with two for loops using Arcpy in python
I wrote some code that works fine with small data size, but when I run it through a 52000 feature dataset, it seems to get stuck in the following function:
def extract_neighboring_OSM_nodes(ref_nodes,cor_nodes):
time_start=time.time()
print "here we start finding neighbors at ", time_start
for ref_node in ref_nodes:
buffered_node = ref_node[2].buffer(10)
for cor_node in cor_nodes:
if cor_node[2].within(buffered_node):
ref_node[4].append(cor_node[0])
cor_node[4].append(ref_node[0])
# node[4][:] = [cor_nodes.index(x) for x in cor_nodes if x[2].within(buffered_node)]
time_end=time.time()
print "neighbor extraction took ", time_end
return ref_nodes
ref_node and cor_node represent a list of tuples as follows: [(FID, point, geometry, references, neighbors)] neighbors is an empty list that will be populated in the above function.
As I said, the last message printed is the first print command in this function. it seems like this feature is so slow, but for 52,000 features it shouldn't take 24 hours, right? Any idea where the problem would be or how to make the function faster?
source to share
You can try multiprocessing, here is an example - http://pythongisandstuff.wordpress.com/2013/07/31/using-arcpy-with-multiprocessing-%E2%80%93-part-3/ .
source to share
If you want to get the K nearest neighbors of each (or some, it doesn't matter) sample dataset or eps-environment of samples, there is no need to implement it yourself. Libraries have been specially created for this purpose.
After they have built a data structure (usually some kind of tree), you can query the data for a neighborhood of a specific pattern. Typically, for large data, these data structures are not as good as they are for small data, but there are also solutions for large data.
I can recommend here KDTree which has Scipy Embedding .
I hope you find it as useful as I did.
source to share