Joblib - how to parallelize changing a variable in memory
I have a question regarding joblib . I am working with networkX plots and want to parallelize edge modification, as iterating over a list of edges is really an awkwardly parallel problem. That being said, I was thinking about running a simplified version of the code.
I have a variable x
. It is a list of lists, similar to an edge list, although I understand that networkX returns a list of tuples for a list of edges and is basically a dictionary based version. Please bring this simple example for now.
x = [[0, 1, {'a': 1}],
[1, 3, {'a': 3}]]
I have two functions that change the value of a dictionary 'a'
like adding the first two values or subtracting the first two values. They are defined as such:
def additive_a(edge):
edge[2]['a'] = edge[0] + edge[1]
def subtractive_a(edge):
edge[2]['a'] = edge[0] - edge[1]
If I do a regular loop, the variable x
can be changed correctly:
for edge in x:
subtractive_a(edge) # or additive_a(edge) works as well.
Result:
[[0, 1, {'a': -1}], [1, 3, {'a': -2}]]
However, when I try to do it with joblib, I cannot get the desired output:
Parallel(n_jobs=8)(delayed(subtractive_a)(edge) for edge in x)
# I understand that n_jobs=8 for a two-item list is overkill.
Desired output:
[[0, 1, {'a': -1}], [1, 3, {'a': -2}]]
When I check x, it doesn't change:
[[0, 1, {'a': 1}], [1, 3, {'a': 3}]]
I'm not sure what's going on here. I can understand the example provided in the joblib documentation - which specifically showed calculating an array of numbers using a simple simple function. However, this does not involve modifying an existing object in memory and this is what I think I am trying to do. Is there a solution to this? How do I modify this code to parallelize the modification of a single object in memory?
source to share
No one has answered this question yet
Check out similar questions: