Joblib - how to parallelize changing a variable in memory

I have a question regarding joblib . I am working with networkX plots and want to parallelize edge modification, as iterating over a list of edges is really an awkwardly parallel problem. That being said, I was thinking about running a simplified version of the code.

I have a variable x

. It is a list of lists, similar to an edge list, although I understand that networkX returns a list of tuples for a list of edges and is basically a dictionary based version. Please bring this simple example for now.

x = [[0, 1, {'a': 1}],
     [1, 3, {'a': 3}]]


I have two functions that change the value of a dictionary 'a'

like adding the first two values ​​or subtracting the first two values. They are defined as such:

def additive_a(edge):
    edge[2]['a'] = edge[0] + edge[1]

def subtractive_a(edge):
    edge[2]['a'] = edge[0] - edge[1]


If I do a regular loop, the variable x

can be changed correctly:

for edge in x:
    subtractive_a(edge) # or additive_a(edge) works as well.



[[0, 1, {'a': -1}], [1, 3, {'a': -2}]]


However, when I try to do it with joblib, I cannot get the desired output:

Parallel(n_jobs=8)(delayed(subtractive_a)(edge) for edge in x)
# I understand that n_jobs=8 for a two-item list is overkill.


Desired output:

[[0, 1, {'a': -1}], [1, 3, {'a': -2}]]


When I check x, it doesn't change:

[[0, 1, {'a': 1}], [1, 3, {'a': 3}]]


I'm not sure what's going on here. I can understand the example provided in the joblib documentation - which specifically showed calculating an array of numbers using a simple simple function. However, this does not involve modifying an existing object in memory and this is what I think I am trying to do. Is there a solution to this? How do I modify this code to parallelize the modification of a single object in memory?


source to share

All Articles