Python Multiprocessing preserves data until further called in each process

Linux uses copy-on-write , which means that when a subprocess forks, the globals in each subprocess have the same memory address while the value changes. Only when the value is changed is it copied.

So, in theory, if the large object is not modified, it can be used by subprocesses without consuming more memory. Let him test this theory.

Here is your code related to memory usage logging:

import sklearn.feature_extraction.text as ftext
import sklearn.linear_model as lm
import multiprocessing as mp
import logging

logger = mp.get_logger()
formatter = logging.Formatter('%(asctime)s: [%(processName)12s] %(message)s',
                              datefmt='%H:%M:%S')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.WARNING)
mp._log_to_stderr = True


def predict(essay="this essay will be predicted"):
    """this method copies the large object in memory which is what im trying to avoid"""
    vectorized_essay = large_object[0].transform(essay)
    large_object[1].predict(vectorized_essay)
    report_memory("done")


def train_and_model():
    """this is very similar to the instantiate_predict method from my first code sample"""
    tfidf_vect = ftext.TfidfVectorizer()
    N = 100000
    corpus = [
        'This is the first document.',
        'This is the second second document.',
        'And the third one.',
        'Is this the first document?', ] * N
    y = [1, 0, 1, 0] * N
    report_memory('Before fit_transform')
    X = tfidf_vect.fit_transform(corpus)
    model = lm.LogisticRegression()
    model.fit(X, y)
    report_memory('After model.fit')
    return (tfidf_vect, model)


def free_memory():
    """
    Return free memory available, including buffer and cached memory
    """
    total = 0
    with open('/proc/meminfo', 'r') as f:
        for line in f:
            line = line.strip()
            if any(line.startswith(field) for field in ('MemFree', 'Buffers', 'Cached')):
                field, amount, unit = line.split()
                amount = int(amount)
                if unit != 'kB':
                    raise ValueError(
                        'Unknown unit {u!r} in /proc/meminfo'.format(u=unit))
                total += amount
    return total


def gen_change_in_memory():
    f = free_memory()
    diff = 0
    while True:
        yield diff
        f2 = free_memory()
        diff = f - f2
        f = f2
change_in_memory = gen_change_in_memory().next

def report_memory(label):
    logger.warn('{l:<25}: {d:+d}'.format(d=change_in_memory(), l=label))

if __name__ == '__main__':
    report_memory('Initial')
    tfidf_vect, model = train_and_model()
    report_memory('After train_and_model')
    large_object = (tfidf_vect, model)
    procs = [mp.Process(target=predict) for i in range(mp.cpu_count())]
    report_memory('After Process')
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    report_memory('After p.join')

      

        
        
        
      

    

This gives:

21:45:01: [ MainProcess] Initial                  : +0
21:45:01: [ MainProcess] Before fit_transform     : +3224
21:45:12: [ MainProcess] After model.fit          : +153572
21:45:12: [ MainProcess] After train_and_model    : -3100
21:45:12: [ MainProcess] After Process            : +0
21:45:12: [   Process-1] done                     : +2232
21:45:12: [   Process-2] done                     : +2976
21:45:12: [   Process-3] done                     : +3596
21:45:12: [   Process-4] done                     : +3224
21:45:12: [ MainProcess] After p.join             : -372

      

        
        
        
      

    

Reported number of changes in KiB free memory (including cached and buffers). So, for example, the change in free memory between "Initial" and "After train_and_model" was about 150 MB. Thus, it large_object

requires 150MB.

Then, after the 4 subprocesses finished, a much smaller amount of memory - just 12MB - was destroyed. The memory consumed can be caused by the creation of the subprocess plus the memory used by transform

and predict

.

So it large_object

doesn't seem to be copied, as if we were to see an increase of around 150MB in consumed memory.

A comment on your run through 20 newsgroups :

Changes to free memory are listed below:

In 20 newsgroups:

| Initial               |       0 |
| After train_and_model |  626804 | <-- Large object requires 627M
| After Process         |       0 |
| After p.start         |   33016 |
| done                  |  200384 | 
| done                  |  -55552 |
| done                  |   -8804 |
| done                  |  -12152 |
| After p.join          | -156240 |

      

        
        
        
      

    

So it looks like it needs 627MB for a LOB instance. I don't know why the extra 200+ MB was used up after reaching the first one done

.

Using load_large_object:

| Initial                  |       0 |
| After loading the object |   33976 |
| After Process            |       0 |
| After p.start            |   21112 |
| done                     |  185256 |
| done                     |     744 |
| done                     | -102176 |
| done                     |     496 |
| After p.join             | -103912 |

      

        
        
        
      

    

Apparently the largest object only needs 34 MB, the rest of the memory, 627-34 = 593 MB, should have been consumed by the methods fit_transform

and fit

called in train_and_model

.

Using one process:

| Initial                  |     0 |
| After loading the object | 34224 |
| done                     | 24552 |
| After Process            |     0 |

      

        
        
        
      

    

This is believable.

So the data you've accumulated seems to support the claim that the LOB itself is not copied by every subprocess. But a new mystery arises: why there is a huge memory consumption between "After p.start" and the first "done". I do not know the answer to this question.

You can try to place calls report_memory

around

vectorized_essay = large_object[0].transform(essay)

      

        
        
        
      

    

and

large_object[1].predict(vectorized_essay)

      

        
        
        
      

    

to see where the extra memory is being consumed. I guess one of these scikit-learn methods is to allocate this (relatively) huge amount of memory.