MRJob: - display intermediate values on the map decrease

Question

MRJob: - display intermediate values on the map decrease

How can I display intermediate values (i.e. print a variable or list) on the terminal while running the mapreduce program using the MRJob python library?

+3

python mapreduce hadoop mrjob

Read Q Jan 24 13 at 12:42

source to share

1 answer

pacodelumberg · Accepted Answer · 2013-02-06T10:50:58+0000

You can output the results to standard error using sys.stderr.write (). Here's an example:

from mrjob.job import MRJob
import sys
class MRWordCounter(MRJob):
    def mapper(self, key, line):
        sys.stderr.write("MAPPER INPUT: ({0},{1})\n".format(key,line))
        for word in line.split():
            yield word, 1

    def reducer(self, word, occurrences):
        occurencesList= list(occurrences)
        sys.stderr.write("REDUCER INPUT: ({0},{1})\n".format(word,occurencesList))
        yield word, sum(occurencesList)

if __name__ == '__main__':
    MRWordCounter.run()

MRJob: - display intermediate values ​​on the map decrease

More articles:

MRJob: - display intermediate values on the map decrease