Can't run python MRJob locally
If I understand MRJob correctly, you can simulate a multiprocessing hasoop process with MRJob by starting it with
python mrfile.py -r local input.txt
I start windows (no choice now) and when I issue the above command, I get a bunch of jumbo mambo and at the end it tells me:
WindowsError: [Error 2] The system cannot find the file specified
This is a complete mistake. Can anyone please help?
C:\Users\someuser\Documents\Python_projects\something>python MRJob_parser.py -r loc
al test2.txt
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.
20150701.211822.496000
writing wrapper script to c:\users\someuser\appdata\local\temp\MRJob_parser.bw401
45.20150701.211822.496000\setup-wrapper.sh
writing to c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211
822.496000\step-0-mapper_part-00000
> sh -ex setup-wrapper.sh 'c:\Users\someuser\Documents\python_venv\something_Project\
Scripts\python.exe' MRJob_parser.py --step-num=0 --mapper 'c:\users\someuser\appd
ata\local\temp\MRJob_parser.someuser.20150701.211822.496000\input_part-00000' > c
:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211822.496000\s
tep-0-mapper_part-00000
Traceback (most recent call last):
File "MRJob_parser.py", line 18, in <module>
Extractor.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 461, in run
mr_job.execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 479, in execute
super(MRJob, self).execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 151, in execute
self.run_job()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 214, in run_job
runner.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\runner.py", line 464, in run
self._run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 173, in _run
self._invoke_step(step_num, 'mapper')
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 260, in _invoke_step
working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 147, in _run_step
procs_args, output_path, working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 250, in _invoke_processes
cwd=working_dir, env=env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 73, in _chain_procs
proc = Popen(args, **proc_kwargs)
File "C:\Python27\Lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Python27\Lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
C:\Users\someuser\Documents\Python_projects\something>
+3
source to share
1 answer
From https://github.com/Yelp/mrjob :
# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
My guess is that you don't need to pass the local runner parameter and throw some error.
0
source to share