Clever Geek Handbook

Hadoop 1.0.1 support on Amazon EMR

I have developed some MR jobs using java and hadoop 1.0.1. However, EMR only supports up to Hadoop 0.20. Is it possible to run Hadoop 1.0.1 jobs in EMR, or do I need to downgrade my library stack to match the EMO version of howop?

+3


source to share


1 answer


Depends on whether you are using any specific 1.0.1 classes or not. The core Mapper and Reducer classes (both new and old API types) did not change between 0.20 and 1.0.1.

You can try changing your hadoop dependency to 0.20.2 and re-creating your MR job - if there are no compilation errors, you're pretty close (0.20 to 1.0.1 errors can be fixed, but I imagine you'll be fine).



If you find that your work is not compiled and it belongs to some I / O formats that are not available in 0.20 (like some of the multiple I / O), you might want to check the Hadoop source for 1.0.1 (or even source Cloudera 0.20.2) to see if you can "pass" the missing formats and add them to your work.

Be sure to re-post compilation errors in your original questions so people can comment on potential work.

+3


source






More articles:


All Articles