Clever Geek Handbook

How to read Mahout generated files in Pig

I am wondering how can I read the generated Mahout files in Pig? I'm guessing there might be a UDF, but I can't find it yet.

+3


source to share


1 answer


I ended up using elephant bird (v2.2.3) like this:



register '/usr/share/dse/mahout/mahout-core-0.6-job.jar';
register './elephant-bird-2.2.3.jar';

%declare SEQFILE_LOADER   'com.twitter.elephantbird.pig.load.SequenceFileLoader';
%declare LONG_CONVERTER   'com.twitter.elephantbird.pig.util.LongWritableConverter';
%declare INT_CONVERTER    'com.twitter.elephantbird.pig.util.IntWritableConverter';
%declare VECTOR_CONVERTER 'com.twitter.elephantbird.pig.mahout.VectorWritableConverter';
%declare TEXT_CONVERTER   'com.twitter.elephantbird.pig.util.TextConverter';

....

sets = LOAD '$INPUT_SETS' USING $SEQFILE_LOADER ( '-c $INT_CONVERTER', '-c $VECTOR_CONVERTER') AS (thing_id:int, recommendations:chararray);

...
+1


source






More articles:


All Articles