Pig + Hbase Aggregate Column Values
I am trying to read in an Hbase atomic increment column in Pig and access it as a long value.
However, the column value uses Hbases Hex as structure: \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x01
Does anyone know of a method to convert this to Pig to become equivalent to the get_counter value:
I posted a solution using UDF:
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.data.DataByteArray;
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.logicalLayer.schema.Schema;
public class ConvertToLong extends EvalFunc<Long> {
@Override
public Long exec(Tuple input) throws IOException {
if (input == null || input.size() == 0) {
return null;
}
try {
long value = 0;
DataByteArray dba = (DataByteArray)input.get(0);
System.out.println( dba.toString() );
byte[] ba = dba.get();
for (int i = 0; i < ba.length; i++)
{
value = (value << 8) + (ba[i] & 0xff);
}
return value;
//return value;
} catch (ExecException e) {
log.warn("Error reading input: " + e.getMessage());
return 3L;
} catch( Exception e ){
log.warn("Error:" + e.getMessage() );
return 2L;
}
}
@Override
public Schema outputSchema(Schema input) {
return new Schema(new Schema.FieldSchema(null, DataType.LONG));
}
}
source to share
You don't need UDFs to load long integers from HBase. You can rely on the parameter -caster=HBaseBinaryConverter
to HBaseStorage.
Example: I have a table called counters, the value is stored in the val: val column (using an increment function that stores data 8 bytes long). To list all counters in a PIG:
counters = LOAD 'hbase://counters' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('val:val', '-caster=HBaseBinaryConverter -loadKey') AS (key:chararray, val:long);
DUMP counters
source to share
I've never used HBaseStorage so I'm not sure, but you can try the following approaches:
Try to read it as long:
data = LOAD 'your/path' USING HBaseStorage(...) AS (x:long);
If that doesn't work, try this:
data = LOAD 'your/path' USING HBaseStorage(...)
data = FOREACH data GENERATE (long) $1 AS x;
Otherwise, you can always write a UDF that will do the conversion:
data = LOAD 'your/path' USING HBaseStorage(...);
data = FOREACH data GENERATE ConvertToLong($1) as x;
source to share