Uncorrected internal error. Java explicit space when using EXPLAIN in pig
When I try to use the following statement
explain -brief A;
I got the following error:
2013-02-06 19:18:34,250 [Low Memory Detector] INFO org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call - Collection threshold init = 526123008(513792K) used = 349984528(341781K) committed = 699072512(682688K) max = 699072512(682688K)
2013-02-06 19:18:34,942 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Java heap space
I tried to increase the size of the heap space by adding the option
-Dmapred.child.java.opts=-Xmx4096m
but it didn't work. Any idea?
Thank. Dan
source to share
-
Increase the PIG heap area with
export PIG_HEAPSIZE=4096
ALTHOUGH, if you are using PIG in a virtual machine, make sure you allocate more than 4GB (or whatever you want to install) to the virtual machine . Let's say if you allocated X
Gb for a virtual machine, then increasing the heap size more than X
Gb will do nothing.
- ALSO, it might be a problem with the PIG edition you are using .
Try using the older version of the pig. I was getting the same reproducible problem with PIG-0.14.0, but not with PIG-0.12.1
source to share