Java 8 Metaspace: OutOfMemoryError: Working with Reflection Inflation
There are many links and open Q&A s on the Internet, still I am missing information.
First things first
Question:
java.lang.OutOfMemoryError: Metaspace
JVM:
java version: "1.8.0_131"
vm: Java HotSpot(TM) 64-Bit Server
vm args: -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=128m
framework:
spring, hibernation, wicket, pier
Suspect 1:
Over the period of use, metaspace gradually increases and the following reflection classes are proportionally loaded into the metaspace [observed with jmap -histo cron jobs]
sun.reflect.GeneratedConstructorAccessor1299
sun.reflect.GeneratedMethodAccessor6929
sun.reflect.GeneratedSerializationConstructorAccessor4220
Possible Solution:
and. Since we use libraries that deal heavily with analytical stuff, we think that 128m is not enough to store all generated XX classes in metaspace. Thus, we plan to double the metaspace limit. -XX: MaxMetaspaceSize = 256m
b. We are not thinking about installing the following
-D sun.reflect.noInflation
-D sun.reflect.inflationThreshold
Suspect 2:
Full GC runs continuously even before it reaches / occupies the fully configured metaspace (128m) and the application becomes unresponsive / slow / sometime OOM since jvm only uses FGC.
[Full GC (Metadata GC Threshold) [PSYoungGen: 224K-> 0K (698368K)] [ParOldGen: 52910K-> 52933K (1398272K)] 53134K-> 52933K (2096640K), [Metaspace: 92733K-> 92733K] (1163264K) 0.1964143 secs] [Time: user = 0.59 sys = 0.00, real = 0.19 s]
...
Metaspace used 147414K, power 155731K, committed 159616K, reserved 1187840K
space class 17242K, capacity 19252K, perfect 20352K, reserved 1048576K
Possible Solution:
-XX: CompressedClassSpaceSize is not explicitly mentioned when starting the vm, which can lead to over-reservation of the address space, resulting in fooled perfect sapce and therefore Full GC. So -XX: CompressedClassSpaceSize = 256m will help vm in proper memory scheduling and reservations.
Questions:
- Suspect 1: Has anyone encountered a similar issue and gotten any fix?
- Suspect 2: Does the -XX: CompressedClassSpaceSize option actually affect metadata scheduling / reservation and does it affect GC? Any pointers?
- Any other suspects? recommendations?
source to share
After spending so much time, it turns out that there is no class leak and we became a victim
package sun.reflect;
/**
The master factory for all reflective objects, both those in
java.lang.reflect (Fields, Methods, Constructors) as well as their
delegates (FieldAccessors, MethodAccessors, ConstructorAccessors).
**/
public class ReflectionFactory {
// "Inflation" mechanism. Loading bytecodes to implement Method.invoke() and Constructor.newInstance() currently costs
// 3-4x more than an invocation via native code for the first invocation (though subsequent invocations have been benchmarked
// to be over 20x faster). Unfortunately this cost increases startup time for certain applications that use reflection
// intensively (but only once per class) to bootstrap themselves. To avoid this penalty we reuse the existing JVM entry points
// for the first few invocations of Methods and Constructors and then switch to the bytecode-based implementations.
I personally confirmed this, every 16th reflexiton generation time this method was used to improve response time.
Finally, we increase the metaspace and everything works fine.
source to share