How to parse google protocol buffers that are in direct memory without allocating a byte array in Java?
I am trying to parse a ByteBuf that is not on the JVM heap into a google protocol buffer object. This is actually a direct memory buffer that Netty passed to me.
This is what I am currently doing:
ByteBuf buf = ...;
ByteBufInputStream stream = new ByteBufInputStream(buf);
Message msg = MyPbMessage.getDefaultInstance().getParserForType().parseFrom(stream);
It might work. However, I found that this type of parsing introduces a new byte array per message and causes a lot of GC.
So, is there a way to avoid this when creating heap array arrays? that is, parse the Google Protocol bytes directly from on-board memory.
source to share
You can do it the way the Guava guys do , store a small buffer (1024 bytes) in the ThreadLocal, use that if that's enough and never put a larger buffer in the TL.
This will work fine as long as most requests can be granted to them. If the mid / mid size is too big, you can go for a soft / weak link, however, without any real testing, it's hard to tell if it helps.
You could combine the approaches, i.e. use a small reference small buffer and a loosely coupled large buffer in TL. You can combine your buffers, you could ...
... but note that everything has a dark side. Lost memory, long buffer life leading to their advancement into older generations where garbage collection is much more expensive.
source to share