Java application is recklessly consuming memory
For a project I am working on, I was tasked with creating a way to convert an image to a non-crisis hash so that it can be easily compared to similar images, however I ran into an issue where the JVMs start recklessly consuming memory despite Java Monitoring and The Management Console does not report any increase in memory consumption.
When I first launched the application, the Task Manager will report these values:
However, after about 30 seconds, these values will double or triple.
I used JMMC to dump the process, but it only reported 1.3MB usage:
The weirdest part for me is that the app does an operation that lasts about 15 seconds, then it waits for 100 seconds (debug) and it is for 100 seconds of the sleeping thread that the memory used is doubled.
Here are my two classes:
ImageHashGenerator.java
package com.arkazex.srcbot;
import java.awt.Color;
import java.awt.Image;
import java.awt.image.BufferedImage;
public class ImageHashGenerator {
public static byte[] generateHash(Image image, int resolution) {
//Resize the image
Image rscaled = image.getScaledInstance(resolution, resolution, Image.SCALE_SMOOTH);
//Convert the scaled image into a buffered image
BufferedImage scaled = convert(rscaled);
//Create the hash array
byte[] hash = new byte[resolution*resolution*3];
//Variables
Color color;
int index = 0;
//Generate the hash
for(int x = 0; x < resolution; x++) {
for(int y = 0; y < resolution; y++) {
//Get the color
color = new Color(scaled.getRGB(x, y));
//Save the colors
hash[index++] = (byte) color.getRed();
hash[index++] = (byte) color.getGreen();
hash[index++] = (byte) color.getBlue();
}
}
//Return the generated hash
return hash;
}
//Convert Image to BufferedImage
private static BufferedImage convert(Image img) {
//Create a new bufferedImage
BufferedImage image = new BufferedImage(img.getWidth(null), img.getHeight(null), BufferedImage.TYPE_3BYTE_BGR);
//Get the graphics
image.getGraphics().drawImage(img, 0, 0, null);
//Return the image
return image;
}
}
Test.java
package com.arkazex.srcbot;
import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
public class Test {
public static void main(String[] args) throws IOException {
//Create a hash
byte[] hash = ImageHashGenerator.generateHash(ImageIO.read(new File("img1.JPG")), 8); //Memory grows to around 150MB here
System.out.println(new String(hash));
try{ Thread.sleep(100000); } catch(Exception e) {} //Memory grows to around 300MB here
}
}
EDIT: The program stopped growing to 300MB after a few seconds for no apparent reason. I didn't change anything in the code, it just stopped doing it.
source to share
I think what you are missing is that some image classes use non-heap memory. It is (presumably) invisible to JMMC because it is only told about cumulus usage. OS level memory monitoring sees this ... because it looks at the total resource consumption of the JVM running your application.
The problem is that non-heap memory blocks are only recovered when the corresponding image objects on the heap are complete. This only happens when they collect trash.
The program stopped growing to 300MB after a few seconds for no apparent reason. I didn't change anything in the code, it just stopped doing it.
I expect the JVM decided it was time to do a full GC (or something like that) and that caused it to free up a lot of space in the non-heap memory pool. This meant that the JVM no longer needed to keep growing the pool.
(I'm deliberately vague because I don't actually know how non-heap memory allocation works under the covers in a modern JVM, but if you want to investigate, the JVM source code can be downloaded ...)
source to share
See explanation in /** comments */
public class Test {
public static void main(String[] args) throws IOException {
//Create a hash
/** Here it allocates (3 * resolution^2 )bytes of memory to a byte array */
byte[] hash = ImageHashGenerator.generateHash(ImageIO.read(new File("img1.JPG")), 8); //Memory grows to around 150MB here
/** And here it again allocates the same memory to a String
Why print a String of 150 million chars? */
System.out.println(new String(hash));
try{ Thread.sleep(100000); } catch(Exception e) {} //Memory grows to around 300MB here
}
}
source to share