Glassfish 4 Grizzly Threads Heavy CPU Usage

I have a Jersey app running Glassfish 4 (4.1 build 13), JDK 1.7 67 update and AWS Linux AMI, and I notice that after a few hours of running it keeps CPU usage and stays even though clients are stopped.

Running "top -H" identifies 2 HTTP-listener-1-core threads with high CPU usage (out of a total of 16). Then I took a stream dump to check these two streams:

"http-listener-1-kernel(3) SelectorRunner" daemon prio=10 tid=0x00007fbc68251000 nid=0xaee runnable [0x00007fbcb55ce000]
    java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000060263ad88> (a sun.nio.ch.Util$2)
        - locked <0x000000060263ad78> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000006025fb068> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
        at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:745)


"http-listener-1-kernel(8) SelectorRunner" daemon prio=10 tid=0x00007fbc6825b800 nid=0xaf3 runnable [0x00007fbcb50c9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x00000006026648b8> (a sun.nio.ch.Util$2)
        - locked <0x00000006026648a8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0000000602664790> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
        at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:745)

      

Interestingly, other threads that do not consume CPU use a different method on sun.nio.ch.SelectorImpl (select instead of selectNow):

"http-listener-1-kernel(7) SelectorRunner" daemon prio=10 tid=0x00007fbc68259800 nid=0xaf2 runnable [0x00007fbcb51ca000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x0000000602665f20> (a sun.nio.ch.Util$2)
        - locked <0x0000000602665f10> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0000000602665df8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:112)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:745)

      

I then brought clients back and left them for a few hours before stopping them again. Of course, "top -H" indicates that the CPU usage has increased again. Another thread dump shows that another thread has switched to "selectNow" and is now taking more CPU time (along with another 2):

"http-listener-1-kernel(6) SelectorRunner" daemon prio=10 tid=0x00007fbc68257800 nid=0xaf1 runnable [0x00007fbcb52cb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x00000006026675c0> (a sun.nio.ch.Util$2)
        - locked <0x00000006026675b0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0000000602667498> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
        at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:745)

      

It looks like something is causing the threads to call the selectNow method on sun.nio.ch.SelectorImpl, and once they get there, the CPU load goes up a lot and doesn't decrease unless the server is restarted.

Is this a known issue? Could this be caused by my code in some way?

Thanks for the help!

+2


source to share


1 answer


Maybe this problem:

https://java.net/jira/browse/GLASSFISH-21211



There is a grizzly patch in the problem, but I haven't tested it yet.

0


source







All Articles