Java and Linux OS freeze randomly for 24 minutes (Linux, arm, Debian, Java 7 ARM)

Background / context:

We run a Java application on one of the CompuLab CoMs:


JVM version: Oracle Java 7 ARM 1.7.0_60

OS link:


The application is not trivial: multiple streams, Ethernet (LAN) access, serial interface, GPRS / UMTS modem, internet access (ppp deamon), GPS, touchscreen, database (SQLite), file system. In other words, make extensive use of OS resources ...

We observe that the Java application (all its threads) and basic OS functionality freeze randomly. I would say it is a Linux kernel bug, but by killing the Java application it recovers and works fine.

This state always takes exactly 24 minutes. Subsequently, he recovers and behaves normally. The average spawn rate is once every 24-30 hours.

When this happens, externally triggered events, such as messages sent to the application via Ethernet or serial, are buffered (probably by the OS) and all are processed immediately upon recovery.

When I set up an SSH connection to the device in advance, after that the connection is either blocked (the whole command is buffered and processed after it is restored - 24 months), or it works like this:

  • Basic OS utilities do not work: for example, "top"
  • jstack -F doesn't work, just hangs and doesn't produce any output
  • killing java application with kill -9 PID released OS and everything starts working fine

While it is in this state, the OS behaves differently each time. Other findings:

  • Basic network utilities do not work (SSH, FTP) - they cannot establish a new connection to the OS from another computer.
  • PING from another machine works until I unplug the Ethernet cable from the device, sometimes PING stops working.
  • Sometimes the OS system time freezes (not always), after 24 minutes it continues to linger for 24 minutes.
  • New USB input devices (mouse, keyboard) cannot be connected during this state (always).

Another strange thing:

The touch screen is used for user interaction (the driver is compiled as a kernel module). And it works even when it hangs. Java application (GUI Swing) can handle events like button click so that I can run some code behind the click handler button. All threads seem to be blocked, but Java Swing can handle some input events, and our application chases them until it needs to interact with already blocked threads or OS (run a bash script on button press) or call a song method. Than it hangs. In other words, the Java application hangs "partially" - it can still process something.

Have already tried:

  • JVM Remote Debugging Tools: Java Mission Control, VisualVM. The connection was established before it hung up. Everything seemed to be OK in terms of thread dump, heap dump, etc. (I can send by email). Even the connection remained and I could see in them the tools that handle the usage reduced to 0% for the JVM.

  • jstack -F (over SSH): doesn't work, just hangs and doesn't produce any output

  • I tried to start the OS without a touchscreen driver and it still happened.

  • I tried to run two parallel Java applications. One of them was very simple - just write log timestamps. And they both hung.

  • I tried to run System.exit (0) in terms of the application click handler. and all threads were hanging, and it didn't work (hung).


Is this a bug in the Linux kernel or a bug in the JVM (its ARM implementation)?

Can Java (JVM) freeze and block basic OS functionality (FTP, SSH, system time, other utilities)?

How can I further diagnose / debug this issue when basic utilities like jstack -F are not working?

Do you have any idea what might be causing this problem and why it always recovers after exactly 24 minutes?

Update 1: 2014-07-10

Finally, I manage to "catch" this strange condition again. Here are my further conclusions.

Based on nos suggestion, I tried running via ssh (installed in an advanced state):

*strace -f -p PID*


Unfortunately the bash script command also hung (same behavior as in jstack).

As for the user limits (ulimit) and OS resources, below I report the numbers taken immediately after restoring the system from the last hanging one . In this state, it worked for 24 hours and I can confirm that these numbers remain roughly the same during long periods of operation (there are no random delays during operation). From my point of view, they are ok and the application in no way oversteps any resource or other limit.

Current Java heap

Usage: 18 MB, Free: 12 MB, Total: 30 MB, Maximum: 230 MB

Java heap

 root@cm-debian:~# /usr/lib/jvm/jdk1.7.0_60/bin/jmap -heap 3242
Attaching to process ID 3242, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.60-b09

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 249561088 (238.0MB)
   NewSize          = 1048576 (1.0MB)
   MaxNewSize       = 4294836224 (4095.875MB)
   OldSize          = 4194304 (4.0MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 12582912 (12.0MB)
   MaxPermSize      = 67108864 (64.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 10092544 (9.625MB)
   used     = 6772088 (6.458366394042969MB)
   free     = 3320456 (3.1666336059570312MB)
   67.09991058745942% used
Eden Space:
   capacity = 9043968 (8.625MB)
   used     = 6620336 (6.3136444091796875MB)
   free     = 2423632 (2.3113555908203125MB)
   73.2016743093297% used
From Space:
   capacity = 1048576 (1.0MB)
   used     = 151752 (0.14472198486328125MB)
   free     = 896824 (0.8552780151367188MB)
   14.472198486328125% used
To Space:
   capacity = 1048576 (1.0MB)
   used     = 0 (0.0MB)
   free     = 1048576 (1.0MB)
   0.0% used
tenured generation:
   capacity = 22134784 (21.109375MB)
   used     = 17650936 (16.83324432373047MB)
   free     = 4483848 (4.276130676269531MB)
   79.7429782915433% used
Perm Generation:
   capacity = 19136512 (18.25MB)
   used     = 19023016 (18.141761779785156MB)
   free     = 113496 (0.10823822021484375MB)
   99.40691386183647% used

9597 interned Strings occupying 729344 bytes.



top - 11:41:29 up 21:59,  2 users,  load average: 1.51, 1.25, 1.22
Tasks:  93 total,   1 running,  92 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.4%us,  8.0%sy,  0.0%ni, 82.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    966780k total,   273080k used,   693700k free,    27216k buffers
Swap:        0k total,        0k used,        0k free,   126352k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                             
 3242 root      20   0  398m  79m  11m S 23.6  8.4 346:16.82 java                                                                                
 3889 root      20   0  2804 1096  848 R  5.5  0.1   0:00.07 top                                                                                 
    1 root      20   0  2124  688  596 S  0.0  0.1   0:02.92 init                                                                                
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.03 kthreadd                                                                            
    3 root      20   0     0    0    0 S  0.0  0.0   0:14.32 ksoftirqd/0                                                                         
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.14 kworker/u:0                                                                         
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                         
    7 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 khelper 


java limits

root@cm-debian:~# java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
    uintx AdaptivePermSizeWeight                    = 20              {product}           
     intx CompilerThreadStackSize                   = 0               {pd product}        
    uintx ErgoHeapSizeLimit                         = 0               {product}           
    uintx HeapSizePerGCThread                       = 67108864        {product}           
    uintx InitialHeapSize                          := 15468480        {product}           
    uintx LargePageHeapSizeThreshold                = 134217728       {product}           
    uintx MaxHeapSize                              := 249561088       {product}           
    uintx MaxPermSize                               = 67108864        {pd product}        
    uintx PermSize                                  = 12582912        {pd product}        
     intx ThreadStackSize                           = 320             {pd product}        
     intx VMThreadStackSize                         = 512             {pd product}        
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) Client VM (build 24.60-b09, mixed mode)


process limits

root@cm-debian:~# cat /proc/3242/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            8192                 8192                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       16382                16382                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us 


system memory information

root@cm-debian:~# cat /proc/meminfo
MemTotal:         966780 kB
MemFree:          694312 kB
Buffers:           27384 kB
Cached:           126364 kB
SwapCached:            0 kB
Active:           140748 kB
Inactive:         107684 kB
Active(anon):      94992 kB
Inactive(anon):     2064 kB
Active(file):      45756 kB
Inactive(file):   105620 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:        524288 kB
HighFree:         301088 kB
LowTotal:         442492 kB
LowFree:          393224 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         94692 kB
Mapped:            21220 kB
Shmem:              2376 kB
Slab:              13268 kB
SReclaimable:       5284 kB
SUnreclaim:         7984 kB
KernelStack:         960 kB
PageTables:          980 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      483388 kB
Committed_AS:     137260 kB
VmallocTotal:     286720 kB
VmallocUsed:        2928 kB
VmallocChunk:     283040 kB

root@cm-debian:~# vmstat -s
       966780 K total memory
       272468 K used memory
       140776 K active memory
       107712 K inactive memory
       694312 K free memory
        27392 K buffer memory
       126404 K swap cache
            0 K total swap
            0 K used swap
            0 K free swap
       726963 non-nice user cpu ticks
            0 nice user cpu ticks
       621187 system cpu ticks
      6371123 idle cpu ticks
         3683 IO-wait cpu ticks
          324 IRQ cpu ticks
         2146 softirq cpu ticks
            0 stolen cpu ticks
       130871 pages paged in
        97520 pages paged out
            0 pages swapped in
            0 pages swapped out
    293822206 interrupts
    494034482 CPU context switches
   1412595732 boot time
         3916 forks


<strong> topics

root@cm-debian:~# cat /proc/sys/kernel/pid_max

root@cm-debian:~# cat /proc/sys/kernel/threads-max

root@cm-debian:~# cat /proc/sys/vm/max_map_count

root@cm-debian:~# ls -l /proc/3242/task/ | wc -l

root@cm-debian:~# ps huH p 3242 | wc -l

root@cm-debian:~# grep -s '^Threads' /proc/[0-9]*/status | awk '{ sum += $2; } END { print sum; }'


open files / file descriptors

root@cm-debian:~# ls -l /proc/3242/fd | wc -l


Update 2: 2014-13-10

This time, I logged all Java thread stacks while the OS was jailbroken (as I said earlier, the touchscreen and its events are still running, so I wrote a stack trace for the log file in terms of the UI button handler).

From my point of view, all threads are in the "correct" state (sleeping, waiting for a UDP datagram, etc.) and obviously the hang is not caused by a Java SW application which would have taken 24 minutes .

10:49:42,293> [INFO ] THREAD stack traces: 

ID: 56, name: Mpg123AudioPlayer_PASSENGER_ctrlLoop
java.lang.Thread.sleep(Native Method)

ID: 11, name: AWT-EventQueue-0
java.awt.EventQueue$ Method)$1.doIntersectionPrivilege($1.doIntersectionPrivilege(
java.awt.EventQueue$ Method)$1.doIntersectionPrivilege(

ID: 34, name: Mpg123AudioPlayer_DRIVER_ctrlLoop
java.lang.Thread.sleep(Native Method)

ID: 26, name: IOTxUdpAccessLoop_IODispatchAccess
java.lang.Thread.sleep(Native Method)$

ID: 29, name: MasterLoop_main
java.lang.Thread.sleep(Native Method)

ID: 27, name: IORxSerialPortAccessPollLoop_IOModemAccess
java.lang.Thread.sleep(Native Method)$

ID: 31, name: UsbUpdateWatchService_ctrlLoop
sun.misc.Unsafe.park(Native Method)

ID: 25, name: IORxUdpAccessLoop_IODispatchAccess Method)$

ID: 2, name: Reference Handler
java.lang.Object.wait(Native Method)

ID: 30, name: VehicleCtrl_ctrlLoop
java.lang.Thread.sleep(Native Method)

ID: 35, name: Mpg123AudioPlayer_INNER_ctrlLoop
java.lang.Thread.sleep(Native Method)

ID: 21, name: IORxSerialPortAccessPollLoop_IOFccAccess
java.lang.Thread.sleep(Native Method)$

ID: 7, name: FileWatchdog
java.lang.Thread.sleep(Native Method)

ID: 8, name: Java2D Disposer
java.lang.Object.wait(Native Method)

ID: 17, name:$Finalizer
java.lang.Object.wait(Native Method)

ID: 10, name: AWT-XAWT
sun.awt.X11.XToolkit.waitForEvents(Native Method)

ID: 32, name: Thread-4
sun.nio.fs.LinuxWatchService.poll(Native Method)

ID: 28, name: IOTxSerialPortAccessPollLoop_IOModemAccess
java.lang.Thread.sleep(Native Method)$

ID: 14, name: DestroyJavaVM

ID: 22, name: IOTxSerialPortAccessPollLoop_IOFccAccess
java.lang.Thread.sleep(Native Method)$

ID: 19, name: TimerQueue
sun.misc.Unsafe.park(Native Method)

ID: 12, name: AWT-Shutdown
java.lang.Object.wait(Native Method)

ID: 23, name: IORxUdpAccessLoop_IOCityScrnAccess Method)$

ID: 3, name: Finalizer
java.lang.Object.wait(Native Method)

ID: 4, name: Signal Dispatcher

ID: 52, name: pool-3-thread-1
sun.misc.Unsafe.park(Native Method)

ID: 24, name: IOTxUdpAccessLoop_IOCityScrnAccess
java.lang.Thread.sleep(Native Method)$

ID: 36, name: RemoteUpdateCtrl_ctrlLoop
java.lang.Thread.sleep(Native Method)

ID: 55, name: Mpg123AudioPlayer_OUTER_ctrlLoop
java.lang.Thread.sleep(Native Method)



source to share

1 answer

This is a problem with the simultaneous use of GPT and local timers.

In the Freescale community, you may see one more questions similar to yours and others from someone with some clarification.

Apply this patch for permission .

From the second post you can go to kernel 3.10.17 from Fresscale or 3.13.3 from

I am currently trying to get a patch to see if a similar problem is resolved.



All Articles