JDBC application hangs for bulk insert when batch size is high

I am trying to bulk insert from a linux server (like X server) using a simple JDBC application.

When I try the app with batch size 5 it works fine.

But when I try it with batch size 500 it hangs. I am not getting any errors or exceptions. When I query the database I see 500 rows inserted into it, but the java app just hangs. The java app won't quit.

The application just hangs when the following method is called

try{
// register driver

// establish connection

// create statement
// add insert commands from inputFile to the batch
statemnt.executeBatch();  //---------------application hangs here
}
//catch block followed by finally block to close resources

      

I tried the same application on another linux server, I was able to successfully execute the application for batch size 500, the application worked fine and exited fine.

The application only hangs on the X server and only for large batch sizes.

Then I checked the strace of the hung process on the X server. I found that it loops through the following system calls

$ strace -f -p [hung-java-process-id]

[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 882169546}) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 882258933}) = 0
[pid 104273] clock_gettime(CLOCK_REALTIME, {1417787058, 859159155}) = 0
[pid 104273] futex(0x7ffe78013754, FUTEX_WAIT_PRIVATE, 1, {0, 49972845}) = -1 ETIMEDOUT (Connection timed out)
[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 932601035}) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 932677888}) = 0
[pid 104273] clock_gettime(CLOCK_REALTIME, {1417787058, 909571241}) = 0
[pid 104273] futex(0x7ffe78013754, FUTEX_WAIT_PRIVATE, 1, {0, 49977759}) = -1 ETIMEDOUT (Connection timed out)
[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0

      

I checked the list of open files opened by the java hover process and got the following output which showed the db connection is still open.

$ lsof -p [hung-java-process-id] | grep TCP

COMMAND   PID       USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java    104273 userid   14u  IPv6 1674188639      0t0  TCP serverX:55744->dbServer:blackjack (ESTABLISHED)

      

I also tried setting the AutoCommit property of the connection to false and then calling commit after the executeBatch method. But still I faced the same problem.

I also tried splitting the package with a smaller batch size. The app works fine up to batch size 29, but when I increase the batch size to 30 it just hangs without any errors or exceptions.

I could not find the cause of the problem. If strace shows the cause of the problem, then please let me know what exactly it means and how to solve it? Any help would be greatly appreciated. Thanks in advance.

Please note that I am using Java 7 .

I suppose the problem might be caused by some configuration on the Linux machine (X server), as the same application works fine on other machines for a large batch size. Any insight related to machine configuration would be very helpful to me.

+3


source to share


1 answer


what database? I've seen some mentions of problems with postgresql drivers, I don't know if other vendors have similar problems.



See GitHub and PostGreSql

0


source







All Articles