Forbidden permission java.net.socketexception - Server Choke

Question

Forbidden permission java.net.socketexception - Server Choke

Recently we had a problem on our production server, the application suddenly starts failing all outgoing requests (public API / local database calls) for a certain period of time and then the requests start to complete successfully and then after a while they start again have a badluck. Every time I restart Tomcat, the problem disappears for a while and then reappears again, and with each repeated tomcat, the problems start to appear for less time until I restart the entire server to ensure a long period of stability.

I have looked all through stackoverflow and google search results, not one of the suggested solutions or research paths led me to solve the problem.

My server setup: - Windows Server 2008 R2 - Tomcat 7.0.59 - JRE 7

I used "netstat -ano" as suggested in one of the answers on stackoverflow.com - I am missing a link, sorry for that - to check if my application is not working, I looked for CLOSE_WAIT status connections and it shows up so there is no leak. connections with CLOSE_WAIT state do not live long, they are 0 longer, maximum 1 or 2.

I installed the preferIPv4Stack

java option successfully setx _JAVA_OPTIONS -Djava.net.preferIPv4Stack=true

, but it didn't work. I suppose it has nothing to do with this, because my server won't be able to send any successful requests, right? In my case, after restarting Tomcat, the server successfully sends requests in x hours and then starts crashing randomly until all requests have succeeded.

Can someone point me in some direction in my investigation of this issue? What should I be looking for?

Any help is appreciated.

EDIT:

Magazine:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://MYURL.COM": Permission denied: connect; nested exception is java.net.SocketException: Permission denied: connect
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:524) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:472) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:420) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at model.RestTemplateClient.AbstractRestClient.doRequest(AbstractRestClient.java:50) [OutswitchCore.jar:?]
    at clients.KigoRestClient.doRequest(KigoRestClient.java:57) [KigoAdapter.jar:?]
    at controller.KigoController.getProperty(KigoController.java:673) [KigoAdapter.jar:?]
    at controller.KigoController.getProperty(KigoController.java:1) [KigoAdapter.jar:?]
    at KigoAdapter.getProperty(KigoAdapter.java:108) [KigoAdapter.jar:?]
    at com.codefish.pcms.channel.processors.PmsActionExecutor.execute(PmsActionExecutor.java:73) [OutswitchCore.jar:?]
    at com.codefish.hibernate.HibernateThread.run(HibernateThread.java:27) [FishTankJava.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_76]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_76]
    at java.lang.Thread.run(Unknown Source) [?:1.7.0_76]
Caused by: java.net.SocketException: Permission denied: connect
    at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.7.0_76]
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.Socket.connect(Unknown Source) ~[?:1.7.0_76]
    at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at sun.security.ssl.BaseSSLSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.https.HttpsClient.New(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) ~[?:1.7.0_76]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509)

OTHER LOGISTICS:

ERROR org.quartz.core.ErrorLogger - An error occurred while scanning for the next triggers to fire.
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.quartzJob': com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168) ~[quartz-2.2.0.jar:?]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3774) ~[quartz-2.2.0.jar:?]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2759) ~[quartz-2.2.0.jar:?]
    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264) [quartz-2.2.0.jar:?]
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.7.0_76]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
    at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1036) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:338) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2232) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
    at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at org.springframework.jdbc.datasource.SimpleDriverDataSource.getConnectionFromDriver(SimpleDriverDataSource.java:140) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:131) ~[spring-context-support-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.2.0.jar:?]
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.2.0.jar:?]
    ... 3 more
Caused by: java.net.SocketException: Permission denied: connect
    at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.7.0_76]
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.7.0_76]
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
    at java.net.Socket.connect(Unknown Source) ~[?:1.7.0_76]
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:213) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:297) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2232) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
    at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
    at org.springframework.jdbc.datasource.SimpleDriverDataSource.getConnectionFromDriver(SimpleDriverDataSource.java:140) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:131) ~[spring-context-support-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.2.0.jar:?]
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.2.0.jar:?]