WCF method - proxy.Open () intermittent timeout

Problem:
Client applications periodically (once a week) get timeouts during proxy.Open()

and only on this call to a specific method. A client that timed out can call the WCF service in a couple of seconds without issue.

Configuration:
All clients running Windows 7 64-bit, .NET 4.5.1, Microsoft Security Essentials
Server running Windows Server 2012R2, .NET 4.5.1, no antivirus installed WCF service runs as a standard Windows service:

[ServiceContract(SessionMode = SessionMode.Required)]  
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]

      

        
        
        
      

    

Total number of clients = 30. No more than 3 access the WCF service simulataniously. The client talks to the service using different methods (synchronizing different tables), but the session lasts no more than 20 seconds. Each message sent is guaranteed to be less than 2MB.

proxy

- this System.ServiceModel.ClientBase

which was created with Visual Studio - Add Service Reference.

app.config
server side:

<system.serviceModel>
   <services>
      <service behaviorConfiguration="secureBehavior" name="MyService.SynchronizationService">
         <endpoint address="net.tcp://localhost:8999/MyService/SynchronizationService" binding="netTcpBinding" bindingConfiguration="secureNetTcp" contract="MyService.ISynchronizationService">
            <identity>
               <certificateReference storeName="Root" storeLocation="LocalMachine" x509FindType="FindByThumbprint" findValue="..." />
            </identity>
         </endpoint>
      </service>
   </services>

   <bindings>
      <netTcpBinding>
         <binding name="secureNetTcp" openTimeout="00:00:25" closeTimeout="00:00:19" receiveTimeout="00:10:10" sendTimeout="00:01:00" 
             listenBacklog="100" maxConnections="100" maxBufferPoolSize="20971520" maxBufferSize="2097152" maxReceivedMessageSize="2097152">
             <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="true"/>
             <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxDepth="2147483647" maxNameTableCharCount="2147483647" maxStringContentLength="2147483647" />
             <security mode="TransportWithMessageCredential">
                <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                <message clientCredentialType="Windows" algorithmSuite="Basic256Sha256" />
             </security>
         </binding>
       </netTcpBinding>
   </bindings>

   <behaviors>
      <serviceBehaviors>
         <behavior name="secureBehavior">
             <serviceMetadata httpGetEnabled="false" httpsGetEnabled="false" />
             <serviceCredentials>
                 <serviceCertificate findValue="..." storeLocation="LocalMachine" storeName="Root" x509FindType="FindByThumbprint" />
             </serviceCredentials>
             <serviceThrottling maxConcurrentCalls="100" maxConcurrentSessions="100" maxConcurrentInstances="100" />
         </behavior>
      </serviceBehaviors>
   </behaviors>
</system.serviceModel>

      

        
        
        
      

    

client side:

<system.serviceModel>
   <bindings>
      <netTcpBinding>
          <binding name="NetTcpBinding_ISynchronizationService" openTimeout="00:00:25" closeTimeout="00:00:19" receiveTimeout="00:01:00" sendTimeout="00:04:00"
             listenBacklog="100" maxConnections="100" maxBufferPoolSize="20971520" maxBufferSize="2097152" maxReceivedMessageSize="2097152">
             <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="true"/>
             <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxDepth="2147483647" maxNameTableCharCount="2147483647" maxStringContentLength="2147483647" />
             <security mode="TransportWithMessageCredential">
                <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign"/>
                <message clientCredentialType="Windows" algorithmSuite="Basic256Sha256"/>
             </security>
          </binding>
      </netTcpBinding>
   </bindings>
   <client>
      <endpoint address="net.tcp://bla.foo.com:8999/MyService/SynchronizationService" binding="netTcpBinding" 
         bindingConfiguration="NetTcpBinding_ISynchronizationService" contract="MyService.ISynchronizationService" name="NetTcpBinding_ISynchronizationService" />
   </client>
</system.serviceModel>

      

        
        
        
      

    

Client side error messages:
Throwing two types of error messages:
1)

System.TimeoutException: Client is unable to finish the security negotiation within the configured timeout (00:00:00).  The current negotiation leg is 1 (00:00:00).   
---> System.TimeoutException: The open operation did not complete within the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout. 
---> System.TimeoutException: Open timed out after 00:00:00 while establishing a transport session to net.tcp://bla.foo.com:8999/MyService/SynchronizationService. The time allotted to this operation may have been a portion of a longer timeout. 
---> System.TimeoutException: Connecting to via net.tcp://bla.foo.com:8999/MyService/SynchronizationService timed out after 00:00:00. Connection attempts were made to 0 of 1 available addresses (). Check the RemoteAddress of your channel and verify that the DNS records for this endpoint correspond to valid IP Addresses. The time allotted to this operation may have been a portion of a longer timeout.

      

        
        
        
      

    

2)

System.TimeoutException: The open operation did not complete within the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout. 
---> System.TimeoutException: Open timed out after 00:00:00 while establishing a transport session to net.tcp://bla.foo.com:8999/MyService/SynchronizationService. The time allotted to this operation may have been a portion of a longer timeout. 
---> System.TimeoutException: Connecting to via net.tcp://bla.foo.com:8999/MyService/SynchronizationService timed out after 00:00:00. Connection attempts were made to 0 of 1 available addresses (). Check the RemoteAddress of your channel and verify that the DNS records for this endpoint correspond to valid IP Addresses. The time allotted to this operation may have been a portion of a longer timeout.

      

        
        
        
      

    

Debugging steps:
At first I thought it was a client side issue. Perhaps the internet connection did not appear after waking up from sleep or something. So I added checks to the client code to check the network connectivity, ping www.intel.com and www.amd.com, wait 10 seconds before calling the WCF service, etc. But it did not help.

Then I added client code validation to actually open the socket for the WCF service manually. A blank dot indicates that a connection exists. I also verified that I am not using the using

c template proxy

, only try { } finally { }

with the correct calls Abort()

and Close()

. However, clients sometimes time out even after checking the connection (I do close the check socket before calling the WCF service).

Both the Event Viewer client and server are cleared of any errors or warnings during the timeout. There is no load on the server and client, pretty much idle all the time.

Since the timeout only happens during a method call Open()

and even on a client that is on the same local network as the server, I think thist must be a server side issue.

Tracing:
My last step was to enable server side tracking and see what happens. Unfortunately, this doesn't provide much insight. At the time of the timeout, I see that the socket check was successful, but then there are no more messages or activity.

You might think that this is a clear sign of no internet connection or packages. But why then the problem only occurs during method invocation Open()

and even inside the local network? It also might not be a hardware issue as the WCF service was being used on a different Server 2008R2 server with the same issues.

Help:
Are there WCF gurus out there who can help me track down this issue?

Could this be some internal WCF error and not handle sockets correctly?

For example, a traceback showed me that even after the client had correctly finished talking to a WCF service - called a method with an attribute [IsTerminating=true]

and called proxy.Close()

- WCF still throws an exception internally. Depending on where you look in the trace log, there are different messages:

1) Activity = receive bytes when connected; Error thrown = System.TimeoutException

-The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.

2) Activity = 0000000000000; Error thrown = System.Net.Sockets.SocketException

-An existing connection was forcibly closed by the remote host.

3) Activity = 0000000000000; Error thrown = System.Net.Sockets.SocketException

-The I/O operation has been aborted because of either a thread exit or an application request.

4) Activity = 0000000000000; Thrown error = System.ServiceModel.CommunicationException

-The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:02:00'.

Bonus points for those who can help get rid of these internal errors (this happens after the client has finished talking to the server). Currently I think I need to use reflection to enter the WCF bowl, find the socket and close it manually ...

Edit:
Bonus points to me :) I found how to fix internal SocketExceptions on the server side, every time the client closes - How to properly close the client proxy (the existing connection was forcibly closed by the remote host)?

Now I need to check if this is affecting the timeouts Open()

. I probably won't be able to release a new version of the client until the new year ... In the meantime, if anyone has any ideas on timeouts Open()

, please share.