High CPU in IIS
I am developing a POS application that has a local database on each POS machine and communicates with the server using WCF hosted in IIS. The app has been deployed by several clients for over a year now.
About a week ago, we started getting reports from one of our clients that the server that hosts IIS is running very slowly. When I checked the issue, I saw the application pool with my tech rocket almost 100% CPU on a server with 8 CPUs.
I checked the Activity Monitor and SQL Network Stream and they showed no significant overload beyond what we usually see.
When inspecting threads in Process Explorer, I saw many threads calling CreateApplicationContext multiple times. I tried to install .Net 2.0 SP1, according to some posts I found on the net, but that didn't solve the problem and replaced the CLRCreateManagedInstance function calls.
I'm going to grab a dump using adplus and windbg of IIS processes and try to figure out what is wrong.
Has anyone come across something like this or have an idea which directory I should check out?
ps The same version of the app is deployed to another client and it works just fine there. I also tried rolling back versions (even very old versions) and it still behaves exactly the same.
Edit: well, the problem is solved, it turns out that I had a SQL query that did not limit the result set, and when the client went through a certain number of rows, it started to take care of the server. It took me two days to find it due to all the ambient noise in the logs, but I waited for the night and took a dump that showed me the query right away.
source to share
This usually has nothing to do with hardware, and it all depends on how IIS is configured, coupled with a few long running requests (100+ milliseconds).
In the application pool settings, set your web garden settings to about 20 or more.
Setting up a web garden is pretty much the number of threads available to handle requests for your application. If it is set to 1, then one request can block the processing of other requests until it completes.
I have an application that processes about 3.5 million requests per day. When the web garden was set to 1, the web server processor remained at 100% and had a lot of requests. When I raised it to 50, the web server CPU dropped to just under 2% and there were no requests.
source to share
We had the same problem. The CPU utilization of some IIS application pool processes was so high that the CPU utilization on the web server was close to 100%.
We first used DebugDiag and ProcMon to narrow down the problem. See here: http://www.iis.net/learn/troubleshoot/performance-issues/troubleshoot-high-cpu-in-an-iis-7x-application-pool
We found many "This thread is waiting in WaitOne messages" messages in DebugDiag analysis. This indicated that some pending requests were waiting for each other. So we went looking for common resources. The only thing we could really find was the database. So we realized that even though the web server has 100% CPU usage, the real problem must be the database server.
My college researched the matter further. He did the following:
1.) Configure SQL Server parallelism
Using ProcMon on the database server, he found that there were too many locks and latches used by SQL Server. Take a look here: http://blog.sqlauthority.com/2011/02/06/sql-server-cxpacket-parallelism-usual-solution-wait-type-day-6-of-28/
It set the number of processors in use per request to 4. The default was 0 (which, in my opinion, is equal to the number of available processors - 24 in our case). You can set this with SQL Management Studio by right clicking on the server node itself and selecting properties.
This has had a dramatic effect, reducing valves and speeding up requests. Our guess is that SQL Server has been overusing parallelization of queries, resulting in too much synchronization on completion.
2.) Creating new and missing indexes in the database
We used a third party forum software on our site which turned out to be using no indexes at all in its database. My college used the following knowledge here: http://www.mssqltips.com/sqlservertip/1634/using-sql-server-dmvs-to-identify-missing-indexes/ to create some new indexes.
Now it looks like the case will be resolved.
source to share
Coming from pure gut and making a complete guess, it looks like some kind of exception might be happening, the exception gets caught in the global exception handler in global.asax, and the exception handler also causes the exception to be thrown and dumped in the process. A virus scanning utility that blocks some files may also be invoked. Although I might be WAY.
source to share