Async rewrites sync code 20 times slower in loops

I try not to panic here, so please bear with me!: S

I've spent a significant amount of time rewriting a large chunk of code from synchronization (i.e. blocking threads) to async (i.e. C # 6+). This code runs inside an ASP.NET application and covers everything from low-level ADO.NET database access to a higher-level workflow pattern and finally a custom HTTP async handler to access the public API - so to speak. The main goal of the rewrite was not optimization, but unraveling, general cleaning and converting the code to something that resembles a modern and thoughtful design. Naturally, it was assumed that the gain was implicitly assumed.

Overall, this is great, and I am very pleased with the overall quality of the new code as well as the improved scalability shown so far in the last couple of weeks of actual testing. CPU and memory load on the server has dropped dramatically!

So what is the problem, you may ask?

Well, I was recently tasked with optimizing a simple data import that still uses the old sync code. Naturally it didn't take me long before I tried to change it to a new asynchronous code database to see what happened.

All things considered, the import code is pretty simple. It's basically a loop that reads items from a list that was previously read into memory, adds each one separately to a unit of work, and saves it to a SQL database using an INSERT statement. After the loop ends, a unit of work is executed, which makes the changes permanent (i.e., the DB transaction is also committed).

The problem is that the new code takes about 20 times longer than the old one when the expectations were completely opposite! I have checked and double checked and there is no obvious overhead in the new code that would guarantee this slowness.

Specifically: the old code is able to consistently import 1100 items / sec, while the new one manages 40 items / sec. AT BEST (on average, this is even less, because the speed drops a little over time)! If I run the same test over a VPN, so the cost of the network outweighs everything else, the bandwidth is somewhere around 25 pps for sync and 20 for async.

I read about several cases here on SO that report 10-20% slowdown when switching from synchronous to asynchronous in situations like this, and I was willing to handle it for hard loops like mine. But a 20x fine in a non-network scenario ?! This is completely unacceptable!

What's my best course of action here? How can I solve this unexpected problem?

UPDATE

I am running the import under the profiler as suggested.

I'm not sure what to do with the results. It would seem that the process spends over 80% of its time just ... waiting. See for yourself:

The 14% spent inside a custom HTTP handler matches IDataReader.Read

, which is a tiny remnant of the old sync API. This is still subject to optimization and will likely be scaled back in the near future. Regardless, it overshadows the cost WaitAny

, which definitely doesn't exist in the entire synchronous version!

Curiously, there are no direct calls from my code to in the report WaitAny

, which makes me think this is probably part of the async / await infrastructure. Am I wrong in this conclusion? I kind of hope I will!

My concern is that I may be reading this all wrong. I know that the cost of async is much more complex than the single threaded cost. After all, it WaitAny

may be nothing more than the equivalent of the "Sytem Idle Process" in Windows - an artificial representation of the processor infrastructure that reflects the percentage of CPU resources that is free.

Can anyone shed some light on me please?