Clever Geek Handbook

The Mystery of EF's Disappearing Call

Today I received an emergency call from users of our ASP.NET production system. Some users (not all) were unable to enter certain information. The user posted the data and then the system froze; the call never returned.

We tried to reproduce the problem in the QA system (which has a new production data recovery) and could not. Then I went from my dev environment and connected directly to the production DB, disguising myself as one of the affected users. Again, no problem. Conclusion: There must be some kind of problem in a production environment, perhaps somewhere in the IIS process that hosts the website.

So, I released Visual Studio to a production server and connected to the IIS process (kids, don't do this at home!), Set a breakpoint in the code for the offending, logged in as a user and tried to save the data. Hit the breakpoint and step by step until I find a line of code like this:

try
{
  ...
  using (var db = new MyDataContext())
  {
    ...
    var fooToUpdate = db.Foos.Single(f => f.ID == fooId); // <-- THIS LINE
    ...
  }
}
catch (Exception ex)
{
  // some error logging
}

After clicking "step" on this line, the stream just disappeared. Disappeared without a trace. I put the sniffer in the database and no request was fired; It goes without saying that there was no DB lock. There were no exceptions. The code injected Entity Framework and never left.

The way of data is that each user has a different and unique one fooId

for each day, so that no other user will have the same fooId

. Most of the users were able to load their Foo, but a few of them were unable to load their personal Foo files. I tried running a request to load Foo in an SSMS window; no problem at all. The only time it fails is with this particular IIS process on the production server.

Now I could just recycle the application pool or restart IIS and that would probably be the problem. But something similar happened a week ago, and we couldn't trace it either. Therefore, we reset IIS, hoping that the problem goes away. And it happened for a week. And now he's back.

Does anyone have any idea how the stream can just evaporate like this? Is the normal Bates hiding behind the EF door?

+3


source to share


2 answers


It turns out that the EF part was a red herring. I went and downloaded Telerik JustDecompile and JustCode, hoping to get into the EF code, but when I stepped into this line, I ended up not in an extension method Single()

, but inside one of my own method calls - this I thought I followed the previous line ... Obviously the code was not completely in sync with the production version.

LESSON 1: If you are attaching to a process, your point of execution may not be what you think if your code is not identical to the code that was compiled into that process.

Anyway, now that I could step into the code without decompiling anything, the first thing I noticed was:

lock (_lockObj)
{
  ...
}


And when I tried to enter it, it froze there. Smoking.

So, some other thread is blocking this object. Looked at other places where the lock gets called, which leads to spaghetti dependencies, as well as another code-locked segment, with multiple DB calls, and even a transaction boundary. It could be blocking transactions with blocking code / db, although a brief check of the code in the DB transaction failed to pick up all the contenders during the lifetime of the transaction to block anything else. In addition, there is evidence that the database does not show any blocking or open transactions. Most likely, it could be the fact that several hundred long queued processes, everything inside the code is locked inside the combination locks, and in the end it all looks like the western highway at 17:05 on Friday, a truck with a trailer car. lying on 3 lanes approaching the GW bridge.

LESSON 2: Code locks are dangerous not only, but especially when used in conjunction with database transactions. Try to find ways to make the code flow safe without using combination locks. And if you really must use code locks, make sure you get in and out as quickly as possible. Don't give your thread a magazine to read while it occupies a single kiosk, so to speak.

0


source


Given the fact that the thread does not magically evaporate, we can assume some of the more likely options:



  • The debugger was having a hard time following production code compiled in Release mode. Just because debugging Release code works 90% of the time, don't fall under the illusion that it's reliable. Optimized code can very quickly throw the debugger out of the way of actual execution. When this happens, it will look like the stream has simply disappeared.
  • Assuming the thread is indeed injecting the call rather than returning (which seems to be corroborated by the initial application freeze complaint), then the most likely scenario is a dead end of some kind. EntityFramework

    dead ends are not common, but they are not unheard of. The most common problems that I am aware of usually include TransactionScope

    or CommitableTransaction

    . Are you using any transactions in the missing sections of the code?
+1


source






More articles:


All Articles