Clever Geek Handbook

Caching large datasets

I am working on an ASP.NET application that has the following requirements:

  • A fairly expensive query of about 20,000 items ( not from the database ) is executed every 15 minutes . It has about 10 columns, all short strings except for one that stores up to 3000 char (usually much less).
  • Process the resulting DataTable using various sorts and filters, and then store the top 100 in additional data tables.
  • Display this information as a form of aggregation of potentially 10,000 people.

It seems to me that this is a great candidate for caching (System.Web.Caching), especially considering that we may wish to support additional filtering on the fly. For example: Filtering a table to rows that are specific to a specific user only.

However, before I start, I would like to understand:

  • If there is any recommendation for storing such a large DataTable in the cache.
  • If anyone has experience that they can share? Large tables that you saved?
  • Any pitfalls to be careful along the way?

Thanks in advance.

+2


source to share


4 answers


This is pretty straight forward, but you can keep an eye on what is actually cached. Rick Strahl has an interesting post on how the cache was actually empty due to memory pressure.



+2


source


Steve Smith won a competition at a Microsoft event with one of his caching solutions. Here is the post


from your blog. Here's an episode of DNR TV where he reviews his methods.
+2


source


You have viewed the Microsoft Enterprise Library Caching Application Block

It is based on best practices and is very easy to use.

http://msdn.microsoft.com/en-us/library/cc309103.aspx

+1


source


Presumably, some rows in the data table are updated between query cycles. Therefore, you will need to refresh the cache before every run, which will give you zero improvement only when reading the file.

The filesystem uses a pretty efficient cache anyway, so why reinvent the wheel.

A better approach might be to rewrite your "request" so that you get everything you need in a single pass of the file.

+1


source






More articles:


All Articles