Cache lookup speed without using too much memory
I need to access a database with 380,000 records. I don't have write access to the DB, I can just read it. I performed a search function using a map to find users by first name. Here's my process: 1 - Load everything from the database 2 - Store everything in Map<Charactere, ArrayList<User>>
, using Alphas to store users according to the first letter of their name.
<A> {Alba, jessica, Alliah jane, etc ...}
<B> {Birsmben bani, etc ...}
When someone searches for a user, I take the first booklet of the first name and use map.get(firstletter)
and then iterate over the ArrayList to find all users.
Map Take a huge amount of memory space I guess (380,000 user objects). I had to increase the heap size I want to make it faster. Use firstname as a key for the Map to make it faster (there are many people with the same name).
I have two solutions:
1 - Still use a map with firstname as key (increasing the heap size again?)
2 - Use files on the disk instead of Map (Alba.dat will contain all Alba for example) and open the right file for each search. No need to incease the heap size, but are there any side effects?
Which one is better? (pros and cons)
Update with more info
This is a database of customers who call our support team by phone. The person making the call must search by customer first name (usually first name followed by last name). Using Db is too slow to search. The solution I have implemented is much faster (1/2 second versus 26 seconds using db), but I want to improve it.
source to share
IMHO, I don't think you need to cache all records in memory, but some of them might be:
- Maybe just use a ring buffer, or
- A more complex and smarter way to implement an LFU cache that only stores the most accessible N-item. See this question to understand how to implement such a cache.
source to share
There are several problems with your approach:
- This means that the number of users does not change, a good design of the application will work with any number of users without changing the software.
- This means that the current problem is the only one. What happens if the next requirement to be implemented is "Caller ID Search" or "ZIP Code Search"?
- Reinvents the wheel, you are currently starting to write a database, index, or information retrieval (but you want to name it) from scratch.
The right thing to do is to export user data to a database engine that provides proper search capabilities. The export / checkout can hopefully speed up if you have timestamps on the change or if you can intercept updates and reapply them to your search index.
What you use to search doesn't really matter, a simple database on a modern system is fast enough. Most also provide indexing capabilities to speed up searches. If you want something that can be built into your application and specialized in searching and solves your problems above, I would recommend using Lucene.
source to share