Best key store for my node.js service
I need to write a node.js service with key-value-store as a backend. My data looks like this:
Item
Name: "Item 1"
Attributes
AttributeA: "ValueA"
AttributeB: "ValueB"
AttributeC: "ValueC"
Item
Name: "Item 2"
Attributes
AttributeA: "ValueC"
AttributeB: "ValueD"
I asked myself what is the best backend for my requirements. I need a fast backend that is easily accessible from node.js and can handle large amounts of data (1-2 million pieces with 15-20 attributes). It is important to note that attributes can differ from element to element, so there are no 20 fix attributes. My later queries would look like this:
- Get all attributes for element 1
- Get all values for attributeA (from all elements)
- Get all elements that contain attributeB
Or maybe noSQL DB like couchDB would be better? I would appreciate some hints.;)
thank
Torben
source to share
While it will probably depend on other parameters (such as the size of the attributes, their type, read / write ratio), I think a document database like MongoDb (ou couchDb, even if I don't use it personnaly) would be not bad. The reason is that you want to query based on the "value" of the attributes, which is not possible in a database like Redis (without changing the schema you suggested, and although Redis is a great application for many use cases).
With docs like this: {name: "Item 1", AttributeA: "ValueA", AttributeB "ValueB" ...}, you can implement the queries you mentioned:
db.docs.find ({name: "Item 2"}); (possibly by specifying an index on the name if you use this query frequently) db.docs.distinct ('attributeA'); db.docs.find ({attributeB: {$ exists: true}}});
etc.
You can find a very good nodejs mongodb driver here: https://github.com/christkv/node-mongodb-native
Hope this helps!
source to share
I think mongoDB will be good for you. It's not really a key / value store, but it's a document store that is slightly different.
Here's what's good about mongo with nodejs:
- Index keys.
- Composite indexes allow fast multi-attribute queries.
- Schema is smaller. You can add / remove attributes as you see fit. This would be useful for you in cases where you do not have attributes for some elements.
- Fast, everything I've ever done with it is amazingly fast.
- Made for a ton of paperwork. Your millions of items correctly indexed will only make mongodb laugh at any doubts.
- Powerful. Things like findAndModify and map / reduce give you amazing flexibility. And most of the time you don't even need them.
- Familiar javascript syntax. this.db.items.findOne ({attrA: 'value'}) just feels good.
Here are some restrictions:
- It seems like everyone started using the node-js / mongodb driver. The most complete one, https://github.com/christkv/node-mongodb-native , has some usability issues. Your code ends up looking very reversed.
- Limit of 16 MB per document. This is usually not a problem, but something to take care of.
- Sharding is a technique / design exercise. It's not too difficult, but it takes a little foresight. I started with one server and expanded as needed.
- You need a 64 bit server. This is usually not a problem on this day, but something to be aware of.
As always, know your tools. I would recommend reading this book: http://manning.com/banker/ . It's very well written.
source to share
If you want to do it with Redis, it's a bit tricky.
Process the search for all elements that have a specific attribute. You have a set for each of the attributes. Every time you add an element, if it has an attribute, add it to the set. When you want to get all elements with an attribute, just query the whole set and then retrieve all elements based on that.
Adding something
HMSET item1 attributeA value attributeB value
SADD set_attributeA item1
SADD set_attributeB item1
For your occasions
1.
HGETALL item1
2.
SMEMBERS set_attributeA
then just iterate and get all values
HGET item1 attributeA
3.
use HGETALL instead of HGET
For the node library use https://github.com/mranney/node_redis , it has some nice extras like creating objects from HGETALL results as well as storing objects in hashes with hmset
source to share