Clever Geek Handbook

Selecting a Database for Stock Data

I'm wondering if NoSQL is an option for this scenario:

The input data represents data on hourly stocks ( sku , quantity, price and some more specific) from several sources. Old versions will simply be lost. So we won't be able to get more than 1 million datasets in the near future, and there won't be any business intelligence queries, for example, in data warehouses. But there will be an aggregation , at least for the minimum price of a group of articles, which should be updated if an article with the minimum price of a group is sold. In addition to these voluminous entries, on a frequent basis, there will be single decrements by the amount of the item, which can occur at any time.

The database will be part of the service that needs to respond quickly to requests via REST. So there must be some kind of caching. No need for a strong consistency, but durability.

Additional wish list:

  • should scale well for growing request load.
  • inexpensive technology in terms of money and complexity (no Oracle cluster)
  • no native languages ​​(no PL / SQL)

MongoDB with its aggregation framework seems promising. Can you think of alternatives? (I'm not sticking with NoSQL!)

+3


source to share


2 answers


I would start with Redis, and here's why:

  • "there must be some kind of caching" => and that's what Redis is best suited for. If, for whatever reason, you decide you want more, you can add more, but keep whatever you've already developed in Redis as a cache for that more.

  • Redis alone is fast. Two reps is faster. Three repetitions are one faster than two, etc.

  • The learning curve is pretty flat and fun => as set theory is really fun.

  • Increments / Decreases / Min / Max is a native Redis conversation

  • Redis integration with XYZ (you mentioned the need for REST API) is all over the google and github world

  • Redis is honest <= actually one of my favorite Redis features

MongoDB will work first, ANY other major NoSQL will work, but why !?

I would go with Redis and if you decide later you need "more", I would look at "Redis + SQL db (Postgre / MySQL / etc ..)" first, it will give you both of the two worlds => "Caching / Speed ​​"and" Aggregation Force "in case you need aggregation of aggregates above Min / Max / Incr / Decr.



Anyone who tells you that PostgreSQL is "not writing fast enough" does not know this.

Whoever tells you that MySQL is "not scalable enough" doesn't know this (for example, Facebook runs MySQL).

Like I already on the video :) => whoever says that MongoDB has "replica sets and outlines", you don't like it, since the replica sets and outlines look only sexy from documents and hype. Once you need the reshard / reorg replica set, you'll find out the cost of choosing the wrong shard key and magic moves ...

Again => Redis FTW!

+2


source


Well, it seems to me that MongoDB

is the best choice.

It has not only aggregation functions, but also the ability to display / shorten queries for the purpose of calculating statistics. It can be scaled with replica sets

and sharding

, has atomic updates for increments (decrements are negative increments only).



Alternative:

  • CouchDB

    - not read quickly enough
  • Redis

    is the db key / value. you will need to program your article logic at the application level.
  • MySQL

    - not scalable enough
  • PostgreSQL

    - can be a good alternative if scaled with pgbouncer

    but not fast enough to write
0


source






More articles:


All Articles