Clever Geek Handbook

Is it good to collect data from devices into a message queue?

Let's say you have 1000 devices all sending data at all times, would a message queue be a good data collection tool for that data?

+3


source to share


2 answers


Obviously it depends:

  • Do you need to process each dataset, no matter how old it is?
  • Will the data appear in the stream or be explosive?
  • Can one application handle all of the data, or will they need to load in a balanced way?
  • Is there a need for messaging features like themes?
  • If your devices are clients, is there a client implementation that runs on them?


If you have a stream of data that can be handled by a single application, and you are tolerant of accidental lost data, I would leave it simple and post the data via REST or similar. I would only look at messaging as soon as you need scalability, durability, fault tolerance, or the ability to load-balancing time.

+1


source


You can't go wrong with the design using queues, but as Chris (another respondent) stated, it might not be worth your effort in terms of infrastructure as web servers handle reasonable load well.



In the "real world" I saw the status of the commercial tools report in the queue for processing, so this is definitely a valid solution.

0


source






More articles:


All Articles