Clever Geek Handbook

Using Azure storage tables as queues with multiple worker roles?

My application will receive 1000+ requests / transactions every second across multiple web role instances. These roles will write a record for each transaction across multiple storage tables (randomly to propagate Azure 500 / sec limit transactions). Now I need a reliable way to process / aggregate this data using multiple workgroup roles and write the results to my SQL database. AKA, it needs to be scaled horizontally.

I need to save / archive all transactions in post-processing of storage tables, so I could have one set of tables for the queues, and when they are processed, move them to archive tables, or maybe a way to do it on one table, not sure.

What would you recommend as a mechanism for distributing the current workload in these queues in my work roles? Obviously each role needs to be aware of what each other role is working on, so they only work on unclaimed transactions. I think each role will fetch 1000 entries from the queue as one workload, and multiple worker roles can run on the same queue.

Should I keep the "Worker Roles" in the cache, perhaps on the SQL server.

Your suggestions are greatly appreciated.

+3


source to share


4 answers


I recommend that you use a proper queuing service to implement this functionality, rather than trying to implement queuing over a table service. This way, you don't have to implement complex logic to know which records have been processed (the logic becomes complex if you consider fault tolerance and possible errors, especially in a service like Storage Table, which has very limited transaction capabilities). Trying to coordinate the work of multiple employees with reliability, accounting for all possible failure scenarios, and at the same time scalability is something that I will not try at the application level.

For example:

  • The web role receives a request that represents a transaction;
  • The web role writes data to multiple tables;
  • The web role sends a message to the queue service representing a transaction with some unique identifier (for example, the request ID if there is no other suitable primary key).
  • The worker role pulls messages from the queue.
  • For each message, the worker role obtains a set of objects from the table store that corresponds to the message's unique identifier.
  • The worker role aggregates data as needed and writes it to the SQL database.


Notes:

  • Use either queue queues (from storage) or Service Bus queues.
  • Distribute load among many queues for scalability.
  • Be sure to apply the correct processing at all levels to accommodate temporary disruptions.
  • Catch the ability to process the same message more than once (processing must be idempotent).
+7


source


I agree with Fernando. Please take a look at my blog post on this topic; this is due to the large scale processing of Azure queues. This is based on a project I did with higher bandwidth requirements than the ones you posted.



+1


source


I also agree with Fernando. The GetMessages method in the Queue API allows you to disable a specified number of messages in a single transaction. If the de-queue logic is implemented correctly, you might not have to worry about the processing being idempotent, but it will make your solution more robust.

+1


source


You might want to consider the following CQRS based approach.

The web role validates the transaction and writes it to one or more queues. (You may have to write in a random or circular fashion across multiple queues if you hit the queue constraints). Note that a queue is just a pipe or pipe to separate your roles from those of workers, and the format of the message doesn't really matter. I would try to model the transaction as an object and serialize it to get the queue message. If the hop is too large to be written as a queue message, you can write to table or block storage and reference this resource in your queue message.

Worker roles, polling one or more queues (randomly or with round mask) and processing transactions, writing to SQL and / or storing tables as needed.

The rationale behind this architecture allows Internet and Worker roles to scale independently and reduce dependencies between them. The web role only needs to know how to validate and serialize a transaction, not how to save or process a transaction.

For more bandwidth on the worker role side, the message can be pulled and processed in parallel. The laser queue ensures that one message is retrieved by only one client at a time (if no timeout has been set). You can maintain idempotency by assuming that the message may have been partially or completely processed earlier.

+1


source






More articles:


All Articles