Clever Geek Handbook

How to develop a spring package to avoid long request queue and restart failed work

I am writing a project that will generate reports. It will read all requests from the database creating a rest call based on the type of request it will make to rest the endpoint, after receiving a response, it will store the response in an object and save it back to the database by calling the endpoint.

I am using spring-batch to handle batch work. So far I have come up with one job (reader, processor, writer) that will do everything. I am not sure if this is the correct design given

  • I don't want to queue requests if some request is taking a long time to get a response. [not sure yet]
  • I don't want to keep the save response until all responses are received. [using commit-internal will help]
  • If the job fails for some reason, how can I restart the job [maybe using batch-admin will help, but what are my other parameters]
+3


source to share


2 answers


By using chunk oriented processing, the Reader, Processor and Writer are done to ensure that the Reader cannot return anything. If you can read one element at a time, process it and send it back to an endpoint that handles persistence, this approach is convenient.

If you have to read ALL information at once, the reader will receive a large collection with all elements and pass it on to the processor. The processor will process all the elements and send the result to the author. You cannot send multiple emails so you can do persistence directly from the processor, and that would be against design.

So, as far as I understand, you have two options:



  • Create a reader who can read one element at a time. Use the processed fragmentation that you've already started reading one item at a time, process it and send it back to save. See how other readers (like JdbcCursorItemReader) are implemented.
  • You create a tasklet that reads the entire set of items, processes it, and sends it back for processing. You can break this down in different tasks.

commit-interval only monitors the number of item transactions. So it won't help you as all processing and persistence is done by calling rest services.

+2


source


I worked out the design and I think it will work fine.

As for the questions I asked, the following are the answers:



+1


source






More articles:


All Articles