Various combinations of Spring steps
I am working on a project where I am going to use Spring Batch and Spring Integration to create a workflow system. This workflow system needs to be able to read messages from the queues, which are actually work requests from clients, and depending on the type of job requests I need to call some 7-8 systems.
Each system reads input files from some location (usually a centralized storage system where all input files are stored as submitted by clients), processes it, and then passes it on to the next system and eventually I can respond to clients like SUCCESS. if it is processed by all systems successfully and INCORRECTLY, if some system cannot process the file, and if that fails, the client should be able to restart the failed job from a step where it actually failed.
I'm going to add each system as a step in Spring Batch and then using Spring Integration. I'm going to simulate a specific flow of systems. FTP get files, send JMS / SMAPI request, get JMS response, FTP files back. and etc.
My questions:
- Is this the correct approach?
- If yes, what are the performance tuning requirements when using Spring Batching and Spring Integration?
- Since all systems will not be called in the same order all the time, how can I write all possible combinations using Spring Batch as potential Spring jobs?
source to share
Question 1: Using Spring-Integration and Spring-Batch together is a good idea.
Question 2: With Spring-Batch, you have a lot of scalability and enhancements to improve performance. You can run entire jobs in parallel, you can run Steps in parallel, you can run chunks in parallel ... Key questions:
- Where is the bottleneck
- Where parallelization improves performance
Question 3: Basically, Spring-Batch is not a workflow system. It supports jobs with a fixed order of steps.
Option 1: you can model each system in its own job ("jobSystemA", "jobSystemB"). In doing so, you can implement the flow logic at your "integration" level and start a new job for each system in the correct order.
Option 2: If you have a fixed number of possible streams like
Flow 1:
Step A (System A)
Step B (System B)
Step C (System C)
Flow 2:
Step A (System A)
Step C (System C)
Step B (System B)
You can define a job for each possible "thread" and implement a call to specific system jobs as a "StepJob" that calls the specific "jobSystem" -job specified in option 1; something like
JobFlow1
Step1 {JobStep call jobSystemA}
Step2 {JobStep call jobSystemB}
Step3 {JobStep call jobSystemC}
JobFlow2
Step1 {JobStep call jobSystemA}
Step2 {JobStep call jobSystemC}
Step3 {JobStep call jobSystemB}
Option 3: If you programmatically define your jobs using the free SpringBatch API, you can create therefore to define the job at runtime.
source to share