Do I need one JVM process for each spring cloud tasktrigger?
We have many batch jobs scheduled today using cron expressions in one application. We would like to isolate these jobs more and hence port them to the spring cloud task.
But after reading the documentation [1], I came to the conclusion that I need to use triggertask
(source), which in turn sends TaskLaunchRequest
to tasklauncher
(receiver), in order to finally start a new process.
This means (if I only have one task / package), I need at least the following JVM processes to start one new process:
- stream server
- triggertask (source)
- tasklauncher (sink)
OK, the thread server and tasklauncher will be shared for any upcoming task, but the triggertask function can only accept a cron definition for a single task and therefore should replicate for any upcoming job. So, do I need at least one "babysitting process" for each task?
in fact??? this sounds like a huge kink ... from my point of view, I would expect cron scheduling to be the core functionality of task definition, so the only thing needed is a stream server.
Am I getting this right or is there something I missed? Is there an easier way to do this in spring cloud environment? I really like the idea of having the thread server start new JVMs as needed, but all these additional processes really feel wrong.
If this should be done on CloudFoundry eg. http://run.pivotal.io then this means I have a cron scheduler for one job which costs 35 $ / Mth (because from Java BuildPack 4.0 JVM Process with only 512 MB will no longer run [2 ]) is an expensive definition of cron ...
[1] https://github.com/spring-cloud/spring-cloud-stream-app-starters/tree/master/triggertask/spring-cloud-starter-stream-source-triggertask [2] https: // www .cloudfoundry.org / just-released-java-buildpack-4-0 /
source to share
TL; DR; Don't do this, and don't write your own scheduling logic or integrate the Cloud Dataflow API with your Enterprise Planner.
Extended version
Let me give a little history on this and then provide my thoughts on what to do.
When the Spring Cloud Task project started, we wanted to create some sample applications that illustrated the use of the task in many different use cases. The ability to easily trigger a task in response to a message received was one of the use cases we identified to create a pattern around. You can see this sample here and here .
When Spring Cloud Data Flow (SCDF) came along, one of the use cases we wanted to address in some way was task scheduling. The problem is that we want the SCDF server to be stateless (since it is itself a cloud native microservice). This means that implementing a scheduler is not an option. From there, we felt that the integration with what every platform provided for planning makes the most sense. However, this also took more work. This approach is actually on our roadmap, but we haven't had user feedback to push this feature higher on the list.
The solution we came across to address this requirement in a shorter time frame was what you found in the documentation today. Reusing these sample apps in conjunction with an app trigger-task
that handles the cron puzzle piece to run tasks at a given time.
My personal recommendation is that if you don't have a scheduler that you are already using, write a boot application that uses Quartz or some other internal scheduler (Spring Scheduler, etc.) to call the SCDF API to start tasks on a given schedule. If available DataFlowTemplate
, writing your own scheduler should be straight forward.
source to share