Translating change log into actual state in Apache Camel
I have two systems, call them A and B. When some significant object changes in A, A sends it to B via Apache Camel. However, I ran into one case where A actually has an object's changelog and B only needs to reflect the actual state of the object. In addition, the changelog may contain "future" entries. This means that a change in the state of the object is planned at some point in the future. System A user can edit this change log, delete change records, add new change records with any time stamp (past and future), and even update existing changes. Of course, A sends these change records to B, but B only needs the actual state of the object.
Please note that I can query objects from A, but A is a system critical system and therefore I am not going to query anything as it might cause additional overhead. Also, the API for querying data from A is more complex and I would like to avoid it whenever possible.
I see two problems here. First, it is aware of whether a specific change to a changelog entry could change the actual state. I'm going to keep a log of the changes in a staging database. As a change log entry, I'm going to add / remove / update it in the staging database and then calculate the actual state of the object and send that state to B.
Second, the change tracking schedule. I couldn't think of anything other than to run intermittent work for a consistent interval (say 15 minutes). This task will scan all records that fall within the time span from the last call to the current call.
What I like about Apache Camel for is its component based approach where you only need to connect the endpoints and get it all done, with a little bit of coding. Are there any pre-existing primitives for this problem in both Apache Camel and EIP?
source to share
I am actually working on a very similar use case where system A sends a snapshot and updates requiring translation before sending to system B.
First, you need to start a mechanism to provide an initial state ("snapshot") from System A, the component timer:
can run one-time startup logic.
You will now receive the snapshot data (you did not specify how it might be a file ftp
or an endpoint jms
). Confirm the data, divide it into elements and store each data element in local memory cache:
, as Sergey suggests in his comment, entered unambiguously. Use an expiration policy that is logical (for example, 48 hours).
From there, continually process the "update" data from the endpoint ftp:
. For each update, you will need to map the update to the data in cache:
and determine what (and when) needs to be sent to system B.
The data that needs to be sent to system B later must be stored in memory or in a database.
Finally, you need a scheduling mechanism to determine every 15 minutes if new data is to be sent, you can easily use timer:
or quartz:
for that.
Thus, you can build the integration of the following components: timer
, cache
, ftp
, quartz
plus some custom beans / processors to perform custom logic.
The main challenges are handling data that is cached and then updated and control mechanisms are designed for what should happen when your camel application initially connects, disconnects, or resumes.
Good luck;)
source to share