Spring Integration 1.0 RC2: Streaming File Content?

I tried to find information on this, but due to the immaturity of Spring's integration framework, I had no luck.

Here is my desired workflow:

  • New files are placed in the Inbox directory

  • Files are collected using file: inbound-adapter

  • The contents of the file are streamed, N lines at a time, to the Stage 1 channel, which parses the line in intermediate (common) representation.

  • This parsed string is routed to multiple "Stage 2" channels.

  • Each "Stage 2" channel does its own processing on the available N lines to convert them to the final presentation. This channel must have a queue, which does not guarantee that the second stage channel will not be overloaded if one channel is processed much slower than the others.

  • The final representation of N lines is written to the file. In step 4, there will be as many output files as were assigned for routing.

** The 'N' above means any reasonable number of lines to read at a time, from [1, whatever I can reasonably fit into memory], but guaranteed to always be less than the number of lines in the full file. *

How can I do streaming (steps 3, 4, 5) in Spring Integration? It's pretty easy to do without file streaming, but my files are big enough that I can't read the entire file in memory.

As a side note, I have a working implementation of this workflow without Spring integration, but since we are using Spring integration elsewhere in our project, I would like to try it here to see how it works and how the resulting code compares for length and clarity.

+1


source to share


1 answer


This is a very interesting case that I wish I had missed for such a long time. It's definitely worth creating a problem. We currently support Spring integration for collecting files and submitting links to them. There is also some rudimentary support for converting files to byte [] or string.



The answer is that you are now doing step 2 in custom java code by sending chunks to a Stage 2 channel. I would recommend not to send links to streams as payload messages.

+1


source







All Articles