Spring Metadatastore Integration
I added a bean called metadataStore to my spring boot + spring integration application and expected the ftp sync to be preserved and not corrupted even after restarting the server. However, my early tests show otherwise; If I start the server and let it collect and process 3 test files and then restart the server, then those same 3 files will be processed and processed again - as if no persistent metadata store was defined at all.
I wonder if some of the configuration details are missing when setting up the datastore ...
@Configuration
public class MetadataStoreConfiguration {
@Bean
public PropertiesPersistingMetadataStore metadataStore() {
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
return metadataStore;
}
}
Also, I see in the spring integration reference for a short example on how to set up an idempotent sink and metadata store. Is this what my implementation is missing?
If so, and if I need to set it like in the example, where will I define my calls to metadataStore.get and metadataStore.put? the outbound adapter I'm using doesn't provide me with an expression attribute ... Here is my naive and incomplete attempt:
<int-file:inbound-channel-adapter id="inboundLogFile"
auto-create-directory="true"
directory="${sftp.local.dir}"
channel="fileInboundChannel"
prevent-duplicates="true">
<int:poller fixed-rate="${setup.inbound.poller.rate}"/>
</int-file:inbound-channel-adapter>
<int:filter id="fileFilter" input-channel="fileInboundChannel"
output-channel="idempotentServiceChannel"
discard-channel="fileDiscardChannel"
expression="@metadataStore.get(payload.name) == null"/>
This is the outbound adapter used in the example:
<int:outbound-channel-adapter channel="idempotentServiceChannel" expression="@metadataStore.put(payload.name, '')"/>
In my outgoing ftp adapter, I cannot insert the above expression :(
<int-ftp:outbound-channel-adapter id="sdkOutboundFtp"
channel="idempotentServiceChannel"
session-factory="ftpsCachingSessionFactory"
charset="UTF-8"
auto-create-directory="true"
use-temporary-file-name="false"
remote-file-separator="/"
remote-directory-expression="${egnyte.remote.dir}"
* NO EXPRESSION POSSIBLE HERE *
mode="REPLACE">
</int-ftp:outbound-channel-adapter>
source to share
By default PropertiesPersistingMetadataStore
, only the state is saved during normal shutdown of the application; it remained in memory until then.
In 4.1.2, we changed it for implementation Flushable
so that users can reset the state at any time.
Consider using the FileSystemPersistentAcceptOnceFileListFilter
on- local-filter
in adapter instead of a separate filter element. See the documentation for details .
Since version 4.1.5
, the filter has an optionflushOnUpdate
to flush()
store metadata on every update.
Other metadata stores that use an external server (Redis, Mongo, Gemfire) do not need to be flushed.
source to share