Spring Metadatastore Integration
I added a bean called metadataStore to my spring boot + spring integration application and expected the ftp sync to be preserved and not corrupted even after restarting the server. However, my early tests show otherwise; If I start the server and let it collect and process 3 test files and then restart the server, then those same 3 files will be processed and processed again - as if no persistent metadata store was defined at all.
I wonder if some of the configuration details are missing when setting up the datastore ...
@Configuration
public class MetadataStoreConfiguration {
@Bean
public PropertiesPersistingMetadataStore metadataStore() {
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
return metadataStore;
}
}
Also, I see in the spring integration reference for a short example on how to set up an idempotent sink and metadata store. Is this what my implementation is missing?
If so, and if I need to set it like in the example, where will I define my calls to metadataStore.get and metadataStore.put? the outbound adapter I'm using doesn't provide me with an expression attribute ... Here is my naive and incomplete attempt:
<int-file:inbound-channel-adapter id="inboundLogFile"
auto-create-directory="true"
directory="${sftp.local.dir}"
channel="fileInboundChannel"
prevent-duplicates="true">
<int:poller fixed-rate="${setup.inbound.poller.rate}"/>
</int-file:inbound-channel-adapter>
<int:filter id="fileFilter" input-channel="fileInboundChannel"
output-channel="idempotentServiceChannel"
discard-channel="fileDiscardChannel"
expression="@metadataStore.get(payload.name) == null"/>
This is the outbound adapter used in the example:
<int:outbound-channel-adapter channel="idempotentServiceChannel" expression="@metadataStore.put(payload.name, '')"/>
In my outgoing ftp adapter, I cannot insert the above expression :(
<int-ftp:outbound-channel-adapter id="sdkOutboundFtp"
channel="idempotentServiceChannel"
session-factory="ftpsCachingSessionFactory"
charset="UTF-8"
auto-create-directory="true"
use-temporary-file-name="false"
remote-file-separator="/"
remote-directory-expression="${egnyte.remote.dir}"
* NO EXPRESSION POSSIBLE HERE *
mode="REPLACE">
</int-ftp:outbound-channel-adapter>
By default PropertiesPersistingMetadataStore
, only the state is saved during normal shutdown of the application; it remained in memory until then.
In 4.1.2, we changed it for implementation Flushable
so that users can reset the state at any time.
Consider using the FileSystemPersistentAcceptOnceFileListFilter
on- local-filter
in adapter instead of a separate filter element. See the documentation for details .
Since version 4.1.5
, the filter has an optionflushOnUpdate
to flush()
store metadata on every update.
Other metadata stores that use an external server (Redis, Mongo, Gemfire) do not need to be flushed.