How to preserve filename when transferring file in Apache Flume?
I am using Flume 1.3.1 ng, I am migrating a file from spoolDir to HDFS Sink and I need the outpufile name to match the input file. For example, if the input file name is sample.gz, then the output is also needed as sample.gz
Could you please provide a procedure on how to achieve this functionality in Flume 1.3.1 ng.
+3
source to share
2 answers
I am using the following settings:
a1.sources.r1.basenameHeader = true
a1.sinks.k1.hdfs.filePrefix = %{basename}
These settings give me the following transformation:
original_file.txt -> original_file.1457085474015.txt
And unfortunately you cannot remove the timestamp from the filename: C
+2
source to share
There is a parameter for spooldir
whether or not to add a header relative to the original filename. It is called fileHeader
and it is set to false by default, so set it ti to true:
agentname.sources.mysource.type = spooldir
agentname.sources.mysource.spoolDir = /path/to/the/dir
agentname.soruces.mysource.fileHeader = true
0
source to share