How to preserve filename when transferring file in Apache Flume?

I am using Flume 1.3.1 ng, I am migrating a file from spoolDir to HDFS Sink and I need the outpufile name to match the input file. For example, if the input file name is sample.gz, then the output is also needed as sample.gz

Could you please provide a procedure on how to achieve this functionality in Flume 1.3.1 ng.


source to share

2 answers

I am using the following settings:

a1.sources.r1.basenameHeader = true
a1.sinks.k1.hdfs.filePrefix = %{basename}


These settings give me the following transformation:

original_file.txt -> original_file.1457085474015.txt


And unfortunately you cannot remove the timestamp from the filename: C



There is a parameter for spooldir

whether or not to add a header relative to the original filename. It is called fileHeader

and it is set to false by default, so set it ti to true:

agentname.sources.mysource.type = spooldir
agentname.sources.mysource.spoolDir = /path/to/the/dir
agentname.soruces.mysource.fileHeader = true




All Articles