Clever Geek Handbook

Duplicate the channel before being intercepted by the interceptor

I am using flume to do something like this

Source --> interceptor --> Channel --> multiplexing --> HDFS Sink
                                          |-----------> Null Sink

I would like to add a channel immediately after the source, but I don't want the event to go through the interceptor. I would like a "raw" event. Like this:

Source --> interceptor (i) --> Channel --> multiplexing --> HDFS Sink
    |                                     |-----------> Null Sink
    |-------> Channel (must no be intercepted by i) --> HDFS

How can i do this?

thank

+3


source to share


1 answer


Since the interceptors are configured for each source, you will need to add a second source (configured without any interceptors and listen on a different Http port), and emit data twice: one instance for the source with interceptors and one copy it to another source.

Another possibility is to chain two agents. The first one contains a single source without interceptors and two sinks: one for storing data, as in HDFS, and the other for feeding the agent that you already have. I mean:



src-->ch-->multip-->sink----------->src-->int-->ch-->multip-->hdfssink
                |-->hdfssink                              |-->nullsink
(________agent1____________)        (_____________agent2_____________)
+1


source






More articles:


All Articles