Apache Kafka: check for message in topic

I do not consider myself an expert in Kafka, but I think you are pretending to be "against" the essence of Kafka.

However, I am getting a solution using the Kafka Streams library for Java. Basically, the process is as follows:

• Match each message with a new key value, where the key is a combination of the previous key and its value: (key1, message1) -> (key1-message1, message1)
  
  

• Group messages using keys, as a result of this operation you get KGroupedStream .

• Apply the decrement function by changing the value to some custom value, such as the Duplicate Value string.

• Convert the resulting KTable after pruning to a KStream and paste it into a new Kafka theme.

There are so many assumptions in the previous explanation, I'm going to provide some code to give some light:

KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> resources =  builder.stream("topic-where-the-messages-are-sent");

KeyValueMapper<String, String, KeyValue<String,String>> kvMapper = new KeyValueMapper<String, String, KeyValue<String,String>>() {
    public KeyValue<String, String> apply(String key, String value) {
        return new KeyValue<String, String>(key + "-" + value, value);
    }
};

Reducer<String> reducer = new Reducer<String>() {
    public String apply(String value1, String value2) {
        return "Duplicated message";
    }
};

resources.map(kvMapper)
    .groupByKey()
    .reduce(reducer, "test-store-name")
    .toStream()
    .to("unique-message-output");

KafkaStreams streams = new KafkaStreams(builder, props);
streams.start();

      

        
        
        
      

    

Keep in mind that this is probably not the optimal solution, and you may not consider it a "graceful" way to solve your problem.

Hope this helps.