Akka-Streams: Behavior "At least sometime" with Kafka and Kassandra

I start with Akka Streams and everything is going well so far. However, I have come across a use case that I don't know how to approach. A script is a stream with an ActorPublisher as a source that uses messages from Kafka and a subscriber as a sink that updates the Cassandra table.

Kafka ~> some display operations ~> Cassandra

The point is, I would like to explicitly validate Kafka every time a message has been successfully processed and inserted into Cassandra so that I can reread the message in the event of a disaster and service failure, i.e. of some kind at least once. How can I approach this in terms of Akka streams ?. Is the script supported?

It's true that I can always set up a Kafka consumer with auto-commit behavior, but I'd rather take control of how I read messages.

Update

Regarding this topic, we are currently evaluating Reactive Kafka where they have enabled manual commit in kafka since version 0.8 (that's prestige for these guys). This feature will allow us to implement the Alof behavior we need.

+3


source to share


1 answer


I think you could use custom stream handling with PushPullStage as described in the akka streams documentation



+2


source







All Articles