How can I write a tuple in redis as well as cassandra using trident topology
I am writing a Trident topology to handle data flow from Kafka and feed to Redis and Cassandra. I can write data to Cassandra. Now I would like to write the same data in Redis.
Is there a way to duplicate tuples and insert it into 2 streams, where one goes into Redis and the other goes into Cassandra?
source to share
For Trident, you can go with smth like this:
TridentTopology topology = new TridentTopology();
Stream stream = topology.newStream("MySpout", spout);
stream.partitionPersist(...); // to Redis
stream.partitionPersist(...); // to Cassandra
This way it will save data from your stream to both databases in parallel.
However, I also think that if such a parallel thing needs to be done within the same topology, or if two different topologies read from the same topic is a better idea. Imagine the Cassandra cluster crashes. In the case of two topologies, you can still continue saving data to Redis. But if there is only one topology, then each tuple failed to hit Cassandra will most likely result in a FailedException to trigger a replay, and each subsequent iteration of the tuple will involve unnecessarily storing the tuple in Redis.
source to share