Reading data from dynamodb streams
I have setup threads for my dynamo tables. I followed the example program in the documentation for reading data from streams ( http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.LowLevel.Walkthrough.CompleteProgram.html ). However, I have a problem - when iterating through shards, it seems that checking that sharditerator is null is not a sufficient condition to exit the loop. My code ends up in an infinite loop. The sample program counts the number of changes and uses it to exit the loop ->
while (nextItr != null && numChanges > 0) {
// Use the iterator to read the data records from the shard
GetRecordsResult getRecordsResult =
streamsClient.getRecords(new GetRecordsRequest().
withShardIterator(nextItr));
List<Record> records = getRecordsResult.getRecords();
System.out.println("Getting records...");
for (Record record : records) {
System.out.println(record);
numChanges--;
}
nextItr = getRecordsResult.getNextShardIterator();
}
But in a real-world environment, I'm pretty sure storing the changelog isn't practical. Is design an endless loop? Why doesn't shardIterator become null?
source to share
I am repeating my answer to the same question that you posted on the AWS Forums here:
The next Imator will not become empty as long as the shard is open and actively receiving possible future updates. Once the shard closes and the last record is passed, your loop may come out and indicate that it is no longer possible to get more records from the shard.
Please refer to our documentation here: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/model/GetRecordsResult.html#getNextShardIterator ()
"If set to null, the shard is closed and the requested iterator will no longer return data."
Hope it helps, thanks for your interest in DynamoDB!
source to share