The parental shard exists, but not the child's shard
I am creating a Spark Streaming project with Kinesis and when I try to connect to a Kinesis stream I get the following error from Spark:
ERROR ShardSyncTask: Caught exception while sync'ing Kinesis shards and leases
com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException: Parent shard shardId-000000000000 exists but not the child shard shardId-000000000002
When I post test data to this stream, or read data from a stream using the Amazon core libraries, I don't get any errors, it only happens when I try to connect to Spark.
Below is the code I am using for my tests:
val conf = new SparkConf().setMaster("local[2]").setAppName("KinesisCounter")
val ssc = new StreamingContext(conf, Seconds(1))
val rawStream = KinesisUtils.createStream(ssc, "dev-test", "kinesis.us-east-1.amazonaws.com", Duration(1000), InitialPositionInStream.TRIM_HORIZON, StorageLevel.MEMORY_ONLY)
rawStream.map(msg => new String(msg)).count.print
+3
source to share
1 answer
How many skulls do you have in Kinesis?
what i would do:
- check the Kinesis realm, make sure your application and stream settings are in the same region.
- delete the DynomoTable that holds the Kinesis streams and start over. below is from the official documentation :
Changing the application name or stream name can lead to Kinesis errors in some cases. If you see errors, you may need to manually delete the DynamoDB table
- check your application code to make sure some parameters are set when running the code.
Hope it helps.
+3
source to share