Moving Cassandra 1.2: Broken Pipe
We have migrated our cluster to Cassandra 1.2 (from 1.1.7) and we have tried post-migration repairs (we run them periodically as recommended).
The repair failed because the SSTable version is different, so we used nodetool upgradesstables
and nodetool scrub
and tried again.
But then the restore ended with this exception:
java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:90)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
We also noticed that it nodetool netstats
shows strange stats before dying (SSTable transmission is 756845% ...).
EDIT
There seems to be a problem with passing compressible SSTables, I'll try to disable compression on that ColumnFamily and try again.
RE-EDIT
Disabling compression fixed the problem, so we tried to re-enable it and got
java.lang.RuntimeException:
Last written key DecoratedKey(97115[...]3136) >= current key DecoratedKey(8...
Any idea?
source to share
The issue was caused by a bug in Cassandra 1.2 affecting the transfer of compressed column families between nodes. This was fixed in 1.2.2 => Jira entry for this bug .
source to share