Cassandra node is almost out of space, but cleaning up nodetool increases disk usage?
One of our nodes was used on 95% of the disks, and we added another node to the cluster to hopefully rebalance, but disk space did not drop on node. I tried to do a nodetool cleanup assuming the redundant keys were on node, but the disk space is increasing! Will cleaning actually reduce the size?
source to share
Yes, but you need to be careful because the compression is calculated and it generates temporary and referenced tmp files that increase disk space until the cleared table is calculated.
So, I would go into your data directory and find out what your keyspace sizes are using
du -h -s *
Then, individually clean up the small keyspaces (you can specify the keyspace in a nodetool cleanup command with a nodetool cleanup) until you have any overhead. To get an idea of how much free space is being freed, go to the log and cat / grep for the flushed copies:
tail <system.log location> | grep 'eaned'
I would recommend that you do not try to clear the keyspace that is more than half of your remaining disk space. Hope this is possible.
If you are running out of space, you will have to disconnect node, attach a large drive, copy data files to a larger drive, translate yaml to new data directories and then restart.This is useful for things like SSDs that are expensive and small, but basic spinning - discs are cheaper and larger.
source to share