Negative vector lengths are not allowed when applying the operation on a huge matrix
I have a matrix M1 (3644069845 elements, 27.2 Gb)
dim(M1)
[1] 5785 629917
Any operation on this matrix that involves parsing the elements ends up with a Few error, for example:
SM1<-as(M1,"dgCMatrix")
Error in .Call(dense_to_Csparse, from) :
negative length vectors are not allowed
Note that my system has 240GB of RAM (ubuntu16, r4.8xlarge) and I am constantly monitoring it to see if the RAM is exhausted, but it seems the RAM is still ok. If I can do this with a sparse matrix, then I could reduce the size to 3GB, because most elements are zeros (95% are zeros).
It would be helpful to use any suggestions to make it a sparse matrix.
Suggestion for next message R - data frame - transform to sparse matrix
allowed, does not work. After 4 hours, I stopped executing.
source to share
You could split the matrix M1 so that it does not exceed the vector length limit (2 ^ 31), making it sparse, and then dismember the matrices and merge them again:
# Split
m11 <- M1[,0:314958]
m12 <- M1[,314959:nrow(M1)]
# Sparse
m11_sparse <- Matrix(as.matrix(m11), sparse = TRUE)
m12_sparse <- Matrix(as.matrix(m12), sparse = TRUE)
# Combine
M1_sparse <- cbind(m11_sparse,m12_sparse)
# Clean
rm(M1,m11,m12,m11_sparse,m12_sparse)
gc()
source to share