Using eml in caret: error for class likelihood
I am trying to compare the standard neural network approach with the extreme learning machine classifier (based on the ROC metric) using the method "nnet"
and "elm"
in the R package caret
. For nnet everything works, but using method = "elm"
I get the following errors:
Error in evalSummaryFunction(y, wts = weights, ctrl = trControl, lev = classLevels, :
train() use of ROC codes requires class probabilities. See the classProbs option of trainControl()
In addition: Warning messages:
1: In train.default(x, y, weights = w, ...) :
At least one of the class levels are not valid R variables names; This may cause errors if class probabilities are generated because the variables names will be converted to: X1, X2
2: In train.default(x, y, weights = w, ...) :
Class probabilities were requested for a model that does not implement them
I also got the first error when method = "nnet"
, but here I could solve the problem by making an estimate of the factor variable. Hence, this cannot be a problem here.
I'm relatively new to R and the error is probably trivial, but I'm stuck right now ... Since elmNN seems to be relatively recently implemented, I also couldn't find anything on the internet about how to use knitting in caret
.
gc <- read.table("germanCreditNum.txt")
colnames(gc)[25]<-"score"
gc_inTrain <- createDataPartition(y = gc$score,
## the outcome data are needed
p = .8,
## The percentage of data in the
## training set
list = FALSE)
str(gc_inTrain)
gc_training <- gc[ gc_inTrain,]
gc_testing <- gc[-gc_inTrain,]
nrow(gc_training) ## No of rows
nrow(gc_testing)
gc_training$score <- as.factor(gc_training$score)
gc_ctrl <- trainControl(method = "boot",
repeats = 1,
classProbs = TRUE,
summaryFunction = twoClassSummary)
neuralnetFit <- train(score ~ .,
data = gc_training,
method = "nnet",
trControl = gc_ctrl,
metric = "ROC",
preProc = c("center", "scale"))
neuralnetFit
plot(neuralnetFit)
nnClasses <- predict(neuralnetFit, newdata = gc_testing)
str(nnClasses)
## start with ELM for German Credit
gc_ctrl2 <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
elmFit <- train(score ~ .,
data = gc_training,
method = "elm",
trControl = gc_ctrl2,
metric = "ROC",
preProc = c("center", "scale"))
elmFit
plot(elmFit)
elmClasses <- predict(elmFit, newdata = gc_testing)
str(elmClasses)
elmProbs <- predict(elmFit, newdata = gc_testing, type = "prob")
head(elmProbs)
source to share
I have no recollection of why I did not include a probabilistic model for ELM (I probably had a good reason). You can use a custom method to get softmax values:
library(caret)
set.seed(1)
dat <- twoClassSim(100)
elm_fun <- getModelInfo("elm")[[1]]
elm_fun$prob <- function (modelFit, newdata, submodels = NULL) {
out <- exp(predict(modelFit, newdata))
t(apply(out, 1, function(x) x/sum(x)))
}
mod <- train(Class ~ ., data = dat,
method = elm_fun,
metric = "ROC",
trControl = trainControl(classProbs = TRUE,
summaryFunction = twoClassSummary))
Max
source to share