2 grams and 3 grams instead of 1 gram using RWeka
I am trying to extract 1 gram, 2 gram and 3 gram portion from a train body using RWeka's NGramTokenizer function. Unfortunately, only 1 gram is obtained. There is my code:
train_corpus
# clean-up
cleanset1<- tm_map(train_corpus, tolower)
cleanset2<- tm_map(cleanset1, removeNumbers)
cleanset3<- tm_map(cleanset2, removeWords, stopwords("english"))
cleanset4<- tm_map(cleanset3, removePunctuation)
cleanset5<- tm_map(cleanset4, stemDocument, language="english")
cleanset6<- tm_map(cleanset5, stripWhitespace)
# 1-gram
NgramTokenizer1 <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
train_dtm_tf_1g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer1))
dim(train_dtm_tf_1g)
[1] 5905 15322
# 2-gram
NgramTokenizer2 <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
train_dtm_tf_2g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer2))
dim(train_dtm_tf_2g)
[1] 5905 15322
# 3-gram
NgramTokenizer3 <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
train_dtm_tf_3g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer3))
dim(train_dtm_tf_3g)
[1] 5905 15322
Every time you get the same result, which is obviously not true.
# combining together 1-gram, 2-gram and 3-gram from corpus
NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))
train_dtm_tf_ng <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer))
dim(train_dtm_tf_ng)
[1] 5905 15322
# A numeric for the maximal allowed sparsity in the range from bigger zero to smaller one
train_rmspa_m_tf_ng_95<-removeSparseTerms(train_dtm_tf_ng, 0.95)
[1] 5905 172
# creat bag of words (BOW) vector of these terms for use later
train_BOW_3g_95 <- findFreqTerms(train_rmspa_m_tf_3g_95)
# take a look at the terms that appear in the last 5% of the instances
train_BOW_3g_95
[1] "avg" "februari" "januari" "level" "nation" "per" "price"
[8] "rate" "report" "reserv" "reuter" "also" "board" "export"
[15] "march" "may" "month" "oil" "product" "total" "annual"
[22] "approv" "april" "capit" "common" "compani" "five" "inc"
[29] "increas" "meet" "mln" "record" "said" "share" "sharehold"
[36] "stock" "acquir" "addit" "buy" "chang" "complet" "continu"
...
Only 1 gram. I tried to rewrite my command like this:
NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))
But it didn't work out. Also tried to add another line:
options(mc.cores=1)
before the NgramTokenizer command, but no change. Any help?
+3
QuestSolver
source
to share
1 answer
I faced the same problem today. It seems that "tm_map" does not work well with SimpleCorpus for some reason.
I changed my code from
corpus = Corpus(VectorSource(pd_cmnt$QRating_Explaination))
to
corpus = VCorpus(VectorSource(pd_cmnt$QRating_Explaination))
And now it works and returns 2 grams.
+2
Frank lin
source
to share