2 grams and 3 grams instead of 1 gram using RWeka

I am trying to extract 1 gram, 2 gram and 3 gram portion from a train body using RWeka's NGramTokenizer function. Unfortunately, only 1 gram is obtained. There is my code:

train_corpus
# clean-up
cleanset1<- tm_map(train_corpus, tolower)
cleanset2<- tm_map(cleanset1, removeNumbers)
cleanset3<- tm_map(cleanset2, removeWords, stopwords("english"))
cleanset4<- tm_map(cleanset3, removePunctuation)
cleanset5<- tm_map(cleanset4, stemDocument, language="english")
cleanset6<- tm_map(cleanset5, stripWhitespace)

# 1-gram
NgramTokenizer1 <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
train_dtm_tf_1g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer1))
dim(train_dtm_tf_1g)
[1]  5905 15322

# 2-gram
NgramTokenizer2 <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
train_dtm_tf_2g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer2))
dim(train_dtm_tf_2g)
[1]  5905 15322

# 3-gram
NgramTokenizer3 <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
train_dtm_tf_3g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer3))
dim(train_dtm_tf_3g)
[1]  5905 15322

      

Every time you get the same result, which is obviously not true.

# combining together 1-gram, 2-gram and 3-gram from corpus 
    NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))
train_dtm_tf_ng <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer))
dim(train_dtm_tf_ng)
[1]  5905 15322

# A numeric for the maximal allowed sparsity in the range from bigger zero to smaller one
train_rmspa_m_tf_ng_95<-removeSparseTerms(train_dtm_tf_ng, 0.95)
    [1] 5905  172

# creat bag of words (BOW) vector of these terms for use later
train_BOW_3g_95 <- findFreqTerms(train_rmspa_m_tf_3g_95)

# take a look at the terms that appear in the last 5% of the instances
train_BOW_3g_95

  [1] "avg"        "februari"   "januari"    "level"      "nation"     "per"        "price"     
  [8] "rate"       "report"     "reserv"     "reuter"     "also"       "board"      "export"    
  [15] "march"      "may"        "month"      "oil"        "product"    "total"      "annual"    
  [22] "approv"     "april"      "capit"      "common"     "compani"    "five"       "inc"       
  [29] "increas"    "meet"       "mln"        "record"     "said"       "share"      "sharehold" 
  [36] "stock"      "acquir"     "addit"      "buy"        "chang"      "complet"    "continu" 

     ...

      

Only 1 gram. I tried to rewrite my command like this:

NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))

      

But it didn't work out. Also tried to add another line:

options(mc.cores=1)

      

before the NgramTokenizer command, but no change. Any help?

+3


source to share


1 answer


I faced the same problem today. It seems that "tm_map" does not work well with SimpleCorpus for some reason.

I changed my code from

corpus = Corpus(VectorSource(pd_cmnt$QRating_Explaination))

      



to

corpus = VCorpus(VectorSource(pd_cmnt$QRating_Explaination))

      

And now it works and returns 2 grams.

+2


source







All Articles