R How to preserve punctuation marks with TermDocumentMatrix ()
I have a large framework where I identify patterns in strings and then retrieve them. I have included a small subset to illustrate my problem. I am creating my templates by creating a TermDocumentMatrix with multiple words. I am using these patterns with stri_extract and str_replace from the stringi and stringr packages to search the punct_prob dataframe.
My problem is that I need to keep the punctuation ticked in "punct_prob $ description" in order to maintain literal values within each line. For example, I cannot have 2.35 mm becoming 235 mm. However, the TermDocumentMatrix procedure I am using removes punctuation (or at least periods) and therefore my pattern search functions cannot match them.
In short ... how can I store punctuation when creating TDM? I tried to include removePunctuation = FALSE in the TermDocumentMatrix control argument, but with no success.
library(tm)
punct_prob = data.frame(description = tolower(c("CONTRA ANGLE HEAD 2:1 FOR 2.35mm BUR",
"TITANIUM LINE MINI P.B F.O. TRIP SPRAY",
"TITANIUM LINE POWER P. B F.O. TRIP SPR",
"MEDESY SPECIAL ITEM")))
punct_prob$description = as.character(punct_prob$description)
# a control for the number of words in phrases
max_ngram = max(sapply(strsplit(punct_prob$description, " "), length))
#set up ngrams and tdm
BigramTokenizer <- function(x) {RWeka::NGramTokenizer(x, RWeka::Weka_control(min = max_ngram, max = max_ngram))}
punct_prob_corpus = Corpus(VectorSource(punct_prob$description))
punct_prob_tdm <- TermDocumentMatrix(punct_prob_corpus, control = list(tokenize = BigramTokenizer, removePunctuation=FALSE))
inspect(punct_prob_tdm)
check the results - no punctuation ....
Docs
Terms 1 2 3 4
angle head 2 1 for 2 35mm bur 1 0 0 0
contra angle head 2 1 for 2 35mm 1 0 0 0
line mini p b f o trip spray 0 1 0 0
line power p b f o trip spr 0 0 1 0
titanium line mini p b f o trip 0 1 0 0
titanium line power p b f o trip 0 0 1 0
Thanks for any help in advance :)
source to share
The problem is not so much with termdocumentmatrix as with RWEKA-based ngram tokenizer. Rweka removes punctuation when performing tokenization.
If you use the nlp tokenizer, it preserves punctuation marks. See code below.
PS I've removed one place in your third text line so that PB is PB as in line 2.
library(tm)
punct_prob = data.frame(description = tolower(c("CONTRA ANGLE HEAD 2:1 FOR 2.35mm BUR",
"TITANIUM LINE MINI P.B F.O. TRIP SPRAY",
"TITANIUM LINE POWER P.B F.O. TRIP SPR",
"MEDESY SPECIAL ITEM")))
punct_prob$description = as.character(punct_prob$description)
max_ngram = max(sapply(strsplit(punct_prob$description, " "), length))
punct_prob_corpus = Corpus(VectorSource(punct_prob$description))
NLPBigramTokenizer <- function(x) {
unlist(lapply(ngrams(words(x), max_ngram), paste, collapse = " "), use.names = FALSE)
}
punct_prob_tdm <- TermDocumentMatrix(punct_prob_corpus, control = list(tokenize = NLPBigramTokenizer))
inspect(punct_prob_tdm)
<<TermDocumentMatrix (terms: 3, documents: 4)>>
Non-/sparse entries: 3/9
Sparsity : 75%
Maximal term length: 38
Weighting : term frequency (tf)
Docs
Terms 1 2 3 4
contra angle head 2:1 for 2.35mm bur 1 0 0 0
titanium line mini p.b f.o. trip spray 0 1 0 0
titanium line power p.b f.o. trip spr 0 0 1 0
source to share
The quanteda package is smart enough to tokenize without treating punctuation characters within a word as "punctuation". This makes creating your matrix very easy:
txt <- c("CONTRA ANGLE HEAD 2:1 FOR 2.35mm BUR",
"TITANIUM LINE MINI P.B F.O. TRIP SPRAY",
"TITANIUM LINE POWER P.B F.O. TRIP SPR",
"MEDESY SPECIAL ITEM")
require(quanteda)
myDfm <- dfm(txt, ngrams = 6:8, concatenator = " ")
t(myDfm)
# docs
# features text1 text2 text3 text4
# contra angle head for 2.35mm bur 1 0 0 0
# titanium line mini p.b f.o trip 0 1 0 0
# line mini p.b f.o trip spray 0 1 0 0
# titanium line mini p.b f.o trip spray 0 1 0 0
# titanium line power p.b f.o trip 0 0 1 0
# line power p.b f.o trip spr 0 0 1 0
# titanium line power p.b f.o trip spr 0 0 1 0
If you want to keep the "punctuation" it will be designated as a separate token when it finishes the term:
myDfm2 <- dfm(txt, ngrams = 8, concatenator = " ", removePunct = FALSE)
t(myDfm2)
# docs
# features text1 text2 text3 text4
# titanium line mini p.b f.o . trip spray 0 1 0 0
# titanium line power p.b f.o . trip spr 0 0 1 0
Note that the argument is ngrams
completely flexible and can take a vector of dimensions ngram, as in the first example, which ngrams = 6:8
specifies that it should form 6-, 7-, and 8-grams.
source to share