Clever Geek Handbook

Calculate sparking match conditions using scala

I want to calculate co-occurance ratios using scala. But I am facing some problems.

This is my code:

val path = "pg100.txt"
val words = sc.textFile(path).map(_.toLowerCase.split("[\\s*$&#/\"'\\,.:;?!\\[\\](){}<>~\\-_]+").map(_.trim).sorted)
val coTerm = words.map{ line =>
    for{ 
        i <-0 until line.length
        j <- (i+1) until line.length
    } {
        ((line(i), line(j)), 1)
    }}

The expected output should be:

coTerm.collect
res48: Array[Unit] = Array(((word1, word2), 1), ((word1, word3), 1), ((word2, word3), 1)...

But my conclusion is as follows:

coTerm.collect
res51: Array[Unit] = Array((), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ()....

I don't know why I can use the println function in .map to print pairs of words, but I cannot emit output.

+3


source to share


1 answer


The reason is that you are not actually returning any records from you map

.

Use yield

to revert records to for

as shown below:



val coTerm = words.map{ line =>
for{ 
    i <-0 until line.length
    j <- (i+1) until line.length
} yield {
    ((line(i), line(j)), 1)
}}
+2


source






More articles:


All Articles