Find all locations / cities / places in text

If I have a text containing, for example, a newspaper article in Catalan, how can I find all the cities from that text?

I was looking at the nltk package for python and I downloaded the Catalan corpus (nltk.corpus.cess_cat).

What I have at the moment: I have installed everything needed from nltk.download (). An example of what I currently have:

te = nltk.word_tokenize('Tots els gats son de Sant Cugat del Valles.')

nltk.pos_tag(te)

      

The city is "Sant Cugat del Valles". What I get from the output:

[('Tots', 'NNS'),
 ('els', 'NNS'),
 ('gats', 'NNS'),
 ('son', 'VBP'),
 ('de', 'IN'),
 ('Sant', 'NNP'),
 ('Cugat', 'NNP'),
 ('del', 'NN'),
 ('Valles', 'NNP')]

      

NNP seems to indicate nouns whose first letter is uppercase. Is there a way to get places or cities and not all names? Thanks you

+3


source to share


3 answers


You can use the geotext python library for this.

pip install geotext

      

is all that is required to install this library. Usage is as easy as:



from geotext import GeoText
places = GeoText("London is a great city")
places.cities

      

gives the result "London"

The list of cities covered in this library is not extensive, but it does have a good list.

+13


source


You either need to train the Named Object Recognizer (NER), or you can create your own reference.

A simple reference I created and use for tasks like yours:

# -*- coding: utf-8 -*-
import codecs
from lxml.html.builder import DT
import os
import re

from nltk.chunk.util import conlltags2tree
from nltk.chunk import ChunkParserI
from nltk.tag import pos_tag
from nltk.tokenize import wordpunct_tokenize


def sub_leaves(tree, node):
    return [t.leaves() for t in tree.subtrees(lambda s: s.node == node)]


class Gazetteer(ChunkParserI):
    """
    Find and annotate a list of words that matches patterns.
    Patterns may be regular expressions in the form list of tuples.
    Every tuple has the regular expression and the iob tag for this one.
    Before applying gazetteer words a part of speech tagging should
    be performed. So, you have to pass your tagger as a parameter.
    Example:
        >>> patterns = [(u"Αθήνα[ς]?", "LOC"), (u"Νομική[ς]? [Σσ]χολή[ς]?", "ORG")]
        >>> gazetteer = Gazetteer(patterns, nltk.pos_tag, nltk.wordpunct_tokenize)
        >>> text = u"Η Νομική σχολή της Αθήνας"
        >>> t = gazetteer.parse(text)
        >>> print(unicode(t))
        ... (S Η/DT (ORG Νομική/NN σχολή/NN) της/DT (LOC Αθήνας/NN))
    """

    def __init__(self, patterns, pos_tagger, tokenizer):
        """
        Initialize the class.

        :param patterns:
            The patterns to search in text is a list of tuples with regular
            expression and the tag to apply
        :param pos_tagger:
            The tagger to use for applying part of speech to the text
        :param tokenizer:
            The tokenizer to use for tokenizing the text
        """
        self.patterns = patterns
        self.pos_tag = pos_tagger
        self.tokenize = tokenizer
        self.lookahead = 0  # how many words it is possible to be a gazetteer word
        self.words = []  # Keep the words found by applying the regular expressions
        self.iobtags = []  # For each set of words keep the coresponding tag

    def iob_tags(self, tagged_sent):
        """
        Search the tagged sentences for gazetteer words and apply their iob tags.

        :param tagged_sent:
            A tokenized text with part of speech tags
        :type tagged_sent: list
        :return:
            yields the IOB tag of the word with it character, eg. B-LOCATION
        :rtype:
        """
        i = 0
        l = len(tagged_sent)
        inside = False  # marks the I- tag
        iobs = []

        while i < l:
            word, pos_tag = tagged_sent[i]
            j = i + 1  # the next word
            k = j + self.lookahead  # how many words in a row we may search
            nextwords, nexttags = [], []  # for now, just the ith word
            add_tag = False  # no tag, this is O

            while j <= k:
                words = ' '.join([word] + nextwords)  # expand our word list
                if words in self.words:  # search for words
                    index = self.words.index(words)  # keep index to use for iob tags
                    if inside:
                        iobs.append((word, pos_tag, 'I-' + self.iobtags[index]))  # use the index tag
                    else:
                        iobs.append((word, pos_tag, 'B-' + self.iobtags[index]))

                    for nword, ntag in zip(nextwords, nexttags):  # there was more than one word
                        iobs.append((nword, ntag, 'I-' + self.iobtags[index]))  # apply I- tag to all of them

                    add_tag, inside = True, True
                    i = j  # skip tagged words
                    break

                if j < l:  # we haven't reach the length of tagged sentences
                    nextword, nexttag = tagged_sent[j]  # get next word and it tag
                    nextwords.append(nextword)
                    nexttags.append(nexttag)
                    j += 1
                else:
                    break

            if not add_tag:  # unkown words
                inside = False
                i += 1
                iobs.append((word, pos_tag, 'O'))  # it an Outsider

        return iobs

    def parse(self, text, conlltags=True):
        """
        Given a text, applies tokenization, part of speech tagging and the
        gazetteer words with their tags. Returns an conll tree.

        :param text: The text to parse
        :type text: str
        :param conlltags:
        :type conlltags:
        :return: An conll tree
        :rtype:
        """
        # apply the regular expressions and find all the
        # gazetteer words in text
        for pattern, tag in self.patterns:
            words_found = set(re.findall(pattern, text))  # keep the unique words
            if len(words_found) > 0:
                for word in words_found:  # words_found may be more than one
                    self.words.append(word)  # keep the words
                    self.iobtags.append(tag)  # and their tag

        # find the pattern with the maximum words.
        # this will be the look ahead variable
        for word in self.words:  # don't care about tags now
            nwords = word.count(' ')
            if nwords > self.lookahead:
                self.lookahead = nwords

        # tokenize and apply part of speech tagging
        tagged_sent = self.pos_tag(self.tokenize(text))
        # find the iob tags
        iobs = self.iob_tags(tagged_sent)

        if conlltags:
            return conlltags2tree(iobs)
        else:
            return iobs


if __name__ == "__main__":
    patterns = [(u"Αθήνα[ς]?", "LOC"), (u"Νομική[ς]? [Σσ]χολή[ς]?", "ORG")]
    g = Gazetteer(patterns, pos_tag, wordpunct_tokenize)
    text = u"Η Νομική σχολή της Αθήνας"
    t = g.parse(text)
    print(unicode(t))


    dir_with_lists = "Lists"
    patterns = []
    tags = []
    for root, dirs, files in os.walk(dir_with_lists):
        for f in files:
            lines = codecs.open(os.path.join(root, f), 'r', 'utf-8').readlines()
            tag = os.path.splitext(f)[0]
            for l in lines[1:]:
                patterns.append((l.rstrip(), tag))
                tags.append(tag)

    text = codecs.open("sample.txt", 'r', "utf-8").read()
    #g = Gazetteer(patterns)
    t = g.parse(text.lower())
    print unicode(t)

    for tag in set(tags):
        for gaz_word in sub_leaves(t, tag):
            print gaz_word[0][0], tag

      



In if __name__ == "__main__":

you can see an example where I am making templates in code patterns = [(u"Αθήνα[ς]?", "LOC"), (u"Νομική[ς]? [Σσ]χολή[ς]?", "ORG")]

.

Later in the code, I read the files from the directory named Lists

(put it in the folder where you have the code above). The name of each file becomes a Gazetteer tag. So, create files like LOC.txt

with templates for locations ( LOC

tags), PERSON.txt

for faces, etc.

+5


source


You don't need to use NLTK for this. Instead, follow these steps:

  • Divide the text into a list with all words.
  • Divide the cities into a dictionary, where {"Sant Cugat del Valles": ["Sant", "Cugat", "del", "Valles"]}. It should be easy to find a list with all the cities in the area somewhere on the internet or from your local local government.
  • Iterate over the items in the text as a list.

    3.1. Iterate over cities, if the first element matches an element in the text, then check the next element.

Here's some sample code you can run:

text = 'Tots els gats son de Sant Cugat del Valles.'
#Prepare your text. Remove "." (and other unnecessary marks).
#Then split it into a list of words.
text = text.replace('.','').split(' ')      

#Insert the cities you want to search for.
cities =  {"Sant Cugat del Valles":["Sant","Cugat","del","Valles"]} 

found_match = False
for word in text:
    if found_match:        
        cityTest = cityTest
    else:
        cityTest = ''
    found_match = False
    for city in cities.keys():            
        if word in cities[city]:
            cityTest += word + ' '
            found_match = True        
        if cityTest.split(' ')[0:-1] == city.split(' '):
            print city    #Print if it found a city.

      

+4


source







All Articles