How to extract tags from HTML using Beautifulsoup in Python

First, if you add convertEntities=bs.BeautifulSoup.HTML_ENTITIES

to

soup = bs.BeautifulSoup(html, convertEntities=bs.BeautifulSoup.HTML_ENTITIES)

      

        
        
        
      

    

then html objects like £

will be converted to the corresponding unicode character eg £

. This will allow you to use a simpler regular expression to determine prices.

Now, given part

, you can find the text content in <div>

with prices using its attribute contents

:

In [37]: part.find(attrs={"class": re.compile('abc')}).contents
Out[37]: [<strike>£3.99</strike>, <br />, u'\xa33.59']

      

        
        
        
      

    

All we need to do is extract the number from each element, or skip it if there is no number:

def parse_price(text):
    try:
        return float(re.search(r'\d*\.\d+', text).group())
    except (TypeError, ValueError, AttributeError):
        return None

price = []
for item in part.find(attrs={"class": re.compile('abc')}).contents:
    item = parse_price(item.string)
    if item:
        price.append(item)

      

        
        
        
      

    

At this point there price

will be a list of 0, 1, or 2 floats. We would like to say

mypart['rrp'], mypart['price'] = price

      

        
        
        
      

    

but it won't work if price

there is []

or contains only one element.

Your method of handling the three cases using if..else

in order is the easiest and arguably the most readable way to proceed. But this is also a bit mundane. If you want something a little shorter, you can do the following:

Since we want to repeat the same price if price

only contains one item, you might think about itertools.cycle .

In the case where price

is an empty list,, []

we want itertools.cycle([None])

, but otherwise we could use itertools.cycle(price)

.

So, to combine both cases into one expression, we could use

price = itertools.cycle(price or [None])
mypart['rrp'], mypart['price'] = next(price), next(price)

      

        
        
        
      

    

The function next

selects the values ​​in the iterator price

one at a time. Since it price

cycles through its values, it never ends; it will just keep entering values ​​in sequence and then, if necessary, starting over again - which is exactly what we want.

A, List, Of, Terms, To, Extract - 1 Feb 2013

can be retrieved again using the attribute contents

:

# jkl
mypart['other'] = [item for item in part.find('p').contents
                   if not isinstance(item, bs.Tag) and item.string.strip()]

      

        
        
        
      

    

So, the complete executable code will look like this:

import BeautifulSoup as bs
import os
import re
import itertools as IT

def parse_price(text):
    try:
        return float(re.search(r'\d*\.\d+', text).group())
    except (TypeError, ValueError, AttributeError):
        return None

filename = os.path.expanduser("~/tmp/file.html")
with open(filename) as fp:
    html = fp.read()

soup = bs.BeautifulSoup(html, convertEntities=bs.BeautifulSoup.HTML_ENTITIES)

for part in soup.findAll('div', attrs={"class": re.compile('(?i)part')}):
    mypart = {}
    # abc
    price = []
    for item in part.find(attrs={"class": re.compile('abc')}).contents:
        item = parse_price(item.string)
        if item:
            price.append(item)

    price = IT.cycle(price or [None])
    mypart['rrp'], mypart['price'] = next(price), next(price)

    # jkl
    mypart['other'] = [item for item in part.find('p').contents
                       if not isinstance(item, bs.Tag) and item.string.strip()]

    print(mypart)

      

        
        
        
      

    

what gives

{'price': 3.59, 'other': [u'A, List, Of, Terms, To, Extract - 1 Feb 2013'], 'rrp': 3.99}