Unable to parse links from xml content
I wrote a script in python combined with xpath to clean up links from a site with xml content. Since I have never worked with xml so I cannot figure out where I am making mistakes. Thanks in advance for providing me with a workaround. Here's what I'm trying:
import requests
from lxml import html
response = requests.get("https://drinkup.london/sitemap.xml").text
tree = html.fromstring(response)
for item in tree.xpath('//div[@class="expanded"]//span[@class="text"]'):
print(item)
xml content that contains links:
<div xmlns="http://www.w3.org/1999/xhtml" class="collapsible" id="collapsible4"><div class="expanded"><div class="line"><span class="button collapse-button"></span><span class="html-tag"><url></span></div><div class="collapsible-content"><div class="line"><span class="html-tag"><loc></span><span class="text">https://drinkup.london/</span><span class="html-tag"></loc></span></div></div><div class="line"><span class="html-tag"></url></span></div></div><div class="collapsed hidden"><div class="line"><span class="button expand-button"></span><span class="html-tag"><url></span><span class="text">...</span><span class="html-tag"></url></span></div></div></div>
The runtime error is shown below:
value = etree.fromstring(html, parser, **kw)
File "src\lxml\lxml.etree.pyx", line 3228, in lxml.etree.fromstring (src\lxml\lxml.etree.c:79593)
File "src\lxml\parser.pxi", line 1843, in lxml.etree._parseMemoryDocument (src\lxml\lxml.etree.c:119053)
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
+3
source to share
1 answer
Switch to which returns bytes instead of which returns unicode : .content
.text
import requests
from lxml import html
response = requests.get("https://drinkup.london/sitemap.xml").content
tree = html.fromstring(response)
for item in tree.xpath('//url/loc/text()'):
print(item)
Notice the fixed XPath expression.
+2
source to share