The scraper picks up a few titles and ignores the rest
I wrote a tiny script in python using xpath to clear the headers displayed in the left side area from yahoo finance. There are multiple titles on this page, but when I run my script, I only get three titles. I don't want to use selenium in this case as I get at least minimal results. I tried to use "time" in my script to make the web page load completely. However, it seems that "time" has nothing to do with it. How can I get all headers from this webpage except using selenium? Here's what I've tried:
import time
import requests
from lxml import html
response = requests.get("https://finance.yahoo.com/").text
time.sleep(5)
root = html.fromstring(response)
for title in root.xpath("//a[u[@class='StretchedBox']]/text()"):
print(title)
+3
source to share
1 answer
I tried my code and apparently I managed to get a lot more than just 3 titles.
It is possible that Yahoo just changed the dynamics of its website so that your code is able to parse more titles.
Output:
This could get 'very ugly, very fast' for Boeing, says one market expert
Toys R Us submits plan to liquidate its US business, will close or sell all US stores
Sears Gains After Tax Benefit Helps Retailer Swing to a Profit
Blood, Fraud and Money Led to Theranos CEO Fall From Grace
Why Cisco Systems Is Looking Like a Must-Own Stock
Realty Income (O) Announces Hike in Monthly Dividend Payment
Tillerson exit has 'very big implications' for oil prices: RBC’s Helima Croft
Kevin O'Leary shares his No. 1 trick for tipping at a res...
For J C Penney Co. Inc. Stock Survival is the Victory
Why Ford (F) Stock Popped Despite Recalling 1.4 Million Vehicles
Toys 'R' Us Liquidation Reveals a Dangerous Trend Forming in U.S. Retailing
Theranos CEO Elizabeth Holmes settles with SEC, agrees to...
Manafort Urges Judge to Dismiss Laundering and Lobbying Case
‘Dow Theory’ Warning Signal Spells Trouble for These 5 Stocks
General Electric Company Stock Is Ugly, But It Has Potential for the Gambler
This popular Dow stock looks ready to join GE on the discard pile
Intel Broadcom Buyout Rumors Put These 3 Stocks at Risk
The SEC isn't sending Theranos CEO Elizabeth Holmes to jail for fraud — here are her punishments
Former Equifax CIO Charged With Insider Trading in Data Breach
Lyft is teaming up with automotive industry giant Magna to develop self-driving cars — and getting a $200 million investment (MGA, GM, GOOGL)
Google, Apple face EU law on business practices
Disney Elevates Two Top Executives in Possible CEO Bake-Off
Nokia Oyj Stock Could Break out over the Next Two Years
Wall Street gains as White House plays down trade war chances
U.S. retail sales falter; inflation creeping higher
0
source to share