Failed to retrieve data using Import.io from Amazon webpage where data is loaded into page via Ajax
Does anyone know how to fetch data from a web page using Import.io where the data is loaded into the page via Ajax. I am unable to retrieve data from the pages below. No problem retrieving first page data. But how to go to the second page and retrieve data from the second page. http://www.amazon.com/gp/aag/main?ie=UTF8&asin=&isAmazonFulfilled=&isCBA=&marketplaceID=ATVPDKIKX0DER&orderID=&seller=A13JB7253Q5S1B ?
source to share
The data on this page is deployed using an interesting combination of technologies; it relies heavily on server side code and Javascript. This type of page can be a problem, however there are always methods to get the data. For example, some sellers have a page like this:
Which extracts data very easily, even using a magic algorithm - https://magic.import.io/?site=http:%2F%2Fwww.amazon.co.uk%2Fgp%2Fnode%2Findex.html%3Fie%3DUTF8%26marketplaceID % 3DA1F83G8C2ARO7P% 26me% 3DA2WO1PQ2OIOIGM% 26merchant% 3DA2WO1PQ2OIOIGM
I had to strip redirect = true from urls before it crashes - just FYI.
While some stores don't have such a URL, it's a bit of a pain and URLs can be tricky to identify.
We help our customers create custom APIs when data is very important to them, so don't hesitate to contact them. I guess the bigger solution would be to create a dataset / API based on the categories you are interested in, and then filter that larger dataset (python or CSV style) by vendor name. This will probably work!
source to share