Clever Geek Handbook

Difference between cURL and web browser?

I am trying to fetch a webpage from the following url:

http://www.medicare.gov/find-a-doctor/provider-results.aspx?searchtype=OHP&specgrpids=922&loc=43615&pref=No&gender=Unknown&dist=25&lat=41.65603&lng=-83.66676

It works when I paste it into my browser, but when I run it through cURL I get a page with the following error: "One or more of the requested URL query string parameters are incorrect or have an unexpected value, please correct and try again."

It doesn't seem to matter if I provide a different userAgent or referrer. There is a redirect, so I am using CURLOPT_FOLLOWLOCATION.

Here is my code:

$ch = curl_init($page);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

$html = curl_exec($ch);
curl_close($ch);
echo $html;

Any thoughts on why such a request would work in the browser and not cURL?

+1


source to share


2 answers


The problem was the use of cookies. The ASP.NET_SessionId cookie had to be set to respond to this particular site. I added the following to my cURL request:

curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIE, 'ASP.NET_SessionId=ho1pqwa0nb3ys3441alenm45; path=/; domain=www.medicare.gov');


I don't know if any session ID will work, but he tried a couple of random ones and they all worked.

+2


source


Your browser is sending cookies that are not supported by cURL. Check the cookies you send to the site using browser tools or Fidler - you will need to send the same.



+2


source






More articles:


All Articles