Difference between cURL and web browser?
I am trying to fetch a webpage from the following url:
http://www.medicare.gov/find-a-doctor/provider-results.aspx?searchtype=OHP&specgrpids=922&loc=43615&pref=No&gender=Unknown&dist=25&lat=41.65603&lng=-83.66676
It works when I paste it into my browser, but when I run it through cURL I get a page with the following error: "One or more of the requested URL query string parameters are incorrect or have an unexpected value, please correct and try again."
It doesn't seem to matter if I provide a different userAgent or referrer. There is a redirect, so I am using CURLOPT_FOLLOWLOCATION.
Here is my code:
$ch = curl_init($page);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
Any thoughts on why such a request would work in the browser and not cURL?
The problem was the use of cookies. The ASP.NET_SessionId cookie had to be set to respond to this particular site. I added the following to my cURL request:
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIE, 'ASP.NET_SessionId=ho1pqwa0nb3ys3441alenm45; path=/; domain=www.medicare.gov');
I don't know if any session ID will work, but he tried a couple of random ones and they all worked.
Your browser is sending cookies that are not supported by cURL. Check the cookies you send to the site using browser tools or Fidler - you will need to send the same.