Clever Geek Handbook

PHP cURL, how to add User Agent value or abort servers blocking cURL requests?

I am transferring an array of objects. I have a cURL client (sender) on its own server and listening to the script on a different server that is not under my control. Then I think they are blocking incoming cURL requests because when I test normal HTML <form>

it works. But not through cURL anyway.

So, I think they made some kind of limitation for cURL.

Then my questions are here:

  • Can the server restrict / block incoming cURL requests?
  • If so, can I cheat / change the HTTP (User Agent) header in my cURL initiation script?
  • Or are there other possible stories?

Thank!

+9


source to share


4 answers


  • On the server side, we can block some requests by recognizing header fields (including Refer, cookie, user-agent, etc.) in the http request, ip-address, access frequency. And in most cases, machine generated requests usually have something other than human requests, like no link and cookie, or with a higher access frequency, we can write some rules to deny these requests.

  • According to 1, you can try your best to simulate real requests by filling in the header fields using random and slow rate using more ip addresses. (sounds like an attack)

  • As a rule of thumb, using a lower frequency and don't put a lot of load on your server, follow their access rules, they rarely block your requests.



+4


source 


  $agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)';
  $curl=curl_init();
  curl_setopt($curl, CURLOPT_USERAGENT, $agent);


+17


source


IF you are still facing the problem, follow these steps.

1.

$config['useragent'] = 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0';

curl_setopt($curl, CURLOPT_USERAGENT, $config['useragent']);
curl_setopt($curl, CURLOPT_REFERER, 'https://www.domain.com/');

2.

$dir                   = dirname(__FILE__);
$config['cookie_file'] = $dir . '/cookies/' . md5($_SERVER['REMOTE_ADDR']) . '.txt';

curl_setopt($curl, CURLOPT_COOKIEFILE, $config['cookie_file']);
curl_setopt($curl, CURLOPT_COOKIEJAR, $config['cookie_file']);

NOTE. You need a COOKIES folder in the directory.

3.

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

If this does not solve the problem, then provide an example of I / O / error / etc. Thus, a more accurate solution can be provided.

+16


source


The server cannot block only cURL requests because they are just HTTP requests. Thus, changing the User Agent of your cURL may fix your problem, as the server will think you are connecting through a browser represented in the UA.

0


source






More articles:


All Articles