Clever Geek Handbook

PHP Curl HTTP requests with proxy fail 95% of the times, why?

Using the php curl command I want to request easy access to https://www.instagram.com ".

$curl->get("https://www.instagram.com");

When I don't use Proxy for my curl it usually exits (sometimes curl breaks the first time and then retries and it goes through the second time).

< Content-Type: text/html
< Date: Fri, 12 Jun 2015 11:35:56 GMT
< Location: https://instagram.com/
* Server nginx is not blacklisted
< Server: nginx
< Content-Length: 178
< Connection: keep-alive
<
* Ignoring the response-body
* Connection #1 to host www.instagram.com left intact
* Issue another request to this URL: 'https://instagram.com/'
* Hostname was NOT found in DNS cache
*   Trying 54.88.218.232...
* Connected to instagram.com (54.88.218.232) port 443 (#2)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
*        server certificate verification SKIPPED
*        common name: *.instagram.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=CA,L=Menlo Park,O=Instagram LLC,CN=*.instagram.com
*        start date: Tue, 14 Apr 2015 00:00:00 GMT

*        expire date: Thu, 15 Oct 2015 12:00:00 GMT

*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3
*        compression: NULL
*        cipher: AES-128-CBC
*        MAC: SHA1
> GET / HTTP/1.1
Host: instagram.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Connection: Keep-Alive
Cache-Control:no-cache
Content-type: application/x-www-form-urlencoded;charset=UTF-8
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0

< HTTP/1.1 200 OK
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Encoding: gzip
< Content-Language: en
< Content-Type: text/html
< Date: Fri, 12 Jun 2015 11:35:59 GMT
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Pragma: no-cache
* Added cookie csrftoken="c44fef8af558485be2ff78da940bdfd6" for domain instagram.com, path /, expire 1465558381
< Set-Cookie: csrftoken=c44fef8af558485be2ff78da940bdfd6; expires=Fri, 10-Jun-2016 11:35:59 GMT; Max-Age=31449600; Path=/
* Added cookie mid="VXrEHwAEAAE0R2ZgV8aoF27i7VD7" for domain instagram.com, path /, expire 2064828781
< Set-Cookie: mid=VXrEHwAEAAE0R2ZgV8aoF27i7VD7; expires=Thu, 07-Jun-2035 11:35:59 GMT; Max-Age=630720000; Path=/
< Vary: Cookie, Accept-Language, Accept-Encoding
< X-Frame-Options: SAMEORIGIN
< Content-Length: 3049
< Connection: keep-alive
<
* Connection #2 to host instagram.com left intact
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en" class="no-js lt-ie9 lt-ie8 lt-ie7 not-logged-in "> <![endif]-->
<!--[if IE 7]>         <html lang="en" class="no-js lt-ie9 lt-ie8 not-logged-in "> <![endif]-->
<!--[if IE 8]>         <html lang="en" class="no-js lt-ie9 not-logged-in "> <![endif]-->
<!--[if gt IE 8]><!--> <html lang="en" class="no-js not-logged-in "> <!--<![endif]-->

The problem is I am using a proxy for my curl, 95% of the time it hangs on "found 173 certificates in / etc / ssl / certs / ca -certificates.crt" and then it gets timeout.

* Rebuilt URL to: https://www.instagram.com/
* Hostname was found in DNS cache
*   Trying 104.144.1.1...
* Connected to 104.144.1.1 (104.144.1.1) port 21269 (#1)
* Establish HTTP proxy tunnel to www.instagram.com:443
* Proxy auth using Basic with user 'proxyusername'
> CONNECT www.instagram.com:443 HTTP/1.1
Host: www.instagram.com:443
Proxy-Authorization: Basic bW9oYW1tYWdoYToyazMzODczMw==
Proxy-Connection: Keep-Alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Connection: Keep-Alive
Cache-Control:no-cache
Content-type: application/x-www-form-urlencoded;charset=UTF-8
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0

< HTTP/1.0 200 Connection established
<
* Proxy replied OK to CONNECT request
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 1

Why would using a proxy do curl act this way on ubuntu? on Windows the same exact code is executed and executed very quickly and selects the instagram.com page, but on Ubuntu it fails 95% of the time.

Here are the curl options:

curl_setopt ( $this->process, CURLOPT_HTTPHEADER, $this->headers );
curl_setopt ( $this->process, CURLOPT_HEADER, 0 );
curl_setopt ( $this->process, CURLOPT_USERAGENT, $this->user_agent );
curl_setopt ( $this->process, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $this->process, CURLOPT_FOLLOWLOCATION, 1 );
curl_setopt ( $this->process, CURLOPT_POST, 0 );
curl_setopt ( $this->process, CURLOPT_ENCODING, $this->compression );
curl_setopt ( $this->process, CURLOPT_TIMEOUT, 40 );
curl_setopt ( $this->process, CURLOPT_SSL_VERIFYHOST, 0 );
curl_setopt ( $this->process, CURLOPT_SSL_VERIFYPEER, 0 );
curl_setopt ( $this->process, CURLOPT_COOKIEFILE, $this->cookie_file );
curl_setopt ( $this->process, CURLOPT_COOKIEJAR, $this->cookie_file );
curl_setopt ( $this->process, CURLOPT_VERBOSE, 1 );

// Proxy settings
curl_setopt ( $this->process, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt ( $this->process, CURLOPT_PROXY, $url);
curl_setopt ( $this->process, CURLOPT_PROXYPORT, $port);
curl_setopt ( $this->process, CURLOPT_PROXYUSERPWD, $userpass);

Here is the output of phpinfo () on curl:

cURL support => enabled
cURL Information => 7.35.0
Age => 3
Features
AsynchDNS => Yes
CharConv => No
Debug => No
GSS-Negotiate => Yes
IDN => Yes
IPv6 => Yes
krb4 => No
Largefile => Yes
libz => Yes
NTLM => Yes
NTLMWB => Yes
SPNEGO => No
SSL => Yes
SSPI => No
TLS-SRP => Yes
Protocols => dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, pop3, pop3s, rtmp, rtsp, smtp, smtps, telnet, tftp
Host => x86_64-pc-linux-gnu
SSL Version => GnuTLS/2.12.23
ZLib Version => 1.2.8
+3


source to share


1 answer


It seems like you have some kind of problem with your certificates ... give the next try, it has worked for me in the past with similar (but different) problems.

update-ca-certificates -f
apt-get install --reinstall ca-certificates


For reference, here's the man for update-ca-certificates

update-ca-certificates is a program that updates the / etc / ssl / certs directory to store SSL certificates and generates Certificates.crt, a concatenated list of files with a single file.

It reads the file /etc/ca -certificates.conf. Each line gives the name of the CA certificate in the / usr / share / ca -certificates directory to be trusted. Lines starting with "#" are line comments and are thus ignored. Lines starting with "!" not selected, which causes the CA certificate to be deactivated. Certificates must have a .crt extension to be included in renew-CA-certificates.

Also, all certificates with the .crt extension found below / usr / local / share / ca -certificate are also included as implicitly trusted.

0


source






More articles:


All Articles