Curl cannot post data to php page
I am trying to submit a value to a form to get data from an aspx page, but I cannot get the data
-
First url to open:
$url="https://www.clarkcountycourts.us/Anonymous/default.aspx"
(this is where cookies are set) -
Second url to find data
$url3 ="https://www.clarkcountycourts.us/Anonymous/Search.aspx"
When I post data, I just get the html of page 3 instead of the data from page 2
Below is my code, what am I missing? please guide me.
<?php
$url="https://www.clarkcountycourts.us/Anonymous/default.aspx";
$cookie="cookie.txt";
$url3 ="https://www.clarkcountycourts.us/Anonymous/Search.aspx";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie); // <-- add this line
curl_setopt ($ch, CURLOPT_REFERER, $url);
$result = curl_exec ($ch);
$SearchBy = "2";
$AttorneySearchMode="Name";
$LastName= "Smith";
$FirstName= "William";
$MiddleName ="";
$CaseStatusType= "0";
$SortBy= "fileddate";
$DateFiledOnAfter = "";
$DateFiledOnBefore = "";
$SearchSubmit ="Search";
$fields = array(
'SearchBy' => urlencode($SearchBy),
'AttorneySearchMode' => urlencode($AttorneySearchMode),
'LastName' => urlencode($LastName),
'FirstName' => urlencode($FirstName),
'MiddleName' => urlencode($MiddleName),
'CaseStatusType' => urlencode($CaseStatusType),
'SortBy' => urlencode($SortBy),
'DateFiledOnAfter' => urlencode($DateFiledOnAfter),
'DateFiledOnBefore' => urlencode($DateFiledOnBefore),
'SearchSubmit' => urlencode($SearchSubmit)
);
$fields_string = "";
foreach ($fields as $key=>$value) {
$fields_string .= $key.'='.$value.'&';
}
$fields_string = rtrim($fields_string, '&');
curl_setopt($ch, CURLOPT_URL, $url3);
curl_setopt($ch, CURLOPT_HEADER, true );
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch,CURLOPT_POST, 1);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
$result2 = curl_exec ($ch);
print_r($result2);
source to share
It took a while to figure it out - but the code below should repeat the results after searching based on your original request from William Smith as an attorney. Hope this makes sense ...
define('ROOT','c:/wwwroot');
$url_base='https://www.clarkcountycourts.us';
$url_login=$url_base.'/Anonymous/Login.aspx?ReturnUrl=/Anonymous/default.aspx';
$url_start=$url_base.'/Anonymous/default.aspx';
$url_search=$url_base.'/Anonymous/Search.aspx?ID=200&NodeID=101,103,104,105,500,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,797,798&NodeDesc=All%20Courts';
$cookiejar=tempnam( sys_get_temp_dir(), 'cookiejar_' );
$lastname='smith';
$firstname='william';
$formparams=array(
'__VIEWSTATEGENERATOR',
'__EVENTVALIDATION',
'__EVENTTARGET',
'__EVENTARGUMENT',
'__VIEWSTATE'
);
$stdparams=array(
'SearchBy' => '2',
'AttorneySearchMode' => 'Name',
'LastName' => $lastname,
'FirstName' => $firstname,
'MiddleName' => '',
'CaseStatusType' => '0',
'SortBy' => 'fileddate',
'DateFiledOnAfter' => '',
'DateFiledOnBefore' => '',
'SearchSubmit' => 'Search',
'CaseSearchMode' => 'CaseNumber',
'CaseSearchValue' => '',
'CitationSearchValue' => '',
'CourtCaseSearchValue' => '',
'PartySearchMode' => 'Name',
'cboState' => 'AA',
'DateOfBirth' => '',
'DriverLicNum' => '',
'chkCriminal' => 'on',
'chkFamily' => 'on',
'chkCivil' => 'on',
'chkProbate' => 'on',
'chkDtRangeCriminal' => 'on',
'chkDtRangeFamily' => 'on',
'chkDtRangeCivil' => 'on',
'chkDtRangeProbate' => 'on',
'chkCriminalMagist' => 'on',
'chkFamilyMagist' => 'on',
'chkCivilMagist' => 'on',
'chkProbateMagist' => 'on',
'DateSettingOnAfter' => '',
'DateSettingOnBefore' => '',
'SearchType' => 'PARTY',
'SearchMode' => 'NAME',
'NameTypeKy' => 'ALIAS',
'BaseConnKy' => 'AT',
'StatusType' => 'true',
'ShowInactive' => '',
'AllStatusTypes' => 'true',
'CaseCategories' => '',
'RequireFirstName' => 'False',
'CaseTypeIDs' => '',
'HearingTypeIDs' => ''
);
/* A fudge I know, manually build this param - I think it might be constructed using javascript on submission */
$stdparams['SearchParams']='SearchBy~~Search+By:~~Attorney~~Attorney||AttorneyNameOption~~Party+Search+Mode:~~Name~~Name||LastName~~Last+Name:~~'.$lastname.'~~'.$lastname.'||FirstName~~First+Name:~~'.$firstname.'~~'.$firstname.'||AllOption~~Case+Status:~~0~~All||selectSortBy~~Sort+By:~~Filed+Date~~Filed+Date ';
/* Initialise curl and set basic options */
$curl=curl_init();
if( parse_url( $url_base,PHP_URL_SCHEME )=='https' ){
curl_setopt( $curl, CURLOPT_SSL_VERIFYHOST, 2 );
curl_setopt( $curl, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt( $curl, CURLOPT_CAINFO, realpath( ROOT . '/cacert.pem' ) );
}
curl_setopt( $curl, CURLINFO_HEADER_OUT, true );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $curl, CURLOPT_AUTOREFERER, true );
curl_setopt( $curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, array('Host: '.parse_url( $url_login,PHP_URL_HOST ),'Connection: keep-alive') );
/* Stage 1 - Get initial page to obtain session cookies etc and store in cookiejar...mmmmm, cookies! */
curl_setopt( $curl, CURLOPT_URL, $url_login );
curl_setopt( $curl, CURLOPT_COOKIEJAR, $cookiejar );
curl_setopt( $curl, CURLOPT_COOKIESESSION, true );
$res=curl_exec( $curl );
/* Step 2 - GET form page and analyse input elements */
curl_setopt( $curl, CURLOPT_URL, $url_search );
curl_setopt( $curl, CURLOPT_AUTOREFERER, true );
curl_setopt( $curl, CURLOPT_COOKIEFILE, $cookiejar );
curl_setopt( $curl, CURLOPT_COOKIESESSION, false );
$res=curl_exec( $curl );
/* utility class that simplifies getting DOMDocument with error checking etc */
$dom=new htmldom( $res );
$html=$dom->gethtml();
$col=$html->getElementsByTagName('input');
/* we need to know values for specific fields - add to array of params to be submitted */
foreach( $col as $index => $node ){
if( in_array( $node->getAttribute('name'), $formparams ) ) {
$stdparams[ $node->getAttribute('name') ]=$node->getAttribute('value');
}
}
/* Not sure how the 'SearchParams' field value is calculated so calculated manually above..... */
/* Stage 3 - POST */
/* Prepare search query */
$info=$res='';
$querystring=http_build_query( $stdparams, '', '&' );
curl_setopt( $curl, CURLOPT_URL, $url_search );
curl_setopt( $curl, CURLOPT_REFERER, $url_search );
curl_setopt( $curl, CURLOPT_COOKIEFILE, $cookiejar );
curl_setopt( $curl, CURLOPT_COOKIESESSION, false );
curl_setopt( $curl, CURLOPT_POST, true );
curl_setopt( $curl, CURLOPT_POSTFIELDS, $querystring );
$res=curl_exec( $curl );
$info=curl_getinfo( $curl );
/* The final search results - you could manipulate the dom to get specific items if you wished */
echo '<pre>';
print_r( $res );
echo '</pre>';
/* Close curl */
curl_close( $curl );
For completeness, the above utility class.
class htmldom{
private $html;
public function __construct( $data=false, $convert=true ){
try{
if( !$data ) return false;
libxml_use_internal_errors( true );
$this->html = new DOMDocument('1.0','utf-8');
$this->html->validateOnParse=false;
$this->html->standalone=true;
$this->html->preserveWhiteSpace=true;
$this->html->strictErrorChecking=false;
$this->html->substituteEntities=false;
$this->html->recover=true;
$this->html->formatOutput=false;
$this->html->loadHTML( $convert ? mb_convert_encoding( $data, 'utf-8' ) : $data );
$parse_errs=serialize( libxml_get_last_error() );
libxml_clear_errors();
}catch( Exception $e ){
die( $e->getMessage() );
}
}
public function gethtml(){
return $this->html;
}
}
source to share
You have two things here.
-
You are not using the cookie jar correctly.
CURLOPT_COOKIEJAR
used when you want to store cookies installed from the server. So cURL doesn't work like a programmable browser. If you want to establish a session (cookie) then the first request usesCURLOPT_COOKIEJAR
and subsequent requests will useCURLOPT_COOKIEFILE
CURLOPT_COOKIEJAR
cookies are placed in the jar from the server.CURLOPT_COOKIEFILE
cookies are sent from the cookie back to the server .Using
CURLOPT_COOKIEJAR
outside of the first connection to the server will just overwrite the cookie jar every time.
Cookies and cURL have been a constant source of frustration for me, so I can make a big deal on this first question.
-
When I manually search this server, the forms end up being submitted to URLs like this:
Also, when I submit the form in my browser, there are many other fields that are submitted as well.
Some of them look important - __VIEWSTATE
, __EVENTVALIDATION
, __VIEWSTATEGENERATOR
.
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE:/wEPDwULLTE3NDQ0MTE0OTgPZBYCZg9kFgICAQ8WAh4HVmlzaWJsZWgWAgIDDw9kFgIeB29ua2V5dXAFJnRoaXMudmFsdWUgPSB0aGlzLnZhbHVlLnRvTG93ZXJDYXNlKCk7ZGSj5Ki6eeFgaBABR8RCqLZPLI/WZw==
__VIEWSTATEGENERATOR:B40748C4
__EVENTVALIDATION:/wEWAgKmuqaTBQKYxoa5CF+xFBuOB81vr2zAb/xD7VZ3+CQV
SearchBy:2
CaseSearchMode:CaseNumber
CaseSearchValue:
CitationSearchValue:
CourtCaseSearchValue:
PartySearchMode:Name
AttorneySearchMode:Name
LastName:Smith
FirstName:William
cboState:AA
MiddleName:
DateOfBirth:
DriverLicNum:
CaseStatusType:0
DateFiledOnAfter:
DateFiledOnBefore:
chkCriminal:on
chkFamily:on
chkCivil:on
chkProbate:on
chkDtRangeCriminal:on
chkDtRangeFamily:on
chkDtRangeCivil:on
chkDtRangeProbate:on
chkCriminalMagist:on
chkFamilyMagist:on
chkCivilMagist:on
chkProbateMagist:on
DateSettingOnAfter:
DateSettingOnBefore:
SortBy:fileddate
SearchSubmit:Search
SearchType:PARTY
SearchMode:NAME
NameTypeKy:ALIAS
BaseConnKy:AT
StatusType:true
ShowInactive:
AllStatusTypes:true
CaseCategories:
RequireFirstName:False
CaseTypeIDs:
HearingTypeIDs:
SearchParams:SearchBy~~Search By:~~Attorney~~Attorney||AttorneyNameOption~~Party Search Mode:~~Name~~Name||LastName~~Last Name:~~Smith~~Smith||FirstName~~First Name:~~William~~William||AllOption~~Case Status:~~0~~All||selectSortBy~~Sort By:~~Filed Date~~Filed Date
Conclusion
I believe you will need to load the search page in order to get the values to be placed in some of these hidden fields (parsing the page) and then submit the search page again as a POST with the hidden fields filled in.
source to share