Problems with creaking and filling out a form with a dropdown menu
I need to fill out a simple form using scrapy, but I just can't figure out how to fill out and submit it.
Here is the HTML of the form:
<form action="#" id="historicalQuoteDatePicker" class="ZEITRAUM" method="get">
<fieldset>
<label for="dateStart">Startdatum:</label>
<input type="text" name="dateStart" id="dateStart" value="" class="hasDatepicker">
<img class="ui-datepicker-trigger" src="http://i.onvista.de/d.gif" alt="Klicken Sie hier um ein Datum auszuwählen" title="Klicken Sie hier um ein Datum auszuwählen">
<label for="interval">Zeitraum:</label>
<select name="interval" id="interval">
<option value="M1">1 Monat</option>
<option value="M3">3 Monate</option>
<option value="M6">6 Monate</option>
<option value="Y1" selected="selected">1 Jahr</option>
<option value="Y3">3 Jahre</option>
<option value="Y5">5 Jahre</option>
</select>
</fieldset>
<span class="button button-purple button-tiny">
<input type="submit" value="Anzeigen">
</span>
</form>
I can just fill out simple search forms. However with this I have tried everything and it still doesn't work. I tried to use the clickdata parameter, but it needs the "name" attribute of the button, which is not listed here.
Here is the code I've tried so far:
def history_popup(self, response):
yield FormRequest.from_response(response,
formxpath="//input[@id='dateStart']",
formdata={"dateStart":"09.08.2013"},
callback=self.history_miner)
I know this is incomplete, but I hope I am on the right track. My question is, how can I click a button and also select one of the options from the dropdown menu?
Any help is greatly appreciated! Thank!
source to share
1) FormRequest clicks on the first item that can be clicked:
The policy is to automatically simulate the default click for any form control that looks like a clickable, such as a.
However, it is possible to select the element to be clicked via clickdata, but it does not require a name attribute, any attribute will work, including the type attribute. In this case, you can do this:
clickdata = { "type": "Submit" }
2) You can "select" one of the options from the dropdown menu in the same way you set the value for the inputs, ei "select_name": "option_text". However, please note that this method sets the value of the combobox as you specify as option_text, even if this parameter does not exist.
formdata = { "interval" : "Jahr" }
3) Finally, the formxpath value must point to the form element, otherwise you will get an error. The FormRequest way is that it finds the form, finds the IN elements that match the names in the formdata, and populates those elements with the corresponding data from the formdata. I believe your formxpath should be:
formxpath="//form[@id='historicalQuoteDatePicker']"
All together now:
FormRequest.from_response(
response,
formxpath="//form[@id='historicalQuoteDatePicker']",
formdata={
"dateStart":"09.08.2013",
"interval" : "Jahr" },
clickdata = { "type": "Submit" },
callback=self.history_miner
)
This has worked for me in the recent past, good luck! Let me know if this works for you. Not very helpful, but sufficient documentation for FormRequest: http://doc.scrapy.org/en/0.24/topics/request-response.html#scrapy.http.FormRequest.from_response
source to share