Nutch error No agents listed in 'http.agent.name'

Question

Nutch error No agents listed in 'http.agent.name'

I am using nutch2.2.1. The log file generates the following error

ERROR protocol.RobotRulesParser - The agent we advertise (nutch-spider-2.2.1) is not listed first in the http.robots.agents property!

My nutch-site.xml (for the property above)

<property>
<name>http.agent.name</name>
<value>nutch-spider-2.2.1</value>
</property>

my nutch-default.xml

<property>
<name>http.agent.name</name>
<value></value>
</property>

Where is the actual problem? Read it clearly (explain correctly). This question is posted here , but I have to forward this question (if necessary) to post it again.

+3

apache web-crawler nutch

Shafiq 24 nov. 14 at 11:48

source to share

1 answer

hqc · Accepted Answer · 2014-12-07T13:33:41+0000

You will add a property "http.robots.agents" and put http.agent.name as the first agent name and keep the default * at the end of the list. Just:

<property>
     <name>http.robots.agents</name>
     <value>nutch-spider-2.2.1,*</value>
</property>

Nutch error No agents listed in 'http.agent.name'

More articles: