Clever Geek Handbook

Can't parse xml input with logstash filter

Hi I am trying to parse the following xml: 
<msg time='2014-08-04T14:36:02.136+03:00' org_id='oracle' comp_id='rdbms'
 msg_id='opistr_real:953:3971575317' type='NOTIFICATION' group='startup'
 level='16' host_id='linux4_l' host_addr='127.0.0.1'
 pid='8986' version='1'>
 <txt>Starting ORACLE instance (normal)
 </txt>
</msg>

using this config:

 input {
   stdin {
    type => "stdin-type"
  }
  }
 filter { multiline {
                       pattern => "^\s|</msg>|^[A-Za-z].*"
                        what => "previous"
                }
                xml {
                        store_xml => "false"
                        source => "message"
                        xpath => [
                                "/msg/@client_id", "msg_client_id",
                                "/msg/@host_id", "msg_host_id",
                                "/msg/@host_addr", "msg_host_addr",
                                "/msg/@level", "msg_level",
                                "/msg/@module", "msg_module",
                                "/msg/@msg_id", "msg_msg_id",
                                "/msg/@pid", "msg_pid",
                                "/msg/@org_id", "msg_org_id",
                                "/msg/@time", "msg_time",
                                "/msg/@level", "msg_level",
                                "/msg/txt/text()","msg_txt"
                        ]
               }
                date {
                        match => [ "msg_time", "ISO8601" ]
                }
                mutate {
                        add_tag => "%{type}"
                }
}
output { elasticsearch { host => localhost } stdout { codec => rubydebug } }

but when i run logstash i get the following error:

{:timestamp=>"2014-09-04T17:28:39.428000+0300", :message=>"Exception in filterworker", "exception"=>#<NoMethodError: undefined method `split' for ["msg_level", "msg_level"]:Array>, "backtrace"=>["/opt/logstash/lib/logstash/util/accessors.rb:19:in `parse'", "/opt/logstash/lib/logstash/util/accessors.rb:15:in `get'", "/opt/logstash/lib/logstash/util/accessors.rb:59:in `store_path'", "/opt/logstash/lib/logstash/util/accessors.rb:55:in `lookup'", "/opt/logstash/lib/logstash/util/accessors.rb:34:in `get'", "/opt/logstash/lib/logstash/event.rb:127:in `[]'", "/opt/logstash/lib/logstash/filters/xml.rb:117:in `filter'"

.... "/opt/logstash/lib/logstash/pipeline.rb: 143: in` start_filters'"] ,: level =>: error} {: timestamp => "2014-09-04T17: 30: 47.805000 + 0300 ",: message =>" Interrupt received. End of pipeline. ",: level =>: warn}

+3


source to share


3 answers


I found my problem, I duplicated the parsing on xpath twice, / msg @level apper.



+1


source


The codec is multiline

not suitable for this type of file, but you would use something like:

multiline {
      pattern => '<msg'
      negate => true
      what => previous
}


The problem is that the last event in the file doesn't get out until the next event (so you lose the last event in the file).

0


source


The problem is that the last event in the file doesn't get out until the next event (so you lose the last event in the file).

Better match the end tag.

multiline {
    pattern => "</msg>$"
    negate => true
    what => next
}
0


source






More articles:


All Articles