Clever Geek Handbook

How to get the final Varnishlog

I want to use Varnishlog with Logstash, Elasticsearch, and Kibana to get an overview of important things like "how long does it take for my server to respond to a request." My problem is that Varnish logs every request on many lines.

eg:.

   14 BackendOpen  b default 127.0.0.1 42697 127.0.0.1 2727
   14 BackendXID   b 722117072
   14 TxRequest    b GET
   14 TxURL        b /images/maske.png
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Host: ecomxenvm01:7272
   14 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   14 TxHeader     b Accept: image/png,image/*;q=0.8,*/*;q=0.5
   14 TxHeader     b Accept-Language: de,en-US;q=0.7,en;q=0.3
   14 TxHeader     b Referer: http://ecomxenvm01:7272/format.css
   14 TxHeader     b X-Varnish: 722117072
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 02 Jul 2015 08:27:38 GMT
   14 RxHeader     b Server: Apache/2.2.15 (CentOS)
   14 RxHeader     b Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   14 RxHeader     b ETag: "60010-111f8-5193f039fed80"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 70136
   14 RxHeader     b Connection: close
   14 RxHeader     b Content-Type: image/png
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 70136
   14 BackendClose b default
   12 ReqStart     c 172.16.145.134 57776 722117072
   12 RxRequest    c GET
   12 RxURL        c /images/maske.png
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: ecomxenvm01:7272
   12 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   12 RxHeader     c Accept: image/png,image/*;q=0.8,*/*;q=0.5
   12 RxHeader     c Accept-Language: de,en-US;q=0.7,en;q=0.3
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Referer: http://ecomxenvm01:7272/format.css
   12 RxHeader     c Connection: keep-alive
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /images/maske.png
   12 Hash         c ecomxenvm01:7272
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 722117072 RFC 120 -1 -1 1435825659 0 1435825658 0 0
   12 VCL_call     c fetch deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   12 ObjHeader    c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   12 ObjHeader    c ETag: "60010-111f8-5193f039fed80"
   12 ObjHeader    c Content-Type: image/png
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.15 (CentOS)
   12 TxHeader     c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   12 TxHeader     c ETag: "60010-111f8-5193f039fed80"
   12 TxHeader     c Content-Type: image/png
   12 TxHeader     c Content-Length: 70136
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 TxHeader     c X-Varnish: 722117072
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Length       c 70136
   12 ReqEnd       c 722117072 1435825658.602955103 1435825658.603794336 0.012183905 0.000789165 0.000050068
   14 BackendOpen  b default 127.0.0.1 42698 127.0.0.1 2727
   14 BackendXID   b 722117073
   14 TxRequest    b GET
   14 TxURL        b /images/arrow.png
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Host: ecomxenvm01:7272
   14 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   14 TxHeader     b Accept: image/png,image/*;q=0.8,*/*;q=0.5
   14 TxHeader     b Accept-Language: de,en-US;q=0.7,en;q=0.3
   14 TxHeader     b Referer: http://ecomxenvm01:7272/format.css
   14 TxHeader     b X-Varnish: 722117073
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 02 Jul 2015 08:27:38 GMT
   14 RxHeader     b Server: Apache/2.2.15 (CentOS)
   14 RxHeader     b Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   14 RxHeader     b ETag: "60009-af-5193f0390ab40"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 175
   14 RxHeader     b Connection: close
   14 RxHeader     b Content-Type: image/png
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 175
   14 BackendClose b default
   12 ReqStart     c 172.16.145.134 57776 722117073
   12 RxRequest    c GET
   12 RxURL        c /images/arrow.png
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: ecomxenvm01:7272
   12 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   12 RxHeader     c Accept: image/png,image/*;q=0.8,*/*;q=0.5
   12 RxHeader     c Accept-Language: de,en-US;q=0.7,en;q=0.3
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Referer: http://ecomxenvm01:7272/format.css
   12 RxHeader     c Connection: keep-alive
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /images/arrow.png
   12 Hash         c ecomxenvm01:7272
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 722117073 RFC 120 -1 -1 1435825659 0 1435825658 0 0
   12 VCL_call     c fetch deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   12 ObjHeader    c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   12 ObjHeader    c ETag: "60009-af-5193f0390ab40"
   12 ObjHeader    c Content-Type: image/png
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.15 (CentOS)
   12 TxHeader     c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   12 TxHeader     c ETag: "60009-af-5193f0390ab40"
   12 TxHeader     c Content-Type: image/png
   12 TxHeader     c Content-Length: 175
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 TxHeader     c X-Varnish: 722117073
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Length       c 175
   12 ReqEnd       c 722117073 1435825658.607635498 1435825658.608191013 0.003841162 0.000532389 0.000023127`

Now my question is, how can I get these lines in one line? Or how can I summarize this query?

Do I need to change something in the config files? Or is there a tool that can help me?

Thanks SM

Config file:

    DAEMON_OPTS="$DAEMON_OPTS -i RxStatus"

    # Configuration file for varnish
    #
    # /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this
    # shell script fragment.
    #

    # Maximum number of open files (for ulimit -n)
    NFILES=131072

    # Locked shared memory (for ulimit -l)
    # Default log size is 82MB + header
    MEMLOCK=82000

    # Maximum number of threads (for ulimit -u)
    NPROCS="unlimited"

    # Maximum size of corefile (for ulimit -c). Default in Fedora is 0
    # DAEMON_COREFILE_LIMIT="unlimited"

    # Set this to 1 to make init script reload try to switch vcl without restart.
    # To make this work, you need to set the following variables
    # explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS,
    # VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE, or in short,
    # use Alternative 3, Advanced configuration, below
    RELOAD_VCL=1

    # This file contains 4 alternatives, please use only one.

    ## Alternative 1, Minimal configuration, no VCL
    #
    # Listen on port 7272, administration on localhost:6082, and forward to
    # content server on localhost:2727.  Use a fixed-size cache file.
    #
    #DAEMON_OPTS="-a :6081 \
    #             -T localhost:6082 \
    #             -b localhost:8080 \
    #             -u varnish -g varnish \
    #             -s file,/var/lib/varnish/varnish_storage.bin,1G"


    ## Alternative 2, Configuration with VCL
    #
    # Listen on port 6081, administration on localhost:6082, and forward to
    # one content server selected by the vcl file, based on the request.  Use a
    # fixed-size cache file.
    #
    #DAEMON_OPTS="-a :6081 \
    #             -T localhost:6082 \
    #             -f /etc/varnish/default.vcl \
    #             -u varnish -g varnish \
    #             -S /etc/varnish/secret \
    #             -s file,/var/lib/varnish/varnish_storage.bin,1G"


    ## Alternative 3, Advanced configuration
    #
    # See varnishd(1) for more information.
    #
    # # Main configuration file. You probably want to change it :)
    VARNISH_VCL_CONF=/etc/varnish/default.vcl
    #
    # # Default address and port to bind to
    # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
    # # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
    # VARNISH_LISTEN_ADDRESS=
    VARNISH_LISTEN_PORT=7272
    #
    # # Telnet admin interface listen address and port
    VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
    VARNISH_ADMIN_LISTEN_PORT=6082
    #
    # # Shared secret file for admin interface
    VARNISH_SECRET_FILE=/etc/varnish/secret
    #
    # # The minimum number of worker threads to start
    VARNISH_MIN_THREADS=50
    #
    # # The Maximum number of worker threads to start
    VARNISH_MAX_THREADS=1000
    #
    # # Idle timeout for worker threads
    VARNISH_THREAD_TIMEOUT=120
    #
    # # Cache file location
    VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
    #
    # # Cache file size: in bytes, optionally using k / M / G / T suffix,
    # # or in percentage of available disk space using the % suffix.
    VARNISH_STORAGE_SIZE=1G
    #
    # # Backend storage specification
    VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
    #
    # # Default TTL used when the backend does not specify one
    VARNISH_TTL=120
    #
    # # DAEMON_OPTS is used by the init script.  If you add or remove options, make
    # # sure you update this section, too.
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
                 -f ${VARNISH_VCL_CONF} \
                 -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
                 -t ${VARNISH_TTL} \
                 -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
                 -u varnish -g varnish \
                 -S ${VARNISH_SECRET_FILE} \
                 -s ${VARNISH_STORAGE}"
    #


    ## Alternative 4, Do It Yourself. See varnishd(1) for more information.
    #
    # DAEMON_OPTS=""

When I find Varnishlog:

[elk@ecomxenvm01 ~]$ sudo locate varnishlog
/etc/rc.d/init.d/varnishlog
/etc/rc.d/rc0.d/K10varnishlog
/etc/rc.d/rc1.d/K10varnishlog
/etc/rc.d/rc2.d/K10varnishlog
/etc/rc.d/rc3.d/K10varnishlog
/etc/rc.d/rc4.d/K10varnishlog
/etc/rc.d/rc5.d/K10varnishlog
/etc/rc.d/rc6.d/K10varnishlog
/usr/bin/varnishlog
/usr/share/man/man1/varnishlog.1.gz
/usr/share/man/man8/varnishlog_selinux.8.gz
+3


source to share


3 answers


You should take a look at this . Special options -I and -i.

The tagged example shows only RxStatus log entries:

varnishlog -i RxStatus



The regular expression example only shows the ReqStart and ReqEnd entries:

varnishlog -I "Req [Start | End]"

+1


source


We are using varnishncsa to generate apache file access log.

varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -P /var/run/varnishncsa.pid -F"%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %D %{Varnish:hitmiss}x"


When adding this parameter to log rotation mode, be sure to use the copy / truncate parameter.

More information https://www.varnish-cache.org/docs/trunk/reference/varnishncsa.html

0


source


Here's some details on how to set this up specifically for varnish: https://www.section.io/varnish-logs-centralized-and-managed-by-elasticsearch-logstash-and-kibana/

Extending effort to varnishlog is exactly the same Start varnishlog as a daemon, write to a file and then push all entries to logstash (possibly via syslog-ng).

Let me know if it helps, would be happy to provide more specific information for your use case.

0


source






More articles:


All Articles