How to split Varnish log for Virtual host

Varnish is a great tool that accelerates the performance of your site with a simple configuration and a low usage of resources, compared to the result you’ll get. Just because Varnish is designed and engineered to be fast by default it does not write any log to disk, outrageous you could think … to be honest the logs are available in a memory segment when this memory is filled, Varnish starts from the beginning and overwrites the oldest data in an infinite loop.

So in reality there are logs and they are available for some time, and this solution is much, much faster then logging to a file and it doesn’t require disk space, but how we can get them on file, possibly divided by virtual host as do Apache or Nginx ?

The first tool that we can use to see the logs is varnishlog, from the varnish site:

varnishlog is one of the programs you can use to look at what Varnish is logging. Varnishlog gives you the raw logs, everything that is written to the logs. There are other clients as well, we’ll show you these later.

In the terminal window you started varnish now type varnishlog and press enter.
….

11 SessionOpen  c 127.0.0.1 58912 0.0.0.0:8080
11 ReqStart     c 127.0.0.1 58912 595005213
11 RxRequest    c GET
11 RxURL        c /
11 RxProtocol   c HTTP/1.1
11 RxHeader     c Host: localhost:8080
11 RxHeader     c Connection: keep-alive

The first column is an arbitrary number, it defines the request. Lines with the same number are part of the same HTTP transaction. The second column is the tag of the log message. All log entries are tagged with a tag indicating what sort of activity is being logged. Tags starting with Rx indicate Varnish is recieving data and Tx indicates sending data.

The third column tell us whether this is is data coming or going to the client (c) or to/from the backend (b). The forth column is the data being logged.

So varnishlog is an interesting tool for a live session when you want to verify if your varnish is doing what you expect, but not so useful when you want to put the logs into a file.

So let’s move to Varnishncsa, another varnish tool that can be much more useful for our goal, Varnishncsa display Varnish logs in Apache / NCSA combined log format

The basic usage from the command line is:

varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid

-a: To append the logs to an already existing file
-w: To write the logs to the /var/log/varnish/access.log file
-D: To run varnishncsa as a daemon
-P: To write the PID file in the /var/run/ folder

The problem wIth this command is that we’ll get just 1 access.log file that is good as long as we manage with Varnish just 1 Virtual hosts, but could be not so useful if we want to have 1 file for every Virtual host defined in our Web server.

A good solution that I’ve found in the blog blox.xcir.net is to have 1 dedicated daemon for every virtual host and use the flag -m that is used to perform a regex match to the tag’s log entry so if we have 2 virtual host: linuxaria.com and linuxaria.org we could start 2 daemons with:

varnishncsa -m "RxHeader:^Host: linuxaria.com$" -a -w /var/log/varnish/linuxaria.com.access_log -D
varnishncsa -m "RxHeader:^Host: linuxaria.org$" -a -w /var/log/varnish/linuxaria.org.access_log -D

And these 2 daemons will generate 2 access_log files with the entry only for these Virtual hosts, now you can use logrotate to rotate the logs every day, just remember to restart the daemon in this case, or statistics software like awstats.

Popular Posts:

None Found