Traffic interception for memcached for statistics / analysis
I want to set up a statistics monitoring platform to look at a specific service, but I'm not sure how. Processing intercepted data is not a problem for me, just how to do it. One idea was to set up a proxy between the client application and the service so that all TCP traffic goes first to my proxy, the proxy then passes the intercepted messages to the waiting thread / fork to pass the message and get the results. Another is to try and sniff traffic between the client and the service.
My main goal is to avoid any serious loss in transfer speed between client and application, but still get 100% complete communication between client and service.
Environment: UBuntu 8.04
Language: c / c ++
In the background, I was thinking about using a sqlite DB running entirely in memory, or a 20-25MB memcache daemon slave to my process.
Update: Specifically, I'm trying to keep track of key usage for the memcache daemon by keeping the set # / gets success / failure on the key. The idea is that most keys have some sort of separator character [`| _- #] to create a kind of namespace. The idea is to step in between the daemon and the client, split the keys with a separately configured separator, and write statistics to them.
source to share
You didn't mention one approach: you can change memcached or your client to record the statistics you need. This is probably the simplest and cleanest approach.
There are several trade-offs between proxy and libpcap approach:
- If you do the packet capture approach, you have to reassemble the TCP
streams into something usable yourself. OTOH, if your monitor program
gets bogged down, it'll just lose some packets, it won't break the cache.
Same if it crashes. You also don't have to reconfigure anything; packet
capture is transparent.
- If you do the proxy approach, the kernel handles all the TCP work for
you. You'll never lose requests. But if your monitor bogs down, it'll bog
down the app. And if your monitor crashes, it'll break caching. You
probably will have to reconfigure your app and/or memcached servers so
that the connections go through the proxy.
In short, a proxy will probably be easier to code, but implementing it can be a royal pain and better be perfect or cached. Changing the application or memcached seems like the most suitable approach for me.
BTW: Have you looked at built-in memcached statistics reports? I don't think it's granular enough for what you want, but if you haven't seen it, take a look before doing the real work: -D
source to share
What are you trying to track down? If you want simple packet or byte counts or basic header information, then this iptables
will write this for you:
iptables -I INPUT -p tcp -d $HOST_IP --dport $HOST_PORT -j LOG $LOG_OPTIONS
If you need more details, take a look at the target iptables ULOG
that sends each packet to user space for analysis.
See http://www.netfilter.org for very detailed docs.
source to share
If you want to go sniffer it might be easier to use tcpflow instead of tcpdump or libpcap. tcpflow will only output the TCP payload, so you don't have to worry about reassembling the data stream yourself. If you prefer to use a library instead of glueing multiple programs together, you may be interested in libnids.
libnids and tcpflow are also available in other flavors of Unix and don't restrict you to Linux only (unlike iptables).
http://www.circlemud.org/~jelson/software/tcpflow/ http://libnids.sourceforge.net/
source to share
iptables provides libipq with a user packet queuing library. From the manpage:
Netfilter provides a mechanism for passing packets from the stack for a queue to user space, and then fetching those packets back into the kernel with a judgment that determines what to do with the packets (such as ACCEPT or DROP). These packages can also be modified in user space before rebooting back into the kernel.
By setting individual iptables rules that forward packets to libipq, in addition to specifying a verdict for them, you can check packets to analyze statistics.
Another viable option is to manually package sniff using libpcap socket or PF_PACKET with socket filter support.
source to share