I didn’t know Linux had builtin support for TIPC, the Transparent Inter-process Communication (TIPC). TIPC is a protocol for inter-process communication in clusters, with interesting things like very small latency, local peers discovery, reliable and unreliable messages, etc…
Another article from The Architecture of Open Source Applications, this time on the internals of Graphite, a framework for enterprise real-time graphing. Metrics are sent to the server by connecting to a port or from AMQP, and they are stored in their custom database backend, carbon (something similar to the rrd database). Then users can directly see the graphs on their web app (a Django application that draws things with Cairo), or it can be used as a service provider for other applications (ie, it returns a graph when you do a HTTP request to some URL).
There are several popular additions in this setup:
- statsd, a network daemon that listens for statistics, like counters and timers, and sends aggregates to Graphite. Check out how they use it at Instagram here.
- diamond, a daemon that sends basic metrics (CPU, net, etc…) to a Graphite server.
- logster, another tool from Etsy for generating metrics by parsing log files
- … and many other tools
I have played just a bit with Graphite in the past, but I gave up on using it because of the dependencies list. Cairo, too many python libraries, a database for Django, a web server… Lots of things to install (and Cairo is specially complicated on a Mac) so I’ll give it another try when I really need it…