A new deadlock detection library, based on the same code already found on the Linux kernel since v2.6, promises to detect wrong mutexes usages in user-space programs. It is mainly targeted to C/C++ programs, as it replaces pthread_mutex_t (and associated functions) with some wrappers, and depends on the Linux kernel for the detection.
I didn’t know Linux had builtin support for TIPC, the Transparent Inter-process Communication (TIPC). TIPC is a protocol for inter-process communication in clusters, with interesting things like very small latency, local peers discovery, reliable and unreliable messages, etc…
Another article from The Architecture of Open Source Applications, this time on the internals of Graphite, a framework for enterprise real-time graphing. Metrics are sent to the server by connecting to a port or from AMQP, and they are stored in their custom database backend, carbon (something similar to the rrd database). Then users can directly see the graphs on their web app (a Django application that draws things with Cairo), or it can be used as a service provider for other applications (ie, it returns a graph when you do a HTTP request to some URL).
There are several popular additions in this setup:
- statsd, a network daemon that listens for statistics, like counters and timers, and sends aggregates to Graphite. Check out how they use it at Instagram here.
- diamond, a daemon that sends basic metrics (CPU, net, etc…) to a Graphite server.
- logster, another tool from Etsy for generating metrics by parsing log files
- … and many other tools
I have played just a bit with Graphite in the past, but I gave up on using it because of the dependencies list. Cairo, too many python libraries, a database for Django, a web server… Lots of things to install (and Cairo is specially complicated on a Mac) so I’ll give it another try when I really need it…
splice() is a Linux system call that can be used for copying data from a socket to another socket or a file. The big advantage of using splice is that it performs the data transfer with the so-called zero-copy: data remains in kernel space and is never copied to user space, and that makes a big difference.
This code shows how to use splice from Python. From the description:
Using the ‘splice’ syscall from Python, in this demonstration to transfer the output of some process to a client through a socket, using zero-copy transfers.
I did’t know Linux implemented this kind of optimization: code generation on the kernel for parsing network packets. When you want to filter some network packets that go through the kernel, this JIT mechanism generates native code for your filtering rules, so parsing is several orders of magnitude faster…
An interesting analysis on how they do asynchronous I/O (AIO) in libtorrent. Things are a bit complicated for them, because they want a truly portable AIO system, so they have finally gone for a threads pool, but I’m not sure this is the best solution if your only target is Linux…
A lengthy introduction to the Linux graphics stack.