Deadlocks

A new deadlock detection library, based on the same code already found on the Linux kernel since v2.6, promises to detect wrong mutexes usages in user-space programs. It is mainly targeted to C/C++ programs, as it replaces pthread_mutex_t (and associated functions) with some wrappers, and depends on the Linux kernel for the detection.

 

image

 

TIPC

I didn’t know Linux had builtin support for TIPC, the Transparent Inter-process Communication (TIPC). TIPC is a protocol for inter-process communication in clusters, with interesting things like very small latency, local peers discovery, reliable and unreliable messages, etc…

Graphite

Another article from The Architecture of Open Source Applications, this time on the internals of Graphite, a framework for enterprise real-time graphing. Metrics are sent to the server by connecting to a port or from AMQP, and they are stored in their custom database backend, carbon  (something similar to the rrd database). Then users can directly see the graphs on their web app (a Django application that draws things with Cairo), or it can be used as a service provider for other applications (ie, it returns a graph when you do a HTTP request to some URL).

image

There are several popular additions in this setup:

  • statsd, a network daemon that listens for statistics, like counters and timers, and sends aggregates to Graphite. Check out how they use it at Instagram here.
  • gdash and Graphene, sexier dashboards for Graphite, based on some javascript libraries (like Twitter’s Bootstrap).
  • diamond, a daemon that sends basic metrics (CPU, net, etc…) to a Graphite server.
  • logster, another tool from Etsy for generating metrics by parsing log files
  • … and many other tools

I have played just a bit with Graphite in the past, but I gave up on using it because of the dependencies list. Cairo, too many python libraries, a database for Django, a web server… Lots of things to install (and Cairo is specially complicated on a Mac) so I’ll give it another try when I really need it…

splice.py

splice() is a Linux system call that can be used for copying data from a socket to another socket or a file. The big advantage of using splice is that it performs the data transfer with the so-called zero-copy: data remains in kernel space and is never copied to user space, and that makes a big difference.

This code shows how to use splice from Python. From the description:

Using the ‘splice’ syscall from Python, in this demonstration to transfer the output of some process to a client through a socket, using zero-copy transfers.

A JIT for packet filters

I did’t know Linux implemented this kind of optimization: code generation on the kernel for parsing network packets. When you want to filter some network packets that go through the kernel, this JIT mechanism generates native code for your filtering rules, so parsing is several orders of magnitude faster…

AIO in libtorrent

An interesting analysis on how they do asynchronous I/O (AIO) in libtorrent. Things are a bit complicated for them, because they want a truly portable AIO system, so they have finally gone for a threads pool, but I’m not sure this is the best solution if your only target is Linux…

The Linux Graphics Stack

A lengthy introduction to the Linux graphics stack.

LTTng 2.0: Tracing for power users and developers

Linux 3.3: Finally a little good news for bufferbloat

Contrary to common sense, large buffers are really a big problem on servers and routers. These are some of the countermeasures included in the latest Linux kernel (see this as well)

Tanenbaum Talks on MINIX and Linux

MINIX can recover from many problems that would cause other operating systems to crash and burn. A very welcome side effect of all this is that all parts of the system, except the tiny microkernel itself, can be updated without rebooting the system or affecting running processes.