DTrace and Erlang: a status report

Sat, Nov 19, 2011

I thought it’d be a good idea to write a status report about my work with Erlang, DTrace, and some SystemTap.

Quick, I Want an Overview and Some Slides….

I gave a presentation at the Erlang User Conference  in Stockholm a couple weeks ago.  The topic was about adding DTrace probes to the Erlang virtual machine.  The presentation slides (and soon video, I hope) are available online at the Erlang Factory website.  Click on the easel icon to fetch a PDF copy of the slides.  When the video is available, it will probably appear as an embedded video player/applet/thingie.

I’ve given a few other talks at EUC, Erlang Factory, and Erlang Workshop venues.  I’ve never had as many people tell me afterward that they both enjoyed the talk and are looking forward to using DTrace (or SystemTap) with Erlang programs.

Getting DTrace into the Erlang/OTP R15 Release

I submitted a set of four patches to the erlang-questions mailing list on Wednesday night, November 16, 2011.  The patches were accepted by the OTP Team and are now sitting in the “pu” branch of the Erlang/OTP GitHub source repository.  The name “pu” stands for “proposed updates”.  The “pu” branch is used for all externally-submitted patches which have passed the initial round of review.  Everything in the “pu” branch is tested regularly by Ericsson on all supported Erlang/OTP platforms (e.g. Solaris, Windows, Linux, …).

Now we wait and see what Ericsson’s tests show.  The DTrace patches were submitted well ahead of the code freeze date for the Erlang/OTP R15B01 release, which is tentatively scheduled for December 14, 2011.  If all goes according to plan, the patches will be in the official Erlang/OTP R15B01 release next month!

What Is Traceable?

Take a look at the “examples” directory in the dtrace application for some simple tracing scripts.  To see the actual definition of the DTrace probes and their arguments, please see the  erlang_dtrace.d definition file.  Please note that this link is a static one and may not always show the absolutely newest, freshest, most current version of the file.

Here’s a short summary of what’s traceable today using DTrace:

  * Processes: spawn, exit, hibernate, scheduled, ...
  * Messages: send, queued, received, exit signals
  * Memory: GC minor & major, proc heap grow & shrink
  * Data copy: within heap, across heaps
  * Function calls: function & BIF & NIF, entry & return
  * Network distribution: monitor, port busy, output events
  * Ports: open, command, control, busy/not busy
  * Drivers: callback API 100% instrumented
  * efile_drv.c file I/O driver: 100% instrumented

As of today, there are 60 probes available.

DTrace & Erlang: Supported Platforms

I’ve been testing the following platforms regularly:

  * OS X 10.6.x, a.k.a. Snow Leopard.  I don't know of any reason why it shouldn't work with 10.7.x, a.k.a. Lion.
  * Solaris 10.  I've run into difficulties with long-running D scripts on my Solaris 10 machine, but I'll write more about that in a later blog posting.
  * Linux, specifically CentOS 5 and CentOS 6 using [SystemTap](http://sourceware.org/systemtap/)'s DTrace compatibility API.  I'll include some recipes for setting up CentOS boxes to be capable of running user-space SystemTap in another blog posting.  Things should also work when using a Fedora Core 16 system, but I haven't tested that yet.

I’m hoping to be able to get FreeBSD 9 supported also.  At the time of this writing, FreeBSD 9.0RC1 is now available for testing.  Andrew Thompson @ Basho has started the FreeBSD 9 work and passed it on to me.  I’ll try to get it hammered into working condition over this weekend.

How do I try it?

First, clone a copy of my source repository at GitHub:

git clone git://github.com/slfritchie/otp.git

Then you need to choose the branch that you wish to build:

  * Run the following if you want to compile the R14B04 version of Erlang/OTP: "cd otp ; git checkout dtrace-r14b04".
  * Run the following if you want to compile the R15A version of Erlang/OTP: "cd otp ; git checkout dtrace-r15".  NOTE that this branch is not fully synchronized with Ericsson's R15A development branch.
  * Run the following if you want to compile the patches as I submitted them to Ericsson for inclusion in Ericsson's source repo: "cd otp ; git checkout dtrace-review3".

I also have a branch called “dtrace-experiment+michal2” that contains the primary copy of all of the changes that I later merge into the three branches listed above.

Here are the instructions for building the Erlang/OTP package.  I assume that you already have the toolchain dependencies (e.g. GCC compiler) and libraries (e.g. OpenSSL) already installed.

./otp_build autoconf ./configure --enable-dtrace {put your other config options here} make make install

If you’re using a Linux box with SystemTap installed, you will still use the configure flag “–enable-dtrace”, despite the fact that you’re going to be using SystemTap.

After you’ve installed everything, then you need two windows:

  1. One window/login session to run an Erlang shell
  2. One window/login session as the superuser (i.e., "root") to run the D scripts.  If you're using SystemTap, please wait until my next blog posting, which will be specifically about Erlang and SystemTap.

Using the first window, start an Erlang shell, e.g. /path/to/your/erlang/installation/bin/erl

Using your second window, start a D script from the examples directory:

cd /path/to/your/erlang/installation/lib/erlang/lib/dtrace-0.8/examples dtrace -qs ./function-calls.d

Then, back in the first window, start typing commands at the Erlang shell prompt.  Watch the output in the second window.