I’ve been wearing a developer hat for too long and not wearing my sysadmin and network manager hats.  And the publications about Ethernet microbursts and the TCP incast pattern have been hiding in conference proceedings & journals that I don’t follow.  (Note to self: change reading habits.)

If you’d rather read a paper about the problem, go read one or more of these:

• http://www.cs.cmu.edu/~vrv/papers/PDSI07-Incast.pdf
• http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf

TL;DR: you can’t pour two buckets of manure into one bucket.  (Credit: my grandfather.)  I’m sure that’s very helpful … now keep on reading.

Assumptions

If you have a recipe using typical, modern, commodity computing hardware like this:

• Computers of fast CPU cores with lots of memory.
• Very efficient network interface cards in those computers, and a reasonably modern OS to take advantage of them.
• Gigabit Ethernet (or 10 Gbit/sec) Ethernet connecting those computers together.  One switch or a tree/fabric/whatever of switches, it doesn’t matter too much which, though multiple switches might be a bit easier.
• An application that has lots of many-to-one or many-to-many communication patterns.  The Riak database by Basho Technologies happens to do a large amount of many-to-one operations, so I’ll use that as my example below.  The same principle applies to Hadoop and many other data-intensive distributed applications.

The Easy Bandwidth Problem

Say you’ve got five machines using “scp” to copy data to a single destination.  Each source machine is capable of outputting 900+ Mbit/sec .  If your five machines each send 900+ Mbit/sec of traffic to a single recipient, your 1Gbit/sec Ethernet switch will soon have no choice but to drop packets.

But the “5 machines sending lots bulk data to 1 machine” is easy to understand: all machines cannot simultanously send 900+ Mbit/sec because the receiver has only 1,000 Mbit/sec of (theoretical) bandwidth.  The switch will drop some packets, then each TCP connection will react and slow down.  Eventually, each sender will reach a steady-state of sending roughly 200 Mbit/sec.  So, 5 * 200Mbit/sec = 1,000 Mbit/sec.  Easy, right?

The More Difficult Bandwidth Problem

What if you have your Riak cluster of 20 machines, and each machine is using (on average) 250 Mbit/sec of both transmit & receive bandwidth on their 1 Gbit/sec Ethernet interfaces?  You’re only using 25% of the theoretical peak rate on each machine’s interface.  You’ve got plenty of room to grow, right?  No, not really.  For clusters of data-intensive applications like Riak or Hadoop, that 25% rate may be halfway to your practical bandwidth limit.  Why?

Because you’re likely to have “microbursts” of traffic sent to a single cluster member.  The switch can’t buffer all of the frames in the microburst, so it drops some.  Then the TCP mechanisms intervene to slow things down.  If your cluster has average utilization of 25-35% and uses Ethernet switches with small buffers, you may already be dropping 40-100 frames per second per machine.  The frame drop rate will rise very sharply as utilization increases.  At 45-55% utilization, you’ll start seeing 100-500 dropped frames per second per machine.  Will TCP be able to run full-speed with frame drop rates that high?  No.  Instead, you’ll be stuck with a cluster that can barely run above 50-60% average utilization.

It sounds a bit crazy, but it’s a real phenomenon.  And it’s finally something that has happened to me.  It took a long time to realize what was really happening … because I never considered a network that was 45-55% utilized was really near its peak usable bandwidth.

An Illustration Using Riak

The Riak database manages its data replication by storing several copies of any single piece of data on multiple machines.  Typically, the number of copies is three.  When your app tries to fetch a key from Riak, Riak will create a new process to coordinate the operation.  That new process will send ‘get’ requests to three different nodes and await their replies.  Each server that receives a ‘get’ request will send a copy of the key, its value, and its metadata dictionary back to the coordinator.

So, what if both of the following were true?

1. All three ‘get’ results are sent to the coordinator at exactly the same time.  (Here, “exactly” means something like “within the same millisecond or perhaps even the same microsecond.)
2. All three ‘get’ results are big, for example, 100 kilobytes each.

Then you have three different computers trying to send their 100 kilobytes of data to a single machine at the exact same time.  The resulting microburst of data creates a problem for the Ethernet switch(es) that these four computers are using.  (For the sake of simplicity, assume that all four computers are using the same Ethernet switch.)

If all three ‘get’ results arrive at the same instant, then the switch must do one of two things:

1. Buffer all of the 300KB of packets.
2. Drop some packets.

If you’re using a typical, low-cost commodity Ethernet switch such as a Cisco Catalyst 3750, you don’t have large amounts of buffer space (compared to other switches on the market).  See a table from a Tolly Enterprises, LLC report number 211127:

But if your Ethernet switch has a lot of buffer space (like the HP or Juniper switches in the Tolly figure), you do not make it impossible to overrun your buffer space: large buffers only make buffer overruns less likely.  Remember, these are fast machines in this cluster.  And your switch may only have buffer space for a few dozen or a few hundred frames, depending on the frame size.

So, now imagine that the Riak cluster is much busier.  Each cluster member is taking in many thousands of queries per second, thus starting several thousand coordinators processes per second.  Each coordinator is getting (typically) 3 replies.  Each reply is big enough to use many Ethernet frames.  And it’s quite likely that enough frames will arrive from all over the cluster to overrun the Ethernet switch’s buffers for the coordinator machine’s Ethernet port.  Boom, “TCP incast” bites you, hard.

Riak in Production at Voxer: a case study

I’ve seen a customer’s cluster (Voxer) do exactly this.  When utilization on a GigE network (fed by Cisco Catalyst 3750 switches) uses more than 50% average utilization (sampled @ 1 second intervals), TCP throughput collapses regularly.  The 1-second utilization rates will see-saw between 900 Mbit/sec down to 200 Mbit/sec.  To make matters worse, the 200 Mbit/sec rate will happen much more frequently than the 900 Mbit/sec rate.

To show that we’re seeing packet drops, we use a couple of methods.  First, using “tcpdump” and “tcptrace”.

root# sh -c 'FILE=/tmp/slf.capt`date +%s` ; echo Putting tcpdump output in $FILE ; time tcpdump -i eth0 -c 200000 -w $FILE ; tcptrace -l $FILE | grep "rexmt data pkts" | head -5 ; echo Pseudo-count of TCP flows with retransmitted packets: ; tcptrace -l $FILE | grep "rexmt data pkts" | grep -v "rexmt data pkts: 0 rexmt data pkts: 0" | wc -l ; echo Total retransmitted packets and bytes: ; tcptrace -l $FILE | awk " /rexmt data pkts/ { p += \$4 + \$8 } /rexmt data bytes/ { b += \$4 + \$8 } END { print p, b }" '
Putting tcpdump output in /tmp/slf.capt1325142154
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
200000 packets captured
200011 packets received by filter
0 packets dropped by kernel
0.22user 0.43system 0:07.50elapsed 8%CPU (0avgtext+0avgdata 17648maxresident)k
136inputs+36960outputs (3major+738minor)pagefaults 0swaps
 rexmt data pkts: 0 rexmt data pkts: 0
 rexmt data pkts: 0 rexmt data pkts: 0
 rexmt data pkts: 0 rexmt data pkts: 0
 rexmt data pkts: 0 rexmt data pkts: 0
 rexmt data pkts: 0 rexmt data pkts: 0
Pseudo-count of TCP flows with retransmitted packets:
99
Total retransmitted packets and bytes:
837 5608410
root# for i in `seq 1 30`; do sh -c 'echo -n `date` " " ; ifconfig eth0 ; sleep 1 ; ifconfig eth0' | grep bytes | sed -e 's/.. bytes://g' | awk 'NR == 1 { rx = $1; tx = $4} NR == 2 { printf "rx %.1f Mbit/sec tx %.1f Mbit/sec ", ($1 - rx) * 8 / (1024*1024) / 1, ($4 - tx) * 8 / (1024*1024) / 1 }' ; date ; donerx 226.3 Mbit/sec tx 293.7 Mbit/sec Wed Dec 28 23:03:17 PST 2011rx 242.8 Mbit/sec tx 265.3 Mbit/sec Wed Dec 28 23:03:18 PST 2011
rx 260.4 Mbit/sec tx 302.1 Mbit/sec Wed Dec 28 23:03:25 PST 2011
rx 226.4 Mbit/sec tx 241.4 Mbit/sec Wed Dec 28 23:03:26 PST 2011
rx 238.1 Mbit/sec tx 277.9 Mbit/sec Wed Dec 28 23:03:27 PST 2011
rx 270.5 Mbit/sec tx 315.9 Mbit/sec Wed Dec 28 23:03:28 PST 2011
rx 257.4 Mbit/sec tx 287.5 Mbit/sec Wed Dec 28 23:03:29 PST 2011
^C

That’s 837 dropped packets in 7.5 seconds of sampling, at an average 1-second throughput of 240-315 Mbit/sec.

Looking at this in a more Linux-specific manner, and a bit more exact measurement of transmit bandwidth during the time period that we’re measuring transmissions:

root# netstat -s | egrep 'segments retransmited|OutOctets|requests sent out' ; sleep 1; netstat -s | egrep 'segments retransmited|OutOctets|requests sent out'
    1542387771 requests sent out
    521663296 segments retransmited
    OutOctets: 1222021825
    1542398375 requests sent out
    521663388 segments retransmited
    OutOctets: 1248437047

That’s 10604 packets sent, 92 packets retransmitted, for 26415222 octets sent (201 Mbit/sec).  The packet retransmission rate is 0.9.  (Yes, all other machines are using roughly the same 1-second bandwidth.)  At 201 Mbit/sec, we’re at 20% of allegedly full bandwidth.  If we double the average utilization of all machines to 400Mbit/sec, the packet rate moves up to the 4-6% range.  And if all machines go up to 500Mbit/sec, things get ugly.

rx 364.4 Mbit/sec tx 916 Mbit/sec Wed Dec 21 15:43:31 PST 2011
rx 339.1 Mbit/sec tx 951 Mbit/sec Wed Dec 21 15:43:32 PST 2011
rx 361.6 Mbit/sec tx 952 Mbit/sec Wed Dec 21 15:43:33 PST 2011
rx 475.4 Mbit/sec tx 491 Mbit/sec Wed Dec 21 15:43:34 PST 2011
rx 529.2 Mbit/sec tx 415 Mbit/sec Wed Dec 21 15:43:36 PST 2011
rx 472.9 Mbit/sec tx 267 Mbit/sec Wed Dec 21 15:43:37 PST 2011
rx 505.3 Mbit/sec tx 269 Mbit/sec Wed Dec 21 15:43:38 PST 2011
rx 393.5 Mbit/sec tx 191 Mbit/sec Wed Dec 21 15:43:39 PST 2011
rx 175.1 Mbit/sec tx 8 Mbit/sec Wed Dec 21 15:43:40 PST 2011
rx 436.8 Mbit/sec tx 246 Mbit/sec Wed Dec 21 15:43:41 PST 2011
rx 487.6 Mbit/sec tx 246 Mbit/sec Wed Dec 21 15:43:42 PST 2011
rx 524.2 Mbit/sec tx 194 Mbit/sec Wed Dec 21 15:43:43 PST 2011
rx 441.9 Mbit/sec tx 699 Mbit/sec Wed Dec 21 15:43:44 PST 2011
rx 382.8 Mbit/sec tx 952 Mbit/sec Wed Dec 21 15:43:45 PST 2011
rx 331.5 Mbit/sec tx 951 Mbit/sec Wed Dec 21 15:43:46 PST 2011
rx 391.3 Mbit/sec tx 949 Mbit/sec Wed Dec 21 15:43:47 PST 2011
rx 538.6 Mbit/sec tx 396 Mbit/sec Wed Dec 21 15:43:48 PST 2011

Bummer.  During about 20 seconds, our transmit bandwidth ranges from a high of 952 Mbit/sec all the way down to 8 Mbit/sec.  EIGHT!  For an entire second!  And it took four more seconds before the transmit rate rises above 500Mbit/sec.

So, one last question about the whole microburst and Ethernet switch buffering and “TCP incast” problem is … are the microbursts really happening?  Are the switch ports really being pushed to full line rate some of the time?

As far as I can tell, the answer is “yes”.  Using an Erlang program (Escript, actually), I have good timer resolution down to about 4 milliseconds, perhaps even 2 milliseconds.  To get polling accurate below that, I’d have to write a demo program in another language.  (Or write the Erlang program so that avoids using timers and instead uses busy-wait loops: Erlang’s time-of-day clock has microsecond resulution.)

Here’s a link to the escript that I used, and here’s some output, taken from an off-peak period last night. The first argument is the number of milliseconds between polling periods, and the second is the path to Linux’s /sys file system file for incoming network octets. (The {22,x,y} stuff is a timestamp, e.g. 22:29:21 Pacific time.  The “ratio” figure divides the maximum bandwidth observed during that second by the average bandwidth for the second.)

root# ./poll-value-mbit.escript 500 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 184.7 Mbit/s Avg 170.3 Mbit/s Ratio 1.1 @ {22,29,21}
Max 206.6 Mbit/s Avg 185.7 Mbit/s Ratio 1.1 @ {22,29,22}
Max 230.1 Mbit/s Avg 206.0 Mbit/s Ratio 1.1 @ {22,29,23}
Max 192.1 Mbit/s Avg 185.4 Mbit/s Ratio 1.0 @ {22,29,24}
Max 183.7 Mbit/s Avg 167.4 Mbit/s Ratio 1.1 @ {22,29,25}
Max 212.5 Mbit/s Avg 191.8 Mbit/s Ratio 1.1 @ {22,29,26}
^C
root# ./poll-value-mbit.escript 250 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 174.0 Mbit/s Avg 157.8 Mbit/s Ratio 1.1 @ {22,29,32}
Max 225.1 Mbit/s Avg 180.2 Mbit/s Ratio 1.2 @ {22,29,33}
Max 210.7 Mbit/s Avg 182.6 Mbit/s Ratio 1.2 @ {22,29,34}
Max 188.3 Mbit/s Avg 172.3 Mbit/s Ratio 1.1 @ {22,29,35}
Max 193.5 Mbit/s Avg 177.5 Mbit/s Ratio 1.1 @ {22,29,36}
^C
root# ./poll-value-mbit.escript 125 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 256.2 Mbit/s Avg 200.5 Mbit/s Ratio 1.3 @ {22,29,44}
Max 245.3 Mbit/s Avg 209.4 Mbit/s Ratio 1.2 @ {22,29,45}
Max 271.8 Mbit/s Avg 212.4 Mbit/s Ratio 1.3 @ {22,29,46}
Max 214.6 Mbit/s Avg 189.4 Mbit/s Ratio 1.1 @ {22,29,47}
Max 261.8 Mbit/s Avg 199.5 Mbit/s Ratio 1.3 @ {22,29,48}
^C
root# ./poll-value-mbit.escript 64 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 458.6 Mbit/s Avg 311.9 Mbit/s Ratio 1.5 @ {22,30,0}
Max 389.1 Mbit/s Avg 236.0 Mbit/s Ratio 1.6 @ {22,30,1}
Max 267.2 Mbit/s Avg 162.8 Mbit/s Ratio 1.6 @ {22,30,2}
Max 276.8 Mbit/s Avg 167.6 Mbit/s Ratio 1.7 @ {22,30,3}
Max 229.3 Mbit/s Avg 172.3 Mbit/s Ratio 1.3 @ {22,30,4}
Max 346.6 Mbit/s Avg 193.1 Mbit/s Ratio 1.8 @ {22,30,5}
^C
root# ./poll-value-mbit.escript 32 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 372.7 Mbit/s Avg 204.2 Mbit/s Ratio 1.8 @ {22,30,9}
Max 305.6 Mbit/s Avg 166.9 Mbit/s Ratio 1.8 @ {22,30,10}
Max 356.0 Mbit/s Avg 192.0 Mbit/s Ratio 1.9 @ {22,30,11}
Max 410.0 Mbit/s Avg 174.4 Mbit/s Ratio 2.4 @ {22,30,12}
Max 349.7 Mbit/s Avg 187.7 Mbit/s Ratio 1.9 @ {22,30,13}
^C
root# ./poll-value-mbit.escript 16 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 441.1 Mbit/s Avg 171.8 Mbit/s Ratio 2.6 @ {22,30,19}
Max 451.1 Mbit/s Avg 179.5 Mbit/s Ratio 2.5 @ {22,30,20}
Max 352.6 Mbit/s Avg 166.3 Mbit/s Ratio 2.1 @ {22,30,21}
Max 424.7 Mbit/s Avg 177.1 Mbit/s Ratio 2.4 @ {22,30,22}
^C
root# ./poll-value-mbit.escript 8 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 598.1 Mbit/s Avg 199.4 Mbit/s Ratio 3.0 @ {22,30,30}
Max 724.1 Mbit/s Avg 205.9 Mbit/s Ratio 3.5 @ {22,30,31}
Max 637.2 Mbit/s Avg 158.8 Mbit/s Ratio 4.0 @ {22,30,32}
Max 727.2 Mbit/s Avg 187.3 Mbit/s Ratio 3.9 @ {22,30,33}
Max 832.3 Mbit/s Avg 221.8 Mbit/s Ratio 3.8 @ {22,30,34}
Max 436.6 Mbit/s Avg 162.6 Mbit/s Ratio 2.7 @ {22,30,35}
^C
root# ./poll-value-mbit.escript 4 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 826.5 Mbit/s Avg 200.2 Mbit/s Ratio 4.1 @ {22,30,41}
Max 858.9 Mbit/s Avg 166.1 Mbit/s Ratio 5.2 @ {22,30,42}
Max 746.8 Mbit/s Avg 176.0 Mbit/s Ratio 4.2 @ {22,30,43}
Max 838.7 Mbit/s Avg 190.0 Mbit/s Ratio 4.4 @ {22,30,44}
Max 674.0 Mbit/s Avg 173.5 Mbit/s Ratio 3.9 @ {22,30,45}
^C
root# ./poll-value-mbit.escript 2 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 950.2 Mbit/s Avg 188.9 Mbit/s Ratio 5.0 @ {22,30,51}
Max 960.0 Mbit/s Avg 197.2 Mbit/s Ratio 4.9 @ {22,30,52}
Max 992.8 Mbit/s Avg 236.1 Mbit/s Ratio 4.2 @ {22,30,53}
Max 997.5 Mbit/s Avg 191.7 Mbit/s Ratio 5.2 @ {22,30,54}
Max 962.5 Mbit/s Avg 172.5 Mbit/s Ratio 5.6 @ {22,30,55}
Max 957.5 Mbit/s Avg 187.6 Mbit/s Ratio 5.1 @ {22,30,56}
^C
root# ./poll-value-mbit.escript 1 /sys/devices/pci0000:00/0000:00:01.0/0000:0a:00.0/net/eth0/statistics/rx_bytes
Max 1645.6 Mbit/s Avg 182.2 Mbit/s Ratio 9.0 @ {22,31,2}
Max 1610.0 Mbit/s Avg 212.0 Mbit/s Ratio 7.6 @ {22,31,3}
Max 1623.1 Mbit/s Avg 206.3 Mbit/s Ratio 7.9 @ {22,31,4}
Max 1522.3 Mbit/s Avg 173.5 Mbit/s Ratio 8.8 @ {22,31,5}
Max 1607.7 Mbit/s Avg 199.0 Mbit/s Ratio 8.1 @ {22,31,6}

It’s clear that as the polling period gets smaller, the ratio between maximum observed incoming rate and average rate gets larger.  It’s also clear that we don’t have good enough timer resolution for 1 millisecond resolution.  2 millisecond resolution might be iffy, however, a separate experiment to test Erlang timer resolution at 2 milliseconds shows that the accuracy is within roughly 10%.

What happens to Erlang programs when TCP incast strikes?

When a few packets are dropped, TCP does a good job of figuring out which packet(s) needs retransmission.  The TCP stack in Linux 2.6.32 seems to do well enough.  At low packet loss rates, throughput isn’t harmed much, and latency penalties are minimal.

At higher packet loss rates, however, you can hit “TCP slow start“.  In the example above, where throughput fluctuates from 952 Mbit/sec down to 8 Mbit/sec, that’s what I believe happened.  (I don’t have proof, alas: I didn’t have a packet capture running at that time.)

Between any two Erlang nodes A and B, there is a single TCP connection that carries all Erlang messages between A and B.  If there is severe congestion between A and B, and TCP slow start happens on the A-to-B TCP connection, messaging between A and node C will not be impacted.

However, there’s a very strong backpressure/feedback mechanism built in to Erlang that is related to output to “ports”.  An Erlang port is a gateway to the outside world, e.g. file descriptors to TCP sockets and to local file systems.  Each port has a buffer associated with it.  If an Erlang process writes data to the port until the buffer is full, then that process will be descheduled: the process cannot run until the buffer is no longer full.  If Erlang system monitoring for busy ports is enabled, a busy_port message will be sent to the system monitor process.

If an Erlang process P on node A attempts to send a message to process R on node B, and if the A-to-B port’s buffer is full, then process P will be descheduled. If Erlang system monitoring for busy network distribution ports is enabled, a busy_dist_port message will be sent to the system monitor process.

In an application like Riak, however, the process scheduling problem caused by a busy_dist_port event is very costly.

• Assume process P on node A is a Riak KV vnode process.  This process is, for discussion purposes, an Erlang/OTP gen_server process.  Like all gen_server processes, it handles all requests serially.
• Assume process R (the soon-to-be-message-receiver in this example) on node B is a Riak client ‘get’ operation FSM (finite state machine) process.
• Assume that the TCP connection between A and B is congested.  In fact, it’s so congested that the buffer for the port for A-to-B messaging is completely full.
• When process P gets R‘s ‘get’ request, it does its normal computation: pass the request to the riak_kv vnode handler, then down to (for example) the Bitcask local storage manager.  Eventually, a reply is calculated and ready to send back to the client, R.
• Our server process P eventually uses the Erlang ! operator to send the reply to R.
• The A-to-B network distribution port buffer is 100% full.  The VM sends a busy_dist_port message to the system monitor process (which will write the event to the Riak application log file), and process P is descheduled.

Given this set of events, then our Riak vnode process P cannot answer any more queries until the A-to-B network link becomes uncongested.   If the cluster has 12 machines in it,  P cannot process any queries from the other 10 machines in the cluster … even if the Ethernet ports to those other 10 machines are 0% utilized.  Service to those other 10 machines will be delayed for as long as it takes for the A-to-B TCP connection’s data to start flowing again.  If TCP slow start is triggered, the Linux TCP stack’s default timeout before starting TCP slow start is 200 milliseconds.  And it can take another fraction of a second longer (at least) before full bandwidth can be utilized … which also assumes that there are no more packets dropped while TCP climbs out of slow start mode … which we now know is a faulty assumption.

(Careful readers will probably draw a connection from the process descheduling problem to a more general head-of-line blocking problem.)

This causes Riak really big headaches.  Suddenly Riak query latencies become extremely unpredictable.  Overall memory usage can rise dramatically, 25% or more, for short periods of time and then fall back to normal as queues eventually drain.

What’s the remedy?

We don’t have a good remedy for this yet.  Options include:

• Using the DCTCP protocol isn’t an option for most Basho customers.
• Enabling support for Ethernet “pause frames” assumes that your switches support them correctly — transmission between switches is apparently not well supported.
• Changing the 200ms TCP “RTO” timer (the retransmission timeout timer) may have a beneficial effect, but it’s difficult to measure because the Linux 2.6.32 kernel’s implemention for changing the RTO timer is buggy.
• Changing other Linux TCP and NIC configuration knobs (e.g. TX queue, firmware ring size & interrupt rates, MTU 9000) have negligible effect.
• Placing limits on all cluster members outgoing bandwidth to 80% of line rate (which might reduce the ability of any single cluster member to cause TCP incast packet drops on any single receiver) appears difficult. Documentation for the Linux tc utility and the “token bucket filter” mechanism suggests that TBF might be 100% accurate up to 10 Mbits/sec.  Um, I need something that can handle a couple orders of magnitude more traffic than that.  And I’d like to be able to configure it without using units like packets/jiffy.

We also have Bug 1309 open, to work around the worst of the head-of-line blocking problem.

Postscript

Many thanks to Matt Ranney at Voxer and his great staff for assisting in troubleshooting this networking problem.  The amount of time that he spent on the phone with various data center support staff is … I don’t want to try to sum it all up.  When we finally stumbled across the theory of the TCP incast pattern, my initial reaction was (paraphrasing), “That’s bullshit.”  But, indeed, I was wrong.  All subsequent research points to TCP incast as our main problem.

Update: 2012-Jan-22

There’s also a good collection of papers about the TCP incast pattern at the CMU PDL Projects page.

I might say something. At any time.

From this day forward, I’ll separate my blog postings by several hours, if not days or months.  You won’t miss anything, really, if you polled less frequently.  Besides, you’re probably a Twitter user.  Follow me on Twitter, @slfritchie, to get your fix.

My firewall might say something.  To you, 50.16.238.123.  At any time.

Except that you’re likely running on an EC2 instance.  Fickle, mercurial.  Poised to multiply a thousandfold like tribbles, overruning my modest ISP service, flooding its virtual storage bins with cute replicas of their fuzzy terror.

The firewall “man” page says nothing about filtering TDP, the Tribble DoS Protocol.  Drat.  I need an upgrade.  At an unknown time.

FreeBSD has supported kernel-space DTrace probes for a while.  However, support for user-space probes, USDT, has not arrived until FreeBSD 9.0RC1.  ”RC1″ means “Release Candidate #1″.  As of this writing, RC2 is the newest release.  The official release of FreeBSD 9.0 (to be called FreeBSD 9.0-RELEASE) will not be ready for at least another month or perhaps longer; see the 9.0 wiki release schedule  for more details.

As I mentioned in the introduction to this article, USDT probes are a bit broken in FreeBSD 9.0RC2.  I’ve added two different kinds of probes to the Erlang virtual machine:

1. Probes that are defined as part of the core virtual machine.  These probes fire in response to events within the virtual machine, e.g. spawning a new process or a garbage collection event.  These probes are defined in erlang_dtrace.d.
2. A probe that can be triggered directly by Erlang code, e.g., dtrace:p(42, "Hello, world!"). This probe is defined in dtrace_user.d and by the Erlang module dtrace.erl.

The probe that can be fired directly from Erlang code must be compiled using position-independent code (PIC) and assembled into a shared library.  The Erlang function dtrace:init() will load the shared library into the virtual machine and initialize the Erlang<->C glue necessary for an Erlang function to call a C function.

Both FreeBSD 9.0RC1 and FreeBSD 9.0RC2 cannot compile and link to PIC code.  So all of the probes in erlang_dtrace.d will work correctly (because they’re not part of a shared library), but the probe in dtrace_user.d will not work correctly.

There is a problem report (PR) for FreeBSD that discusses this PIC problem: kern/159046: [dtrace] [patch] dtrace library is linked with a wrong flags on -CURRENT.  I emailed an update to that ticket this afternoon, asking that Alex Samorukov’s fix be folded into the 9.0-RELEASE of FreeBSD.  I have no idea if it will happen in time or not … but I have my fingers crossed.

In the meantime, I have a fix for those who are using one of the FreeBSD release candidates where the PIC bug is still present.  You have a couple of options:

1. If you’re familiar with building & installing FreeBSD kernels and FreeBSD user-space programs, then following the advice in the FreeBSD problem report (linked above) will work just fine.
2. Download PIC-compiled copies of the system libraries that are required for PIC-compiled trace code and install them manually.

In theory, any Linux system that supports user-space SystemTap should work just fine.

In practice, I highly recommend that you use a CentOS 5 machine.  Why?

• CentOS 5 is what I use for my Linux + SystemTap testing
• CentOS 6 ought to work, but I’ve noticed strange problems when trying to run some SystemTap scripts … and have seen incorrect/weird behavior from other scripts.
• Fedora Core 16 ought to work as well as CentOS 5, but I haven’t tried it yet.

First, Read the README File

Installing SystemTap on a CentOS 5 Machine

I started with a a CentOS 5.6 32-bit machine, freshly installed via the installer CD. Using a 64-bit machine is fine.  My 32-bit machine is actually a virtual machine, running inside VMware Fusion.

First, edit the /etc/yum.repos.d/CentOS-Debuginfo.repo file to change the “enabled=0” line to be “enabled=1” instead.

Next, run the following “yum” commands:

 yum -y install kernel kernel-devel kernel-headers yum -y install kernel-debuginfo kernel-debuginfo-common systemtap systemtap-sdt-devel systemtap-runtime systemtap-testsuite yum -y install make gcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel

Those three commands take care of the following:

• Upgrade your kernel
• Install the kernel & user packages required for SystemTap
• Install the minimum packages needed to build the Erlang/OTP package.

rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
yum install --enablerepo=webtatic git

When that’s done, reboot your machine.  The default kernel in your GRUB boot loader config should be the latest stable kernel for CentOS 5.

Now you’re ready for the next step … skip down to the section “Build Erlang/OTP With SystemTap Enabled“.

Installing SystemTap on a CentOS 6 Machine

Setting up CentOS 6 is slightly easier than CentOS 5.

So, I’ve had a more difficult time using SystemTap on CentOS 6.  There seem to be two (or three?) main variations of the SystemTap kernel & user-space packaging.  Whichever one that CentOS 6 uses doesn’t work well.  Or, doesn’t work very well for me.  If you have a different experience, please contact me!

OK, here’s the packages to install via “yum”:

yum install kernel kernel-devel kernel-headers
yum install systemtap systemtap-sdt-devel systemtap-runtime systemtap-testsuite
yum install make gcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel
yum install git

Reboot your machine, then you’re ready for the next phase.

Build Erlang/OTP With SystemTap Enabled

First, please make certain that the kernel that’s currently running supports user-space SystemTap

See the “How Do I Try It?” section of the status report for instructions on how to compile the Erlang/OTP distribution.  Both the R14B04 and the still-under-development R15A releases are supported.

Checking to See If User-Space SystemTap Works

Please run the following command to make certain that user-space SystemTap is installed correctly on your system.

    grep CONFIG_UTRACE /boot/config-`uname -r`

The output should be:

    CONFIG_UTRACE=y

If not, you’re using the wrong kernel.

Next, check to see if your Erlang build has been compiled successfully with SystemTap via the DTrace compatibility mode:

    PATH=/path/to/beam:$PATH stap -L 'process("beam").mark("*")'

You should see output that looks like this:

process("beam").mark("aio_pool__add")
process("beam").mark("aio_pool__get")
process("beam").mark("bif__entry")
process("beam").mark("bif__return")
process("beam").mark("copy__object")
process("beam").mark("copy__struct")
[...]

Congratulations!

Running Your First SystemTap Script on CentOS 5

You will need two windows/login sessions:

1. One window/login session to run an Erlang shell
2. One window/login session as the superuser (i.e., “root”) to run the SystemTap scripts.

On my machine, I have close to a dozen different versions of Erlang installed.  To keep my SystemTap/DTrace builds separate from everything else, I used “./configure --prefix=/usr/local/erlang/R15A.dtrace“.  Therefore, my startup command is “/usr/local/erlang/R15A.dtrace/bin/erl -smp disabled“.

Your paths are probably different.  For the rest of this little tutorial, I assume that you know what the top-level path is.

Using your second window, use “ps axw | grep beam” to find the full path to the BEAM virtual machine.  This path is NOT THE SAME as the path used to start Erlang via the “erl” command.  On my system, the result is:

[root@localhost ~]# ps axw | grep beam
 9606 pts/2    S+     0:00 /usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin/beam -- -root /usr/local/erlang/R15A.dtrace-review3/lib/erlang -progname erl -- -home /home/fritchie -- -smp disable

The directory that contains the “beam” executable is what’s necessary for the “env PATH” commands that we will use later.  In my case, that directory is “/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin“. Your directory name is probably different.

In the second window/login session, start a SystemTap script from the examples directory: 

cd /usr/local/erlang/R15A.dtrace/lib/erlang/lib/dtrace-0.8/example
env PATH=/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin:$PATH stap ./garbage-collection.systemtap

Then, back in the first window, start typing commands at the Erlang shell prompt.  For example: 

Erlang R15A (erts-5.9) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false] [dtrace]

Eshell V5.9  (abort with ^G)
1> "this is a test of the SystemTap system".

Over in the second window, you should see output that looks like this:

GC minor start pid <0.24.0> need 3 words
GC minor end pid <0.24.0> reclaimed 3 words
GC minor start pid <0.22.0> need 14 words
GC minor end pid <0.22.0> reclaimed 14 words
GC minor start pid <0.24.0> need 8 words
GC minor end pid <0.24.0> reclaimed 8 words

Congratulations, go have fun exploring Erlang and SystemTap!

 WARNING: Erlang SMP Support Uses a Different Executable Name!

All of the SystemTap scripts in the “examples” directory contain the name of the Erlang virtual machine executable, “beam“.  We started Erlang using the “-smp disabled” flag because we wanted to make certain that we were using “beam“, the executable that does NOT support SMP, i.e., symmetric multi-processing, i.e., multi-CPU and multi-core systems.

The executable that supports SMP is called “beam.smp“.  If you use the SMP version of Erlang, then you must also change the name of the executable in your SystemTap scripts!

Running Your First SystemTap Script on CentOS 6

So, all of the directions given in the CentOS 5 section are exactly the same for CentOS 6 also.

Why is there a separate CentOS 6 section?  Because things don’t work correctly for me, and I’m hoping that someone else will notice and then email me with a fix.  (Hint, hint, …. :-)

Here’s what I see when I run the same garbage collection script on my CentOS 6 machine:

[root@localhost ~]# env PATH=/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin:$PATH stap ./garbage-collection.systemtap
semantic error: no match while resolving probe point process("beam").mark("gc_major-start")
semantic error: no match while resolving probe point process("beam").mark("gc_minor-start")
semantic error: no match while resolving probe point process("beam").mark("gc_major-end")
Pass 2: analysis failed.  Try again with another '--vp 01' option.

Actually, I know how to fix that one.  The problem comes from the DTrace convention of using underscore characters and hyphen characters in probe names.  With DTrace, if a probe definition file uses a name has two underscores, e.g., “gc_major__start”, then the DTrace probe name is “gc_major-start”.

The version of SystemTap that CentOS 5 uses seems to be aware of the double-underscore-becomes-a-hyphen convention.  The CentOS 6 version of SystemTap does not.

So, we fix our file with a bit of editing, so it now looks like this:

[... comments at top of file omitted ...]
probe process("beam").mark("gc_major__start")
{
    printf("GC major start pid %s need %d words\n", user_string($arg1), $arg2);
}

probe process("beam").mark("gc_minor__start")
{
    printf("GC minor start pid %s need %d words\n", user_string($arg1), $arg2);
}

probe process("beam").mark("gc_major__end")
{
    printf("GC major end pid %s reclaimed %d words\n", user_string($arg1), $arg2);
}

probe process("beam").mark("gc_minor__start")
{
    printf("GC minor end pid %s reclaimed %d words\n", user_string($arg1), $arg2);
}

Now we try to run again.  This time we see garbage instead of the Erlang PID strings that we ought to see:

[root@localhost ~]# env PATH=/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin:$PATH stap /tmp/garbage-collection.systemtap
GC minor start pid ???? need 8 words
GC minor end pid ???? reclaimed 8 words
GC minor start pid ???? need 14 words
GC minor end pid ???? reclaimed 14 words
GC minor start pid need 16 words
GC minor end pid reclaimed 16 words
^C

Other SystemTap scripts in the examples director have bigger problems, such as the following (after changing the probe name hyphens to double-underscores):

[root@localhost ~]# env PATH=/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin:$PATH stap /tmp/messages.systemtap
semantic error: failed to retrieve location attribute for local 'arg4' (dieoffset: 0x193897): identifier '$arg4' at /tmp/messages.systemtap:79:9
        source:     if ($arg4 == 0 && $arg5 == 0 && $arg6 == 0) {
                        ^
Pass 2: analysis failed.  Try again with another '--vp 01' option.

[root@localhost ~]# env PATH=/usr/local/erlang/R15A.dtrace-review3/lib/erlang/erts-5.9/bin:$PATH stap -L 'process("beam").mark("*")' | grep message
process("beam").mark("message__queued") $arg1:char* $arg3:int $arg4:Sint $arg5:Sint $arg6:Sint
process("beam").mark("message__receive")
process("beam").mark("message__send") $arg1:char* $arg2:char* $arg4:Sint $arg5:Sint $arg6:Sint
process("beam").mark("message__send__remote") $arg1:char* $arg2:char* $arg3:char* $arg5:Sint $arg6:Sint $arg7:Sint

So, the probe is there, it’s named correctly, and it’s got the proper number of arguments but … SystemTap doesn’t like it.  Perhaps there’s a different version of SystemTap syntax also?  Hrm, I don’t know the answer, but I’ll try to find out in the next week….

Conclusion

I hope you’ve enjoyed this little tutorial.  Now go have some fun and explore Erlang and SystemTap!

 UPDATE 1, Saturday Nov 19, 2011:

It’s quite possible that despite the number 6 being greater than the number 5, CentOS 6 is using a version of SystemTap 1.2, and CentOS 5 is using a version of SystemTap 1.3.  If that’s true, then it’s probably better for CentOS 6 users to roll their own SystemTap support.  Many thanks to Andreas Schultz for pointing this out.

Another tip from Andreas: Newer versions of SystemTap are available using the CentOS continuous release (CR) repository.  Beware that packages there may not be 100% stable.

UPDATE 2, Tuesday Nov 29, 2011:

There appears to be an additional problem, this time with CentOS 5 and its SystemTap version 1.3: probes defined by shared libraries do not work correctly.  To demonstrate, run this little demo in two different windows/login sessions.

In window number #1, run this command using a DTrace-enabled Erlang build. Depending on where your Erlang package was installed, you must change the directory paths to match your environment.

/usr/local/erlang/R15A.dtrace-review-20111127/bin/erl -eval '{dtrace:init(), [{dtrace:p(42), io:format("."), timer:sleep(1000)} || _ <- lists:seq(1,2000)]}.'

This 1-liner will load the “dtrace.so” shared library into the Erlang virtual machine and then try to fire a probe within that library once per second.

In window/login session #2, as root/superuser, run the following two commands. Also, note the output from the first command should match what I show here.

# env PATH=/home/local/erlang/R15A.dtrace-review-20111127/lib/erlang/lib/dtrace-0.8/priv/lib:/usr/local/erlang/R15A.dtrace-review-20111127/lib/erlang/erts-5.9.pre/bin:$PATH stap -l 'process("dtrace.so").mark("*")'
process("dtrace.so").mark("user_trace__i4s4")
process("dtrace.so").mark("user_trace__s1")

# env PATH=/home/local/erlang/R15A.dtrace-review-20111127/lib/erlang/lib/dtrace-0.8/priv/lib:/usr/local/erlang/R15A.dtrace-review-20111127/lib/erlang/erts-5.9.pre/bin:$PATH stap -e 'probe process("dtrace.so").mark("*") {printf("Shared lib probe fired");}'

If the probe is working correctly, then you should see the “Shared lib probe fired” message printed once per second. Unfortunately, on my CentOS 5 machine, the probe does nothing.  :-(

Quick, I Want an Overview and Some Slides….

I gave a presentation at the Erlang User Conference  in Stockholm a couple weeks ago.  The topic was about adding DTrace probes to the Erlang virtual machine.  The presentation slides (and soon video, I hope) are available online at the Erlang Factory website.  Click on the easel icon to fetch a PDF copy of the slides.  When the video is available, it will probably appear as an embedded video player/applet/thingie.

I’ve given a few other talks at EUC, Erlang Factory, and Erlang Workshop venues.  I’ve never had as many people tell me afterward that they both enjoyed the talk and are looking forward to using DTrace (or SystemTap) with Erlang programs.

Getting DTrace into the Erlang/OTP R15 Release

I submitted a set of four patches to the erlang-questions mailing list on Wednesday night, November 16, 2011.  The patches were accepted by the OTP Team and are now sitting in the “pu” branch of the Erlang/OTP GitHub source repository.  The name “pu” stands for “proposed updates”.  The “pu” branch is used for all externally-submitted patches which have passed the initial round of review.  Everything in the “pu” branch is tested regularly by Ericsson on all supported Erlang/OTP platforms (e.g. Solaris, Windows, Linux, …).

Now we wait and see what Ericsson’s tests show.  The DTrace patches were submitted well ahead of the code freeze date for the Erlang/OTP R15B01 release, which is tentatively scheduled for December 14, 2011.  If all goes according to plan, the patches will be in the official Erlang/OTP R15B01 release next month!

What Is Traceable?

Take a look at the “examples” directory in the dtrace application for some simple tracing scripts.  To see the actual definition of the DTrace probes and their arguments, please see the  erlang_dtrace.d definition file.  Please note that this link is a static one and may not always show the absolutely newest, freshest, most current version of the file.

Here’s a short summary of what’s traceable today using DTrace:

• Processes: spawn, exit, hibernate, scheduled, …
• Messages: send, queued, received, exit signals
• Memory: GC minor & major, proc heap grow & shrink
• Data copy: within heap, across heaps
• Function calls: function & BIF & NIF, entry & return
• Network distribution: monitor, port busy, output events
• Ports: open, command, control, busy/not busy
• Drivers: callback API 100% instrumented
• efile_drv.c file I/O driver: 100% instrumented

As of today, there are 60 probes available.

DTrace & Erlang: Supported Platforms

I’ve been testing the following platforms regularly:

• OS X 10.6.x, a.k.a. Snow Leopard.  I don’t know of any reason why it shouldn’t work with 10.7.x, a.k.a. Lion.
• Solaris 10.  I’ve run into difficulties with long-running D scripts on my Solaris 10 machine, but I’ll write more about that in a later blog posting.
• Linux, specifically CentOS 5 and CentOS 6 using SystemTap‘s DTrace compatibility API.  I’ll include some recipes for setting up CentOS boxes to be capable of running user-space SystemTap in another blog posting.  Things should also work when using a Fedora Core 16 system, but I haven’t tested that yet.

I’m hoping to be able to get FreeBSD 9 supported also.  At the time of this writing, FreeBSD 9.0RC1 is now available for testing.  Andrew Thompson @ Basho has started the FreeBSD 9 work and passed it on to me.  I’ll try to get it hammered into working condition over this weekend.

How do I try it?

First, clone a copy of my source repository at GitHub:

git clone git://github.com/slfritchie/otp.git

Then you need to choose the branch that you wish to build:

• Run the following if you want to compile the R14B04 version of Erlang/OTP: “cd otp ; git checkout dtrace-r14b04“.
• Run the following if you want to compile the R15A version of Erlang/OTP: “cd otp ; git checkout dtrace-r15“.  NOTE that this branch is not fully synchronized with Ericsson’s R15A development branch.
• Run the following if you want to compile the patches as I submitted them to Ericsson for inclusion in Ericsson’s source repo: “cd otp ; git checkout dtrace-review3“.

I also have a branch called “dtrace-experiment+michal2” that contains the primary copy of all of the changes that I later merge into the three branches listed above.

Here are the instructions for building the Erlang/OTP package.  I assume that you already have the toolchain dependencies (e.g. GCC compiler) and libraries (e.g. OpenSSL) already installed.


./otp_build autoconf
./configure --enable-dtrace {put your other config options here}
make
make install


If you’re using a Linux box with SystemTap installed, you will still use the configure flag “--enable-dtrace“, despite the fact that you’re going to be using SystemTap.

After you’ve installed everything, then you need two windows:

1. One window/login session to run an Erlang shell
2. One window/login session as the superuser (i.e., “root”) to run the D scripts.  If you’re using SystemTap, please wait until my next blog posting, which will be specifically about Erlang and SystemTap.

Using the first window, start an Erlang shell, e.g. /path/to/your/erlang/installation/bin/erl

Using your second window, start a D script from the examples directory:


cd /path/to/your/erlang/installation/lib/erlang/lib/dtrace-0.8/examples
dtrace -qs ./function-calls.d


Then, back in the first window, start typing commands at the Erlang shell prompt.  Watch the output in the second window.

Professionally, things at Basho are very good.  It’s a great bunch of smart, ego-firmly-in-check folks.  Basho sent me to the Erlang User Conference  in Stockholm a couple weeks ago to present my work on adding DTrace probes to the Erlang virtual machine.  Flying from Stockholm to San Francisco was a very long Sunday, but I managed to survive.  The meetings in San Francisco with my fellow Bashonians were great, as was a lunch with the old crowd at Gemini Mobile in Foster City.

I’m fighting a bad head cold at the moment.  That has made work on DTrace & Erlang stuff difficult this week.  (More on DTrace in a later posting.)

We’re quite looking forward to the Thanksgiving break next week, then travel to Salt Lake City the following weekend to visit our new niece Autumn as well as budding musician Thai and proud parents Victor and Carla.  And perhaps someday we’ll make plans for Christmas….

Meanwhile, it’s about time for the 2011 Minneapolis/St. Paul International Film Festival to start.  I bought my gold pass today and had a chance to visit with a couple of the organizers for a couple of minutes.  Almost all films have been chosen and scheduled, but Web site is still undergoing a lot of development. One missing features is easy browsing films by date & time.

The printed program will likely be done in a week or so.

I haven’t looked at the whole program yet, but (strangely enough) the title The Lutefisk Wars (IMDB) managed to jump out at me….

Seven Programming Languages on the Rise: From Ruby to Erlang, once niche programming language are gaining converts in today’s enterprise. http://infoworld.com/d/developer-world/7-programming-languages-the-rise-620

Installing CentOS 5 From Scratch

I was installing on virtual hardware, using VMware Fusion for MacOS.  The virtual machine’s configuration summary: 256MB RAM, 1 CPU, and a 20GB expanding disk.  A Hibari server can consume far more than 256MB of RAM if you’re storing a lot of keys, but for experimentation & development work, 256MB is fine.  Besides, it’s easy to change the amount of RAM when using virtual hardware.

Booting the CentOS 5 server DVD, I chose the graphical installation option.

Partitioning via LVM:

• LogVol00, /, 19584 MB
• LogVol01, swap, 768

Clock does not use UTC.  Most people I know don’t use UTC like that, so why does Red Hat (and therefore CentOS) make it the default?

Software package choices:

• un-select: Desktop – Gnome
• select: Server
• Customize later

After first reboot, “Setup Agent” runs automatically:

• Authentication: change nothing
• Firewall config: Security level disabled, SELinux disabled
• Network config: DNS config: set appropriately to your environment
• Services config: turn off: avahi-daemon, bluetooth, cups, isdn, nfslock, pcscd, portmap, rpcgssd, rpcidmapd, sendmail, yum-updatesd.
  • There are a few other services that could be turned off, but those are mostly small quibbles not worth worrying about.
  • The SSH service is on by default, hooray.
  • The default sshd config allows “root” to log in, which I frequently use.  Your opinion may not agree.

Once booted fully to a login prompt, log in, install VMware tools, use defaults for everything, and then reboot.

Building Erlang

The blog article at ErlangExamples.com has the longer explanation.  A really short summary of how to install the prerequisite software packages is:

sudo yum -y install make gcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel

I put Erlang in a non-standard place because it’s easier to deal with multiple releases of the Erlang runtime on the same machine. By default (i.e., without using the “--prefix” argument to “./configure“), the main Erlang commands “erl” and “erlc” are installed in /usr/local/bin.  With the commands below, those commands will be available instead at /usr/local/erlang/R13B04/bin.

  tar zxvf otp_src_R13B04.tar.gz
  cd otp_src_R13B04
  ./configure --prefix=/usr/local/erlang/R13B04
  make
  make install

Building Hibari

As root:

  yum install libtool ncompress

Bummer, “yum install git” doesn’t know about Git.  Have to build it from scratch.

  mkdir -p /usr/local/src/git
  cd /usr/local/src/git
  wget http://www.kernel.org/pub/software/scm/git/git-1.7.2.tar.bz2
  tar jxvf git-1.7.2.tar.bz2
  cd git-1.7.2
  ./configure
  make
  make install

As non-root (use only one of the 1st or 2nd line, depending on your login shell).

  set path = ($path /usr/local/erlang/R13B04/bin) (csh/tcsh style)
  export PATH=${PATH}:/usr/local/erlang/R13B04/bin (sh/bash style)
  mkdir -p ~/work/hibari
  cd ~/work/hibari
  git clone git://hibari.git.sourceforge.net/gitroot/hibari/bom .
  env BOM_GIT=git://hibari.git.sourceforge.net/gitroot/hibari/ \
  ./bom.sh co src/top/hibari/GDSS
  ./bom.sh make
  make

As root:

  cd /path/to/hibari/pkg
  ./installer-gdss.sh -o silent

And because I install Erlang in a slightly weird place, I needed a symlink to point to the place where /etc/init.d/gdss expects to find it. (As mentioned above, I install it at /usr/local/erlang/R13B04.)  As root:

  ln -s /usr/local/erlang /usr/local/hibari/ert

Now to start Hibari and then configure the local node to be a standalone/1-node system.  Again as root:

  /etc/init.d/gdss start
  /etc/init.d/gdss provision-standalone

That’s it, we’re all done.  You can point your Web browser at http://localhost:23080/ (or substitute the Hibari machine’s DNS hostname or IP address, if that machine isn’t the localhost) to verify that the server is indeed running.

NOTE: You’ll need Git and Mercurial in order to check out the basho_bench source.

You’ll also need the (very cool) statistics package R installed in order to create basho_bench’s graphs. For CentOS 5 users, I don’t have any good advice, sorry. For Mac OS, once I’d installed MacPorts, I think it was just a matter of “port install R”.

Now make a copy of the example config file:

cp examples/hibari.config ./my-hibari.config

… and edit it to reflect your local environment. Sorry, I don’t have any extensive documentation on it right now. There is basho_bench- specific documentation in the “docs” subdirectory. All of the Hibari- specific stuff is (hopefully clear?) in the comments of the example config file.

The ‘code_paths’ section of the config will require pointers to a bunch of code directories in the Hibari source. *ALSO*, there is one patch that you need to install in the …/hibari/src/erl-apps/gmt_util__HEAD directory. Download the patch file and put it in /tmp. The URL is: http://www.snookles.com/scott/hibari/gmt_config.erl.patch

Then apply it and rebuild using:

cd /your/path/to/hibari/src/erl-apps/gmt-util__HEAD
patch -p1 < /tmp/gmt_config.erl.patch
rm .bom*
cd ../../../
make


OK, now back to editing “./my-hibari.config”. Hopefully the comments are clear enough for you to figure out what to do. As a first experiment, I suggest that you ignore the “Erlang native client” and instead use the EBF client. Most of the entries in examples/hibari.config are good enough.

I suggest changing ‘duration’ to 1 minute and ‘concurrent’ to something small’ish like 10 (until you’re familiar with the tool). Change the code_paths to reflect where you’ve got the Hibari source. Then change the ‘hibari_servers’ list to point to your server(s). For a single node system, make only one entry in this list. :-)

Now, time to run the tool.

./basho_bench ./my-hibari.config


The output data will go into a local subdirectory called “tests”. When the run is finished, you’ll see something like:

=INFO REPORT==== 16-Sep-2010::01:21:31 ===
application: basho_bench
exited: stopped
type: permanent
Test completed after 1 mins.


The command “make results” will create a graph. The graph is a PNG file that is located in the same directory as the test results. Note that there is a symbolic link in the “tests” subdirectory that points to the most recent test run results. For example, “tests/20100916_012028″. That file can be viewed by most bitmap graphic image viewing utilities; for MacOS, I usually use:

open ./tests/current/summary.png


I’ll include a sample config and graph. The config is for a 10 minute run with 100 load generator threads. This isn’t a very good test environment: the server is a small RAM, single disk machine. The basho_bench machine is a MacBook Pro that uses a WiFi network to talk to the server. This is not the kind of environment that you want to try to optimize, so don’t bother trying. This is just meant to be a quickie example, to be emailed before I fall asleep.

Happy hacking!

-Scott

(Click on the image below to see a full-size, hopefully-undistorted image).