drk’s Electronic Music Blog

Bonjour El Capitan! Hello? Hello?

We (Delora Software) recently chased down a problem a few of our KymaConnect customers encountered on OSX 10.11. KymaConnect, for those of you outside the Kyma world, is a kind of communication center between MIDI, networked devices, and Symbolic Sound's Pacarana hardware sound engine family (it also works with the previous generation sound engine, the Capybara). KymaConnect usually connects to the Pacarana' Ethernet port to the Mac's Ethernet port using a direct cable connection. No network hub, switch or router is necessary. This connection forms a private IP network between the Mac and the Pacarana.

The Pacarana uses DHCP to obtain its IP address. When it does not find a DHCP service, it self-assigns its IP address in the link-local (169.254.XX.YY) range. It uses Bonjour to advertise its offered OSC UDP service.

KymaConnect relies on this Bonjour information to locate and connect to the Pacarana. The customer symptoms were that the Bonjour information was no longer working in a consistent, reliable manner. We set out to replicate and troubleshoot the problem. What we discovered was a bit surprising: El Capitan was behaving differently than previous OSX releases. And by previous releases, I mean every release all the way back to Leopard (10.5).

As it turns out, OSX 10.11 sends certain Bonjour-related transactions a bit differently than it previously did, and those differences were causing problems for the Pacarana. I contacted the wizards at Symbolic Sound and they quickly analyzed the problem and then created a firmware workaround. The work around seemed to address the underlying problem and all seemed well. Except we noticed intermittent failures when the Mac's Ethernet port's IP address was configured by manually assigning a link-local IP address instead of using DHCP (and yes we are aware that manually assigning a link-local IP address is coloring outside the lines).

What follows dives a bit into the details and is probably of limited interest to a typical KymaConnect user. So here's the bottom line: if you are running KymaConnect on OS X 10.11, we no longer recommend that you use a manual IP address for the Mac's Ethernet port, as was previously recommended here. You should instead set the IP address to DHCP.

A direct Ethernet connection from the Mac to the Pacarana depends on both devices self-assigning their IP address; there is no DHCP service present. This works fine, and in our testing it worked well with El Capitan, once the Pacarana received its new firmware. Delora Software, though has advised, in certain situations, that users instead manually assign the Mac Ethernet port's IP address in the link-local range.

Why use a manually assigned IP address? Because OSX takes awhile before it decides that a DHCP server is unavailable, and during that time it is impossible to establish a connection with the Pacarana. The typical delay is rarely noticed in the studio but on stage it can be critical if the Ethernet cable is accidentally disconnected.

Somewhere around OSX 10.9 (Mavericks), the self-assign delay grew longer, much longer under certain circumstances. Some users were noticing it in their daily studio work and wanted a way to shorten the delay. So we started recommending the manual IP work around. OSX seemed happy with it, the Pacarana seemed happy with it, and all was well, until El Capitan.

I will be the first to admit that manually setting an IP address in the link-local range is not completely playing by the rules. Still, El Capitan's behavior was surprising because it not only deviated from the previous six major OSX releases, but also because its own behavior was inconsistent. It worked most of the time but sometimes it did not. Worse yet, when it failed. it would not self-recover.

Here's a recipe for "encouraging" a failure:

  1. Have a working connection Ethernet between the Mac and the Pacarana.
  2. Break that connection by some combination of powering off the Pacarana, putting the Mac to sleep, or pulling the cable.
  3. Attempt to re-instablish the connection by reversing step #2.

One side note. In this particular application it is very common for the Pacarana to be power cycled. Many users power their Pacarana each time they use Kyma, and when finished, they power it off. Kyma even encourages this behavior by asking if you want to do this when you quit.

Usually El Capitan sails through this without issue. But sometimes it does not. When it fails, OSX's Bonjour service no longer properly resolves the Pacarana's IP address. The failure is outside the application (like KymaConnect), and can be confirmed with no application running simply by using OSX's command line interface to Bonjour services.

I spent a lot of time making friends with mDNSResponder, OSX's low-level Bonjour service manager, and its brother, the command line program dns-sd. Between the two of those, plus inspecting logs, here is what I discovered. mDNSResponder knows of the Pacarana's OSC UDP service but it never seems to issue a request to resolve the IP address when asked. It smells like a caching problem but that was not consistent with the behavior. mDNSResponder reported that there was no resolution, and the error code even implied that the request timed out. Yet there was no evidence that the request was ever made.

mDNSResponder would continue in this fashion until some external event jostled it back. Sometimes the problem clears itself if you ask for a resolution enough times. Other times it clears if there is some type of other network event, like connecting the Mac to Wi-Fi on its Airport. The following Terminal command also consistently cleared the condition:

sudo killall -HUP mDNSResponder

Yep, forcing the mDNSResponder daemon to restart, re-read its configuration, and (a side effect) clear its cache, gets everything back on track. It's a mystery.

This has only been observed when the Mac's Ethernet port has a manually assigned IP address in the link-local range. It has not been observed if the manual IP is outside the link-local range. For example manually assigning 192.168.1.XX work fine. So it seems to be link-local related.

In our testing the failure rate was not high, perhaps less than one time in ten per opportunity. The above Terminal command seemed to work reliably to clear a failure. So that's a possible work around for those who absolutely believe that they need to continue using manual IP assignment. And when would that be?

The manual IP technique was originally a work around for the long start up delay caused by the way OSX treated DHCP assignment when no DHCP server was present. So the tradeoff on OSX 10.11 would seem to boil down to which is worse, the start up delay every time you power up the Pacarana, or running the risk that it won't connect some small percentage of times, and you then have to run the magic Terminal command.

We measured the delay from Ethernet transition event until KymaConnect was happily talking with the Pacarana. This is on only one system so it's only one data point. What we observed was around 30 seconds of delay when DHCP was used and the Pacarana was power cycled. Interestingly if the cable was removed, or the Mac put to sleep, the delay was more on the order of 20 seconds.

When a manual IP address was used, the delay after powering the Pacarana was around 20 seconds, and for other transitions around 12 seconds. Neither one of these represents a significant improvement over the more reliable DHCP approach. So it would seem that using the manual IP trick was a dubious choice given the small absolute time savings.

Furthermore, here is some additional context. The time to launch Kyma 7 was around 40 seconds, and that was with the application residing on a fast SSD volume. So in the most common scenario of starting up Kyma to do some work, the connection is already up and running before Kyma finishes launching. Seems like staying with DHCP is the right call.