Using Docker and userspace networking to simulate real-world networks

This is a guest post from Vanessa Pyne and Max Rottenkolber. Vanessa works at Daily and (among other things) assembled Daily’s custom automated test framework. Max is a freelance consultant at Inters with expertise in software networking.

Daily provides real-time video and audio APIs to help developers build and iterate on their apps quickly and efficiently. As we have all likely experienced first hand, real-time communication is closely tied to the quality of the underlying network. To make a reliable real-time communication product, adapting to diverse network conditions is key.

Unsatisfied with existing network conditioning solutions, we decided to create a solution to help test Daily’s products under varying network conditions. As a result, we created Synthetic Network, a tool based on Docker and packet processing in userspace to simulate arbitrary network conditions to Linux applications.

Introducing Synthetic Network

Synthetic Network ships as a Docker image containing:

• a Rust program that forwards packets between two Linux network interfaces, and introduces artificial latency, bandwidth limits, packet loss, and jitter
• a frontend that exposes a JSON API and a Web UI used to configure synthetic network conditions
• and an entrypoint script that starts both of the above, and performs some setup to ensure network traffic of applications running within the container are routed via the synthetic network

The idea is that you derive an image FROM syntheticnet and add the application you want to test. You can then use the resulting image to spin up containers with individually adjustable network conditions.

As an example (and a useful tool for ad-hoc testing), we include a Dockerfile that derives from the synthetic network image and adds a Chrome web browser exposed over VNC. The resulting image can be used to test any web application under arbitrary network conditions. It's really quite magical.

Once the Chrome-on-Syntheticnet container is running, you can point a VNC client at localhost:5901. It will then be greeted by a Chrome window showing the web UI used to configure the synthetic network interactively, aka Quality of Service. If the current Quality of Service settings please you, pop open a new tab to browse the web with the network conditions set in the web UI.

The web UI lets you tweak the network quality separately for user-defined flows, so you can adjust quality of service individually for specific hosts, protocols, and port ranges. In addition, a monitor for currently active traffic provides visibility into real-time network activity, lets you quickly add matching flow conditioning rules, and helps cross-reference activity with the rules affecting it.

The magical part really sparkles when you see adjustments in the UI affect network conditions in real-time. For example, in the screenshot we join a Daily meeting in a new tab via the VNC and we join the call from a local browser. Via the Quality of Service UI, we can choose UDP flow from a particular IP and limit the ingress bandwidth. Then we see the quality of the remote video degrade (in real time!) in the local browser window.

Scripting and automated tests

While manual fiddling with the network is infinitely satisfying, sooner or later you’ll want to automate your network testing. To that end the synthetic network frontend comes with a small JavaScript library that wraps the JSON API in a programmatically friendly object.

const SyntheticNetwork = require('synthetic-network/frontend');

// Connect to JSON API endpoint of a syntheticnet container
const synthnet = new SyntheticNetwork({hostname: "localhost", port: 3000});

As mentioned earlier, Daily has a custom modified automated test framework built in Node, which can now easily consume the synthetic network module. We write a test with two participants (browsers). One a local webdriver instance and a synthbot™ —our affectionate name for a container running synthetic network and Daily-flavored selenium-webdriver. An example of an automated, fairly sophisticated network condition test is as follows:

1. First, we select only UDP flow and initialize it to a reasonable bandwidth

synthnet.addFlow('webRTCudp', { protocol: 'udp' });
synthnet.flows.webRTCudp.link.egress.rate(1e6); // 1 Mbps
await synthnet.commit();
await assert(localWebDriverRecvVideo(), true);

2. We assert the local webdriver instance receives a video stream from the synthbot.We then set the egress bandwidth for that UDP flow to an egregious 100kb for 10 seconds.

synthnet.flows.webRTCudp.link.egress.rate(100); // 1 Kbps
await synthnet.commit();
await driver.sleep(10 * 1000);
await assert(localWebDriverRecvVideo(), false);

3. Now we can assert the other participant ceases to receive a video stream from the synthbot. The high level question we are attempting to answer is will the app recover from a network interruption as such. We return bandwidth to a more reasonable 1Mbps.

synthnet.flows.webRTCudp.link.egress.rate(1e6); // back to 1 Mbps
await synthnet.commit();
await assert(localWebDriverRecvVideo(), true);

4. Finally, we can assert that the local webdriver instance receives video again to confirm the app has recovered from the network interruption.

Looking under the hood

This works by creating a new veth(4) pair inside the container. Then we can update the container’s default route to send packets via the bridge network we want to condition using one end of the virtual ethernet device pair instead.

Next, our userspace packet processing program opens both the original network interface created by Docker for the bridge network, as well as the other end of the veth pair with the SOCK_RAW and AF_PACKET options for the socket(2) system call. It then forwards packets at layer 2 between both interfaces.

The forwarder itself is a userspace Rust program based on Rush, a Rust port of the Snabb kernel-bypass networking toolkit. Rush inherits Snabb’s philosophy to demystify software networking, representing packets as simple flat byte arrays. The forwarder can delay, drop, or even mutate packets, and all of the network conditioning happens within.

While synthetic network containers need to run as privileged in order for us to create veth interfaces and open devices with SOCK_RAW, Docker’s bridge networking allows us to set up our forwarding scheme without affecting network configuration on the host. Most importantly, Docker provides us with a stable API across developer machines which might use Docker for Mac, Docker for Windows, or a Docker installation on Linux.

More information

For more information on this tool you can visit the Synthetic Network GitHub page.

Max journalled the design and development of Synthetic Network in a series of reports, which have also been included in the GithHub repo to explore.