Update on the RIPE Atlas Anchor VM pilot

We advertised our plans to evaluate RIPE Atlas anchor VMs in December last year. Although we had a group of willing partners right from the outset, we since received calls from volunteers in the community who also offered to participate. Since then, we installed a number of these anchors in various places, helping us get an insights into what goes well and where the challenges lie.

Besides setting up our own anchor VM, we started the pilot with three other VM anchor hosts: Sander Steffann, Jared Mauch and Amazon. While we already had agreements with Amazon to work on the pilot together, the other two individuals expressed their interest when we first advertised the upcoming pilot. We're very grateful for their assistance!

Installation Experiences

Our approach to setting up the VM anchors was to take the requirements that we already had for the hardware anchors and adapt these to the VMs. In the case of physical anchors, the hardware requirements are straightforward, since we use the same servers in all cases and take care of the OS installation ourselves. In the case of VM anchors, providing the "hardware" with enough capacity is the responsibility of the hosts.

Also, at least for the pilot phase, we wanted to avoid dealing with the plethora of management interfaces, consoles, subscription and payment methods, etc., used by those who could provide VMs for this purpose -- so we delegated this problem to the volunteering partners. It's entirely possible that we would continue to adopt this approach during the production phase, otherwise much of our resources would be spent on one-time administration tasks.

Due to the diversity of virtualisation platforms the RIPE NCC cannot generate a universal installation image as we do for the hardware anchors. Therefore we request the host to pre-configure the VM with similar characteristics as the ones found in the hardware anchors. This includes disk partition layout, OS version, CPU, memory and so on.  When the host delivers the VM to us, we check if the requirements are met. If they are met, we configure the VMs in a very similar way that we do with the hardware anchors. This includes installing the RIPE Atlas probe software and anchor services on it, performing the network calibration measurements and adding them to our monitoring system. When all this is finalised, the anchor is added to the mesh measurements (anchoring measurements) and made public.

Early Results

Sander and Jared created their VMs in virtualisation infrastructure hosted in their premises. Amazon created the VM on their public cloud. Sander’s and Jared’s anchors were installed quickly and without notable issues: in this case our process did not need significant customisations.

The installation from Amazon was more challenging: after some back-and-forth about what virtualisation technology fits best, we managed to bring the VM up and configure it as usual. One challenge was addressing the network configuration as, in this case, the machines don't simply get publicly reachable IP addresses. However, our contact at Amazon was quick to help us overcome these hurdles. As a result, the whole process of the installation of this anchor required some further customisation to our current deployment workflow.

Our most recent addition comes from Digital Ocean who, as of the time of writing, have just finished setting up their first VM. Since Digital Ocean also has physical anchors, this gives us the additional benefit of being able to compare, much more directly, the performance of physical anchors against that of VMs.

All in all, these machines are up and running, and are almost indistinguishable from physical anchors. Here are the links to the probe pages for the first five anchor VMs:

1. nl-ape-as203993 (Sander)
2. us-chi-as2914 (Jared)
3. us-qas-as16509 (Amazon)
4. nl-ams-as3333-4 (RIPE NCC)
5. ca-tor-as14061 (Digital Ocean)

These all have a special system tag "system-virtual" that makes them recognisable as VMs. And you can explicitly include or exclude these as probes in your measurements. In the future, if and when we make VMs a production service, we will make it more visible in the list of anchors and we might even create a separate list for them.

Next steps

We are entering the last two months of the pilot and we want to install more anchors from Amazon so that we can validate the installation process. We will stop installing new VM anchors by end of May as we want to use June to gather all results and make a decision whether we should go into production or not. The decision will be based on a few key factors:

• Changes required on our system to support the VM anchors
• Feedback from the community on whether they find this useful or not
• Feedback from the hosts themselves
• Expected impact on the deployment process and operations (complexity) for the RIPE NCC

We will also report on the above mentioned physical vs. VM comparison if we get the chance to do so in time.

We plan to publish a new RIPE Labs article with the results and the decision by the end of June.

Feedback

Feedback in all its forms is welcome. Leave your questions, suggestions and statements of enthusiasm in the comments below, or send an email to us at atlas@ripe.net.