Working Together While Apart

On 18 March 2020, our staff in Amsterdam left the familiar surrounds of the office and began life working remotely, followed shortly afterwards by our Dubai office. By now, this is not news – just change the date, city and company name and this basic story would probably apply to most of you reading this article. Many of us are in the same boat, adapting to a new situation while trying to maintain or even improve the services we offer. And the services we offer are now more essential than ever, at a time when staying connected over the Internet is widely recognised as being important to people’s well-being.

In this article, we wanted to give you a look into how the RIPE NCC is managing its work and adapting, and especially some of the changes we’ve had to take make to continue dealing with all the old challenges in addition to some new ones.

Supporting our Members

One of the more active steps we took since the pandemic started was to extend the payment schedule for our existing members. Early on, we were contacted by a small number of members who let us know they would have trouble paying their invoice on time due to the crisis. Such contact is usually indicative of larger underlying concerns within our 20,000-strong membership, and with the pandemic spreading rapidly throughout the service region we needed to act quickly to support our members, and the deadline for payment was extended by three months following a Board resolution on 26 March.

Although extending a payment schedule might look like an easy matter on the face of it, there are quite a few pieces that we needed to take into account. We’ve worked hard over the years to automate systems, and our financial administration is no different. Normally, members who don’t pay within 60 days receive no approval of requests, and from there it moves to suspension of service after 90 days and eventually termination of the Standard Service Agreement. The financial administration is therefore closely connected to our registration services, and changing the schedule required quite a bit of engineering work and close cooperation from our Finance, Legal, Registration and Customer Service teams to properly understand and mitigate the impact. Navigating this complex process without teams being able to get together in a room and thrash out the issues was a good early test of how well we can collaborate in this new environment.

Our registry and customer services teams carry out the all-important work of reviewing resource requests, handling the processes around membership applications, and dealing directly with members to help resolve their registry- and account-related issues. So, what are we seeing there? The work for our staff here is largely unchanged in nature – there has been no decrease in number of LIR accounts since IPv4 run-out; the number of tickets we are dealing with has only decreased very slightly since we started working remotely; we continue to carry out due diligence investigations; and we talk to our members during scheduled Assisted Registry Checks (ARCs). The graphs below give some idea of what's been happening in terms of LIR account applications and the amount of support requests that our IP Resource Analysts and Customer Service Analysts have been handling.

Although the number of new LIRs being activated is at historically low levels, the weekly number already started trending downwards in late January. Though the lockdowns and economic outlook may play a role, we also had the IPv4 run-out in November. People may no longer find 256 IP addresses per LIR account to be worth rushing into, and as our waiting list wait time for a /24 has been at zero days since mid-January, there could be much less urgency about acquiring these blocks.

Perhaps most surprising is how little our work has changed overall. We handle the same emails, phone calls, and the same printing and sending of documents – only this is now done from home. Anecdotally, from our colleagues who carry out the Assisted Registry Checks (ARCs) with our members, the topics that have been brought up more frequently in the last two months have been requests for help with documenting usage, RPKI queries and requests for demos of our tools like RIPEstat, RIS and RIPE Atlas. In the ARCs, one of things we check is whether the LIR has made assignments from their IPv4 or IPv6 allocation. If they haven’t, then we offer to work with the LIR to create assignments, and by doing so, document the resource usage in the RIPE Database (keeping in line with the relevant policy).

Our IP Resource Analysts have many stories from pre-coronavirus days about having ARC calls with members who are at home with the kids shouting, or who are sitting in their cars in traffic jams, or even taking a break from a wedding party to get their resources and account checked with us on an ARC (true story!). Now we join our members in calling from our kitchen tables and helping to make sure everything is working well for them. So far, this is all going smoothly, and it even gives us a welcome chance to connect with people while we live in a locked-down world.

Moving from Physical Engagement

Since the end of February, in coordination with various hosts and partners, we’ve had to postpone MENOG, SEE, ENOG and a RIPE NCC Day. The last community event we sent staff to was DKNOG on 12 March, which became a virtual event at the eleventh hour (see Alan Eising’s RIPE Labs article on this). We also had to start planning for our first virtual RIPE Meeting. All of these postponements required a lot of work, especially from our event organisers, external relations and communications teams. We also needed legal support to review contracts with venues and other parties, and in the case of RIPE 80 we needed to organise a lot of refunds for registered attendees. And keeping both RIPE and the other regional communities informed about developments was a key aspect during this time.

In many ways, the work involved in postponing the first of these events – the MENOG 20 Meeting and Peering Forum in Bahrain – informed much of our approach to dealing with issues arising from COVID-19 in the weeks that followed. Bahrain was one of the earlier countries outside China to have to deal with a COVID-19 outbreak, and staff from around the organisation had to come together in February to decide how we would deal this situation, bearing in mind our responsibilities to both the community and to our staff. Many of the discussions then around events and travel, including with external health experts, translated into the wider guidelines and processes we would implement for staff travel and for managing our work in this period. It also helped to establish early on a regular pattern of communication among the staff who would make decisions on how we would move from office-based to home-based work.

With hindsight, the decisions to postpone MENOG 20 and to some extent SEE 9 (postponed on 6 March) were tough ones to make – at the time it was difficult to know if the COVID-19 situation would develop into a full-blown pandemic, and a lot of staff and external parties had worked for a long time to prepare these events for their communities. Subsequent meeting postponements were clearly the only course of action to take.

While we were reaching a decision on MENOG 20, we already formed a taskforce centred around our events team to begin work on preparing for a virtual RIPE 80 and creating scenarios for how this might work. For a short while, preparing for a virtual RIPE Meeting while simultaneously preparing for a physical meeting was quite a lot of overhead for our events and technical teams. And while staff worked on things like reviewing technical platforms (see Kaveh Ranjbar’s RIPE Labs article on the decision to use Zoom), the RIPE Chair discussed possible scenarios with the working group chairs, the RIPE Programme Committee and the RIPE NCC Executive Board, who made the decision on 18 March to go ahead with a virtual RIPE 80 and a RIPE NCC General Meeting that would be facilitated remotely. Looking back, it was a good example of how well our staff and the wider community can coordinate in a pinch. 

A New Way of Engaging

Engagement with our members and the wider Internet community is one of our most important functions as a coordination centre and the secretariat for the RIPE community. The pandemic has forced us to take a fresh look at how we engage with our members and the community. In pre-coronavirus days, we met with people at RIPE, roundtable and regional meetings, face-to-face training courses, member lunches, and by sending our staff to the events and meetings held by so many others in the industry and community. In this way, we educate our members and make them aware of our services, we establish connections and help to develop communities, we get valuable input on what's important to our members and community, and we represent the interests of our community to governments and other important stakeholders.

COVID-19 has changed how we do all this. To say our travel costs will come in under budget this year is probably something of an understatement. But removing the physical element of how we engage and educate does not reduce the need to do this. With this in mind, we’re looking at how online engagement in the short-term can meet the shortfall in physical engagement, but we’re also looking at how the current situation might change how we engage over the longer term.

We’ve been able to follow virtual events in the industry that have already taken place and get input from their organisers on what makes for a good virtual meeting. Our philosophy is that we don’t want to simply put a physical RIPE Meeting online but rather work to create a virtual event that engages an online audience and brings the community together in a unique and worthwhile way. At time of writing, over 1,500  people have registered to attend RIPE 80, which is a record for a RIPE Meeting. We hope to use this opportunity to bring in new people who normally would not be able to attend a physical RIPE Meeting and ensure that it still enables the community to share experiences, learn from each other and get valuable work done in a typically enjoyable RIPE way. You can still register for the meeting (and you should tell your colleagues about it too!). Capacity is unlimited, it’s free to attend and we hope it will go some way towards satisfying our need to connect with each other during these times. And yes, to answer the question on many people’s mind, one key element of the traditional RIPE Meeting will still be available to you – the RIPE Meeting t-shirt!

Online Learning

One service that has not had to adapt dramatically is our training portfolio, which has developed a much greater virtual presence in recent years to meet the demand from a much-increased membership (see the RIPE Labs article from Rumy Spratley-Kanis). Our work to create an online learning environment with the RIPE NCC Academy in the past few years means we have the experience in providing quality learning resources online, and we have a substantial bank of material that we can use to deliver training remotely.

The webinars that our Learning and Development Team deliver were already well-established too, and the main change we needed to make here was to increase capacity and frequency, which we have been able to do. And there has clearly been a lot of demand for spots – registrations have increased fourfold in the last two months. Our face-to-face training courses were always much appreciated and requested by our members, and the team has worked to convert these courses to shorter, two-hour remote learning sessions that use adapted material from the in-person courses. These two-hour sessions are held in real-time, using our webinar platform. Members can take part in activities and have the chance to ask us questions and engage with our trainers, who have also been working to convert their living rooms, kitchens and bedrooms into functional course delivery environments. The first of our remote learning sessions was on Measurements and Tools and was held on 30 April – keep an eye out for more of these in the coming weeks and months.

A big project that launched during the pandemic on 1 April was RIPE NCC Certified Professionals. This was the culmination of a huge effort from our Learning and Development Team (as well as practically all other departments), and it was helped by a large and diverse group of community members who provided valuable input. This programme, which allows people to take exams to certify their RIPE Database skills, was largely completed before the pandemic broke out, but a lot of coordination was needed to bring it online and get vouchers to all our members while we worked remotely. Although the cliché of people having more time during lockdown is not one we’re seeing personally, we do think this programme is ideal for operators who would prefer to be away from the office when taking an exam, and we encourage you to take some time to do this. We’re planning to add IPv6 certifications later in the year. Although creating these will not be as easy as it was before we started working remotely, we often tell our members that they can learn from anywhere; we’ll adapt this advice for ourselves as we work to build this programme into something great for our community.

A Certified Professionals call

Engaging Members and Community

One factor that makes our engagement work somewhat easier is the fact that everyone else is more or less in the same boat. There is a willingness from people to engage remotely that perhaps was not there before and the same desire is there for interacting and sharing knowledge and experience. We’ve noted a huge increase in the number and diversity of stakeholders who have been actively requesting remote engagement and events. And in many ways, our regular engagement activities were well-suited to moving to a virtual environment.

As noted above, we have had to postpone our regional events and in-person member lunches. But we’ve already been able to meet the shortfall by organising virtual member lunches that are still targeted at members in specific regions. So far, we’ve hosted two of these and members have been able to discuss a wide range of topics and services with our staff, albeit in a DIY lunch setting. Attendance at these lunches has been about what we expect for physical lunches, and we believe that the virtual version gives more people the chance to take part. Indeed, we are looking at possibly continuing with the virtual lunch format once we are again able to travel to meetings.

Regionally, we were also delighted to be able to host the Southeast Europe Country Report and Discussion as an online event. This report gives insight into the state of the Internet in eight countries in that region, and while we originally intended to launch it ahead of the SEE 9 meeting in Slovenia, we were able to host a valuable discussion via Zoom Webinar about relevant Internet governance issues in Southeast Europe, including IPv6 deployment, IXPs and cybersecurity.

The SEE Country Report and Discussion

We collaborated to create a remote training that was given to the UAE regulator, CERT and Law Enforcement Agencies on the RIPE Database and measurement tools. This was organised as an outcome of last year's roundtable in UAE where governments were asking for more support on using our tools to produce national reports on the status of their Internet. We are also planning to host more events in the SEE, MENOG and ENOG regions as initial feedback from participants has been very positive.

We foresee more of these types of virtual events in the coming months, and the online nature and ease of set up means we can host them according to demand and reach out to people who might not have been able to attend an in-person event. As well as our own events, we’ve also been attending other events online such as ICANN, IETF and LACNIC. And we’re happy to see virtual events such as ALNOG in Albania take place, and of course we’re always eager to be part of the discussions or to contribute presentations.

RIPE NCC Hackathons have also been a very effective way for us to stay connected with and support the technical community. On 24 March, we started a five-week long ongoing hackathon that was aimed at measuring the health of the Internet during the pandemic. The results of this will be presented during RIPE 80 and on RIPE Labs. And speaking of RIPE Labs, we’ve published 17 articles relating to specifically related to the Internet during the pandemic, and contributions from staff and community members have doubled over the comparable period before we started working remotely. Online engagement is perhaps moving people to the writing desk as much as to the conference call.

Other programmes such as RACI and the RIPE Fellowship will need to be more closely looked at – giving students and new community members the chance to come to our events is not possible at the moment, but we still want to give a platform to researchers and the academic sector, and also ensure we are bringing new and diverse voices into the RIPE community.

If all of this is not yet completely normal for us, we’ve been encouraged by how engagement is working so far, and we plan to take the lessons from the current period into account for the long term as we look to engage more effectively with members and stakeholders.

Critical Services and Measurements

The Internet and its operations have come under unprecedented scrutiny in the past few months. Topics that were once the preserve of industry specialists have moved into the mainstream – governments, media and the general public have been asking how the Internet will hold up under the pressure of a remote-working society. The RIPE NCC has been contacted by all these groups to ask about our ability to function at this time. And this is understandable, as the Internet enables the infrastructure that provides for people under lockdown, allows children to be educated and entertained, lets us connect with family and friends, and eases the mental burdens associated with being isolated.

Our teams running critical services such as K-root, DNS services, the RIPE Database, the LIR Portal, our websites and RPKI have all managed to keep things running with some extra effort needed to coordinate when issues arise. Having technical teams work together during an outage or similar has always relied on people getting in a room to rapidly diagnose issues and come up with solutions, as well as running to various other people who need to get involved such as comms people to inform our members. Working remotely is a very different story, and our engineers and other staff can no longer look each other in the eyes at critical moments, and this has been one of the more difficult aspects of remote work.

And now more than ever before, our tools that allow people to measure and monitor the Internet, as well as troubleshoot their networks have taken a new level of important. RIPE Atlas, RIPEstat and the Routing Information System (RIS) have played a big part in not only letting people see what's going on with the Internet, but also providing a high level of assurance to people that the internet continues to function as it should. We’re also happy to see projects such as the Internet Health Report, which began with a RIPE Atlas Hackathon and was funded by the RIPE Community Projects Fund, use RIPE Atlas data to monitor network delays during the pandemic.

Although RIPE Atlas relies on physical probes, we send them out via a third party, and this has not been affected by our remote working. However, various restrictions apply in various countries, so if you want to use RIPE Atlas and are concerned you won’t be able to receive a probe, then look at the software probes that we launched in February this year.

The teams managing our measurement tools have not experienced difficulties working remotely; in fact, the only piece of choreography that was needed was during an upgrade to the RIPE Atlas firmware, which requires a physical presence by design and needed staff to come to the office to perform a socially-distant upgrade.

Technical Support

In terms of the crucial IT support that allows us to work remotely, this has gone very well so far. We moved to our current office in Amsterdam in January 2017, and at this time our IT team decided while making the switch to set up our systems so that physical presence in the office would be barely necessary. Although this decision was not made with a pandemic in mind, it has certainly paid off with IT staff being able to fully support their colleagues through remote systems.

Much of the IT team has been working for the past couple of months to prepare for the RIPE 80 virtual meeting, and in that period, we changed plans to travel to Sweden to end our colocation there and this was handled very nicely by Netnod. And we also have no need to attend our colocation facility in Amsterdam. In fact, the two biggest issues we’ve experienced have been out of our hands to a large extent – shipping laptops and equipment to new staff members, and substandard connectivity that our staff experience with home connections.

The People

Putting the health and well-being of staff at the top of our priority list during the past few months is something we have felt obliged to do. And this is also something that has been necessary for the organisation to do in order to meet its commitments to members and stakeholders in terms of being 100% operational during the current period. Asking staff not only to work harder to meet the same service levels, but to do so while working from home and dealing with all the personal issues that this pandemic brings such as looking after children or being isolated from their families, means that as an organisation we also have to work much harder to ensure our people are coping with the new situation.

From the beginning, management and HR have made sure that any change in the current situation is quickly and clearly communicated to staff. All relevant information on COVID-19 and how it relates to our working life is published in a centralised location for staff, and management will use all channels to make sure staff are aware of any changes. This includes how any government decision affects us as well as information on scenario planning for an eventual return to office life. Former quarterly staff meetings are now shorter and Zoom-friendly and take place monthly to try keep everyone up to speed with current issues, And HR is continually assessing and adjusting policies to reflect reality rather than the old office-based world we lived in.

At the RIPE NCC, we have a situation where the vast majority of staff are not working in their home country, which brings its own challenges. We also need to actively push people to take vacation time to make sure they don’t become overworked at a time when travel and the ability to visit our home countries is not possible. As an indicator on staff well-being we’ve been looking at sickness rates daily, and so far there has been no increase during the period we’ve been working remotely. HR has started a "recharge your battery" campaign that asks people to take the breaks they need as well as days off to rest and recuperate. They also created online workshops that help people to manage work from home in a healthy way, and they are also giving lots of extra attention to our colleagues who are struggling with balancing work and family needs. We even have a daily exercise session on Zoom that's proven to be a fun way to get colleagues connected.

One area that has been difficult is in the onboarding of new staff. Previously, new staff would have a full programme of mentoring with people from across the organisation to help them get up to speed on the job and also socially. This is not possible, so we’ve found ways to change the onboarding process to an online one - we arrange an online coffee with people from all departments and we have created introduction videos. There is no substitute for getting to know each other in person, so we will do all we can until we're able to meet in person. We are already looking forward to meeting new colleagues who have joined since mid-March, including our new Managing Director Hans Petter Holen who has taken the reins at a difficult time and has had to hit the ground running while working from Norway.

We are also putting a lot of effort into making sure staff stay connected with each other – not just from a work perspective but also socially. Apart from frequent and regular check-ins with colleagues over Zoom, we’ve taken steps to help people maintain some of the social bonds that make the RIPE NCC a very nice place to work. We now have an always-on Zoom room where people can drop in and have a coffee or drink (after 5PM) with each other. We use that to arrange online staff social events, although much of the old social activity has organically moved to an online arena. We have monthly lightning talk sessions where staff can present on anything they like, and we now run these as soon we have enough content. Smaller events like this give us a chance to have a shared experience and help us remember that we’re still part of a bigger group and we’re all going through this thing together.

Lessons Learned (and Remembered)

In looking at what we’ve learned about ourselves in the past few months I’m struck by the multitude of positives we can take from it all. We’ve been incredibly adaptable and supportive of each other during a very difficult time. This was helped by being prepared from an early stage and being willing to act quickly and communicate transparently both internally and externally. We also found that technical setups and processes we’ve had for many years have now come to prove their worth.

Right now, we don’t know how long we’ll be working away from each other. But we do need to look at this not as a temporary situation but one that will impact us all and change how we work in the coming years. So while this article covers much of what we’ve gone through recently I feel it only scratches the surface and we’ll all be coming back to look at those longer-term impacts in the months and years ahead.

We also recognise that we are indeed part of something bigger. We often talk about the community we’re involved in and we have perhaps taken that somewhat for granted when life was relatively normal. But we are seeing now on mailing lists and at virtual events and through our personal connections that we are all willing to help each other and go the extra mile to share our experiences and offer whatever support we can to each other and to the people who rely on our work. If there has been an upside to this terrible situation, then for me it has undoubtedly been the truly global nature of this event and the push that has given us to come together within the RIPE NCC and as a much wider community who are willing to support each other when we need it most.