Note: this article is mostly about Europe. Considerations and conditions in the US and the rest of the world are very different.

It turns out that through PowerDNS and its parent company Open-Xchange, we know a lot about how large-scale European communication service providers work - most of whom are our customers in some way.

Problem statement

There is an ongoing discussion whether European telecommunication companies should use equipment from Huawei. The main concern (even not always explicit) is that since Chinese companies are heavily influenced and often owned (indirectly) by their government (or ruling political party), using Huawei equipment for 5G might be bad for us.

The theory is that our telecommunications infrastructure could then be spied upon by China, or that it could be shut down in case of a conflict with China or a Chinese ally. A less urgent aspect is that getting all our telecommunications equipment from very far away would impact our ‘digital sovereignty’, meaning that even if we wanted to, we would no longer be able to autonomously build up a communication infrastructure without Chinese help.

These three worries - spying, availability and sovereignty (or autonomy) - are supposedly on the balance when it comes to “5G decision”.

One European government attempted to procure a terrestrial emergency communication network a few years ago. This attempt failed since vendors only offered cloud operated services - no vendor was able to quote an actually independent network.

In this post, I argue that this particular Rubicon has long been crossed, and that buying yet more telecommunications infrastructure and services from potential geopolitical foes is not the main concern we should have.

Instead, we should work very hard to regain some semblance of control over our current telecommunications infrastructure - something we have long lost.

Note: many of the arguments about China work just as well for the US, as noted in this (paywalled) Financial Times article:

"As an icebreaker, [telecommunication operators] were asked if they thought the Chinese could eavesdrop through “backdoors” in Huawei equipment. Every single hand went up. One of the bankers then asked, for balance, if they thought the US could access communications through key Cisco equipment. “All the hands went straight back up without hesitation.”

The assumed provider security model

In the 5G discussion, the assumption is that national, large-scale telecommunication service providers are currently in good (or even full) control of their networks. The idea is that these providers (think Vodafone, Deutsche Telekom, Proximus, Orange, Telefónica, KPN etc.) procure equipment, which is then shipped by the vendor to the operator.

The provider’s employees would then get trained on this new equipment, unpack it, perform tests, configure it and use it to build new networks. Subsequently, other provider employees would operate and monitor the actual network.

If the equipment behaves strangely, for example by sending data to outside servers, telco staff would be able to pick this up and investigate. Similarly, if software upgrades come in, these would be tested by the service provider to see if nothing bad is in there, and would then be installed on the network.

Highly privacy sensitive areas, like call detail records, can then be used within the service provider to perform activities like billing or to resolve customer disputes. Similarly, if local government agencies show up with warrants, the data they need is then extracted from these locally operated systems under full service provider control.

All this would then be possible because the provider has experienced staff with a lot of telecommunication expertise.

Governments also believe in this model and require key personnel within national service providers to hold security clearances, so that police and intelligence agencies can ask questions and be sure their interest does not leak to third parties.

In this model, the 5G discussion is then framed as one where picking the wrong vendor upsets this model of good local control. Suddenly things would change.

The reality

In reality, most service providers have not been operating on this model for decades. Driven by balance-sheet mechanics and consultants, service providers have been highly incentivised to outsource anything that could possibly be outsourced, and then some more even.

In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor. Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers.

The host service provider often has no detailed insight in what is going on, and would have a hard time figuring this out through their remaining staff. Rampant outsourcing has meant that most local expertise has also left the company, willingly or unwillingly.

We recently asked a large European service provider why only part of their customers get IPv6 service, and how they pick which parts do or do not get such service. They could not tell us, and informed us they too would like to know.

Since the early 2000s at least, most billing has been outsourced. This works by sending all Call Detail Records (CDRs) to a third party, often from Israel or China. A CDR stores who called whom and for how long. More data might be attached, for example the location of the customer, or where the customer was roaming abroad etc.

CDRs are powerful metadata which frequently get used in criminal and intelligence investigations. If these contain country or regional information (cell tower IDs, coordinates), they form a virtual trace of a subscriber’s activities.

It turns out however that customer invoicing is such a challenge that billing was among the first services to be fully outsourced to third & frequently foreign parties. In this way, there is no need to plant backdoors - data willingly gets sent out.

Wholesale outsourcing

In a typical large-scale service provider, the mobile and/or fixed access networks are operated by the vendor and not the provider. The vendor however still needs technical input on what needs to be done, which means that the service provider does need to have IT staff.

However, over time, such IT staff also tends to get outsourced. At one major mobile provider the chain is now that the company has outsourced IT to Tech Mahindra and that Tech Mahindra in turn talks to Ericsson, who then finally operate the network.

Meanwhile, Ericsson and other vendors in turn have outsourced or shipped many functions to different countries where staff is more affordable.

In another example, one large Dutch mobile provider has handed over most of their technical staff to Huawei. Half of their freshly built and well designed headquarters has since stood empty - what remains in the other half are IT Architects who do not get closer to actual operations than an Excel sheet or a Visio diagram.

In summary, the idea that telecommunication service providers are currently autonomous and able to guarantee the privacy of their subscribers is highly questionable.

Similarly, any worries about “the Chinese” being able to disrupt our communications through backdoors ignore the fact that all they’d need to do to disrupt our communications is to stop maintaining our networks for us!

Security and safety by service level agreement

Service providers are of course well aware of the risks they run. Such risks are addressed in contracts and service level agreements. In this vision, Chinese companies would not disrupt our communications because there is a contract that says this will not happen.

Similarly, vendors commit to only use data they have access to for the stated purpose. Or put differently, our billing records will be ring-fenced from foreign intelligence agencies, because the contract says so.

Everyone with even a modicum of experience in national security knows such agreements aren’t worth the paper they are written on. In fact, most European countries’ laws (like American and Chinese ones) contain provisions that government warrants trump any customer privacy commitments.

The comforts of “security by contract” are such however that we frequently want to believe it is possible to send our innermost secrets abroad and not harm our safety that way.

The security department

All service providers have a security department, and I know many of these people well, and I feel their pain. In the Financial Times article mentioned above we find this quote:

“We don’t trust anybody whether they sit in China or the US and no operator should,” said Scott Petty, chief technology officer at Vodafone UK. “Our job is to protect our customers.”

It is then up to the security department to deliver on this bold claim. Sadly in all providers I know, security departments struggle to get their recommendations implemented, especially those that might harm profits, revenues or merely delay to “go to market”.

One large-scale provider has taken the somewhat unique step of selling its own security department, making it even easier to ignore their demands. Another provider lost their CISO after an embarrassing confrontation during an upper-management all-hands.

The reality is that in order to not have to trust vendors, wherever they come from, the whole company must skill up and be in actual control of the network and its operation.

It is not enough to have a security department full of good intentions.

Huawei, Nokia, Ericsson?

In other less discussed news, the choice between Ericsson, Nokia and Huawei is not as stark as frequently assumed.

If Europe (or ‘the west’) wants to retain autonomous communication capabilities, it would surely be helpful to pick a European vendor from time to time.

But if communication service providers want to be in control of their networks, outsourcing their operations to any third party, especially one staffed from far away, is not going to help much.

It should be noted that core Ericsson software components appear to be developed in China, which may bring worries of its own.

All large scale outsourcing companies have been thoroughly compromised. It may fundamentally be very difficult to deliver low cost services, very far away from your customer, securely.

To operate a network securely may mean actually employing lots of people locally and building up a relation with them so the service provider can build up a culture and knowledge base that is robustly able to maintain control over the communication network.

How did we get here?

If we compare European telecommunication service providers, some are still holdouts that perform many of their operations in house, without wholesale outsourcing, notably in the UK. This shows it is certainly possible to still operate a network somewhat autonomously.

Similarly, most American service providers have managed to retain far more expertise and are able to run their networks much more independently of their vendors. US providers may leak less customer data, but to compensate, they flat out sell it.

European service providers however had reasons beyond balance-sheet gymnastics to outsource: we have traditionally not valued (telecommunication) engineering expertise. Instead for years service providers have glorified their marketing and finance departments.

One even went so far as to state during an all-hands meeting with technical staff that ‘running a communication network’ was by no means a core competence for them.

With such statements, service providers are no longer attractive places for technical talent to work. And not only is there a lack of appreciation for skilled engineers, there is also a remarkable lack of payment.

Service providers have interpreted their struggles to recruit talent as ‘bad labour market conditions’ and have instead punted for outsourcing - thus accelerating the trend, because who would want to join a company that is rapidly outsourcing your kind of work to the lowest bidder? In the end, outsourcing is now almost the only possibility to survive.

As a case in point, one European 15-million subscriber network now relies on a core team of 4 people (one of whom is their manager) to provide all addressing and numbering services. After years of failed attempts, these four people will now also be outsourced.

Now what?

From this it is pretty clear what needs to happen. Service providers should first regain autonomous technical capabilities by any means necessary. 

Technical expertise is the first line of defense against malicious vendors attempting to spy and destabilise. Having strong local knowledge of telecommunications helps assure the future autonomy of vital capabilities.

With sufficient local skills, service providers would also not have to hand over entire networks to single vendors. It may then again be possible to run best-of-breed solutions where a vendor has to behave or run the risk of being phased out in short order.

Open source?

Open source is frequently touted as being more transparent. My company and its parent mostly live from deploying and supporting open source, so it is no surprise that I think open source does have a role to play.

Open source is only transparent however to people with the right kind of skills. It is not enough to base your network on open source technologies if these are not paired with the kind of staff able to detect and smoke out problems.

New vendors?

As noted, the main vendors in the mobile/telco networking arena are giants like Huawei, Ericsson and Nokia. Because of a relative lack of skills, operators are forced to buy integrated solutions, often from single vendors. There is little credible ability left to integrate solutions from different (partially competing) suppliers.

This means that it is currently difficult for new entrants to thrive in the 5G (or even 4G) market - either you show up with all the goods or customers will have a hard time to integrate your technologies, if they are good or not.

Reportedly, the US has offered more than $1bn in research money to help smaller vendors grow into serious 5G players.

Europe might similarly decide that having a functioning communication solutions ecosystem is in its best interests.

Automation

Typical service providers have hundreds of thousands of network elements. Surprisingly perhaps, many of these are actually maintained manually (!). Thousands of networking engineers labour to keep all this infrastructure operating well.

Meanwhile, modern large scale internet companies (like Google, Netflix, Facebook) have automated all such maintenance. Automation in this context means that no configuration states are edited manually but instead, entire networks get provisioned and configured from central templates.

With such automation, small teams of engineers can control and operate vast networks with relative ease - especially if good use is made of continuous integration and real life testing.

In this way, clever automation is an alternative to large-scale outsourcing, or put differently, it allows for the possibility to in-source maintenance without having to attract thousands of people no longer available for hire.

Summary

Telecommunication service providers are by and large currently not in good control of their networks. Through rampant outsourcing they have become utterly dependent on network vendors and other third parties. Picking Huawei at this stage is not specifically a sea change but simply a continuation of existing policy for most providers.

If we really care about our privacy and the stability of our communication networks, and if we take the long term view we should be able to build such networks autonomously, it is far more important for European service providers to skill up again.

Doing so will require retooling the companies so engineering departments are again empowered and not fodder for outsourcing. Such skilled-up employees are a powerful line of defense against nefarious vendors.

Regaining technical capabilities may again make it possible to operate multi-vendor networks, where all vendors are at credible risk of being displaced if problems are uncovered. This would also allow room for smaller vendors to enter the market.

In addition, through use of open source and advanced automation, it is possible to regain control of networks without having to hire thousands of people that the job market is no longer supplying.

And most importantly: the discussion should not be about the choice for Huawei or not. It should be about how we’ll control our vital communication infrastructure - only (not) choosing a specific vendor will do very little to change that.

This was originally published on Bert Hubert's website.