In my last blog, I talked about the need to plan strategically and implement tactically. The logic behind that recommendation is that it’s all too easy to take short-term measures you think will lead somewhere, and then go nowhere beyond the short term. Looking at operators’ actions in 2017 and their plans in 2018, I see that risk already biting some critical projects.
Based on my research and modeling, I’ve developed a picture of the trends in what I call “process opex”, meaning the non-capital costs associated with network and service operations. In 2017, they ran about 24 cents on each revenue dollar across all network operators, which is more than the total capital budget of operators. SDN, NFV, and zero-touch automation all promise to reduce process opex, and in fact the savings that could have been achieved in 2016 and 2017 amounts to about six cents per revenue dollar. What actually happened, according to operators themselves, was a savings of just a bit more than two cents per revenue dollar.
I was curious about the problem here, and what I learned when I asked operators for the reason for the savings shortfall was pretty simple and also sadly expected. According to operators, they did not start their projects with the target of complete zero-touch automation. Instead, they attacked individual subsystems of their network and operations process. The also didn’t consider any broad automation mission in the way they did their per-subsystem attacks. Many even said that they didn’t group multiple subsystems and try to find a solution across the group.
Only about five percent of operators seem to understand there’s a risk associated with this approach, and none of them seem to have quantified it or even fully determined what its scope could be. I had a couple who I’d worked closely with, and I opened with them the touchy question of whether this might end up locking them into a path that didn’t go where they wanted. Only one had a comment, and it was “I do understand that might happen, but we have no options for a broad strategy being presented to us at this point, and we need to do something now.”
What made this particular comment troubling was that this operator was also assessing ONAP/ECOMP, and so it could be argued that they did have a possible broader pathway to automation already under review. Pushing my luck, I asked why ONAP wasn’t seen as a possible basis for a long-term automation approach, and the response was “Oh, they’re not there yet and they don’t really know when that might be possible.”
The urgency in addressing automation, for the purpose of reducing operations costs and improving both accuracy and agility, is understandable. Operators have been under pressure to improve profit per bit for over five years, and it’s only been in the last two that they’ve really done much. What they’ve done was to attack areas where they had a clear problem and a clear path forward, and as my specific contact noted, they weren’t being offered anything broad and systemic at the time their initiatives kicked off.
They still aren’t being offered anything, of course. If you look at the things operators have already attacked (you can find a list of some in a Light Reading piece on this topic), you find mostly stuff in the OSS/BSS area. You also see areas that are undergoing changes because of the changing nature of services, customer support, and competition.
I think the fact that the current opex-reducing automation initiatives really got started late in 2015 or early 2016 is also a factor. There was no zero-touch automation group in ETSI then. Most NFV work being done at that point didn’t really address the broader questions of automating services that consisted of virtual and physical functions either. The notions of service events and an event-driven models for service and network management were still in the future, and AT&T’s ECOMP wasn’t even really announced until March of 2016. It’s also true that ECOMP as it was defined, and largely as it still is, was targeted to deploy “under” OSS/BSS rather than as a tool to orchestration OSS/BSS processes. Once operators got started with OSS/BSS-linked automation projects, they were out of scope to the ECOMP evaluation people. They were, in most cases, even run by a different group—the CIO for OSS/BSS and the CTO for ECOMP.
Operators, as I’ve said, do understand that they’re likely to lose something with their current approach (but see no alternative). How much they might lose is something they’ve not assessed and can’t communicate, but if I tweak my model I can make a guess.
Today process opex is running about 24 cents per revenue dollar across all operators. The incremental growth in process opex through 2020 is projected to be four cents per revenue dollar, and automation can relieve only two cents of that, making 2020 process opex 26 cents per revenue dollar. It could have been 23 cents if optimum automation practices had been adopted and phased in beginning in 2016. The model says that achieving optimality now would lower the 2020 number to 25 cents, which is still two cents more than optimum. To put things into perspective, cutting capex by ten percent would save that same two cents.
Losses are accumulative, and so are shortfalls on savings. We could have hoped to save an accumulated total of twenty-six cents on each revenue dollar with zero-touch automation from 2016 through 2020. What operators now think they will save with their incremental approach is less than half that amount. There is a LOT of savings left on the table, enough to fund more than half-a-year’s capex budget. That’s the price of not having a grand strategy to adopt.
Where things could get really complicated is when you take this beyond 2020. IoT, 5G, increased reliance on Internet-only customers because of streaming competition, and interest in expanding the use of SDN and NFV could combine to add another three or four cents to process opex by 2025. For some possible services, the lack of automated tools in management and operations could threaten the business case, or kill it.
I got a LinkedIn message that might give us some insight into why we didn’t have that grand strategy. Norman Dorn sent me a LINK to a new-gen functional architecture for telecommunications systems that was developed by Bellcore before the iPhone came out. The insight the list of goals demonstrates is dazzling, but Apple came along with a commercial product that swept the market without dealing with most of the issues, particularly insofar as they related to the relationship between network technology and appliance technology. Comprehensive strategy will always support optimality, but it may not support practicality or competitive advantage.
There has to be a middle ground for operators in areas like zero-touch automation, 5G, IoT, and even SDN/NFV. We could still find it for all of these things, but to do that we may have to abandon the traditional standards-driven approach and try to mold open-source projects into the incubators of future technology. The only thing in that direction that’s worked so far is the “ECOMP model” where the buyer does the early work and when critical mass has been developed, the result is open-sourced. That puts too many functional and strategic eggs in that first buyer’s basket. Maybe it’s time to work out a way for technical collaboration of network operators of all sorts, without vendor interference and regulatory barriers.