Month: March 2023

I doubt that anyone but Wall Street short-sellers were happy about the problems with Silicon Valley Bank and Signature Bank. The broad problem for each of these banks has been discussed, but what about the linkage? Is there a reason why both banks, with different exposure to different risks, failed in a period of less than a week? What does this tell us, what must we learn, to protect not only tech and startups, but economic health overall.

SVB’s problem was that unlike most banks, it didn’t really do a lot of short-term lending. Tech companies stockpiled their cash there, cash from VC funding rounds, IPOs, and operations, and the bank held the cash in bonds rather than lending it out. When the Fed started raising interest rates, the price of bonds dropped because the bonds’ own interest payments were less attractive given the new rates. That meant that the bank’s reserves fell. Had they loaned out more of the money, the higher interest rates set by the Fed could have given them more money, not less. In addition, the tech sector and startups and VCs that largely create it were pressured by the same rise in rates, borrowing less and pulling money from their accounts. When the bank had to sell some assets at a loss, the run started. The perfect storm.

Signature Bank’s problem was crypto. Almost a quarter of the bank’s deposits came from the crypto sector, which even at the end of 2022 was showing some warning signs. As a result, the bank announced it would shrink the share of deposits represented by the crypto space in 2023. However, that had apparently not gone far enough when fears of bank problems were raised by the SVB problem’s emergence. The SVB weekend closure meant a stablecoin lost access to what backed it, and suddenly there was no stability in crypto anywhere.

In both cases, the banks suffered both a confidence crisis among depositors and among investors. This in turn caused regulators to step in. Bank failures aren’t unusual; there are usually a dozen or so in a given year, but it was hoped that regulations put in place after the 2008 crisis would prevent them in the future. It has worked so far among the major banks, but smaller regional and specialized banks like SVB and Signature have continued to be vulnerable.

Given that the “cause” of the two takeovers were very different, why did we see them almost one on top of the other? Part of the reason is the contagion created by publicity. Depositors who have more in a bank than the FDIC insurance would cover were fearful after SVB, the great majority of whose depositors were largely unprotected until March 13^th, when the government stepped in with guarantees. Short-sellers jumped on every bank stock to make money, and some investors were fearful and joined them in selling. But while these were also “causes” of the problems, what started the perfect storm? Speculation, meaning bubbles. One thing that’s been true of the financial industry for at least a century is that it loves bubbles, loves to speculate and leverage to gain more.

Crypto is a self-contained bubble; there is no real foundation of value behind it, only the “technical” buy/sell relationship of the market. Even “stablecoins” that are supposed to be pegged to assets like the US dollar are really bubbles. Why would you invest in something that’s pegged to something else? Answer, you hope that the value appreciates despite the peg but that the peg protects the downside. Well, one stablecoin was temporarily caught in the SVB mess, and it may not be out of the woods. That’s because nobody reserves 100%; if enough people bail on a stablecoin, it fails too.

The tech VC world is also, I would argue, a bubble. For decades, the VC community funded multiple startups in the same space to create “buzz”, the media attention that tended to make the space seem hot and induce established firms to buy up the startups or be left out. The current focus is on social-media startups, despite the fact that ad sponsorship is a zero-sum game. What you do is to raise money, spend a bit on cloud resources to host your startup, spend more to promote it, and then try to “flip” it quickly. We’ll see AI playing along these lines shortly, and in great volume.

Banking once was regulated enough to prevent most bubbles, but those regulations were weakened in 2018. The real problem, though, was Wall Street’s appetite for more risk, and their search for new bubble opportunities. Even dot-com bubble and the credit-default-swap bubble didn’t stimulate addressing the systemic risk of bubbles, just minimal local ways of reducing their impact. As a result, banks were exposed to problems that really existed outside banking…what might be called “inherited bubbles”. While there are banks that want to issue crypto, the majority who are at risk there are at risk because they’ve banked crypto firms’ assets. SVB banked startup assets, tech assets. Both held reserves in bonds, reserves required by law, and at the same time the Fed was taking steps to make the bonds worth less.

The global economy has a bunch of moving parts, but they’re not separate parts. Instead, they all move in a complex way, and so something that distorts one segment will almost inevitably bite you in a segment that seems on the surface to be totally unrelated. Should the Fed have asked what the impact of higher rates would be on bonds and those who hold bond reserves? Surely, but if they did, it didn’t show. They left it to the market, and the market loves bubbles, until they burst.

There are a lot of reasons why it’s said that the US is losing its top spot in tech innovation, but I think the biggest reason is one that’s not talked about at all. It’s bubbles. Startups that are actually aimed at creating a real, new, value have almost disappeared in the VC bubble. The thing about bubbles, including startup bubbles, is that they enrich capital and not labor. You don’t need many real engineers to launch a social-media bubble, far fewer than you’d need to start a company that was going to build a product and employ real people. Bubbles have enriched the wealthy rather than creating jobs for the average worker, and it’s real stuff that advances technology, and pretty much everything else.

I’ve worked with telcos for decades, consulting with them, cooperating on international standards bodies, and sharing meals and drinks with many of their executives and planning staffs. There’s a lot of good things you can say about them, but even I have to admit that they have a tendency to take root and become trees. That’s one reason why I have to wonder about a story (quoting an analyst study) that says that each big telco will spend $1 billion on cloud network transformation. The other is that I’m hearing nothing from operators to support the statement.

This story suggests a huge boom in cloud usage as part of the telco network, one that will generate enormous investment and necessarily generate even more enormous benefits to justify that spending. This, at a time when enterprises, who have way longer and broader experience with public cloud services than telcos do, have decided that cloud cost overruns are often crippling and some companies (Basecamp, for example) have saved millions moving higher-level features off the public cloud and into the data center. So are telcos behind the enterprises in recognizing that the cloud’s savings are often illusory, or are they ahead of their usual pace in accepting new technologies?

The devil, they say, is in the details, and it’s not just this general prediction I have a problem with. According to the article, “46% of telco network capacity will be entirely cloud native in the next three to five years.” There is simply no way that’s going to happen, unless somebody proposes that we redefine both “cloud-native” and “cloud”. I don’t know a single telco who plans to do anything like that. Network capacity is based on routers and optical paths. A big Tier One in the US told me over a decade ago that they’d looked at hosted router capabilities to save money, not in the cloud but in their own facilities, and had determined that they couldn’t perform as the task required. To host virtual routers in the cloud, as a “virtual network function”, has also been examined and rejected.

I talked to a telco CTO on this point two weeks ago, obviously not relating to the story/study but to the question of transport evolution. They were excited about “convergence” on more optical capacity, but they were not excited about the use of VNFs for traffic. The problem was cost. Cloud providers charge for traffic, and so about the silliest thing you can do in the cloud is push a lot of featureless bits through it. “We ran the numbers on a carrier Ethernet business connection, and the cost of VNFs even providing security was outlandish.”

The comments cast doubt on the benefits of hosting real network elements in the cloud. According to the article, early adopters, defined as companies with “a comprehensive telco cloud strategy with well-defined goals and timelines; advanced players in terms of the proportion of network functions that have been virtualized; and those that expect more than 50 percent of their network capacity be on cloud”, will recover 47% of their investment in three to five years. Let me get this straight. These people are going to go to their CFO with a project that won’t pay back even half its cost within three to five years? I don’t know any CFOs who wouldn’t throw them out of the office with that story.

Finally, we have the notion that a BT project that’s saving them a lot of money with the cloud. The problem is that it’s not using the cloud to push bits, but to host what are really OSS/BSS applications now run on mainframes. Cloud network transformation doesn’t happen because somebody moves record-keeping on customers or assets to the cloud. None of that has anything to do with network capacity. To move network capacity to the cloud, you’d have to move service data plane handling to the cloud, and there is absolutely no operator I’ve talked with that’s looking to do that. Even if, contrary to my CTO comment, virtual functions might support business point-of-connection services, NFV’s support of those is focused on universal CPE (uCPE), meaning open white-box devices on the customer premises and not in the cloud. And the initial NFV “Call for Action” white paper made it clear that routers and big iron traffic-handlers were not targets of NFV.

That doesn’t even get to the point about “cloud-native”. Like a lot of concepts that get good ink, the term is smeared across a bunch of things it rightfully shouldn’t be applied to. I believe that “cloud-native” means “created from stateless microservices”, and that structure is reasonable for GUIs but totally unreasonable for data-plane handling for the simple reason that there’s too much message traffic between components.

The story introduces comments about 5G and Open RAN, which suggest that all this is moving traffic to the cloud, but the majority of virtual-function usage defined in 5G relates to control-plane rather than data-plane behavior, and while O-RAN does define data-plane elements as hosted functions, operators are interested in this more as a means of opening up the implementation to white boxes and local servers, not pushing the functions to the cloud. RAN is an edge function, and you can’t backhaul a data plane to a cloud hosting point.

The story, and apparently the study, also talk about 5G and Open RAN as sources of “potentially lucrative services and use cases”, but where are they? We’ve had 5G discussions for five years, and there have been no new lucrative services, only continued speculation. Yes, there are still operators (and 5G vendors) who believe in these new services and use cases, but there’s also plenty of people who believe in elves and the tooth fairy, and some who think the earth is flat. Wishing won’t make it so.

I think that the main problem here is conflation. People think of telcos as network operators, as though all they operated were networks. A big telco has the same business management issues as any other big company. They have, hire, fire, and pay employees, manage real estate, file tax reports, handle their stock activity, and so forth. All this stuff is lumped into “OSS/BSS”, and while things even relating to the network are only a small part of that, even OSS/BSS gets lumped into “network”. The problem with this particular story/report is that it says that “46% of telco network capacity will be entirely cloud native in the next three to five years.” As I said earlier, there is simply no way that’s going to happen and I don’t know any real telco operations type who believes otherwise.

Another conflation problem is treating virtualization and hosting of features as being the same as public cloud hosting. The NFV ISG didn’t propose to move features to the public cloud. The initial work focused on deploying features on carrier cloud hosting. My own work with the ISG was directed at making virtual functions compatible with the platform software used to build clouds, so as to take advantage of the capabilities being developed there. I’m not suggesting that there was, and is, no value in public cloud hosting of some functions, only that saying that VNFs make up things like 5G doesn’t mean that they’re hosted in public clouds. A “cloud” VNF isn’t necessarily a public cloud VNF, and that connection is made way too often.

I don’t have access to the report the article cites, so I can’t say whether it’s responsible for the claims, whether it has been summarized inaccurately by somebody, or whether the reporter didn’t understand the details. In a sense, I’m surprised, but in another sense maybe not. Is this another example of pushing hype? If so, it would be really disappointing.

The economics, and opportunities, associated with broadband deployment have always been complicated. It’s been almost 20 years since I started using “demand density”, a measure of economic value that could be “passed” by a mile of broadband infrastructure and available to be hooked up and monetized. Since then, I’ve seen the maturing broadband market changing even that dynamic a bit. Overall demand density, over the operators’ footprints, by state, by metro area, may be useful in broad-strokes planning, but we’re now seeing a need for some micro-focusing as we consider what technologies are best in satisfying broadband needs and operator profit goals.

I pointed out in past blogs that within any given service area we tend to have pockets of concentration of demand/opportunity, and pockets where there’s not much going on. In New Jersey, my home state and a state where overall demand density is among the highest in the nation. We have large state-owned land areas where there’s nearly-zero residential or business population. But today, in the world where smartphone connectivity is as important (or even more important) than wireline broadband, even those areas may require broadband connectivity to support highways that move through them.

It’s also true that all of those pockets of concentration I mentioned aren’t the same. Obviously an important measure of the economic value of a “community” is the total spending that it can generate from both residential and business users. The question is how that can be realized, and that depends on a number of complicated factors.

One such factor is the distribution of residential income. Take two communities, one of twenty thousand residents and one of fifty thousand. Suppose the first community has twice the household income of the second. Is the second still a better opportunity because it offers more total income? No, for three reasons. First, higher-income communities are usually more spread out, so the “pass cost” to prepare to connect customers will be higher. Second, zoning rules in the first community will likely limit businesses locating there, which reduces the total network opportunity. Finally, households tend to spend a given percentage of disposable income on broadband, and higher-income households have more disposable income as a percentage of total income.

Another, growing, factor in opportunity measurement is the value of mobile coverage in the area. Going back to my New Jersey example, a mobile operator might have virtually no “direct” opportunity from residents in an undeveloped state-land area, but if a major highway supporting commuting or leisure travel passes through, or if there’s a popular business or recreational destination in the area, then lack of mobile broadband there will discredit an operators’ service to all who need it there, regardless of where they live.

One of the things that these factors have done is change the dynamic of fiber broadband deployment. Twenty years ago, Verizon dominated residential fiber broadband in the US because it had a high regional demand density and could therefore afford to push Fios to much of its service footprint. In the years since, AT&T started deploying fiber in its higher-density pockets, and new players have started to offer fiber in areas where the incumbent operator didn’t.

5G Fixed Wireless Access (FWA) is another development related to the intricacies of demand density. An FWA deployment is a single fiber-fed tower (fiber to the node or FTTN) that supports users in a rough one-mile radius. Because user connections are made via RF, the pass cost is limited to the cost of the tower and fiber feed; no trenching of fixed media is required. It’s no wonder that many sources say FWA is the fastest-growing form of broadband in the US.

Then there’s satellite. If you look at mobile coverage maps, you see that there’s a good chunk of most geographies where no mobile service is available, and it’s highly probable that there’s no fixed-media broadband there either. Satellite broadband is often the only broadband available in undeveloped areas, because the “node” is up in space and can serve a very large area without any other infrastructure deployment needed. Even in the US, there’s growing interest in satellite because it’s available everywhere, and it’s likely that some smartphones will support satellite broadband directly in the near future, supplementing their normal mobile broadband connections.

OK, so we have really two “broadband infrastructures” deploying. One is the physical-media “wireline” form that includes fiber, CATV, and some copper loop. The other is the RF form, which includes mobile broadband, satellite broadband, and FWA. The most significant thing happening in broadband is that second form, and in particular the mobile and FWA pieces. The reason is that these are “seeding” broadband infrastructure further into thin areas, making them “thick enough”.

If you have to push glass to a 5G tower or FWA node, wouldn’t it be possible to branch out from that point with fiber PON? Couldn’t the ability to serve a given thin location be enough to make that location more suitable for residential and business development? The more mobile and FWA we deploy, the more we reduce the incremental pass cost for even fiber to the home or business, because we have a closer feed point to exploit.

I’ve talked with some of the proponents of the “fiber broadband for all” thesis, and this is how the more thoughtful of the group see their goal being achieved. Is it realistic? I’ve tried to model that question, and while I’m not completely confident about the result for reasons I’ll note here, they’re interesting.

My model says that the current mobile broadband and FWA trends will, within 10 years, create enough “seed” points of fiber feed that almost 70% of the US households could be offered PON connections, and that would rise to almost 80% in 20 years. The qualification I’ve noted is that it’s nearly impossible to project economic, political, and technical trends out that far. The biggest caveat my modeling reveals is dependence on the cost and performance of FWA.

It is credible to assume that table stakes for home broadband in 10 years will be at least 500 Mbps download and 200 Mbps upload. If FWA can achieve that, that eliminates one risk to the model results. If technology advances in FWA could extend range (subject to topology) to 2 miles, that would eliminate one risk, and create another. At the 2-mile range level, FWA that meets basic service goals would be so much cheaper than PON that an FWA provider could undercut a PON/CATV provider on price, even in areas where PON and CATV are now dominant. In other words, we might have a kind of reversal of the impact of seeding, one where FWA starts to cannibalize the low-end piece of the fiber space, to the point where fiber growth would likely stop well short of the model predictions.

One thing that seems clear, more so for sure than the future of universal FTTH, is that broadband availability based on terrestrial tools is going to expand significantly in the next decade, in the US and in other developed economies. That likely means that satellite broadband will remain confined to under-developed countries and areas, and that terrestrial broadband will continue to improve, limited only by the willingness of operators to invest.

Just how difficult is “carrier cloud”? There’s been a lot of talk about how hard it is for operators to deploy their own cloud resources, particularly when many services have a much bigger footprint than the operators’ own real estate holdings. There’s been a lot of talk about how public cloud partnerships are favored by operators, in part because of the footprint problem and in part because most are uncomfortable with their ability to sustain their own cloud resources. Now there’s talk about whether even public cloud hosting could raise unexpected costs. Maybe, but it’s not clear just whose problem that is.

The article cited here is about the question of the portability of network functions, meaning the hosted functions that were created under the specifications of the Network Functions Virtualization ISG of ETSI. According to the article, “Network functions (or NFs) developed for one vendor’s cloud cannot enter another’s without expensive repurposing by their suppliers.” There are three questions this point raises. First, is it true? Second, whose problem is it? Finally, does it matter?

The best we can say regarding the first question is “Maybe”. Any software that runs in a public cloud could be designed to use cloud-specific web service tools, which would make the function non-portable. However, nearly any function could be written to run in a virtual machine without any specialized tools, an IaaS application. These applications would be portable with a minimum of problems, and in fact that was a goal of the NFV ISG from the first. Subsequent efforts to align with “CNFs” meaning “Containerized Network Functions” admit to a bit more risk of specialization to a given cloud, but it’s still possible to move properly designed VNFs between clouds with an acceptable level of effort.

The second question has a bit of its own ambiguity. The author of a VNF determines whether it uses non-portable features, which means that a VNF “supplier” could in theory write either largely portable or thoroughly non-portable VNFs. In that sense, it would be the VNF supplier who made the bad decision to render a function non-portable. However, many of these suppliers are beholding to a particular cloud provider, via their development program. Cloud providers would love to have VNFs run on their cloud alone, so in a sense their proprietary goals are at fault. But operators themselves have the final word. Nobody puts a gun to your head and demands you buy a non-portable VNF. If operators, either individually or collectively through their participation in the NFV ISG, demanded that all VNFs forswear the use of all but the most essential non-portable features and take steps to make it easier to port those that must be used, VNF authors would toe the line because they couldn’t sell otherwise.

But the third question is likely the critical one. Does any of this matter? It’s the hardest question to answer because it’s a multi-part question.

The first and most obvious part is “Does NFV matter at all?” The NFV ISG got started over a decade ago, and quickly aimed well behind the proverbial duck in terms of pooled-resource hosting (despite my determined efforts to bring it into the cloud era from the first). The initial design was off-target, and because most groups like this are reluctant to admit they did a bunch of the wrong stuff, the current state of NFV is still beholding to much of that early effort. Are there operators who care about NFV? Sure, a few care a little. Is NFV going to be the centerpoint of carrier cloud? Doubtful, no matter who hosts it.

The second part of the question is “Does cloud portability matter?” The majority of operators I’ve talked with aren’t all that excited about having to integrate one public cloud provider into network services, and are considerably less excited about the integration of multi-cloud. In the enterprise, the majority of users who purport to be multi-cloud are talking about two or more clouds in totally different areas (a Big Three provider plus Salesforce is the most common combination). Slopping application components between cloud provider buckets isn’t something anyone likes much, and so operators get little encouragement for the idea outside the tech media. So, the answer to this question is “not a whole lot.”

The third piece of the question is “Does cloud portability in carrier cloud actually mean function portability?” The majority of interest by operators in cloud partnerships at the service feature level comes in areas like 5G, where some elements of 5G are provided by the cloud. These elements integrate with the rest of the network through interfaces defined by the 3GPP or O-RAN Alliance, which means that they’re essentially black boxes. Thus, if Cloud Providers A and B both offer a given 5G feature, connected via a standard interface, the feature itself doesn’t need to be portable because the same black box is available from multiple sources.

The final piece? “What about features of more advanced services?” The truth is that the hosting of basic network functions isn’t going to move the ball much for operators, or for cloud providers. The big question is whether something bigger, like “facilitating services” or even OTT services like the digital-twin metaverse, might lie ahead. If it does, then broader use of better features could provide operators with the incentive to self-host some things.

The problem is that this question has no clear answer at the moment. Some operators are committed to facilitating services, some are even looking at OTT services, but nobody is doing much more at this point than dabbling. One reason is that there are no real standards for how these new features/functions would work, which means that there’s a risk that they wouldn’t be implemented and interfaced consistently.

That’s the real risk here, not the VNV risk. VNFs were designed to be data-plane functions, and were gradually eased over into standardized control-plane elements for 5G. IMHO, generalized facilitating services or OTT service features would likely be “real” cloud components, unlikely to obey the management and orchestration rules set out for NFV. Still, though, they’d need to have some standardization of APIs and integration or providers of the new features/functions wouldn’t be able to write to anything that would ensure they could be linked with operator networks and services. The NFV ISG, in my view, is not the appropriate place to get that standardization done, and until we have such a place, then the risk the article describes exists, just in a form different from what’s described.

There aren’t all that many enduring questions in networking, but one I recall coming up almost thirty years ago came up again last week. It was “Is it getting cheaper to provide more capacity than to try to optimize it?” There’s a new flavor to this question, though: “Should we be using more optical paths to substitute for MPLS LSPs?” I’ve blogged about the potential benefits of a metro-network model where metro centers are meshed in some way via optical trunks. Might we actually be heading there? Ciena’s quarterly numbers, reported on March 6^th, suggest that maybe we are, but not conclusively.

The company’s revenue grew by just over 25%, and their premier product area, “Converged Packet Optical” jumped from $541 million in fiscal 1Q22 to $736 million in the first fiscal quarter of 2023. Routing and switching also gained (from $86 million to $120 million), and platform software and services gained just slightly. All the other areas showed a decline y/y, including Blue Planet.

The Blue Planet story might be the most direct indicator of a shift in thinking. Blue Planet is Ciena’s operations automation software framework, and the fact that revenue there declined while the revenue for optical/packet equipment was up significantly suggests that most of the gear is going into sites with Blue Planet already in use, or where it is still not being considered. That could be an indication that optical networks generally are lower-touch than router networks, and require less operations automation. Indeed, that’s one of the value propositions for the “more bits not more bit management” theme.

Another interesting point from their earnings is that the company got about 40% of its revenue from non-telco sources. Does this mean they’re seeing some data center missions, things that might lead to success in the metro of the future? Maybe. While the “webscale” players are the largest segment of the non-telco space, the other two segments (cable and government/enterprise) combine to be larger, and they’re similar to the telco space in terms of requirements.

The question of data centers is critical because data center coupling to the network is essential in injecting new service features. As I’ve noted in the past, these features have to be injected close enough to the user to allow for personalization and a high QoE, but deep enough to benefit from economies of scale. The only place that can happen is “metro”, the major metropolitan-area concentration points. There are, for example, about 250 metro areas in the US. Metro areas also represent a great meshing point, which would expand the optical convergence opportunity.

The key piece of the Ciena story that relates to metro is their “Coherent Routing”, which is a software-managed multi-layer model that combines IP/MPLS at the top layer with an optical base. Metro is the logical place where the layer transition would occur, and where their new 8140 router (announced last week) is targeted. The 8140 is a logical candidate for both metro aggregation and metro service injection, since you could make data center connections to it. You can tie it to business Ethernet connections, to residential broadband (FTTH, cable, etc.) and to mobile and 5G networks, and you can also connect to a metro data center owned by the operator, an interconnect/COLO player, or a cloud provider, as well as to traditional public cloud regional data centers. Obviously it supports connection to the Internet and to CDNs. Ciena shows it interconnected with the 8190 for many of these missions, too.

I think that Ciena has a very good, perhaps even great, metro story, but they have an issue they share with another metro pioneer, Juniper, in that their documentation and positioning isn’t exactly evangelistic. They can do what’s needed, but you won’t find material that shows the full potential and role of metro in the network of the future. If an operator has already recognized just how important the metro is, Ciena’s stuff can connect with their thinking, as Juniper’s can. If the operator hasn’t made the connection yet, it’s not likely that the Ciena material will push them over the threshold into metro-land.

Ciena (and Juniper) aren’t simply ignoring the obvious here. The importance of marketing to operators, even to cloud providers, is a matter of debate among vendors. These are giant buyers, large enough to support dedicated sales resources with technical sales support backup. Given that, most network vendors don’t really provide what might be called “educational” or “mission-focused” marketing material on the theory that their sales relationships can carry that. However, my contacts with operators suggest that this isn’t a great strategy. Salespeople tend to shy away from educational/evangelistic activities; they’re time-consuming and they may not pay off for many quarters, when sales organizations have to focus on making quota in the present. Thus, they may be slow taking advantage of strategic superiority when they have it, simply because it’s not translating to tactical sales success. Next year, it would.

For Ciena, I think the problem is particularly acute because “packet/optical convergence”, the concept that’s in play to shift focus from bandwidth management to bandwidth creation, is very transport-centric rather than service-evolution-centric. Operators can view that convergence either as a cost-management strategy or as a new-service strategy. The operators’ own biases and legacy push them toward the former, and ultimately it’s the new service approach that has the potential of driving significant spending by operators, and thus revenue for vendors. And convergence tends to focus toward the bottom layer, which of course is less visible and tends to develop only when the layer above it runs out of gas. What happens with convergence if the packet/router players in that top layer step up? They push the optical piece downward, which constrains its growth. For Ciena, the convergence has to take place at the metro level to maximize their equipment sales.

Cisco, who has an even-less-specific metro positioning than Ciena, nevertheless is looking to make its routers into the optical on-ramp through its Acacia acquisition, which lets routers with highly effective optical interfaces be the on-ramp to packet/optical convergence. Since those routers can connect with each other efficiently via optical trunks, they let Cisco take a bigger bite of the top piece of the convergence story. The more routers do at the metro level, the more it is that Ciena is stuck in a pure aggregation role, because service injection deeper than the metro is highly unlikely.

Router vendors aren’t the only players who need to understand how critical the metro is, optical players like Ciena may have even more at stake. With a strong metro story, they can make metro a barrier to router expansion and an outpost in the services space. That’s a role operators really need some vendor to play, and so the position is perhaps the strongest out there for expanding vendors’ opportunities. I suspect we’ll see clear signs of just what vendors might be going after the space by the end of this year.

We’ve started to see more stories about companies not “moving everything to the cloud” but instead moving everything off the cloud. The most publicized and analyzed examples of this is collaborative service provider Basecamp a who dropped cloud services costing $3.2 million in favor of in-house servers. This is a very radical example of a trend I’ve noted in the past. Most enterprises have identified at least one cloud application that broke the bank, cost-wise, and in most cases their problems seem to have stemmed from the same factors that Basecamp reported.

To start with, the notion of moving everything to the cloud has had a significantly negative impact on CIOs and IT planners. It’s led to the assessment that anyone who plans or maintains a data center application is a Luddite, and it’s contributed to what one CIO I chatted with called “gaming the numbers to get the right answer”. That means that many cloud application plans make all manner of favorable and unjustified assumptions to make the cloud look cheaper, when in fact it is almost surely not going to be.

Here’s a basic truth the article I’ve cited illustrates. There are no situations where the cost of a VM in the cloud will be less than the cost of the same VM in a well-designed data center. What makes the cloud cheaper is a lack of hosting scale that’s unusual for an enterprise, a need for scalability or elasticity, or a need to distribute hosting over a wide geography. The economies of scale a cloud provider can generate are, according to enterprises, only minimally better than that available in an enterprise data center. The difference is more than covered by the profit margins of the cloud provider. So to start with, cloud benefits are small enough to require protection, which leads to the second truth.

Cloud development practices that are widely promoted are usually drawn from OTT players rather than enterprises. They’re supporting global user bases and message-based exchanges among users rather than transaction processing. Enterprise IT is the opposite, but the “move everything” mantra encourages enterprise IT architects and developers to forget that. The first big problem that creates is database costs.

The article shows that the largest component of Basecamp’s cloud costs was the cost of cloud RDBMS. Enterprises often move data to the cloud because having a cloud application access databases stored on the premises creates traffic charges. The problem is that hosting the data in the cloud is costly, and unless the data is fairly static you end up paying to keep it synchronized with mission-critical databases associated with transaction processing that’s not been moved.

Then there’s the problem of componentization. Scalability is difficult to achieve without explicit state control, because user interactions are typically multi-step and where you are in the stages of the interaction has to be maintained somewhere. Sometimes this results in each piece of a multi-step interaction being handled by a different component, and that increases hosting costs. Sometimes a database is used to hold state so the components themselves are stateless and scalable, but that increases database costs again.

Scalability is itself the biggest source of cloud cost overruns. Yes, one of the big benefits of the cloud is that you can accommodate changes in load by scaling components, but of course that results in scaling of costs. Enterprises admit that in many cases they’ve adopted scaling strategies that did little to really change user quality of experience but increased costs significantly. What they realized they needed to do was to assess the way a multi-step interaction evolved. In many cases, there was considerable “think time” associated with a given step, and such steps didn’t gain much from scaling. The role of the user, and the role of software, have to be considered throughout a multi-step interaction to make a good decision on what benefits from scaling, and what limits should be imposed.

One enterprise described their decision to shift the front-end processing of a single application to the cloud. The application was being run using a maximum of four hosted containers in their data center. When they shifted it to the cloud, even with constraints on scaling and database costs, it ended up costing them six times as much because they allocated more resources than necessary and because their database costs weren’t properly assessed. The CIO said that even if they simply adopted a cloud container hosting service for four containers, it would have cost twice as much as it did in the data center.

Given stories like these, it’s hard not to wonder if the whole public cloud thing is a scam on enterprises. It’s not. The problem is our tendency to gravitate to simple things rather than to address complexity. There are things that the cloud is great at. There are applications where OTT development practices will enhance application performance and enhance user QoE. There are also applications that result in the opposite outcome, and it’s those stories we’re now hearing. The challenges of proper cloud usage, and the penalties for not ensuring it, haven’t really changed. What’s changed is the way they’re being discussed.

How it’s discussed isn’t changing enough, though. We’re accepting that the cloud isn’t naturally cheaper than the data center for simple VM hosting. We’re starting to accept that everything is not moving to the cloud, but we’re not yet discussing when the cloud is a better choice, why it is, and what’s really happening with cloud adoption. That’s making it harder to address the really important question, which is “What role will the cloud play in the future of IT?”

Simplistic answers to questions are comforting, which is why we fall into the “simplicity delusion” all too often. The cloud is a bunch of different things, ranging from a way to avoid building data centers even if it’s more expensive, to a way to provide global, elastic, reach to critical applications. There are things we could not do without the cloud, and other things that we should never consider doing there. There is no simple answer to the critical question of the cloud’s future, neither a highly positive or highly negative one. But there is an answer if we look hard, and objectively, at the question.

MWC is almost over, and it’s been a mixture of the “Old Telco” and the “New Cloud.” The combination could have possibilities, but I think the show was almost two shows because the combination was either absent or at least trivialized. I blogged yesterday about the GSMA API initiative, and that’s only one example of the fact that, as the song goes, “Two different worlds, we live in two different worlds….”

According to Omdia as quoted HERE, “5G has disappointed pretty much everybody—service providers and consumers, and it has failed to excite businesses.” I wonder how much any of these groups really expected from 5G, frankly. What I think we had, in the end, was the only budgeted telco infrastructure initiative, and you either believed in it as a vendor or telco, or you started looking for another job. Well, reality is. But the point here is that while 5G has failed to live up to the hype, it’s going to take time (and maybe 6G) to move on from it. Meanwhile, the telco community is trapped in a fading delusion.

Also meanwhile, players who haven’t traditionally figured in MWC have shown up and are making waves. It’s not that Intel or Microsoft or Google or Amazon were pushing a story that was totally disconnected from 5G, but that they were trying to influence the whole “moving on” thing. Don’t fall into the “G” trap yet again, dear telcos, lift your eyes from the traditional muck at your feet and see the sky!” For those who find this satire a bit too obscure, I mean that these players are putting one foot in the “hosting and API” piece of 5G, and just hinting a bit at where that might take telcos if they elevated their planning out of the realm of simple connection.

The GSMA APIs that were announced are trite, proven failures. The concept of APIs, the promise of APIs, is perhaps the only way that operators can move on. APIs mean software components assembled to create features. That means hosting, the cloud. Since operators have demonstrated that they really don’t want to deploy their own “carrier cloud”, the New Cloud players are looking at promoting hosted features and offering a place to host them.

If you look at the offerings that the Big Three cloud vendors made, you see that they’re far from futuristic in service terms. I’ve touted my notion of the “digital-twin metaverse”, and there’s nothing that even approaches that in terms of defining what might be a broad and credible service model that adds to the top-end revenue pie because users want it bad enough to pay. Instead, what we have is a series of things that add up to “take a baby step in this direction, and eventually you’ll See the Light.”

In a way, this makes a lot of sense for all concerned, because telcos wouldn’t have let themselves become so mired in delusional 5G if they weren’t deathly afraid of anything that wasn’t an evolution of things familiar to them. Facilitating services assembled from APIs? Give them a Valium! A few harmless, familiar, OSS/BSS services exposed as APIs? Yeah, just a brief nap will let telcos recover from that. Add in the “You don’t need your own cloud” and you have something that might even raise a cautious smile.

So what’s the problem? Isn’t the MWC story proof that cloud and data center vendors are leading telcos out of the dark ages? No, just that they’re trying to lead them away from a 6G rehash of 5G disappointment. Why? Because 6G won’t come along for three or four years, and nobody wants to wait that long for a revenue kicker in these challenging economic times. We could easily see the “G” delusion replaced by the “API” delusion, a different but familiar telco dodge-change maneuver.

There are some signs, even at MWC, that at least some players may recognize that something real and different needs to be done. Evolution is good if it leads you to a state better than the one you’re in. Revolution is faster and may, in the end, be less painful, so who might be the most capable of creating a revolution? It would have to be the cloud providers. Of the cloud providers, the one I think could be the most influential in breaking the “G” deadlock is Google. While Google didn’t announce any specific new service/benefit target either, you could infer that their GDC Edge and Nephio stories could be designed to evolve into a platform for that next big thing. It’s harder to pick out a path to that happy destination in either Amazon’s or Microsoft’s story, though both might also have something in mind.

Among the network vendors, I’m watching Nokia here because they may well be the only player at the whole show that actually sees how different the future of telecom must be. But look at all the qualifiers I’ve had to use regarding Nokia. They are committed to transformation, and they’re a player whose business model is clearly threatened by the “G” delusion. I think they’ve shown that they realize that, but do they have a specific plan, a target approach, a new benefit model in mind? That’s hard to say.

The risk that the cloud provider approach of one-foot-in-the-G might present is that it might simply facilitate telcos’ putting both feet there. APIs and hosted carrier cloud are still a general strategy, one that could be seen by operators as enough progress and enough discomfort. What gets them to pull the anchor foot out and step forward instead? A clear promise, and that is not being offered by anyone at this point, not Google or even Nokia. We’ll watch for signs, though, and with hope.

What “interfaces” are to hardware, APIs are to software. Thus, any transformation in network services that’s based on adding features to connectivity is very likely to rely on APIs. But this new mission for APIs is hardly the first mission. Operators have long exposed connection services through APIs, and APIs are the basis for application connections over traditional services. With MWC just starting, it’s not a surprise that API-related announcements are coming out. The question is whether they’re pushing us toward the new API missions, or taking root in the old.

According to one announcement from the GSMA, 21 operators have joined forces to define an Open Gateway initiative, which is a set of open-source APIs that provide access to features of a mobile network. These APIs, unlike those that have been offered in the past, expose more OSS/BSS than network features, including identity verification and billing. The article notes that eight services are initially targeted; “ SIM swap (eSIMs to change carriers more easily; “quality on demand”; device status (to let users know if they are connected to a home or roaming network); number verify; edge site selection and routing; number verification (SMS 2FA); carrier billing or check out; and device location (when a service needs a location verified).”

On the surface, this looks very conventional in that it targets things already provided by operators rather than seeming to present new features or capabilities. There’s also been plenty of talk in the past about operator revenue opportunities associated with exposing OSS/BSS features, and none of it has generated more than a pimple on the operators’ bottom lines. Despite that conventional bias, something good might come out of this.

The first potential good thing is the federation concept I blogged about earlier. The GSMA is playing the role of an API harmonizer here, and it’s also presenting its offerings in open-source form, meaning other operators would be free to adopt it. One of the challenges operators face in “higher-layer” services is a lack of consistency of features and interfaces when such a service has a footprint larger than that of a single operator.

The second potential good thing is that you could characterize these targeted features as “facilitating services” of the kind that AT&T says it’s interested in offering. AT&T is one of the operators who signed on to the initiative, which I think means that they’re seeing this link to facilitation too. It’s long been my view that the best way for operators go gain profits from higher-layer services is through offering mid-layer features that would make those services easier for OTTs to develop and deploy.

The third potential good thing is the most general, the most potentially important, and the most difficult to assess in terms of real operator commitments. It may be that this initiative represents a kind of capitulation by operators, an admission that connection services can’t pay off and that in order to make higher-level services pay off, you need an application/mission ecosystem to develop the framework in which they’re used, before the services themselves can be commercially successful. If operators are finally seeing that, then they may take realistic steps toward a new revenue model instead of trying to resurrect Alexander Graham Bell.

One must always take the bad with the good, of course, and the bad here is that the new APIs and services may be too low a set of apples. Yes, it makes sense to launch an initiative with service features that an operator can already deliver out of their software. However, what’s facile isn’t necessarily what’s valuable. An initiative that doesn’t deliver the desired incremental revenue or other benefits is not only crippled, it’s potentially destructive of further evolution of capabilities.

That’s particularly true of international bodies like the GSMA. Recall that when the NFV ISG was launched, it got off on the wrong foot and was never able to admit it and make the major adjustments that generating relevance would have required. High-intertia processes not only take a long time to put into place, they also take a long time to change. In the market for higher-level features, speed of offering and pace of response to market changes are both critical. That’s particularly true when some of the players most likely to avail themselves of these APIs are the cloud providers, and they have their own aspirations in the “facilitating services” space.

The challenge of mid-layer opportunities is that they can be attacked from below or from above. Cloud providers could build a “telco PaaS” set of web services that would be designed to let OTTs build services faster, cheaper, and better. In fact, most or even all the feature areas the new Open Gateway initiative wants to offer could be offered by cloud providers too. Given their well-known agility and the fact that they’re facing their own revenue growth pressure, cloud providers could well grab all the truly high-value facilitation for themselves, and leave the operators to handle the stuff that’s more trouble than it’s worth.

Telefonica, one of the operators in the initiative, announced it was expanding its relationship with Microsoft “to the area of programmable networks through APIs in the context of the GSMA Open Gateway initiative.” It’s not difficult to see how this might represent the classic “fox in the hen house” challenge, focusing Telefonica on delivering little pieces of value out of its OSS/BSS, but leaving Microsoft in control of the way that truly high-value stuff like a digital-twin metaverse, would be facilitated.

If, of course, Microsoft is really that much better at moving away from comfortable, incumbent, missions. The biggest problem operators have in any advanced services concept development may be their bias toward evolution. If you’re winning the game, you have less interest in changing the rules, and the rules need to change if operators are to restore their profit growth.

Saving profits for telcos globally would require telco advanced services revenues that total over $190 billion annually, according to my model. If telcos were to be the dominant providers of facilitating features, my model says that the total service revenues for all players involved would have to be about $350 billion. If telcos provides a smaller piece of the facilitating pie, the top-line revenues would have to be higher in order for them to meet their own revenue goals. Once total revenue goals hit roughly the $650 billion level, the benefits needed to justify that level of spending become difficult to achieve.

This may be the critical point that telcos and their vendors miss. It’s not enough to expose a few easy and limited-value OSS/BSS or connectivity services via APIs, you have to expose things valuable enough to induce OTTs to wholesale them. Otherwise, even major new revenue opportunities won’t contribute enough telco revenue to move the needle on declining profit per bit, and something more radical may be needed, even perhaps more radical than subsidization.