Month: July 2015

I said in a comment on an earlier blog that I thought all the IoT approaches touted so far were irrational.  In earlier blogs I’ve noted my view that IoT had to be viewed more as a big-data application than as a network.  A few of you have asked me to expand on my own view of IoT, and so that’s what I propose to do here.

“Classic” IoT is a vague model for device connection to the Internet whereby sensors and controllers of various kinds are directly connected to the Internet.  Once there, they’re available to fuel a whole series of new applications.  For proponents of this vision, the question is how we support LTE or WiFi interfaces to all these gadgets.  There are a lot of issues associated with this model, from a public policy perspective, from the perspective of ROI on IoT applications, and from a simple technology-ecosystem perspective.  We have to start an IoT discussion by addressing these issues.

In policy terms, it’s clear that just putting a bunch of sensors and controllers on the Internet would create a massive challenge in security and privacy.   Imagine how much fun hackers would have diddling with the traffic lights in New York, shutting down lanes on bridges and tunnels, and perhaps even impacting pipeline valves and the power grid.  Imagine how much easier it would be for stalkers (or worse) to track prospective victims by looking at the security and traffic cameras.  Happy fishbowl, everyone.  Obviously there’s no chance this could be allowed.

Perhaps, then, it’s fortunate that it’s far from clear who’d have an interest in deploying this stuff.  State and local governments in many areas have found they can’t even get permission or funding to set up traffic cameras.  Public utilities already have sensor and controller connectivity, but it’s shielded from the very open environment the IoT proposes to foster and they’d hardly be looking at magnifying their vulnerability.  Private companies would look at the IoT model and ask how they could possibly earn a return by just publishing data or allowing control openly.

The technical challenges fall into two groups, one relating to both policy and ROI and the other to utility.  On the policy/ROI side, the problem is that the more sophisticated you make a sensor or controller the more it costs and the more power it will need.  If you have a home security system you probably use inexpensive wired sensors for your doors and windows, and maybe for motion detection.  These probably cost about twenty bucks a pop.  Imagine an IoT world, where each of these sensors is online through WiFi or LTE, and each is equipped with a firewall and network-based DDoS protection to prevent attack.  You’re probably looking at five times the cost, plus you’ll either have to power the stuff or change batteries a lot.

The utility issue arises from the fact that a given sensor is just an IP address in the classic IoT model.  How is that sensor put into a useful context?  For example, if it’s a traffic sensor, what road and milepost is it located at, and what format is its information in?  Is it counting cars, measuring speeds, or both? How do we know it’s even what it purports to be?  It might be spoofed by some hacker or presented by an enterprising rest stop owner who wants to divert traffic by making an expressway look jammed with traffic.

In my opinion, IoT isn’t a movement at the network level, but rather an architecture built around a big-data model.  Imagine a database where information from known and authenticated contributors is collected and structured.  The contributions could include traffic sensors, home sensors, even locations of mobile devices.  All the data would be contributed based on policy-defined limits on use.  Those who wanted to use the data would do a big-data query that would be policy-validated to insure it meets security and privacy rules, and would receive what they needed—historically or in real time.  Control elements would be represented by write-enabled variables, and accessing them would also be policy controlled.

Where’s the network?  Behind the database.  Any owner of sensors could contribute information into the big-data repository, but they would control the contribution and be able to state policies on how their data could be used.  The “network” connecting their sensors could be anything that’s convenient, meaning that all of the current sensors and controllers that are networked using any protocol could be admitted to the IoT repository through a gateway.  No need to make sensors directly visible online, or to change sensor technology to support direct Internet visibility.

This sort of IoT could be visualized as a collection of “device subnets” that would use any suitable technology to attach sensors and controllers.  Each would have a gateway through which the data was pumped into the IoT repository, and the gateway would manage the policies and formatting.  The IoT repository would be an online database query service—a web service.  It might be linked onto a company VPN, to a cloud application set, or made available on the Internet.

You can probably see the similarities between this model and the web.  Anyone can put up a website; anyone could “put up” a device subnet directly, or contribute to one of any number of IoT repositories subject to their policies.  Anyone could access what’s put up, subject to whatever policy limits the owner imposes.  The commercial terms of any of these relationships could be whatever the market sets.

IMHO, it would be the IoT repositories that would establish the value of the whole picture.  Any cloud provider could establish one, of course, including Amazon, Apple, Google, and Microsoft.  Interconnect players like Equinix could build them, and network operators could as well.  For some of the players like Amazon, Apple and Google, you could see their repository exploiting the mobile devices they offer (directly or as a platform).  Auto manufacturers could join somebody’s repository or start their own.  Same with home security companies, federal, state, and local government, and even public utilities.

What about standards?  Well, if we presume the IoT Repository model, and if we presume that we’re accessing primarily those devices with large installed populations, standards shouldn’t really be much of an issue.  A query can format data in any way that’s convenient, unlike an interface.

This model is also easily federated.  We have hotel and airline sites today, and discount travel sites that create front-ends to their models, and even a couple who front-end the front-ends.  We could build gateways between IoT repositories, high-level repositories that culled specialized data from others, or did specialized analytics.  Think cottage industry.

One of the most interesting points about this model is that it raises what might be called the “utility IoT” approach.  A company deploys a bunch of sensors and controllers and pays for the effort by 1) contributing the data to repositories and/or 2) developing and deploying their own IoT repository where they charge for access.  Doing this would be easier for telcos and public utilities who have historically low internal rates of return and tolerance for high first costs, but in theory any player could bootstrap into it.

This isn’t classic IoT, it’s not a universe where new OTTs mine sensors that somehow magically appear, magically create ROI, and magically generate traffic and equipment revenues.  It’s somewhere I think we could get to, and that seems a better approach to me.

In prior blogs I looked at the NFV deployment model and the way that management as ETSI defined it would presumably work within a “typical” deployment.  The question this last of my more detailed explorations of NFV management will deal with is how “NFV management” relates to management and operations in a broader sense.  You can’t, after all, support services by managing only NFV infrastructure.  You almost certainly can’t built them that way either.

There’s no single management and operations model in play today among operators, but whatever is out there has to deal with those two areas in some way.  “Management” is normally applied to the physical resources used to build services, and “operations” to the business processes and commercial tasks related to service sale and maintenance.  It wouldn’t be unreasonable to say that operations is a customer-facing process and tool set, and management faces resources.  Since the TMF links these two in its SID data model, it should be clear that many view management to be “under” operations.  The fact that many services today are still provisioned through NMSs says that many see them separated.

Another TMF concept is useful in understanding management integration.  The Enhanced Telecommunications Operations Map or eTOM is a picture of the steps associated with creating, selling, sustaining, and terminating a service.  There are a number of eTOM references, depending on whether you are or are not a TMF member, but here’s a basic public vision.  eTOM is divided into levels or layers, and at the most detailed level it’s a pretty comprehensive picture of what has to be done from soup to nuts, service-wise.

In the real world, most eTOM activities are intermingled between human and automated tasks, and between operations and management tools (using my previous division of the two).  From low-level eTOM, one could almost picture service operations as a modular function, where different pieces might be implemented different ways and in different places.  As part of a service, NFV has to integrate in some way with eTOM.

How?  NFV, in the strict construction of the ETSI ISG, is a set of specifications that define how real network functions hosted in traditional devices could instead be deployed as cooperative software elements on some agile resource set.  The operative part of “NFV” that threatens the traditional management/operations model is the “virtual” part.  In effect, virtualization of any sort creates an intermediary.  We used to have customer-facing and resource-facing pieces, remember?  Well, now we have this “virtual” piece that might look like a resource from the customer side, a customer from the resource side, or all or none of the above.

In the ETSI E2E architecture, there is an implicit vision of how virtualization and management combine.  We have an Element Manager that’s almost cohabiting with VNFs and is responsible for management of the VNFs themselves in the “customer direction”.  We have a VNF Manager that is (via some intermediary elements) responsible for managing the resource relationships with the VNFs.  Presumably, though this isn’t stated explicitly, we have resource management tools and practices aimed at the NFV Infrastructure as a pool of devices.

IMHO, the ETSI activity has focused most of its specification work on the VNF Manager piece as the “management” approach.  This is consistent with what I’ve called a “black-box” view of network functionality.  A VNF is a function.  A function is managed as a function, not as a collection of chips (today) or software (under NFV).  What happens to make software into the manageable function we expect is largely the VNFM’s problem?  And largely what ETSI worries about.  We could draw this out if you like.  Make a box all the way on the right and call it “traditional management/operations”.  Draw a box to the left of that with a bidirectional arrow connecting it, and call the new box “ETSI Element Manager”.   Draw another right-working box called “VNFs”, then one more called “VNFM” and finally one called “VIM/NFVI” and you have the picture.

This picture doesn’t necessarily represent a break in any management model.  If we assume that the ETSI EM depicts the functional model of the underlying structure completely and accurately then we could substitute a VNF implementation for a real device 1:1 and nobody would care.  The devil is in the details.

Here’s an example.  We can horizontally scale components in NFV, right?  That’s supposed to be one of the benefits.  You don’t horizontally scale chips or devices on demand, so the current management model for Real_Widget wouldn’t have the properties of Virtual_Widget I’d like to sell, whatever a widget is.  However, I could in theory build a new Widget-MIB that had the fields necessary to represent incremental NFV functionality, and if my management system could contend with that extra data I’d still be fine.

Another issue less easily fixed is in the concept of FCAPS, which is traditionally seen as the high-level vision of “network management”.  All of the letters in the acronym represent something that had a single logical meaning in the old device days, but has two meanings in the world of NFV.  What’s a “fault?”  Is it a failure of the virtual device, meaning that we’ve exhausted the automatic remedies for replacement/reconfiguration of VNFCs that NFV might offer, or a failure of an underlying resource?

We could assume operations integration with FCAPS would work if we applied the acronym to the virtual world.  In the real world, downward to the resources, we have a problem of correlation because the relationship between resource faults and virtual device faults depends on how we’ve allocated resources and the extent to which we attempt automatic remediation.

Which raises the challenge of virtualization.  If we want operations to know about real problems, real resources, real capacities and cost accounting, then we have to dip below the virtual.  We have to somehow tie operations processes to the deeper reality.  That’s also true of management processes, because as we travel down the traditional service-network-element-management stack in a virtual world, we find there’s a basement, which is the virtual-to-resource mapping.

ETSI talks in general terms about operations/management relationships with the NFV software, but the interfaces for these are not defined nor are there any solid rules for how the relationships would be structured.  The TMF has a good opening approach in its customer/resource-facing service model and the (NGOSS Contract, now part of GB942, the TMF Business Services Suite) notion of steering service events to suitable processes through the intermediary of a service contract data model, but the specifics of this aren’t real clear even for the real-device world and that part of the TMF model is (according to my operator sources) rarely implemented.

In a standards sense, then, we’re not solving the problem yet.  Unfortunately, we can’t just ignore management integration because there will surely be no pure NFV service early on, and likely never a pure NFV service even down the line.  There are going to be legacy devices in networks for a very long time, likely forever.  Given that, and given that operations efficiencies and service agility isn’t very meaningful if you confine either or both to just a piece of a service, we need to harmonize management completely.  Here and there, federated and solo, NFV and legacy, applications and services, transport and connection.  Services to users have few boundaries even now, and management can’t have them either.

So here’s where I think we are.  There are only two ways to make a management connection from top to bottom.  One is to build “virtual-device MIBs” that could be based on current “real” MIBs but that would reflect data elements that represented any new service features, costs, or conditions that would arise in an NFV world.  We’d then have to populate these fields from real resource information as the service progressed through its lifecycle.  The other is to provide operations/management coupling through the virtual layer into the real resources.  My own work has always focused on the second of these approaches because I’m leery of having resources living behind a perpetual mask, but there’s no question that it would be easier to attack the former approach than the latter.

If this second approach is taken, then the service data model could be supplemented with the information collected when binding service components to each other and to resources.  These bindings could be traversed to dive into more detail on service state.  You could also, at any level of “object” in the model, describe the state/event relationships that would fulfill the TMF concept of mapping events to services.  It’s obviously more complicated, but if you did this you could define any current or newly developed operations process at any state/event intersection, and provide full integration of management components from top to bottom.

We have to do either a virtual-device-MIB or data-coupled management model; I don’t believe any other options even exist.  Unfortunately, I don’t think we have a convincing model for either in place; not in the ETSI ISG or TMF.  So I’d like to see operators and vendors cooperating (perhaps even in PoCs and lab trials) to explore the consequences of each approach and the alternatives for implementation.

NFV is turning out to be a lot more complicated than it first appeared, and that’s particularly true in the area most critical to vendors—the business case.  While the question of making a broad business case for NFV is weeding out a lot of secondary players, it’s not deciding a market leader yet.  In fact, it’s not even clear what a winning strategy will be.  We have three options out there, and now’s a good time to look at them.

What most operators want from NFV is what I’ll call a systemic model for deployment, something that can justify a broad commitment to NFV (and almost always SDN, collaterally) and bring NFV to the largest collection of services and customers.  The average operator I’ve talked with thinks that systemic NFV could touch as much as 75% of all customers.

In order for systemic NFV to work, you have to be able to deliver operations efficiencies and service agility, because operators (particularly CFOs) say that capex improvements won’t create enough momentum or even fully justify NFV complexity.  That means you need to extend ETSI-modeled NFV both into legacy infrastructure and into OSS/BSS orchestration.  You also have to be able to host a large number of diverse VNFs.

From a sales perspective, systemic NFV is definitely a “hang in there” proposition.  The sheer scope of the success goal means that nearly everyone who signs anything will have to sign off on systemic NFV.  IT will touch every piece of the network, every major vendor, every craft practice and operations/management software tool.  It’s also so big that it’s hard to grasp it, and many proponents of this model are trapped in small-scale on-ramp projects that might or might not lead to a realization of the broad goal.

The second approach to NFV success is the magic bullet model.  Rather than trying to build up NFV to a broad base through a wide range of services, magic-bullet proponents seek to identify a killer app, a single service that has so profound a benefit case that it can carry NFV into deployment by itself.  Once this app has greased the NFV skids, other applications can then follow along.

Magic bullets, to succeed, have to be both accurate enough and massive enough, and that’s the current rub.  The obvious candidate for a magic-bullet attack is mobile services, because mobile infrastructure is still the capex focus.  It’s easier to deploy a new technology where money is still being spent on a large scale, than to displace already-bought stuff elsewhere.  The question is whether mobile is the right target.

The risks of mobility lie in extensibility in the service domain.  Yeah, we can apply NFV to manage costs in mobile networks, and perhaps even to improve operations efficiency, but service agility goals demand hypothetical services.  IMS and EPC are candidates for early NFV exploitation, but they’re specialized multi-tenant applications.  Services built to demonstrate agility would have to be built both on IMS and on NFV to be relevant, and right now we use both IMS and NFV only for efficient hosting—we don’t have a model of service-building.

The third NFV strategy is what I’ll call (given my penchant for quoting old poems and music) the September-Song approach.  “…I let the old earth take a couple of whirls…” is the relevant theme.  Septemberish NFV advocates are essentially saying that NFV is inevitable, that somebody will hit on the magic formula for deployment.  That somebody will then spawn explosive NFV growth, which will create an explosive growth in demand for something NFV consumes a lot of—servers, data center switches, software licenses.

It you’re a platform (server, OS) vendor, there’s something said for the wait-and-see approach, because 1) you don’t have to go out and create and merchandise a full NFV solution and 2) you don’t face the risk of alienating the players who do manage to make a business case.  It’s a kind of arms-merchant approach to the NFV wars, because you have something everyone will need.

The obvious problem the Septemberists face is the risk that an NFV magician who’s able to make the business case will sell servers and software too.  That could happen both for a systemic NFV player or a magic-bullet player, and the result would be that Septemberists would have to fight their way into a deal whose business case is under the control of another vendor.

We’ve had examples of competitive evolution for all these approaches recently, which I think proves that none of them are off the table yet.  That also means none are winning convincingly.

HP is the paramount player in the systemic camp.  Their OpenNFV has legacy device orchestration, OSS/BSS and NMS integration, a strong ecosystem, a good on-boarding model, and good engagement in a variety of trials to prove out service breadth.  Their problem has been that they’ve become perhaps a bit obsessed with the trials and have underplayed their systemic assets.  That’s easy to do because it’s hard to make something like NFV operations efficiency exciting.  In the service agility area, services are VNFs and you can’t be seen to favor a given partner if you’re a partnership-driven ecosystem.

HP doubled down on VNF partnerships this week with a big NEC announcement.  One thing this shows is that larger players like NEC see HP as a viable platform going forward, an endorsement that’s likely to play well with operators and with other prospective partners.  But the press release on the deal didn’t mention any specific services, which means that it doesn’t add a lot of near-term impetus to HP’s drive.

In the magic-bullet class of NFV player, Alcatel-Lucent has been making news through NFV-ready IMS and IMS-related offerings.  A highly focused mobile drive has given Alcatel-Lucent a presence even in accounts where another vendor (HP, for example, with Telefonica) already had a win.  Alcatel-Lucent has, in its Rapport collaboration framework, an application platform to facilitate service creation that’s NFV- and IMS-compatible, and so it addresses the limitations of early mobile-service targeting I noted above.

The challenge is that platforms do not a service make.  IMS has been a theoretical platform for rich communications services for a decade and it’s not killing off OTT competitors.  Part of the problem is that it’s not entirely clear what platform capabilities Alcatel-Lucent’s Rapport and IMS actually bring to service developers or VNF developers, nor is it clear how NFV and IMS cooperate to be greater than the sum of the parts.  Alcatel-Lucent needs to make all that clear.

The Septemberist giant is of course Intel.  An optimal deployment of NFV could generate over a hundred thousand new data centers, ten times that number of new servers, and a heck of a lot of new CPU chips.  Intel is the clear leader to pick up the NFV financial marbles because they’re a part of almost any credible winning strategy.

To address some of the risks on the platform side, Intel has been pushing its Wind River Titanium Server strategy, and recently won a Nokia validation that might signal a firm link with the leading magic-bullet player—Alcatel-Lucent—when/if the Alcatel-Lucent/Nokia deal closes.  Wind River is also a platform partner for systemic leader HP.

For all of this, Intel still hasn’t taken a step toward making the business case.  Yes, they have the right hardware to deploy NFV.  Yes they have the right software platform.  They don’t contribute much to the direct business case, though, and so they are still at risk for a slow-roll NFV that undershoots potential, or the introduction of a competitor who is able to take advantage of the slow roll to get into the game.

So where does this leave us?  I think that it will be difficult, though not impossible, for any player—even HP—to make a pure systemic run at the NFV opportunity.  It’s probably too late to socialize that complex story, though I think they need to try.  I think that mobile is going to be hard to use as a truly universal magic bullet because it doesn’t hit enough operators, and doesn’t hit hard enough to push universal adoption, unless you build a service framework on it.  And I think that waiting for somebody else to win and hoping to ride on their coat-tails is always an unacceptable risk.

Something evolving from mobile has to be the answer, and I think that “something” is the always-overplayed Internet of Things.  All three of our giants are trying to come to terms with a real IoT architecture, and I think that whoever wins it can win NFV too, as long as they make what should be the obvious connections.  That would create a truly massive win, perhaps the largest in networking since the early days of IP convergence.

Hopefully your interest in NFV management prompted you to read yesterday’s blog and you’re ready to follow up.  If not, you may want to review it before you read this one because I’m building on the last one with only a very brief level-set!

Let’s assume we have a VNF with four components, one of which is horizontally scalable and sits in front of the other three, which are in line.  You can draw this out as four boxes left to right, with the user on the extreme left and the “service” interior on the right.  This is supported using a subnet and it’s got a private IP address (like the usual 192.168.1.x).  The leftmost VNF has a port exposed via something Google-Andromeda-or-Amazon-Elastic-IP-like mechanism which for brevity here I’ll just call “SuperNAT”.  Similarly the rightmost has an exposed port for service connection.

Let’s assume that we have a lot of load on our VNF on-ramp element.  The first obvious question is how we know that.  In the ETSI model we have Element Managers (EM) that are associated with the VNFs and we also have a VNF Manager.  It would seem logical that if the VNFs themselves were capable of understanding their own load profiles, EMs could communicate a need for scaling.  If not, it would have to come from “outside” meaning that the state of actual network and hosting resources might be used.

Whatever the source, scaling would have to be initiated as a lifecycle process, and the VNFM would drive the VIM to allocate additional instances.  That much is clear.  What is less clear is what would happen in a case like my example here, where in order to support multiple instances of our head-end VNF we’d likely need to load-balance.  We now need an additional functional component not part of the original picture.  How does that get instantiated?  Normally, the NFV Orchestrator would be responsible for this sort of decision.  Remember that we have a coordinated need to deploy the load balancer and to then reconnect the front-end elements, including the connection to the user.  (Note that for service availability reasons we might presume that every service with scaling had a predefined load-balancing element in the configuration to prevent interruptions during this reconnect phase.)

Faults are more complicated.  If something in NFVI breaks, then we have two possible paths forward.  First, we could assume that the fault would be recognized by the VNFs themselves based on conditions that would be visible to them on their interconnected pathways.  So if a VNF fails in our hypothetical service, the VNFs connected to it would presumably recognize the problem.  The other possibility is that the fault would be recognized by the infrastructure management systems, whatever they are.

The ETSI spec suggests that a VIM could notify a VNF manager of “changes in state”, and one might presume that this would mean that the VNFM could either undertake to replace the item on its own, or could notify an EM in the VNF, which would then start remediation.  It seems to me that if you have VNFMs and EMs in the picture, you have to let both of them know what’s being managed.

In a fault situation, we’re assuming that everything in real remediation terms is getting done by the VNFM, just as we did in the scaling example above.  That’s reasonable given that the VNFM seems to have all the parametric data on the service, but it kind of makes the Orchestrator look like a rump function.  I’d like to see a model where all of this was collected into a single software structure; I think it’s going to be difficult to build something with the separation of functions and the exposure of interfaces that the ETSI model defines, given the ambiguity of roles.

You can see the security ambiguity I talked about in the last blog more clearly now.  VNFMs have the ability to command resources, which means that to control a VNFM is to control infrastructure, at least in some sense.  The challenge here is that if the VNFM is specialized to the service itself, meaning we have per-service or per-VNF VNFMs, or even just if we have proprietary VNFMs, we’ve relaxed security on the network.  A single service, or worse a single service instance and its associated customer, has the ability to call on infrastructure.

I understand that this could in theory be prevented, meaning that you could “authenticate” requests.  The problem is that it’s hard to know what’s authentic.  Remediation or scaling requests additional resources, which obviously impacts the shared pool.  Under what conditions does a VNFM have the right to do that?  Who enforces the conditions?  If we say that the VNFM enforces its own security, we’ve just justified having no security at all because that principle runs afoul of all the firewall and management integrity checks traditional built into networks.

Then there’s operations integration.  We are spinning up additional VMs in scaling, and we’re replacing components due to a fault that would very possibly create an SLA violation in remediation.  It’s hard to see how both these conditions wouldn’t have to be reflected broadly, but in three specific places—the service model for NFV, the network operations center, and OSS/BSS.

Even for “NFV operations”, we have to maintain an accurate model of the service or we can’t respond to future change requirements.  Imagine the challenges of fault management if scaled components didn’t show up on the service model?  How does that happen, though?

I also wonder how a NOC finds out about a problem with a VNF.  You could say that overloading of a VNF isn’t a NOC problem, but if NOCs are still expected to respond to customer complaints, how do they see the conditions that the NFV service itself is responding to?

Then there’s how this gets integrated with OSS/BSS.  If a customer calls and asks something about the service, can a customer service rep dig into the details of the current service model and state?  Right now there’s no interface expressed for that, or any specific detail on the model itself.

You might get the idea from this that I’m saying that NFV won’t work as described.  I’m not saying that, but I am saying that I doubt that the ETSI model could be taken literally.  Operators tell me that all of their PoC and lab work is building out from basic ETSI descriptions into what’s essentially ad hoc extensions of NFV to cover all the bases.  That’s not necessarily a bad thing; innovation and multiple approaches can be valuable.  It does tend to negate the standards, though, because these innovations could very well be PoC-specific, service-specific, vendor-specific, and thus generate a bunch of silos.

What’s needed here?  Well, the simple answer is that we need to define the abstractions themselves—the service models that MANO would use, the models that are used by MANO to drive the VIMs—and we need complete flow diagrams to describe explicitly what happens under the kind of conditions I’ve outlined here.  You can’t define an architecture without testing your structure with the kind of things it’s expected to handle.  That’s routine in software design.  It needs to be done with NFV, and quickly.

NFV management has never been my favorite part of NFV, and I’ve groused about it here fairly regularly.  It’s probably time to talk about the issues in more detail, and so I’m going to do an as-yet-undetermined number of blogs in a series about the issue.

To get this straight, we have to set the stage.  NFV presumes that virtual network functions (VNFs) are collections of components that are hosted and connected during the deployment process by the NFV Orchestrator.  The management, meaning lifecycle management, of this collection is the responsibility of the VNF Manager or VNFM.

VNFs would have to be collected in some sort of subnetwork, and this is shown in the ETSI End to End Architecture Document’s Figure 3.  The easiest way to think of this would be as an IP subnet, though no specific reference to a network structure is provided in the document.  I’m presuming one here because it’s difficult to talk about management issues when you don’t have any specific way to reference the things you’re managing.

In our hypothetical NFV subnet we’d have a bunch of hosted software components (VNFCs) that are linked somehow.  The ETSI material calls the relationship a forwarding graph, but I’m not sure that doesn’t presume simple linear service chains.  Even if you had chains, you’d need to have something to chain through, meaning a network service that offered connectivity to the component.  Using this the elements inside a subnet would all be able to talk to each other, presuming they had an address reference.  Our components will also have to be visible to the real world, at least in part.  The ETSI Figure 3 shows endpoints connected to VNF1 and VNF3, which presumes that these endpoint connections on the VNFs are “visible” in the service address space, outside the VNF subnet.

Security, compliance, and sanity seem to dictate the presumption that our subnet is based on something like an RFC 1918 address space (assuming IPv4), so the VNFs would all be invisible to the outside world.  To make some of the ports on VNFs visible, we’d have something like NAT to translate between one of the private addresses and a public address.  We’d also need a DHCP function to assign addresses and a DNS to allow the components to see each other.  If we do this, then every VNF lives in its own private universe, secure from visibility to other VNFs and even to its own service address space.  It shares only ports it is designed to share, and only to other subnets it specifically elects to link with.

So where we are is as follows:  Something like Figure 3 would probably be set up by defining a private subnet with its own DNS and DHCP, and with NAT to convert between the internal addresses it wants to publish in the service data space, and addresses in that space.

We’re not done with subnets, though.  We have to be able to deploy this stuff, right?  Thus, we have to presume that there is an infrastructure network where all of the resources live.  We also have to assume that either in this network or in yet another network we have all the elements of NFV software, which means MANO, EMSs, and OSS/BSS connections (the actual OSS/BSS could be elsewhere but we have to be able to reach it).

You’re probably wondering why I’m getting into this, and the answer is that the framework we presume has to be there to deploy NFV will also have to support management of NFV services once they’re deployed.  We have to be able to harmonize the role of VNFM within this structure, and if we have any issues we have to get them addressed.

Management starts with the presumption that there is, included with a VNF, an Element Manager that performs all the VNF’s typical management functions.  This EM links with the VNFM for resource information and to provide lifecycle management.  The VNFM would go to the Virtual Infrastructure Manager to deploy something, such as a scale-out.  However, it also appears that the NFV Orchestrator also goes to the VIM for deployment.  To start with wouldn’t it be logical to say that “deployment” was a lifecycle stage?  Yes, but if an EM has to request lifecycle management that can’t happen till the EM, which co-deploys with the VNF, presumably, is actually deployed to do the requesting.

Apart from this we have some challenges of addressing and security.  It’s reasonable to assume that the EM talks to the VNFM through one of those NATted interfaces, so we can at least make the connection.  As long as there’s some record of the EMS address so that the VNFM could contact it, presuming it needs to, we are fine.

An issue arises if we look deeper into the VNFM proposal.  There’s a goal of supporting multiple VNFMs, so that VNF-specific VNFMs could be offered.  The reason given is that the task of lifecycle managing a VNF could be pretty specialized, and that may be true.  However, we now have to look at the addressing and security issues.

If a VNFM is provided by a VNF vendor, where does it live?  You have three options.   You can put it inside the VNF, inside the private subnet where MANO and the rest of the software lives, or in some third disconnected subnet.  What are the implications?

If you put the VNFM inside the VNF then we’re letting VNFs manage their own lifecycles, allocate resources, etc.  We have to give the VNFM a link to the VIM, which means that the VNF can “see” infrastructure directly and control real resources.  I think this is a serious security and stability problem.

If we put the VNFM inside the MANO subnet, we’re letting vendors add service-specific software inside NFV’s control software, where there are no barriers to what it could do.  That is IMHO a far worse issue with security and stability.

If we put the VNFM in its own subnet we’re still giving that subnet access to a VIM, and while that could be made more secure than the first (and second) options, it’s still not ideal.  The VNFM still can directly control resources.

My conclusion here is that we need to be looking at NFV deployments like cloud applications in a multi-tenant world.  Amazon and Google both provide a mechanism much like I’ve described to create subnets where components are hosted using private IP and then use NAT or “elastic IP” addresses to map to addresses visible outside.  We have to be able to draw a picture of an NFV deployment as a set of IP subnets which interconnect in some way.  Google offers such a picture in its own Andromeda architecture.  If we can draw the subnet structure of NFV we can see at least what connections between private spaces and public spaces are, and whether these connections are sufficient to permit NFV software to function as needed and secure enough to be acceptable by operators.

Functionality is yet another matter.  The best way to look at how this would work is to look at a deployment, a horizontal scaling, and a fault response.  That’s what I’ll do in later blogs.

 

 

Since the dual United Airlines and NYSE outages I’ve gotten a lot of email about the stability of new network architectures.  While I don’t have any special insight into those incidents and so can’t (and won’t) speculate on how they were caused or how they could be prevented, I do have some experience with network outages.  Are SDN and NFV going to make things easier for us, harder, or what?

The core reality of networking today is that it’s adaptive in operation.  Devices exchange information via control protocols, and the propagation of this information establishes forwarding rules that are then used to move packets around.  At the same time, management information moves to and from specific devices using the same data paths and forwarding rules.  The adaptive nature of today’s networks make them vulnerable in two distinct ways.

First, a device could be cut off from the rest of the network by a failure.  In this case, the device wouldn’t be accessible to management commands and thus could not be controlled, at least not until a pathway to that device was restored.  If the device had been contaminated by bad parameterization or software, the problem might prevent paths from ever being established, in which case you’d have to send someone to manually fix things (or provide an out-of-band management connection to every device).

The second issue is the bad apple problem.  You know (maybe) the old saying that “One bad apple spoils the barrel”.  The fact that devices in a legacy network derive most of their knowledge of topology and status from an exchange of information with adjacent devices means that a single device that’s contaminated could contaminate the whole network with bad information.  In most cases this means either that the device advertises “false” routes that are suboptimal or perhaps can’t even work, but it might also mean that the device floods partners with nonsense, ignores requests, and so forth.

Both these problems tend to happen for two reasons.  First, the device is parameterized incorrectly, meaning that there’s a human error dimension.  The largest network outage I ever saw in my career, which tool over fifty sites down hard for over 24 hours and caused failures of at least a quarter of sites at any given time for a week, was caused by a parameter error.  The second issue is a software problem.  We’ve all heard about how software updates to a device cause it to behave badly with its neighbors.

Logically, the questions to be asked for NFV and SDN are, first, how susceptible they’d be to the current pair of issues and second whether there might be new issues arising.  Let’s look at those points.

We have to set the stage here.  In SDN, we have a number of models of service in play these days.  Classic OpenFlow SDN says that white-box switches have their forwarding managed entirely by a central SDN controller.  In some cases classic SDN is overlaid on legacy forwarding, meaning that there’s still adaptive topology management being done by the device but explicit forwarding control via OpenFlow is possible.  Some other models (Cisco’s preferred approach) would utilize legacy adaptive behavior completely and use policies to add software control over the process.

In any model that retains adaptive behavior, we have the same risks that we have today.  If the model adds central SDN forwarding control, then we add the risks that such control might add.  Primarily, the risk of central control is the failure of the central controller.  If a controller fails, then it can’t respond to network conditions.  That doesn’t mean the network fails, only that it can’t be changed to reflect failures or changes in traffic or connection topology.  The big question for an SDN controller, IMHO, is whether it’s stable under load.  My biggest-network-failure example was caused by a parameter error, but the reason it exploded was that the error caused a flood of problems that swamped a central control mechanism.  When that failed under load, everything broke, and since everything now had to be restored, the controller never came up.

Bad-apple device problems in SDN wouldn’t impact the topology and forwarding of the network, but if a device went maverick and didn’t do forwarding updates correctly or at all, the central controller might not “understand” that the route it commanded hadn’t really worked.  I’ve not yet seen a demo/test of a controller that involved checking the integrity of routes and perhaps flagging a device as being down if it’s not doing what it’s supposed to do.

The cutoff problem in SDN has the same kind of risk.  A device could be cut off because an adjacent device killed the route to the controller.  If the device is functional enough to do what it would likely be supposed to do (try other paths) and if there were other paths available, you still might be able to restore contact.

Overall, my feeling is that purist OpenFlow SDN is less at risk to traditional adaptive-behavior-related outages for the obvious reason that it relies on central control.  If the controller is designed properly, hosted reliably, and if the devices are set up to deal with path loss to the controller in a reasonable way, then I think you can say that classic SDN would be more reliable than legacy networks.

NFV is a bit more complicated.  NFV doesn’t aim at changing network control-plane behavior, so if you hosted VNFs that did switching and routing via NFV you’d simply substitute a software version of a device for an appliance version.  All the adaptive risks would be the same.  If you hosted SDN VNFs and centrally controlled them, you’d now have the SDN risks.  Where NFV is different is first in the issue of node reliability and second in the management plane.

Servers, data center switches, and intra-VNF paths in an aggregate configuration make NFV more complex and likely generate a lower MTBF than you’d have with a ruggedized appliance.  NFV could potentially have an improved MTTR because you could fail over components, but you’d see an outage in most cases.  We also don’t really have much data on how fast service could be restored and how an extensive failure like a data center drop would impact the ability to even find alternative resources.  Thus, it’s hard to say at this point just what NFV will really do to network availability.

On the management side it’s even more complicated.  In traditional networks, management and data paths are handled equally, meaning that you have connectivity for both or for neither.  In NFV, the presumption is that at least a big chunk if not all of the management data is carried on a subnetwork separated from the service data paths.  It’s not unlike the SS7 signaling network of the old phone network (which we still have in most of the world).  If we presume that VNFs are isolated to secure them from accidental or malicious hacking from the data plane, we now have a subnet for every VNF and management connections within (and likely to and from) those subnets.  Because NFV depends on better remediation for its availability more than reliable appliance strategies, loss of management integrity could hurt it more.

The net for NFV is that we don’t know.  We have not built an NFV network large enough to be confident we’ve exposed all the issues.  We haven’t identified possible issues fully, and tested them in credible configurations.  I think that it would be possible to build NFV networks that were less susceptible to both the bad-apple and cut-off network problems, but I’m not sure the practices to do that have been accepted and codified.

The net, IMHO, is this.  If we do both SDN and NFV right, we could reduce the kind of outages we’ve seen this week.  If we do them badly, deploying either or both would make things worse.  Since we have far less experience managing SDN and NFV than managing legacy networks, that tells me that we have to be graceful and gradual in our evolution, or we’ll make reporters a lot happier with dramatic stories than we’ll make customers happy with reliable networks.

One of the ironies of NFV is that its greatest success may be coming from deployments that are actually not NFV at all.  A part of this is due to normal market dynamics; you always try to pick the low apples first.  Another part is due to the scope limitations I’ve blogged about before; holistic benefits demand a holistic solution, and standards-based NFV doesn’t cover enough ground yet.  One interesting question is whether the current dynamic could help or hurt long-term NFV deployment.

Most of the publicized NFV service strategies are based on a CPE-hosted functions model, rather than the cloud-or-virtualization-hosted model that operators first envisioned and that the ISG is working to define.  In the CPE-hosted model, a user is given a premises box that is capable of being loaded with feature software from a central management system.  This box then provides the “virtual functions” on demand, easily updated to new versions, deleted when no longer needed, or augmented when conditions demand.

Operators tell me that the primary reason for this shift is the problem with “first cost” in NFV deployments.  If you presume central hosting of VNFs in NFV you need something to host them on, and unless you want to hairpin traffic a considerable distance you’ll need those hosting points at least proximate to the points of user connection.  For a network operator rolling out an NFV-based service over three or four or thirty or forty metro areas, this means an early commitment to multiple cloud data centers with enough servers to create suitable economy of scale.  That cost will rack up immediately, and the operator will then have to wait while marketing and opportunity combine to create buyers, which is why it’s called “first cost”.

When you use CPE hosting, you deploy a hosting point on the customer premises, and the cost is incurred only when you have compensating revenue.  Costs continue to scale with revenues through the life of the service.  When you use CPE hosting, you also eliminate some of the problems associated with shared infrastructure.  Security is easier to address because you don’t have multiple users sharing servers.  There’s less chance of one user’s service impacting another by having them load the server hosting a component of both users’ services.  Management is easier because there is a real box that contains everything.

If you look a bit deeper into the CPE-first movement, you see it’s a reflection of those shifting value propositions for NFV.  We started saying that shared-server efficiencies would reduce capex, and clearly the CPE-driven approach wouldn’t have shared servers at all.  That means that the justification, the business case, has to be developed by reduction of opex or improvement in the revenue flow.  That’s good news for operators because it suggests that the two NFV benefits that seem now to be the most credible are in fact credible enough to overcome any need for capex reduction at all.

CPE-first deployment really doesn’t need “NFV” at all, it needs only a management system to push software images into the CPE.  We have other successes with “NFV” in IMS and EPC, but these are deployments of multi-tenant assets that are actually likely to look more like cloud computing mediated by NFV management than like the kind of per-user-and-service NFV everyone expects.  The bad news for NFV, then, is that while the benefits are being proven we’re not really validating the full architecture.

If you look at the NFV specifications, you see a significant amount of work put into the details of picking the right place to host stuff, into creating high levels of availability, and so forth.  Tuning, in short.  If NFV is either about CPE-hosted VNFs or largely persistent multi-tenant VNFs, then are these microtunings useful?  We’re not making a resource pool selection in most cases.  That suggests that we may be paying too much attention to optimizing something that’s not currently proving to be important at all.

The big question, of course, is whether we can get to NFV deployment in a more “as-we’ve imagined” sense from what’s actually happening.  It’s a hard question to answer because you can look at the evolution two different ways.

If vendors who support either of the current strategies are capable of delivering centrally hosted NFV and supporting the standards, and if they are not tempted to under-commit to traditional NFV implementation points by the fact that they are making money doing something different, then everything we have going on could grease the skids for NFV progress.  If those vendors focus on the limited needs of their early service successes, then the broader features of NFV may end up becoming at best “options” to be offered later or at an additional cost.  We’d create a pseudo-NFV, or even (as some of my blog readers have suggested recently) a whole bunch of walled garden strategies because we don’t need the NFV standards much at all.

It seems to me that the answer to what happens in NFV evolution is going to come from the way that the NFV business case is made.  The current successes, if you look at them in benefit-harnessing terms, are successful because they address a special case of something.  The CPE-hosted approach, for example, addresses service agility in the context of business users whose “agility” needs focus on connection-point features and whose service value is high.  The IMS/EPC examples address operations improvements in a multi-tenant service set that simplifies the operations changes needed, versus per-user-per-service deployments.

Open, multi-vendor, revolutionary NFV has to be more pervasive than our current applications.  I think we’ve validated the notion that operations efficiencies and service agility can justify something that’s at least NFV-like if not a full NFV deployment.  We have to go the rest of the way, which means that we have to be able to exploit the benefits of service agility and operations efficiency more broadly.  IMHO, this is what all the NFV efforts should be directed at doing.

It’s an “enoughness” problem.  It’s not that we can’t help operations efficiency today, but that we can’t help enough of it in scope terms, and help it enough in terms of cost impact. In theory we could get there by expanding NFV’s scope into operations orchestration and legacy equipment, but I’m not sure we have time for that gradualism.  We need to find a broader trigger for NFV opportunity, one that exposes benefits on a broad scale but can still be implemented at least somewhat gracefully.  That trigger is probably IoT.

Something like IoT, truly and fully modeled as an NFV application and supported with a credible set of products, is the key to NFV’s broad success because such a service set would be a model for broad NFV deployment as well as a source of near-term drivers.  So what we may be proving here is that a truly comprehensive IoT implementation is going to be the thing that gets NFV moving, that keeps it from becoming nothing more than a series of vague specifications to guide specialized deployments.

We have to do more with NFV to get more from it.  We also have to get out of gardens or we’ll inevitably get walled in.

Alcatel-Lucent has collected a number of important NFV deals recently.  The company won a pair in China (China Mobile and China Unicom) and also won an expanded deal with European innovator Telefonica.  There seems to be a common element in these deals—voice services and mobility.  I think that says a lot about NFV and how it will deploy.

Up to now, we’ve seen what I’ll call an NFV-directed vision of NFV deployment, meaning that NFV benefits were expected to directly drive a roll-out.  Operators might differ in terms of the specific benefit targeted or their path to realizing it, but in all cases it was an NFV benefit doing the heavy lifting.

I’d suggest that Alcatel-Lucent’s recent success is due in large part to a shift in NFV drivers.  What we’re seeing is actually more a cloud-centric vision of the future, a future where what we’re trying to achieve is cloud hosting of multi-tenant elements or services rather than an NFV-centric deployment of service-specific functional instances.  NFV’s benefit is not in “enabling” this but in making it efficient and agile.

In my view, the original mandate for NFV—capital cost efficiencies—has been largely ineffective at moving the ball.  There’s not enough savings to be had even under ideal conditions, and those ideal conditions could be achieved only if we somehow made the more complex VNF-based services operationally effective enough to be as cheap or cheaper to run as legacy equivalents.  That leaves operations efficiency and service agility as the benefits essential to NFV success.  The point now seems to be that these benefits aren’t conclusively linked to NFV.

There is little credibility to the notion that customers would pay for NFV itself—they’d pay for services that NFV deploys.  Those services, in the form of VNFs, provide what’s sold to users and NFV provides how the sale is realized through deployment and lifecycle management.  A lot of this view has been expressed through past assertions that VNFs were critical in NFV—you need functions and features to build service value.

If there’s little chance customers would buy NFV there is zero credibility to the notion that you can create agility or operations efficiency for just a piece of a service.  We’re going to have legacy elements forever.  We’re going to have services made up of many VNFs, even many of what could be called “images” that are cooperative collections of VNFs assembled to create services quickly.  Because of scope limitations in the ISG from the first, management of legacy elements is out of scope and management of complex services has yet to be conclusively addressed.

The reason these points are relevant is that operators absolutely need relief from their revenue/cost-per-bit margin compression.  That induces them to look for solutions to the new-service and efficient-operations problems.  Alcatel-Lucent offers (in its virtual IMS and Rapport platform) a strong voice and collaboration services framework that builds easily on mobile investment but also introduces a true value proposition for IP voice services for non-mobile applications.  They have, in Motive, an operations framework that can orchestrate and manage services end to end regardless of the mix of legacy and VNF elements.  So what we’re seeing is Alcatel-Lucent exploiting its strengths, and pulling NFV along with it.

You might wonder why that’s true, and here I think we come to an important point.  Hosted service or application features need to have as much automatic management and deployment features as you can introduce or they quickly fall into the same margin-compression problem we have for bit services.  Furthermore, it’s clear that in cloud computing as in NFV overall, you can’t make a durable value proposition if you can’t build highly dynamic services that are increasingly directed at people not at sites.  That means extemporaneous service delivery, which by even the standards of “traditional” standardized NFV could be difficult to achieve.  Alcatel-Lucent knows, as most NFV leaders do, that some superset of ETSI NFV is going to be critical, so they want to control as much of the early opportunity as possible so they have a commanding lead when things get moving.

This isn’t a shift that’s limited to a single vendor and a couple operators.  I’ve blogged all this year about the fact that NFV is taking a more operations-centric tack in the real world, even though that shift is much harder to detect in the progress of current PoCs or in vendor announcements.  The vendors who have the greatest NFV traction (Alcatel-Lucent, HP, and Oracle in alphabetical order) are all vendors who have an exceptionally strong operations story.  An operations story that is in fact outside the scope of official NFV.  Oracle, who in my view is functionally in third place on the list, is nevertheless the vendor growing fastest in recognition by operators, because they’ve made an operations-first strategy their cornerstone.

The shift Alcatel-Lucent’s recent wins highlights just may be a leading indicator of a face-off on NFV in the near future.  Alcatel-Lucent is the only network vendor of the leaders, and thus Alcatel-Lucent has network/service assets (like Rapport and IMS) that can easily be pushed into near-term opportunities for new services and new revenues.  That’s harder for the other two vendors because they lack the incumbent service infrastructure position.  However, vendors like HP and Oracle could frame a broader story through partnerships.  I think we can see a little of that in HP’s IoT approach, which I think has tremendous potential both as a service element and as a blueprint for handling multi-tenant features deployed through NFV.

This year is critical for the operators because most say that in 2017 their revenue-and-cost lines will cross over for connection services.  That can’t be remedied without some significant shift in infrastructure policy in 2016, and it’s hard to see how an operator could drive that sort of change without having some direct and positive field trial experiences in 2015.  These guys have got to get going, they’re not going to wait for mature standards because they can’t, and they’re going to follow the path of greatest reward and least risk.  That path may lead away from traditional NFV issues into OSS/BSS, service creation, mobility, content delivery, and (yes) IMS.  If it does, then Alcatel-Lucent is going to have a lot of reasons to be happy.

In the idealistic vision of the future network (a vision I still hope can be realized), NFV forms an operational- and feature-enhancing umbrella over SDN to create agile services that improve efficiency and add greater value than the basic connection services of today.  This vision would require some significant expansions in scope for NFV; primarily, NFV would have to be given the ability to orchestrate and manage legacy elements of infrastructure.  It’s not a pipe dream that this could happen because we have vendors who already offer this broader scope.

There will always be real network elements, and likely there will always be legacy L2/L3 services as well.  Most of the “SDN” and all of the “NFV” to date are hybrids with legacy technology.  We’ve started to hear about how legacy technology extends NFV hosting of features by connecting the feature subnetworks with the users of the service.  It also creates the underlayment for many of the virtual connectivity features, and this raises the interesting question of whether NFV should be considered to be multi-layered and vertically integrated.  NFV, via SDN, makes vSwitch connections.  Could it also control the real switches underneath?

One reason this is important is that fault management in even today’s services has to contend with the problem of “common cause”.  If a fiber trunk is attacked by the classic “cable-seeking backhoe” then there’s a physical layer outage, a Level 2 outage, and a Level 3 outage even if we limit ourselves to classic OSI.  The break could generate a flood of failures at the service level.  If NFV is responsible for remediation of service faults, does NFV have to “know” that fiber trunk outages should be addressed at the fiber level through rerouting and not by re-framing every service over the trunk individually to use a different path?  In a network operations center today, we’d see fault correlation activity to try to prevent this fight-the-symptoms-not-the-disease syndrome.  How would that work in NFV?

Even at a more mundane level, we have to wonder whether a universally capable and operationally optimizing NFV implementation wouldn’t be used to provision underlying facilities so their operation would be optimized too.  In our hypothetical data center with vSwitches, why wouldn’t we use NFV to provision the physical devices?  Don’t say there aren’t any either; even SDN would demand white-box facilities and SDN depends on having control paths to the switches.

Then there’s multi-tenant.  Suppose we decide to set up a kind of super-IMS using NFV, as many are already proposing and as Alcatel-Lucent is already being contracted to do.  IMS isn’t single-service-per-instance.  You set up an IMS for an operator, not for every call, but how does NFV’s deployment of a multi-tenant resource provide for integrating that resource with other services and applications?  If super-IMS exposed APIs for other services to use, how would those APIs be made available to the other services?

One concept that I think is absolutely critical in addressing these kinds of issues is that all network services have to be framed as network-as-a-service abstractions.  A service at any level is a black box, known by its properties not by its contents (which are invisible).  What follows from that is that the NaaS abstraction has management properties and state which are derived from those invisible contents.  The user of the service “knows” whether the black box has failed, but not how the failure happened.  I think this vision is at least somewhat accepted in both SDN and NFV, but not completely, because we don’t address the notion of layered services even though all services today are layered.

In layered services, a given level (the “retail service” for example) is composited from lower layers.  The user layer is a NaaS, but so are the lower layers.  The retail service would not then have visibility down to the bottom of the stack of infrastructure actually used, but only to the level of the black box combination below.  It would be responsible to remedy faults reported by its own black box and present a fault to its own (retail) user if that isn’t possible.

In fault management terms, this could have profound implications.  If a “retail NaaS” sits on a couple of “component NaaS” services that in turn exercise “transport NaaS” services, then a lower-layer fiber fault causes not a retail fault but a transport NaaS fault.  The lower layer would be given the opportunity to correct the problem, in which case you’d have a report of an interruption but not a failure at the retail level.  If the lower layer (transport) can’t fix things, then the problem would escalate to the “component NaaS” level for remediation, reaching the retail level only if nothing can be done below.

Our visions of SDN and NFV are both, by this measure, too vertically integrated.  We are expecting to allocate resources at a primitive level and not through adopting the NaaS services created by lower layers.  One thing that I think the “right” vision of NaaS would have done is make the whole SDN/NFV integration question moot.  NFV does not, ever, exercise SDN.  It exercises NaaS abstractions that can be fulfilled by SDN.  We need not, should not, focus on the implementation because that would mean our NFV principles would violate the principles of layered networking that are the foundation of packet communication of all types today.

In this framework, the absolutely critical element is that black box.  A black box is defined by its properties, as seen from the outside, so what we need to be thinking about is how we describe this in technology terms.  The most logical answer, I’ve suggested in the past, is the notion of a “recipe”.  If you’re making Margarita Shrimp, you have a black-box abstraction in hand.  The recipe name is the name of the abstraction, and the recipe is a procedure that realizes the outcome (produces the dish) when invoked.

We don’t need to name every possible dish to cook, nor does NFV or SDN have to name all its possible abstractions.  We have to be able to assign a “dish name” and provide a recipe for it no matter what it is.  The biggest hole in the notions of SDN and NFV as they are today is that we’re not focused on the notion of either producing or consuming black-box abstractions.  Without that notion we can’t do layers, and we vertically integrate services so that a common low-level fault blows up into an avalanche of service failures before anything really tries to deal with it.

To me, the lessons of our layered past dictate we have a layered future.  That means that we have to think about the basic principles of isolation, and adhere to them on one hand while making sure that we don’t constrain future services by depending on fixed service models based on older technologies.  The way to accomplish that is simple.  Named abstractions with recipes.  If that concept can be brought successfully into both SDN and NFV, we’ll take a giant step toward saving ourselves from a lot of problems—including those scalability issues I blogged about yesterday.

Over the holiday weekend I got an email from a network operator friend who offered a comment on the state of SDN and NFV.  The point was simple; it’s not completely accurate to say that the PoCs and trials so far have validated either SDN or NFV functionality.  The reason isn’t that these efforts have left functions out, but that they’ve not addressed operation at scale.

SDN provably works in a data center.  There are a few NFV models that could in theory deploy in a kind of minimalist way (service chaining using edge hosting, for example) but nobody thinks SDN or NFV can survive on those models alone.  We have to make both SDN and NFV work in massive deployments.  How?

The first step, on which my operator friend and I agree, is to formalize the process of “domains” and “federation”.  If we think of SDN as being a domain of switches under a controller, we’ve defined an SDN domain.  If we think of NFV as being a collection of NFVI under a single MANO instance, we’ve defined an NFV domain.  The point is that we know that there will be multi-domain networks built using both technologies, so we need to understand how services can cross domain boundaries.

The current mechanisms for SDN federation rely mostly on legacy protocol features—BGP is an example.  This isn’t a long term approach to the issue because it doesn’t let services be built across multiple domains using the full set of SDN features.  In SDN it seems obvious that we need to have a control hierarchy, and/or define an “interdomain” protocol set that would let controllers cooperate to establish services across a connected set of domains.

NFV has no real federation capability as yet, though the NFV ISG has deliberated the question of interconnection of domains.  The sense I get from operators is that they are seeing cooperation as being primarily an exchange of resources rather than cooperative functionality.  An operator might deploy on NFVI provided by a partner, for example, but they’d use their own MANO.  That doesn’t seem to be a good long-term approach either, in no small part because of the next point.

Which is that the limits of domain size and performance have to be established.  How many switches can an SDN controller manage, and how many service creations and changes can be managed/orchestrated by NFV MANO?  It’s possible to extrapolate based on controllable events like service setups just how many events a controller or MANO instance could support.  The problem is that it’s difficult in the context of limited trials to understand how many events might be created under a full set of real-world network conditions.

Suppose we have ten thousand customers with service-chained cloud-hosted components.  We’re using NFV to orchestrate the hosting and SDN to make connections.  Even ten thousand customers probably generate only a few moves, adds, and changes in a day so this isn’t much of a problem for either technology.  Now suppose that we have a major trunk or data center failure.  We have a good number of those ten thousand service customers looking for automatic remediation.  Few would believe that a single instance of an SDN or NFV controller could absorb that load, particularly when the users would be scattered about a metro area or larger.

You can’t have multiple co-equal controller instances trying to allocate the same pool of resources.   How does Guy A know that Guy B took capacity on a given trunk?  What happens if we had conflicting assignments of capacity?  There has to be some higher-level process that “knows” that when you have a massive failure you have to start by trying to replace the massive facilities that failed, and then move upward for remediation that will ultimately reach the service level.  What process is that in either SDN or NFV?

Even if we have controllable, federated, domains we still have to be able to engineer and test the combination without creating a national-scale communications disaster to prove we can handle one.  That means that both SDN and NFV domains have to be engineered to appear as true black-box functional elements so that domain management doesn’t have to dig into the details of what’s out there to understand how to hand off to it or replace it.  It also means that element behavior has to be designed to meet domain behavior standards.  We’re talking a lot about SDN or NFV performance, but we don’t really have a standard against which we can measure it.  We don’t know what a domain has to do, and thus don’t know what elements of that domain have to do to make the domain work.

Part of this engineering is dealing with the impact of various technology options and proposed specification elements on “domainability” of the whole.  For example, we know that SDN has two modes of route control.  One says the controller simply lays out routes based on a central topology understanding and analytics on device behavior.  The other says that when a packet is presented at a switch with new forwarding needs, the switch asks the controller for handling.  I think everyone understands that there are strengths and limitations to both these modes, but do we know what they are to the point where we could size controllers and domains?  I doubt it.

Part of the problem is goes back to federation.  We know based on cloud computing and OpenStack experience that there’s a limit to the size of a domain that a given instance of OpenStack can support.  Some of the elements are single-thread, and it’s hard to see how you avoid that when (as I noted above) resource grants have to be coordinated to make sure several control points don’t grab the same thing.  How does that serialization happen, and what’s the performance implications for the mechanism we’ve selected?

My operator friend is right; we’ve not really dug into at-scale SDN or NFV as strongly as we should have, and as a result most prospective users (and some actual users) don’t understand what might happen if an event creates a flood of changes.  I’ve had plenty of experiences in the networking industry where a small situation caused a cascade of problems that swamped the whole of a network.  The worst example I ever saw of an enterprise network failure, one that impacted almost sixty locations and over ten thousand workers in a mission-critical field, started with a link error.  The escalation to disaster wasn’t caused because the error spread, but by the fact that the remediation overloaded critical management elements way past the point of surviving or failing gracefully.  Neither SDN nor NFV can afford that, and while I think both technologies can be made to scale, federate, and survive, I don’t think we’re as close to being able to do that as we should be.