Old Town Square, Prague, Czech Republic [© F2014]

New Carrier Network Part 1 — Bell Lab’s Future X Network and Nokia’s Network Function Interconnect

Introduction

The last major carrier network transformation happened in 2000s when carriers all over the world replaced their multiple network silos comprising TDM, ATM, Frame Relay, and IP etc… for their voice, video and data traffic by a single QoS-capable networking technology called MPLS. The new QoS-enabled MPLS technology allows a carrier to:

• Save capex and opex
• Speed up new network service introduction

The MPLS technology revolution in the last carrier network transformation serves the carriers and the customers very well. Also, the telecom industry as a whole learned a lot about label switching to transport network services with different QoS requirements and it is reflected in many IETF’s RFCs such as:

• RFC 5439 — An Analysis of Scaling Issues in MPLS-TE Core Networks
• RFC 8402 — Segment Routing Architecture

A few years ago, Bell Lab introduced a new cloud-native carrier network blueprint called the Future X Network. The following shows its high level architecture:

Bell Lab’s Future X Network is designed to be the new cloud-native carrier network to support all current and foreseeable wireless and wireline (e.g., 4G and 5G wireless, IoT, Cloud and NFV etc..) traffic capacity, service QoS and latency requirements. With the popularity of network function virtualization (NFV) and Cloud-Native Network Functions (CNFs) to replace many physical network functions for cost-effectiveness, elasticity, and scalability, Clouds supporting NFVs and CNFs become part of the new cloud-native carrier networks.

In Future X Network, it also defines higher value networking functions for the new cloud-native carrier networks such as programmable network interface, Analytic and machine learning for carriers to effectively control, manage, and monetize their network resources. Future X Network attempts to resolve all upcoming carrier networking challenges in a holistic rather than piecemeal approaches.

One of the key elements in the Future X Network is the Smart Network Fabric (SNF) showed as Domain 3 in the above diagram. SNF comprises all the end-to-end network and compute entities such as Edge/Core Clouds, Access/Core Networks and all packet and optical network interconnections to form the new cloud-native carrier network. SNF attempts to resolve the following problems that carriers are facing now and in the near future in their networks such as:

1. A new network core or underlay network that is 10X to 100X larger in traffic capacity than the current carrier networks (i.e., SNF’s Driver — Scale the Core Network)
2. Support down to 1ms traffic latency for new network applications such as 5G autonomous driving and real-time robotic (i.e., SNF’s Driver — Scaling Out the Edge)
3. Network provisioning and control automation due to the significant increase of network complexity and the numbers of service endpoints with the increase in the size of core networks, and the popularity of private and public Clouds and IoTs etc… (e.g., SNF’s Driver — Scale the Dynamism)
4. Support timely setup and tear down of end-to-end network services with deterministic service QoS to fulfill their Service Level Agreement (SLA). The end-to-end network services can span from wireless radio access, access/core network, to compute and storage resources of VMs in Data Centers (i.e., SNF’s Driver — Scaling New Service SLA with Slicing)

The above four problems identified by SNF for the future cloud-native carrier networks become some key drivers along with the higher value networking functions such as programmable network interfaces for the next carrier network transformation after MPLS in 2000s.

This article describes the details of each of the four SNF drivers and their corresponding technology enablers. Unlike most articles in my blog where you can use your home PC to verify the configurations, this article presents theory-only information but should be useful for readers to visualize what the future carrier networks look like. If there are any differences between SNF presented in this article and the official Future X Network’s SNF information from Bell Lab, the later is of course the correct version 🙂

1) Scale the Core Network

In the near future, the growth of network traffic generated by the new 5G wireless, IoT, Clouds, robotic and Internet etc… will demand a new carrier network core that is 10X to 100X larger capacity than the current carrier networks. A core network can comprise multiple domains or networks such as:

• Wireless Radio Access Network (RAN)
• Access and Core packet networks
• Edge and Core Clouds
• Optical transport network supporting the packet networks
• All network connections interconnecting the clouds and the networks

Within the Access and Core packet networks, we can further divide a packet network into the Overlay and Underlay packet networks as follows:

The following shows an example how Overlay network services such as Virtual Private Routed Network’s Hub & Spoke and Service Function Chaining can be setup easily over the rather static Underlay network:

Another example is an Overlay ePipe service deployed between R1 and R4 in the below diagram I used in my Segment Routing article. This Overlay network service can be supported by different Underlay network tunnels such as the shortest-path (R1 — R2 — R4) and the Traffic Engineering (TE) path (R1 — R3 — R2 — R4).

To achieve 10X to 100X more traffic capacity in the Core network, some new packet and optical technology are needed such as:

• Advanced in silicon and component packaging technology such as higher speed (e.g., > 400Gbps) network processors, and switching fabric interconnections etc…
• Optical Colourless, Directionless, Contentionless Flexgrid (CDC-F) for flexible optical wavelength routing
• Optical Probabilistic Constellation Shaping to improve Signal to Noise ratio for successful optical signal detection over a long distance
• Space Division Multiplex (SDM) using wireless’ Multiple Input and Multiple Output (MIMO) idea can potentially offer 10X increase in the amount of traffic that can be sent and received over a fiber
• Ultra wideband wavelength routing using both the optical C and L bands’ channels

Each of the above technology deserves a lengthy discussion and we will leave them for future articles for the time being.

2) Scaling Out the Edge

In the last network transformation with MPLS, packetized voice was one of the most latency and jitters demanding network applications for carriers and the maximum round-trip latency should not exceed 250ms. In this new carrier network transformation, the new carrier networks need to support latency sensitive wireless and wireline applications down to 1ms round-trip latency for network applications such as autonomous driving and real-time robotic etc… Note that a 100km optical fibre with zero hardware and software processing delay already incur 1ms round-trip latency. Therefore, unless we could send network traffic faster than light, the only way to support this kind of latency sensitive network applications is to place the processing systems such as VMs and Containers close to the transmission sources — hence, Edge Clouds.

Using the new 5G wireless deployment as an example. An Edge Cloud can be deployed say, within 10km radius of its supported gNBs (e.g., 5G new radio) so that 5G software running as Containers (i.e., NFV) inside the Edge Cloud can have 900us round-trip processing budget to meet the 1ms round-trip latency requirement. It is likely that latency sensitive 5G traffic is tagged with say, APN = LOW_LATENCY in the PDU Session Establishment request for the traffic to be processed at the Edge Clouds and all other latency insensitive traffic is forwarded to the Core Clouds for economic traffic processing.

In short, the Scaling Out the Edge strategy calls for:

• Strategic placement of processing VMs/Containers in the Edge and Core Clouds for latency sensitive and insensitive traffic processing respectively
• Use of Network Function Virtualization (NFV) running on VM and Containers on x86 servers in the Clouds instead of proprietary hardware networking platforms for cost-effective and flexible network traffic processing through VM/Container (i.e., VNF) orchestration

Luzerner See, Lucern, Switzerland [© F2014]

3) Scale the Dynamism

The new 10X to 100X core network, 5G wireless, public and private Clouds etc… significantly increase the number of service endpoints and network control and provision complexity. Therefore, a new approach to control and provision the new carrier network is needed such as:

• Automated network control and provisioning systems with assistance from intelligent analytic systems to understand the current and historical network traffic patterns so that necessary preventive and optimization actions can be performed
• End-to-End Centralized instead of manual node-by-node network control and provisioning such as a centralized Controller advocated in Software Defined Network (SDN)

The original Bell Lab’s Future X Network does not go into details to define protocols and technology for the Scale the Dynamism strategy. Nevertheless, Nokia has published a white paper called Network Function Interconnection (NF-IX) and it can be used as an implementation example of the cross-domain Overlay and Underlay packet networks for SNF’s Scale the Dynamism (this section) and Scaling New Service SLA with Slicing strategies (next section).

Segment Routing Interconnection Controller (SRIC) is the packet network SDN Controller referenced in Bell Lab’s Future X Network. The following shows the control protocols and operations of SRIC:

There are a number of technical challenges that SRIC needs to resolve for centralized provision and control of the cross-domain packet carrier network such as:

1. Major Network architecture decisions

• Why centralized network control and provisioning? Why not maintain the existing manual nodal router control?
• Why all packet vendors decide that Segment Routing (SR) is the future Underlay network data plane? What are SR’s advantages?

2. Overlay network service offering, topology discovery and provisioning

• What Overlay network services should be offered to subscribers, VNFs and PNFs?
• Which protocol can SRIC use to provision the Overlay network services onto the routers?
• How an Overlay service, say EVPN VPLS with EVI 1234 discover its member routers in Access, Core, and DC networks?
• How Overly network services know its Underlay network binding?

3. Underlay network discovery (particularly shared networks with different admins), TE info distribution, and SR-TE Tunnel signaling?

• How shared networks exchange their topology and TE info with SRIC without exposing their complete Underlay network topology?
• When each network runs its own IGP for scalability and security, how SRIC can signal a contiguous SR-TE tunnel across all domains/networks?
• How SRIC obtains Underlay network TE info from all shared networks for its PCE to compute the contiguous SR-TE tunnels?
• Which protocol SRIC uses to signal SR-TE policy and SR Candidate Path to head-end routers for traffic forwarding using source routing?
• How different Underlay transports used in different domains such as IP in DC and SR-MPLS in Access/WAN can interconnect?
• What happens when bandwidth of an existing SR-TE tunnel is exhausted and cannot accommodate new Overlay service requests?
• Which SR-TE path for a given destination should SRIC choose to satisfy the Overlay service’s SLA?
• How an Overlay network service’s SLA being mapped to and satisfied by the Underlay SR-TE path?

SRIC offers the following design and protocols to resolve the above Overlay network services and discovery issues:

• BGP EVPN (RFC 7432) offers highly scalable L2 and L3 Overlay network services such as Virtual Lease Line (VLL), Virtual Private LAN Service (VPLS) and Virtual Private Routed Network (VPRN) through the use of a dedicated control plane protocol, MP-BGP EVPN. MP-BGP EVPN comprises 5 EVPN route types for distributing overlay service topology information such as Ethernet Auto-Discovery, MAC/IP, Inclusive Multicast, segment, and prefix. Please refer to RFC 7432 for more information
• A SRIC can use NetConf / YANG to provision EVPN L2 and L3 services onto the service endpoints upon network service requests from API or routers (e.g., Path Computation Clients) to the SRIC
• Service Function Chaining (SFC) (RFC 7665) allows composing different virtual and physical Service Functions to argument the BGP EVPN services for VNFs and PNFs. The following shows how SFC can offer enhanced Overlay L2 and L3 EVPN network services for subscribers, VNFs and PNFs

The Overlay BGP EVPN and SFC network services need Traffic Engineering (TE) tunnels on the Underlay network to physically transport the traffic in order to meet their SLA or QoS requirements. Therefore, SRIC needs a centralized TE-capable Underlay tunnel signalling protocol.

With the experience learned in designing and operating MPLS-TE label switching networks in the past decade (RFC 5439), a new source routing Underlay Network TE-capable signaling protocol called Segment Routing (RFC 8402) is used by SRIC to signal end-to-end cross domain Underlay TE or SR-TE tunnels. Similar to MPLS LSPs, a SR-TE Underlay Network tunnel or LSP can support multiple Overlay network services such as VLL, VPLS and VPRN etc…..

SRIC offers the following design and protocol to resolve the above Underlay network topology and TE info discovery issues:

• Some access networks such as a Radio Access Network (RAN) may be shared among multiple carriers for cost reason and thus running a single instance of IGP such as OSPF or ISIS for end-to-end cross domain Underlay Network topology discovery is not allowed for security reason. Instead, BGP-LS (RFC 8571) is run on each domain’s gateways and peer with SRIC using eBGP. A gateway running BGP-LS replicates its domain’s IGP Link-State information and converts it into BGP routes. Through eBGP peering with BGP import and export policies between the gateways and SRIC, selective Underlay network information is exchanged to protect the WAN and the shared networks. In this way, SRIC of a carrier can control only the Underlay network resources of the shared networks that are relevant to the carrier to support its Overlay BGP EVPN and SFC network services
• In order for each domain or network to exchange Traffic Engineering (TE) information, new IGPs supporting TE info distribution such as OSPF-TE (RFC 7471) and ISIS-TE (RFC 7810) are used in each domain or network
• To setup an end-to-end Underlay traffic tunnel across multiple domains when each domain is running their own IGP for scalability and security, we need to run BGP-LU (RFC 3107) at each domain’s Gateways and routers to stitch an end-to-end contiguous Underlay Network tunnel or SR-TE tunnel. This is very similar to Seamless MPLS for extending MPLS from WAN to Access networks covered in my previous article
• When deploying Segment Routing (SR) network, the existing IGP protocol needs to be updated to use either SR-ISIS or SR-OSPF so that the IGP protocol can now distribute Segment Identifiers (SIDs) (i.e., topology and router instructions) in addition to the regular Link-State information of the Underlay Network. A SID can represent a network prefix, a node, a node’s adjacent interface or even an instruction for a router

For extending the BGP EVPN network services from Access/WAN networks to Data Centers (DC), there are two popular choices such as:

1. If the gateways, and aggregation and leaf routers in the DC support Segment Routing, BGP-LS and BGP-LU can be used for SRIC to extend the SR-TE tunnel to VMs inside the DC. Since most DCs support only IP routing, SR-MPLS over UDP (RFC 7510 SRoUDP), which supports ECMP can be used to encapsulate the SR-TE tunnel to the VMs in a DC
2. Use the draft RFC draft-ietf-bess-dci-evpn-overlay to create EVPN VPLS at the DC Gateways to bind Access/WAN’s EVPN Instance (EVI) to DC’s VxLAN ID (VNI) and vice versa

4) Scaling New Service SLA with Slicing

Upon a new network service request with SLA to SRIC, SRIC needs to convert the service request to a SR-TE Policy (e.g., Headend, Colour, Endpoint). SR-TE Policy’s Colour is a 32 bit value representing the SLA of the network service request.

SRIC will then compute a Candidate Path comprises a list of Segment IDs (SIDs) representing the path from the Headend to the Endpoint routers and signal the SR-TE Policy and the Candidate path to the Headend router so that it can place the list of SIDs onto each packet to forward the packets to the Endpoint (i.e., source routing).

Segment Routing enables a Headend router to have more granular per flow or per application routing as only the Headend router needs to keep track of the tunnel information to the Endpoint. All intermediate routers do not need to keep any information for this SR-TE tunnel.

The programming of the SR-TE Policy and the Candidate Path to the Headend router can be via:

• Draft RFC — Advertising Segment Routing Policies in BGP — draft-ietf-idr-segment-routing-te-policy-07
• RFC 5440 — Path Computation Element (PCE) Communication Protocol (PCEP)

The Candidate Path for the Overlay packet network service constitutes a flow in the SR-TE Tunnel in the Underlay packet network. Similar to an MPLS LSP, a SR-TE Tunnel can aggregate multiple service flows from the Headend to the Endpoint routers as long as the QoS criteria of the SR-TE Tunnel such as bandwidth and latency can satisfy the SLAs of each of the Overlay service flows. SRIC ensures that the aggregated Overlay service flows on a SR-TE Tunnel does not exceed the total assigned bandwidth of the tunnel. Otherwise, SRIC modifies the existing or creates a new SR-TE tunnel to accommodate the new Overlay service request.

The mapping of an Overlay network service’s SLA to SR-Policy’s Colour, and the Underlay network SR-TE Tunnel selection to satisfy the Colour or SLA of the Overlay network service is vendor implementation specific.

A Network Slice is defined in 3GPP’s 5G specifications as a virtual end-to-end network partition or slice of a carrier’s network that can comprise wireless, wireline, access/core networks, and storage/compute resources in the Edge/Core Clouds etc… The following shows a mobile Network Slice of a carrier network involving 5G wireless radio (e.g, gNB), Access/Core packet networks supporting SR, and compute and storage resources in Edge/Core Clouds for processing the 5G wireless traffic.

In the future, a carrier will be able to sell you a virtual partition of its end-to-end network (i.e., Network Slice) for any duration and the carrier can setup and tear down the network slice in seconds to maximize the usages and revenues of its network and cloud resources.

Similar to any existing network services, a Network Slice can be a hard Network Slice with SLA/QoS guaranteed end-to-end or it can be a cheaper soft Network Slice using non-guaranteed shared network resources.

The most common network devices’ QoS implementation for Underlay packet network in the Access/Core networks are DiffServ QoS defined in RFC 2474. Commonly offered DiffServ QoS constructs for a networking device are Committed Info Rate (CIR), Peak Info Rate (PIR), Forwarding Classes, and Hierarchical Scheduler etc… For an introduction of DiffServ QoS implementation on Nokia’s 7750, please refer to my articles End-to-End Network Service QoS.

Using the above SNF’s Data Plane and Network Slice diagram, let say we need to provision ten VLL Overlay network services between VNF1 and VNF2 with the following SLA criteria:

• Less than 10ms latency
• High Priority traffic
• CIR 100Mbps
• PIR 500Mbps

The following shows the SRIC’s operational sequences:

• SRIC provisions the two routers or VNFs on Access and Core networks with 10 EVPN VLL network services using NetConf / YANG or CLI
• The EVPN VLL configs trigger the MP-EVPN control plane to distribute the new setup to SRIC
• SRIC creates a SR-TE policy (i.e., Headend, Colour, Endpoint) and assigns an appropriate value to the Colour attribute according to the SLA of the VLLs
• SRIC computes a Candidate Path between the Headend and Endpoint and signal the SR-TE Policy and the Candidate Path to the Headend router using protocol such as SR-BGP or PCEP
• The Headend router tries to identify any existing SR-TE tunnels that can support 1Gbps (i.e., 10 x 100Mbps) CIR and 5Gbps (i.e., 10 x 500Mbps) PIR with the matching SR-TE Policy such as the Endpoint and the Colour attribute. If no existing SR-TE tunnel that can satisfy the requirements is found, the Headend router will inform SRIC to signal a new SR-TE tunnel between the Headend router to the Endpoint and updates its TE database accordingly
• When the new or existing SR-TE tunnel is identified, the Headend router pushes the Candidate Path (i.e., a list of SID) computed by SRIC to the Endpoint

Conclusion

This article discusses the Scale the Dynamism and Scaling New Service SLA with Slicing initiatives of the Smart Network Fabric (SNF) defined in Bell Lab’s Future X Network. Implementation examples of the two SNF’s initiatives are shown using Nokia’s Network Function Interconnect (NF-IX). architecture. NF-IX highlights the use of Segment Routing Traffic Engineering (SR-TE) as the underlay transport network for the new cloud-native carrier networks. In fact, All major packet vendors agree that SR-TE is the future cloud-native carrier network’s Underlay network and BGP EVPN and Service Chaining are the Overlay network services.

Part 2 of this article will address some detailed network design issues such as IGP short-cut, multiple IGP instances, BGP Fast Re-route, EVPN service multi-home and BFD timers etc… for network scalability, resilience and redundancy. Stay tuned.

You can follow me on www.medium.com for all my current and future articles on 5G Core, IP/MPLS and software technology.

Munsterbrucke, Zurich, Switzerland [© F2014]

Originally published at http://clcnetwork.wordpress.com on September 23, 2019.