A Tale of Two Layers – Correlating Overlay and Physical Network Data for better OpenStack Network Analytics

For an overview of this blogpost, check the below video. Also you can find similar videos in the Video section of this site.

Openstack networking can help us deliver the benefits of network virtualization to applications – lower lead times for deploying services, better resilience to failures and easier scaling. But virtualization presents a new paradigm for visibilty and debugging of network elements, physical and virtual.  Applications live their lives in the overlay network. Sometimes they face connectivity, latency and other performance problems. Is it a network problem? How do we isolate it and solve it?

To do this, we really need to map what’s happening in the overlay network with the networking entities that are operating in the physical network. In this presentation, we will demonstrate an Analytics application using the OpenContrail system’s open API’s to look at a traffic flow in the overlay, and examine what physical routers and network interfaces it is traversing in the underlay. This can be co-related to interface statistics and other health indications in the underlay network.  We can also look at the physical routers being traversed by tunnels that go across from one underlay entry point to another. Correlating physical and virtual network information is these ways is a big step towards identifying, solving and even anticipating the connectivity problems of applications in OpenStack.

Collecting Information from Physical Nodes

Discovered topology information and interface statistics are ingested via SNMP.  We rely on lldpTable, ipMib, ifTable and ifXTable.  Each physical router is represented by the pRouter UVE (see Operational State in the OpenContrail system: UVE – User Visible Entities through Analytics API), which exposes these MIBs. We also correlate this information against vRouter UVEs to present physical topology in the pRouter UVE – a list of all other pRouters and vRouters that are connected this pRouter, and the links between them.

overlay_to_physical_blogpost_image1

OpenContrail Analytics’ collector can take in flow information from physical nodes in the form of sFlow and IPFIX. The collected records can then be accessed via the Query REST API (see OpenContrail Analytics Query API) , with the following schema:

http://127.0.0.1:8081/analytics/table/StatTable.UFlowData.flow/schema

 Physical Topology

Using the information listed above, the OpenContrail UI presents an interactive view of the Phyisical Topology of the network. Along with the switches/routers, we display details about the links between them, including traffic statistics along that link.

overlay_to_physical_blogpost_image2

For TORs, we show the vRouters hanging off them, and the VM instances that run on those vRouters. The interfaces that connect them and their traffic statistics can be shown as well.

overlay_to_physical_blogpost_image3

Search Flows

We can list historical overlay flows records seen on the system, according to the flow parameters used for REST API Flow Record queries, i.e. Source-VN/Source-IP, Dest-VN/Dest-IP, vRouter, Protocol/Source-Port and Protocol/Dest-Port.

Then we can map these flows onto the Physical Nodes to see what path they took. The underlay flow parameters used for a given overlay flow depend on the type of encapsulation being used. MPLS-over-UDP and VXLAN encapsulations try to add entropy (for better load-balancing between paths) by varying the UDP source-port based on overlay flow parameters.  Multiple overlay flows are expected to hash onto the same underlay flow.  When looking at sFlow or IPFIX information of underlay flows to infer the path of a given overlay flow, we can exploit all samples of the underlay flow, even if they were actually due to other overlay flows.

overlay_to_physical_blogpost_image4

Trace Path

The vRouterAgent supports an active probe-based Trace Path mechanism via its HTTP Introspect API.

http://127.0.0.1:8085/diag.xml#Snh_TraceRouteReq

The UI displays all recently active overlay flows for a given vRouter or Virtual Machine. We can then trace the corresponding underlay flow using active probes that have the same outer header as regular packets of this underlay flow.  The active probes are sent out using escalating TTLs, to elicit ICMP packets from the physical routers that the underlay flow would traverse. The destination vRouter detects this as a special probe packet based on the inner header, and replies back to the originating vRouter, thus completing the illumination of the path.

overlay_to_physical_blogpost_image5

Conclusion

OpenContrail Analytics exposes detailed information about overlay flows in the network, and operational information about vRouters. We have also seen the physical network information that can be ingested into OpenContrail Analytics and presented northbound. Using all these APIs, an application can be built that displays the physical topology, and shows the physical path taken by an overlay flow. We have built one such application – other applications can be built on top of these APIs as well.