Tracing Flows through Virtual Networks and Service Instances in OpenStack and OpenContrail

This is a guest blog by Johnny Chen from AT&T with contributions by Qasim Arham, Henon Huang, Vijay Kamisetty from Juniper Networks.

Overview

The purpose of this is to lay out a method of tracing flows through OpenStack & Contrail environment with a scaled-out Service Instance (virtual firewall) in the path.

Environment Versions:
OpenStack == Juno (2014.2.4)
Contrail == 3.0.0

 


Table of Contents

Overview

Table of Contents

Background & Drawings

Tracing: Outside => In

Step 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table
Step 2: Verify Traffic to vFW Service Instance VM Compute
Step 3: Trace Flow into vFW Service Instance VM
Step 4: Verify Traffic is leaving vFW towards DST VM
Step 5: Verify Traffic is getting to DST VM TAP Interface
Step 6: On DST VM, Verify Traffic Inbound

Tracing: Inside => Out

Step 1: On DST VM Compute, Trace Return Flow
Step 2: Verify Return Traffic hitting same vFW as Inbound Path

References


Background & Drawings

Tracing Flows through Virtual Networks and Service Instances_background drawings

 

 


Access to some useful elements for this exercise:

RESOURCE
DESCRIPTION
OpenStack Horizon
OpenStack Horizon Web GUI
Juniper Contrail
Contrail Web GUI
SDNGW
SDN Gateway
Compute
Contrail vRouter Commands and tcpdump
Splunk or Juniper JSA/STRM Console
SEIM Log and Correlation (Optional)

Caveat: There is often more than one way to get to the end result so while this is an attempt to document one method, there could be other ways of achieving to a diagnosis. For this example, we will use a lab network depicted in the drawing below as a reference and trace the packet flow between 12.12.0.102 (SRC) and 10.10.0.100 (DST) where the destination is a VM in the Overlay Network.

The flow path is as follows:
12.12.0.102 <> Physical Network <> SDN-GW <> VN_A <> FW Service Instance <> VN_B <> 10.10.0.100

 


Tracing: Outside => In


STEP 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table.

1a. Single VM or Compute (Single Next-Hop)

In a single VM architecture for FW, you would see a single next-hop in the table for that routing-instance. However, in the case of a scaled-out service instance, there might be multiple next-hops pointing to the computes where the FW VM’s instantiated on. When dealing with Composite Next-Hop, identifying which FW on which Compute can get tricky (either log into all the FW’s and look through the session tables for the flow in question or use a SEIM or Log Correlator like Splunk or STRM/JSA). Here, we will focus on Single VM or Compute (Single Next-Hop).

 

On SDNGW:

SDNGW> show route 10.10.0.100

vntest.inet.0: 50 destinations, 150 routes (50 active, 0 holddown, 0 hidden)
 @ = Routing Use Only, # = Forwarding Use Only
 + = Active Route, - = Last Active, * = Both

10.10.0.0/24 *[BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.5
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 [BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31

SDNGW> show interfaces gr-0/1/0.19283
 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526)
 Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL
 Gre keepalives configured: Off, Gre keepalives adjacency state: down
 Input packets : 4332961
 Output packets: 7699862
 Protocol inet, MTU: 9062
 Flags: None
 Protocol mpls, MTU: 9050, Maximum labels: 3
 Flags: None

Useful Information from Output:
172.20.0.23: NEXT-HOP Compute
31: Label of NEXT-HOP on Compute

What we find is the GRE interface (Label 31) for the NEXT-HOP points to the COMPUTE at 172.20.0.23. You can find which COMPUTE this is via the Contrail WebGUI (Monitor => Virtual Routers => [Search Field]172.20.0.23 or search for the VHOST0 IP’s of your computes. In this case, the COMPUTE “Hostname” matches with the NEXT-HOP IP 172.20.0.23 matches with COMPUTE cmpt001. cmpt001 hosts the VM of the vFW Service Instance that is the next element in the path to the DST VM. From here, go to Step 2.

1b. Multiple VM / Compute (Composite Next-Hop)

Here, we will focus on Multiple VM’s or Computes (Composite Next-Hop).

On SDNGW:

SDNGW> show route 10.10.0.100

vntest.inet.0: 70 destinations, 150 routes (70 active, 0 holddown, 0 hidden)
 @ = Routing Use Only, # = Forwarding Use Only
 + = Active Route, - = Last Active, * = Both

10.10.0.0/24 @[BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.4
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.74659, Push 62
 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.5
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 [BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.74659, Push 62
 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 #[Multipath/255] 1w1d 21:13:23, metric 100, metric2 0
 via gr-0/1/0.74659, Push 62
 > via gr-0/1/0.19283, Push 31

SDNGW> show interfaces gr-0/1/0.19283
 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526)
 Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL
 Gre keepalives configured: Off, Gre keepalives adjacency state: down
 Input packets : 4332961
 Output packets: 7699862
 Protocol inet, MTU: 9062
 Flags: None
 Protocol mpls, MTU: 9050, Maximum labels: 3
 Flags: None

If your environment utilizes logging correlation software like the Splunk or Juniper JSA/STRM, that tool would be useful to figure out which “next-hop” scaled-out service instance vFW. Otherwise, we will have to log into each vFW and look at the flow or session table entries to figure out which compute is next in path.

Based on the above information, we can see the traffic transit a particular vFW in the scaled-out Service Instance and we can go directly to Compute where that vFW VM resides. If you know which vFW is taking the traffic (we will assume VFW001 in this case), you can look up with COMPUTE that VM is instantiated on and go from there.

Figure out which COMPUTE the ServiceInstance vFW is on:

toolsvm:~$ nova list | grep -i vfw
 | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q | VFW001 | ACTIVE | - | Running | ADMIN=192.168.0.194; VN_A=11.11.0.64; LOGNET=192.168.11.65; VN_B=10.10.0.64 |
 | z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j | VFW002 | ACTIVE | - | Running | ADMIN=192.168.0.196; VN_A=11.11.0.66; LOGNET=192.168.11.67; VN_B=10.10.0.65 |

toolsvm:~$ nova show a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q
 +------------------------------------------+------------------------------------------------------------+
 | Property | Value |
 +------------------------------------------+------------------------------------------------------------+
 | VN_A network | 11.11.0.64 |
 | LOGNET network | 192.168.11.65 |
 | ADMIN network | 192.168.0.194 |
 | VN_B network | 10.10.0.64 |
 | OS-DCF:diskConfig | MANUAL |
 | OS-EXT-AZ:availability_zone | default |
 | OS-EXT-SRV-ATTR:host | cmpt001 |
 | OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt001.test.net |
 | OS-EXT-SRV-ATTR:instance_name | instance-00000eef |
 | OS-EXT-STS:power_state | 1 |
 | OS-EXT-STS:task_state | - |
 | OS-EXT-STS:vm_state | active |
 | OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 |
 | OS-SRV-USG:terminated_at | - |
 | accessIPv4 | |
 | accessIPv6 | |
 | config_drive | |
 | created | 2016-06-01T00:35:36Z |
 | flavor | flv_vsrx (12345678-012a-3bcd-e4f5-6789g01h2345) |
 | hostId | asdflkijhbalk1h23k132427896r98a7asfhio1u241234iuhewnoafs |
 | id | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q |
 | image | vsrx15.1x49d30.3 (6baishdu-ewiu-9238-sibh-02394isfoayx) |
 | key_name | - |
 | name | VFW001 |
 | os-extended-volumes:volumes_attached | [] |
 | progress | 0 |
 | security_groups | default |
 | status | ACTIVE |
 | tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy |
 | updated | 2016-06-01T00:35:36Z |
 | user_id | d8923923hisuadfhf893923h2hfdasfh |
 +------------------------------------------+------------------------------------------------------------+

toolsvm:~$ for x in $(nova list | grep -i foam | awk {'print $4'}); do nova show $x | sed -n '10p;24p;28p' | awk {'print $4'} | xargs | sed 's/ / || /g'; done
 cmpt001 || a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q || VFW001
 cmpt003 || z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j || VFW002

We see that VFW001 with ID a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q is on COMPUTE cmpt001.

TOP


STEP 2: Verify Traffic is making it into vFW ServiceInstance VM Compute

27440195245_4ffd43cd65_b

 

2a. Go to COMPUTE

toolsvm ~]$ ssh user@cmpt001
 password for [user]:
 Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-61-generic x86_64)

2b. Find COMPUTE interfaces

cmpt001:~# cat /etc/contrail/contrail-vrouter-agent.conf | grep -A13 -i virtual-host-interface
 [VIRTUAL-HOST-INTERFACE]
 # Everything in this section is mandatory

# name of virtual host interface
 name=vhost0

# IP address and prefix in ip/prefix_len format
 ip=172.20.0.23/32

# Gateway IP address for virtual host
 gateway=172.20.0.1

# Physical interface name to which virtual host interface maps to
 physical_interface=p1p1

cmpt001:~# ifconfig vhost0
 vhost0 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
 inet addr:172.20.0.23 Bcast:172.20.0.31 Mask:255.255.255.240
 UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
 RX packets:84487487 errors:0 dropped:182627 overruns:0 frame:0
 TX packets:82063519 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:253984497954 (253.9 GB) TX bytes:67502412941 (67.5 GB)

cmpt001:~# ifconfig p1p1
 p1p1 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
 UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
 RX packets:194126327 errors:0 dropped:0 overruns:0 frame:0
 TX packets:125130748 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:286638778868 (286.6 GB) TX bytes:94956347917 (94.9 GB)
 Interrupt:40 Memory:f3000000-f37fffff

2c. Verify vRouter is exchanging XMPP information with the CONTRAIL CONTROLLER(s)

cmpt001:~# tcpdump -D | grep -i vhost0
 1.vhost0

cmpt001:~# tcpdump -nei 1 port xmpp-server
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on vhost0, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:04:23.420090 c9:36:dd:24:po:p0 > b0:ob:ab:ba:0a:a0, ethertype IPv4 (0x0800), length 66: 172.20.0.5.5269 > 172.20.0.23.58699: Flags [.], ack 3731677826, win 8748, options [nop,nop,TS val 1268013680 ecr 1267113680], length 0
 12:04:23.420120 b0:ob:ab:ba:0a:a0 > c9:36:dd:24:po:p0, ethertype IPv4 (0x0800), length 66: 172.20.0.23.58699 > 172.20.0.5.5269: Flags [.], ack 1, win 6792, options [nop,nop,TS val 1267114931 ecr 1268012429], length 0
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

2d. Verify ICMP packets from example (12.12.0.102 => 10.10.0.100) is making it into the COMPUTE cmpt001

cmpt001:~# tcpdump -D | grep p1p1
 7.p1p1

cmpt001:~# tcpdump -nei 7 proto 47 | grep 11.11.0.60
 tcpdump: WARNING: bond0.2004: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on bond0.2004, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:07:02.093007 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20030, length 64
 12:07:03.094702 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20031, length 64
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel
We see that ICMP is making it into the Compute in the above TCPDUMP output. (TCPDUMP can process PROTO 47 for MPLSoGRE but if the tunnels are MPLSoUDP, you’ll have to decode this in another packet analyzer like WireShark).

TOP


STEP 3: Trace the flow into vFW Service Instance VM

3a. Find NEXT-HOP (mpls –dump will output the entire Label => NextHop table) – we found the MPLS Label 31 from the Next-Hop from the SDNGW

cmpt001:~# mpls --get 31
 MPLS Input Label Map

Label NextHop
 -------------------
 31 49

3b. Find NEXT-HOP Interface and VRF that the FW VM interface that is created on for this flow on VN_A.

cmpt001:~# nh --get 49
 Id:49 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:8 Vrf:11
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:24 Len:14
 Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 24
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/24 OS: tap93d2dja8-22
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:11 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:728634 bytes:71113479 errors:0
 TX packets:756736 bytes:89113687 errors:0

3c. [IGNORE FOR NOW] On Vrf:11 (VN_A), we see the routing-table point to the FW’s “RIGHT” interface as the NEXT-HOP to 10.10.0.0/24 Network (VN_B)

cmpt001:~# rt --dump 11 | grep 10.10.0
 Vrouter inet4 routing table 0/13/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 10.10.0.0/24 24 P - 164 -

cmpt001:~# nh --get 164
 Id:164 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:11
 Flags:Valid, Policy, Ecmp,
 Sub NH(label): 95(31) -1 14(62)

root@cmpt001:~# nh --get 95
 Id:95 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:25 Vrf:11
 Flags:Valid,
 EncapFmly:0806 Oif:24 Len:14
 Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 24
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/24 OS: tap93d2dja8-22
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:11 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:733959 bytes:71635018 errors:0
 TX packets:762144 bytes:89646655 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

(Interfaces are enumerated in the order MGMT, LEFT, RIGHT, OTHER… so the TAP interface above it the “RIGHT” interface in VN_A)

3d. Check for “Discards” on Vrf:11

cmpt001:~# watch vrfstats --get 11

Every 2.0s: vrfstats --get 11 Fri Jun 03 19:13:13 2016

Vrf: 11
 Discards 0, Resolves 0, Receives 0, L2 Receives 7783, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 26818, Fabric Composites 24593, Encap Composites 24593, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 4000360, Gre Mpls Tunnels 7780, Vxlan Tunnels 0
 L2 Encaps 3321946, Encaps 32480
 GROs 2662900, Diags 0
 Arp Virtual Proxys 654, Arp Virtual Stitchs 1539, Arp Virtual Floods 600, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Flo
 ods 0

Based on the above output, it does not appear that there are any discards on Vrf:11 that would be causing the connectivity failure (Counter at “0” and not incrementing).

3e. Look for “Discards” via dropstats and check relevant counters (“Flow Action Drop” & “Discards” in this case) to see if they increment

cmpt001:~# watch dropstats

Every 2.0s: dropstats Fri Jun 03 19:16:59 2016

GARP 0
 ARP no where to go 0
 Invalid ARPs 0

Invalid IF 0
 Trap No IF 0
 IF TX Discard 0
 IF Drop 0
 IF RX Discard 0

Flow Unusable 0
 Flow No Memory 0
 Flow Table Full 0
 Flow NAT no rflow 0
 Flow Action Drop 12040
 Flow Action Invalid 0
 Flow Invalid Protocol 0
 Flow Queue Limit Exceeded 132

Discards 8372
 TTL Exceeded 0
 Mcast Clone Fail 0
 Cloned Original 348762341

Invalid NH 20637113
 Invalid Label 0
 Invalid Protocol 0
 Rewrite Fail 0
 Invalid Mcast Source 0

Push Fails 0
 Pull Fails 0
 Duplicated 0
 Head Alloc Fails 0
 Head Space Reserve Fails 0
 PCOW fails 0
 Invalid Packets 0

Misc 764
 Nowhere to go 0
 Checksum errors 0
 No Fmd 0
 Invalid VNID 0
 Fragment errors 0
 Invalid Source 12
 Jumbo Mcast Pkt with DF Bit 0
 ARP No Route 0
 ARP Reply No Route 0
 No L2 Route 136423

“Discards” and “Flow Action Drop” counters stayed steady at “12040” and “8372” respectively and did not increment.

3f. Find which VM is associated with TAP Interface tap93d2dja8-22 (there is more than one way to get this information)

CLI Method:

– From previous discovery, we know that the ID of the VM VFW001 on cmpt001 is a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q. Map the TAP Interface to the VM:

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

Contrail GUI Method: Find the Compute where the VM resides (cmpt001) via Monitor => Virtual Routers = cmpt001 // Interfaces. From there, do a search for the interface tap93d2dja8-22. From there, you should be able to find which VM the TAP Interface is mapped to.

3g. From the above information, we know that this is mapped to the vFW”RIGHT” interface which is on the VN_A Network. We can do a TCPDUMP to verify if the traffic is making it to the FW’s Ingress/VN_A interface:

cmpt001:~# tcpdump -nei tap93d2dja8-22
 tcpdump: WARNING: tap93d2dja8-22: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap93d2dja8-22, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:09:41.995714 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10575, length 40
 12:09:42.995769 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10593, length 40
^C
2 packets captured
4 packets received by filter
 0 packets dropped by kernel

3h. From the above, we see ICMP echo requests going into the FW, but no replies. Let’s check for what the vFW sees:

toolsvm:~$ ssh 192.168.0.194

Password:
 --- JUNOS 15.1X49-D30.3 built 2015-12-17 04:39:24 UTC
 VFW001>

VFW001> show security flow session destination-prefix 10.10.0.100
 Session ID: 206346, Policy name: allow_restricted/6, Timeout: 2, Valid
 In: 12.12.0.102/34054 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34054;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,

Session ID: 206347, Policy name: allow_restricted/6, Timeout: 4, Valid
 In: 12.12.0.102/34058 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34058;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,

Session ID: 206350, Policy name: allow_restricted/6, Timeout: 2, Valid
 In: 12.12.0.102/34057 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34057;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,
 Total sessions: 3

The vFW sees packets inbound through it to the VM 10.10.0.100 but no return traffic (which matches what we see on the TCPDUMP on the FW’s GE-0/0/1 or “RIGHT” interface). At this point, we need to look at the traffic leaving the FW on the “LEFT”/VN_B interface.

3i. Check the COMPUTE vRouter Flow Table to see if the traffic is being forwarded.

cmpt001:~# flow --match 10.10.0.100
 Flow table(size 68157440, entries 542386)

Entries: Created 898146 Added 898144 Processed 898146 Used Overflow entries 0
 (Created Flows/CPU: 368496 140472 62462 39956 28374 22527 25634 19742 20169 15999 6767 6693 6514 6322 7041 6249 6200 6272 6324 6235 7985 9035 7329 9146 6741 6608 5981 7374 10025 8246 663 941 721 734 700 671 771 837 1120 4070)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([10.10.0.100]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------

168624<=>23140 12.12.0.102:1393 1 (2->3)
 10.10.0.100:0
 (Gen: 5, K(nh):33, Action:F, Flags:, S(nh):94, Stats:257760/25260480, SPort 53824)
 --
 430164<=>119628 12.12.0.102:1393 1 (13->5)
 10.10.0.100:0
 (Gen: 11, K(nh):49, Action:F, Flags:, S(nh):88, Stats:257760/21651840, SPort 60610)
 --

TOP


STEP 4: Verify traffic is leaving vFW Service Instance VM towards 10.10.0.100

28163632680_7f52eebbe3_b

 

4a. Verify next TAP Interface for FW (“LEFT” / VN_B) sees traffic exit the vFW destined for the VM at 10.10.0.100

cmpt001:~# nh --get 33
 Id:93 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:12 Len:14
 Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 12
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/12 OS: tap13a2d42c-5d
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:12 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:1236404 bytes:135113415 errors:0
 TX packets:1231952 bytes:120251878 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

cmpt001:~# tcpdump -nei tap13a2d42c-5d
 tcpdump: WARNING: tap13a2d42c-5d: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap13a2d42c-5d, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:24:31.175890 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11410, length 40
 12:24:32.176227 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11413, length 40
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

4b. Traffic (ICMP echo requests) is leaving vFW egress interface (GE-0/0/0) “LEFT” or VN_B interface destined towards the VM 10.10.0.100. Check to see if there are any discards on the VRF.

cmpt001:~# watch vrfstats --get 12

Every 2.0s: vrfstats --get 12 Fri Jun 03 19:13:13 2016

Vrf: 12
 Discards 0, Resolves 0, Receives 0, L2 Receives 12369, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 18742, Fabric Composites 12190, Encap Composites 12168, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 17, Gre Mpls Tunnels 4010, Vxlan Tunnels 0
 L2 Encaps 71982, Encaps 1302371
 GROs 2687, Diags 0
 Arp Virtual Proxys 8658, Arp Virtual Stitchs 4858, Arp Virtual Floods 3748, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 30

No discards.

4c. Verify routing on Vrf:10 (VN_B) FIB table for 10.10.0.100

cmpt001:~# rt --dump 12 | grep 10.10.0.100
 Vrouter inet4 routing table 0/2/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 10.10.0.100/32 32 LP 31 88 2:38:c8:ea:9a:21(211664)

4d. Find the NEXT-HOP COMPUTE to get to the VM on VN_B at 10.10.0.100

cmpt001:~# nh --get 88
 Id:88 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:144 Vrf:0
 Flags:Valid, MPLSoUDP,
 Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 74 c0 8c dc d4 10 75 a0 08 00
 Vrf:0 Sip:172.20.0.23 Dip:172.20.0.24

Next-Hop 88 points to a destination IP of 172.20.0.24 (vhost0 Interface IP of the destination Compute on Default Vrf0) with MPLS Label 31

TOP


STEP 5: Verify traffic is getting to Destination VM at 10.10.0.100 on TAP Interface

5a. Find NEXT-HOP COMPUTE at 172.29.2.82

CLI Method (From toolsvm – use the IP of the Destination VM to find Compute):

toolsvm:~$ for x in $(nova list | grep -i 10.10.0.100 | awk {'print $4'}); do nova show $x | sed -n '7p;21p;25p' | awk {'print $4'}; done
 cmpt002
 98asdfhjkn213-sdai-ncxv-3421987kjlsm
 testvm-VN_B_1

toolsvm:~$ nova show 98asdfhjkn213-sdai-ncxv-3421987kjlsm
 +------------------------------------------+-------------------------------------------------------------+
 | Property | Value |
 +------------------------------------------+-------------------------------------------------------------+
 | MNS_shared_OAM_PROTECTED_NET_1_1 network | 10.10.0.100 |
 | OS-DCF:diskConfig | AUTO |
 | OS-EXT-AZ:availability_zone | nova |
 | OS-EXT-SRV-ATTR:host | cmpt002 |
 | OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt002.test.net |
 | OS-EXT-SRV-ATTR:instance_name | instance-00000f28 |
 | OS-EXT-STS:power_state | 1 |
 | OS-EXT-STS:task_state | - |
 | OS-EXT-STS:vm_state | active |
 | OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 |
 | OS-SRV-USG:terminated_at | - |
 | accessIPv4 | |
 | accessIPv6 | |
 | config_drive | |
 | created | 2016-06-01T00:35:36Z |
 | flavor | m1.tiny (2) |
 | hostId | a0s9dfuwqehrlkqwerl89ydfkj1394874329y2nlkadfasygas98ydfs |
 | id | 98asdfhjkn213-sdai-ncxv-3421987kjlsm |
 | image | ubuntu-14 (1f062a45-4a90-437d-a2b8-b2b5a565d95a) |
 | key_name | - |
 | metadata | {} |
 | name | testvm-VN_B_1 |
 | os-extended-volumes:volumes_attached | [] |
 | progress | 0 |
 | security_groups | default |
 | status | ACTIVE |
 | tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy |
 | updated | 2016-06-01T00:35:36Z |
 | user_id | d8923923hisuadfhf893923h2hfdasfh |
 +------------------------------------------+-------------------------------------------------------------+

GUI Method (From Contrail GUI)

Monitor => Infrastructure => Virtual Routers => Search

5b. Go to COMPUTE cmpt002. Based on Label 31 for NEXT-HOP on this COMPUTE, we can find where the traffic is destined to.

cmpt002:~# mpls --get 31
 MPLS Input Label Map

Label NextHop
 -------------------
 31 73
 root@cmpt002:~# nh --get 73
 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:14 Len:14
 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00

cmpt002:~# vif --get 14
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/14 OS: tap83f2ki4w-72
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:13 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:2272968 bytes:214007093 errors:0
 TX packets:2294863 bytes:287437540 errors:0

5c. Verify the above output matches the VM libvert.xml

cmpt002:~# cat /var/lib/nova/instances/98asdfhjkn213-sdai-ncxv-3421987kjlsm/libvirt.xml | grep -i tap83f2ki4w-72
 <target dev="tap83f2ki4w-72"/>

5d. Check for Vrf:13 discards

cmpt002:~# watch vrfstats --get 13

Every 2.0s: vrfstats --get 13 Fri Jun 03 19:33:33 2016

Vrf: 13
 Discards 0, Resolves 0, Receives 0, L2 Receives 3299, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 3364, Fabric Composites 330, Encap Composites 739, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 4, Gre Mpls Tunnels 0, Vxlan Tunnels 0
 L2 Encaps 3338, Encaps 2421364
 GROs 40513, Diags 0
 Arp Virtual Proxys 2906, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

5e. Check vRouter Flow Table for “Action:F”

cmpt002:~# flow --match 10.10.0.100
 Flow table(size 68157440, entries 532480)

Entries: Created 2101736 Added 2100485 Processed 2101736 Used Overflow entries 0
 (Created Flows/CPU: 367906 175642 111664 82241 69167 55275 51234 46499 43885 43268 16593 19378 19266 18389 18108 17846 17739 18159 17939 18510 31225 119065 99542 54060 37515 32004 30342 31058 41341 28021 7123 86430 156010 50283 20184 11714 9180 8086 7844 12001)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([10.10.0.100]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------
 14325<=>438965 12.12.0.102:61 1 (6->5)
 10.10.0.100:0
 (Gen: 10, K(nh):73, Action:F, Flags:, S(nh):22, Stats:1166/69960, SPort 61800)
 --

5f. Verify Traffic is being forwarded on TAP Interface on the VM

cmpt002:~# tcpdump -nei tap83f2ki4w-72
 tcpdump: WARNING: tap83f2ki4w-72: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap83f2ki4w-72, link-type EN10MB (Ethernet), capture size 65535 bytes
 13:02:32.939608 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59313, length 40
 13:02:33.935583 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59317, length 40
 13:02:34.952637 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59321, length 40
 13:02:35.960168 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59322, length 40
 ^C
 4 packets captured
 6 packets received by filter
 0 packets dropped by kernel

The traffic flow packets are making it all the to the TAP Interface on the vRouter of the Compute that is assigned to the VM 10.10.0.100. We need to get the DST VM owner to verify that they can see the traffic and at this point it could be any number of problems:
– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,

TOP


STEP 6: On VM, run TCPDUMP to inspect inbound traffic

6a. Access VM (this example will utilize Link-Local Address from Compute)
Find Link-Local Address of TAP interface of VM via Contrail Web GUI

On Compute, SSH to Link-Local Address 169.254.0.14

cmpt002:/etc# ssh 169.254.0.14
 169.254.0.14's password:
 Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)

testvm-vn_b:~$

6b. Verify Traffic is making it to the VM

testvm-vn_b:~$ ifconfig
 eth0 Link encap:Ethernet HWaddr 7b:23:a9:tu:c3:p0
 inet addr:10.10.0.100 Bcast:172.20.50.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB)

eth1 Link encap:Ethernet HWaddr 02:8a:db:64:fb:45
 inet addr:192.168.0.58 Bcast:192.168.0.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB)

lo Link encap:Local Loopback
 inet addr:127.0.0.1 Mask:255.0.0.0
 UP LOOPBACK RUNNING MTU:65536 Metric:1
 RX packets:34 errors:0 dropped:0 overruns:0 frame:0
 TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:7130 (7.1 KB) TX bytes:7130 (7.1 KB)

testvm-vn_b:~# tcpdump -D
 1.eth0
 2.eth1
 3.any (Pseudo-device that captures on all interfaces)
 4.lo

testvm-vn_b:~$ tcpdump -nei 1 proto 1
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
 13:16:15.056859 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 65, seq 109, length 40
 13:16:15.056963 7b:23:a9:tu:c3:p0 > bk:2a:n6:00:02:00, ethertype IPv4 (0x0800), length 74: 10.10.0.100 > 12.12.0.102: ICMP echo reply, id 65, seq 109, length 40
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

testvm-vn_b:~$
At this point, we can safely say that the traffic flow packets are making it all the to the TAP Interface on the vRouter of the Compute that is assigned to the VM 10.10.0.100. There are any number of reasons why traffic may fail to/from a VM:

– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,

TOP


Tracing: Inside => Out


STEP 1: On Compute where VM is being hosted on, Trace flow back out to Source Host

29831101064_328e78b9ee_b
>>*Assume traffic is fixed on the DST VM and now we are tracing the return flow back to the SRC IP

1a. Get vRouter Flow Table information for return traffic from 10.10.0.100 to 12.12.0.102

cmpt002:~# flow --match 12.12.0.102
 Flow table(size 68157440, entries 532480)

Entries: Created 2105344 Added 2104093 Processed 2105344 Used Overflow entries 0
 (Created Flows/CPU: 368319 176011 111801 82325 69226 55341 51282 46532 43914 43306 16597 19384 19280 18394 18114 17851 17745 18169 17948 18521 31269 119556 99662 54124 37564 32051 30401 31112 41397 28061 7134 86678 156797 50372 20220 11723 9190 8099 7854 12020)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([12.12.0.102]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------
 463620<=>89580 10.10.0.100:65 1 (6->5)
 12.12.0.102:0
 (Gen: 14, K(nh):73, Action:F, Flags:, E:0, S(nh):73, Stats:3/222, SPort 51745)
 --
E:0 == ECMP Index 0 (for ECMP Multi-Path, Next-Hop starts and increments from 0)

1b. Find Return Next-Hop Information

cmpt002:~# nh --get 73
 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:14 Len:14
 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00

cmpt002:/etc# rt --dump 13 | grep 172.20.212
 Vrouter inet4 routing table 0/6/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 172.20.212.0/24 0 P - 102 -

cmpt002:~# nh --get 102
 Id:102 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:6601 Vrf:13
 Flags:Valid, Policy, Ecmp,
 Sub NH(label): 22(27) 22(31)
Here we see 2 Next-Hops and since we saw E:0 in the vRouter Flow Table, it means we will take the first Next-Hop(Label) of 22(27). Remember this MPLS Label 22 as we will need this to get the VHOST0 IP of the Compute where the vFW resides on.
cmpt002:/etc# nh --get 22
 Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:100 Vrf:0
 Flags:Valid, MPLSoUDP,
 Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 75 a0 8c dc d4 10 74 c0 08 00
 Vrf:0 Sip:172.20.0.24 Dip:172.20.0.23
172.20.0.23 (cmpt001) is the Compute where the vFW VM resides.

TOP


STEP 2: On Compute where Next-Hop vFW VM is being hosted on, verify return flow is hitting same egress vFW as ingress

2a. Trace path of return flow back (MPLS Label 27) to vFWand verify it is the same vFW that was utilized for the inbound traffic.

The COMPUTE with Vhost0 IP of 172.20.0.23 == cmpt001 and VFW001 has a ID of a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q from previous discovery.

cmpt001:~# mpls --get 27
 MPLS Input Label Map

Label NextHop
 -------------------
 27 65
 root@cmpt001:~# nh --get 65
 Id:65 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:29 Len:14
 Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 29
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/29 OS: tap13a2d42c-5d
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:12 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:1311527 bytes:145819268 errors:0
 TX packets:1305396 bytes:125789289 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>
This is the same vFWas was used for the Ingress flow so this symmetric flow is now verified. If traffic was SOURCED from VM out, then you would follow the same procedure outbound the rest of the way as we used to trace the flow inbound into the VM 10.10.0.100.

Continue the same process to trace the flows outbound from the vFW to the VM.

TOP


References

RESOURCE
DESCRIPTION
Juniper Contrail – Life of Packet Life of packet Blog
Juniper vRouter Commands Juniper GitHub for vRouter CLI Commands
Juniper vRouter Utilities
Juniper TechPubs vRouter CLI Utilities
OpenStack CLI Reference
OpenStack CLI Reference

TOP