OpenStack Neutron, OpenVSwitch, and Jumbo frames

MTU has always been a touchy subject in Neutron. Who manages it? Should instances have info about the underlying infrastructure? Should this all be on the operator to configure properly? Luckily these questions appear to have been answered for the most part in the Mitaka release of OpenStack. This bug for OpenVSwitch (and this one for linuxbridge) more or less solve this issue for us.

Now we can configure MTU in Neutron and Neutron will be intelligent about how to use it. My end goal is to have an infrastructure with 9000 mtu, while the instances themselves can live with the standard 1500 mtu. To achieve that I did have to pull in one patch early, though it looks like it will make it into Neutrons mitaka-rc2 release. The patch applies global_physnet_mtu value to br-int and br-tun so the operator doesn’t have to. Beyond that, it was all just a matter of Neutron config options, which is fantastic!

Here are the changes I had to make to properly get Neutron using my larger 9000 MTU without my intervention.

# /etc/neutron/neutron.conf
[DEFAULT]
global_physnet_mtu = 9000

# /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
physical_network_mtus = physnet1:9000
# The default value for path_mtu is 1500, if you want your instances to have
# larger mtus you should adjust this to <= global_physnet_mtu
path_mtu = 1500

That was it! My interfaces we’re properly configured.

160: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether a0:36:9f:67:32:c6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a236:9fff:fe67:32c6/64 scope link tentative 
       valid_lft forever preferred_lft forever
161: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether 66:fd:95:70:37:4a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::64fd:95ff:fe70:374a/64 scope link tentative 
       valid_lft forever preferred_lft forever
162: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether 76:37:eb:d7:3b:48 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7437:ebff:fed7:3b48/64 scope link tentative 
       valid_lft forever preferred_lft forever

A shoutout to Sean Collins (sc68cal) for fixing the br-int/br-tun issue and Armando Migliaccio (armax) for targeting the bug for inclusion in mitaka-rc2 if all goes well!

3 thoughts on “OpenStack Neutron, OpenVSwitch, and Jumbo frames

  1. I have installed mitaka release and was hitting this error where virbr0 mtu is still 1500 bytes.I think its got to do with the missing patch that you mentioned.I was looking for help in applying that patch

    i executed following into a temp directory
    git fetch git://git.openstack.org/openstack/neutron refs/changes/17/295917/2 && git checkout FETCH_HEAD

    I see bunch of files related to neutron but i am not sure how to apply this patch or whats the next step after this.I also see devstack folder but i havent used devstack for installation.

    Any help is appreciated

  2. i updated the ovs_lib.py in /usr/lib/python2.7/dist-packages/neutron/agent/common dir file with ovs_lib.py i got after downloading the patch.

    Then I have set global_physnet_mtu as 9000 in neutron.conf and path_mtu as 9000 and physical_network_mtus as 9000 under [ml2] in ml2_conf.ini file

    I also had to recreate my virtual networks and instances for the mtu to take into effect in instances.But i am still unable to ping larger sized packets from vm to vm.

    In compute nodes i see that virbr0 is still 1500 and i am not sure why
    tap8eca5f1d-bb Link encap:Ethernet HWaddr fe:16:3e:d5:46:35
    inet6 addr: fe80::fc16:3eff:fed5:4635/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:8950 Metric:1
    RX packets:45 errors:0 dropped:0 overruns:0 frame:0
    TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:3694 (3.6 KB) TX bytes:4967 (4.9 KB)

    virbr0 Link encap:Ethernet HWaddr 52:54:00:d9:25:2d
    inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
    UP BROADCAST MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

    Also for some some strange reason i am unable to ssh to my instances using the floating ip after i updated mtu values.I am able to ping the floating ip though?The instances mgmt interfaces have come up with 8950 as mtu

    • Unable to ssh but able to ping is a classic MTU mismatch issue given the subject matter at hand.

      The 8950 MTU is normal for vxlan/encapsulated networks to account for the overhead of the protocol. Only vlan and “flat” networks would receive 9000 MTU.

      “virbr0” is not used by OpenStack at all and can be completely ignored here.

      If the instance is getting 8950 set, you will likely want to look elsewhere for issues (is the path between your ssh client and the instance 9000 MTU?).

      In my experiences since this article setting up a management interface and network that is only 1500 or 1450 MTU is greatly preferred for management, but does complicate the setup further.

      I will be writing an updated article about MTU in OpenStack in the coming week as this one is a bit outdated and has some misconceptions on my part.

Leave a Reply

Your email address will not be published. Required fields are marked *