☁️ OpenStack Error: Neutron Overlay MTU Mismatch Dropping/Fragmenting Tenant Traffic

Exact Error Message

MTU mismatches rarely raise a clean API error — they surface as silent connectivity failures. Large packets are dropped while small ones (ping, SSH handshake) succeed. Typical symptoms captured from inside an instance:

$ curl https://example.com
* Connected to example.com (93.184.216.34) port 443
* TLS handshake, Client hello (1)
  (hangs, then) curl: (28) SSL connection timeout

$ ping -M do -s 1472 10.0.0.5
PING 10.0.0.5 (10.0.0.5) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450

On the network node you may see the effective MTU Neutron assigned:

$ openstack network show private -c mtu
+-------+-------+
| Field | Value |
+-------+-------+
| mtu   | 1450  |
+-------+-------+

What It Means

Tenant networks in OpenStack usually run over an overlay encapsulation (VXLAN or GENEVE). Each encapsulated packet carries extra header bytes, so the usable payload inside the overlay is smaller than the physical link’s 1500-byte MTU — commonly 1450 (VXLAN) or 1442 (GENEVE). Neutron computes this and advertises the reduced MTU to instances via DHCP.

Trouble starts when something in the path does not agree on that number: an instance hard-codes 1500, the physical fabric can’t carry the full encapsulated frame, or global_physnet_mtu is misconfigured. Packets larger than the smallest link get dropped, and because TCP handshakes and pings are small, basic connectivity “works” while real payloads (TLS, file transfers) silently black-hole. This is a classic path-MTU / MTU-mismatch failure.

Common Causes

• Instance interface MTU is 1500 but the overlay only supports 1450, so large frames are dropped.
• The physical underlay does not support jumbo frames, so encapsulated 1500-byte payloads exceed link MTU.
• global_physnet_mtu in neutron.conf (or path_mtu) is set incorrectly for the fabric.
• DHCP MTU option is not being honored by the guest (static config, cloud image quirk).
• ICMP “fragmentation needed” (Path MTU Discovery) is blocked by a security group or firewall, breaking PMTUD.
• A router or provider network segment along the path has a smaller MTU than the tenant network.

Diagnostic Commands

Check the MTU Neutron assigned to the network and its ports:

openstack network show private -c mtu
openstack port show <PORT_ID> -c mtu

From inside the instance, confirm the interface MTU and find the real path MTU:

ip link show eth0 | grep -o 'mtu [0-9]*'
ping -M do -s 1422 -c 3 10.0.0.5    # 1422 payload + 28 = 1450; increase until it fails
tracepath 10.0.0.5

Inspect Neutron’s configured MTU values on the controller:

sudo crudini --get /etc/neutron/neutron.conf DEFAULT global_physnet_mtu
sudo crudini --get /etc/neutron/plugins/ml2/ml2_conf.ini ml2 path_mtu

Check the ML2/OVS agent log for MTU-related warnings:

sudo journalctl -u devstack@q-agt --no-pager | grep -i mtu
# package installs: /var/log/neutron/neutron-openvswitch-agent.log

Step-by-Step Resolution

1. Establish the real path MTU with ping -M do (do-not-fragment). Increase the payload size until packets stop getting through; the largest that succeeds plus 28 bytes is your path MTU.

2. Confirm the underlay. If you intend to run VXLAN/GENEVE without shrinking guest MTU, the physical fabric must support jumbo frames (MTU 1550+). Verify with the same DF ping between compute hosts.

3. Set Neutron’s global physical MTU to match the underlay in neutron.conf, and path_mtu in the ML2 config:

# /etc/neutron/neutron.conf
[DEFAULT]
global_physnet_mtu = 1500

# /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
path_mtu = 1500

4. Restart Neutron so it recomputes overlay MTUs, then confirm the network’s advertised MTU:

sudo systemctl restart devstack@q-svc devstack@q-agt
openstack network show private -c mtu

5. Ensure guests actually pick up the advertised MTU. New instances get it via DHCP; existing ones may need a renew or a manual set:

sudo dhclient -r eth0 && sudo dhclient eth0
# or force it directly:
sudo ip link set dev eth0 mtu 1450

6. Allow ICMP so Path MTU Discovery works, which lets endpoints adapt when a smaller link exists mid-path:

openstack security group rule create --protocol icmp default

7. Re-test with a large-payload DF ping and a real TLS request; both should now succeed:

ping -M do -s 1422 -c 3 10.0.0.5
curl -sI https://example.com | head -1

Prevention

• Decide an MTU strategy up front: either enable jumbo frames on the underlay (keep guests at 1500) or accept a reduced overlay MTU (e.g. 1450) advertised via DHCP.
• Set global_physnet_mtu/path_mtu to match the physical fabric and keep them in configuration management.
• Always permit ICMP in default security groups so PMTUD can function.
• Bake DHCP-driven MTU handling into cloud images; avoid hard-coding 1500 in guest network config.
• Test large-payload connectivity (not just ping/SSH) as part of network validation, since small packets hide MTU faults.

Related Errors

• curl: (28) ... timeout on HTTPS while ping/SSH work — the signature symptom of an overlay MTU black hole.
• message too long, mtu=1450 from ping -M do — the kernel reporting the effective path MTU.
• No IP on the instance — a DHCP agent problem rather than MTU (different failure mode).
• Intermittent NFS/large-transfer stalls over a provider network — a mid-path segment with a smaller MTU.

Frequently Asked Questions

Why does ping and SSH work but HTTPS or file transfers hang? Small packets (ping, TCP handshake) fit within the smallest MTU, so they pass. Large data packets exceed the mismatched MTU and are dropped, so the connection stalls after the handshake.

Should I lower guest MTU or raise underlay MTU? Either works. Enabling jumbo frames on the physical fabric lets guests stay at 1500; otherwise let Neutron advertise a reduced overlay MTU (e.g. 1450) and ensure guests honor it via DHCP.

Why does blocking ICMP make this worse? Path MTU Discovery relies on ICMP “fragmentation needed” messages. If a security group drops ICMP, endpoints never learn to shrink packets and large frames silently black-hole.

How do I find the real path MTU quickly? Use ping -M do -s <size> and increase <size> until it fails; the largest successful payload plus 28 bytes is the path MTU. For MTU-troubleshooting prompts, see the OpenStack prompts.

Do I have to restart Neutron after changing global_physnet_mtu? Yes. Overlay MTU is computed at service start, so restart the Neutron server and agents, then confirm the new value with openstack network show. For more networking guides, see the OpenStack guides.