[Bug 1988018] Re: [SRU][mlx5] Intermittent VF-LAG activation failure

I tested netplan.io 0.107.1-3ubuntu0.22.04.2 from jammy-proposed, all looking good!
The intermittent failures reported in comment #12 are resolved.

First of all, the eswitch/switchdev functionality is not available on Jammy's GA 5.15 kernel,
so I upgraded to the HWE kernel and installed Netplan from proposed:

ubuntu@akis:~$ sudo devlink dev eswitch show pci/0000:86:00.0
kernel answers: Operation not supported
ubuntu@akis:~$ sudo apt-get install --install-recommends linux-generic-hwe-22.04
ubuntu@akis:~$ uname -a
Linux akis 6.8.0-51-generic #52~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Dec 9 15:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@akis:~$ sudo apt install -t jammy-proposed netplan.io
ubuntu@akis:~$ apt list *netplan*
Listing... Done
libnetplan-dev/jammy-proposed 0.107.1-3ubuntu0.22.04.2 amd64
libnetplan0/jammy-proposed,now 0.107.1-3ubuntu0.22.04.2 amd64 [installed,automatic]
netplan-generator/jammy-proposed,now 0.107.1-3ubuntu0.22.04.2 amd64 [installed,automatic]
netplan.io/jammy-proposed,now 0.107.1-3ubuntu0.22.04.2 amd64 [installed,automatic]
python3-netplan/jammy-proposed,now 0.107.1-3ubuntu0.22.04.2 amd64 [installed,automatic]

Next, I identified the Mellanox ConnectX-5 NIC (enp134...0/1) and confrimed that LAG is disabled:
ubuntu@akis:~$ sudo lshw -c network -businfo
Bus info Device Class Description
============================================================
pci@0000:06:00.0 enp6s0 network I210 Gigabit Network Connection
pci@0000:35:00.0 enp53s0np0 network MT27800 Family [ConnectX-5]
pci@0000:3a:00.0 enp58s0np0 network MT27800 Family [ConnectX-5]
pci@0000:58:00.0 enp88s0np0 network MT27800 Family [ConnectX-5]
pci@0000:5d:00.0 enp93s0np0 network MT27800 Family [ConnectX-5]
pci@0000:86:00.0 enp134s0f0np0 network MT27800 Family [ConnectX-5]
pci@0000:86:00.1 enp134s0f1np1 network MT27800 Family [ConnectX-5]
pci@0000:b8:00.0 enp184s0np0 network MT27800 Family [ConnectX-5]
pci@0000:bd:00.0 enp189s0np0 network MT27800 Family [ConnectX-5]
pci@0000:e1:00.0 enp225s0np0 network MT27800 Family [ConnectX-5]
pci@0000:e6:00.0 enp230s0np0 network MT27800 Family [ConnectX-5]
ubuntu@akis:~$ sudo cat /sys/kernel/debug/mlx5/0000:86:00.0/lag/state
disabled
ubuntu@akis:~$ sudo cat /sys/kernel/debug/mlx5/0000:86:00.1/lag/state
disabled


I changed the Netplan configuration according to the test plan above, and rebooted the system:
ubuntu@akis:~$ sudo netplan get

** (process:3196): WARNING **: 14:22:14.283: `gateway4` has been deprecated, use default routes instead.
See the 'Default routes' section of the documentation for more details.
network:
version: 2
ethernets:
enp134s0f0np0:
optional: true
virtual-function-count: 8
embedded-switch-mode: "switchdev"
delay-virtual-functions-rebind: true
enp134s0f1np1:
optional: true
virtual-function-count: 8
embedded-switch-mode: "switchdev"
delay-virtual-functions-rebind: true
[...]
bonds:
bond0:
interfaces:
- enp134s0f0np0
- enp134s0f1np1
parameters:
mode: "active-backup"


After the reboot the link-aggregation (LAG) is activated and the bond0 is up:
ubuntu@akis:~$ sudo cat /sys/kernel/debug/mlx5/0000:86:00.1/lag/state
active
ubuntu@akis:~$ sudo cat /sys/kernel/debug/mlx5/0000:86:00.0/lag/state
active
ubuntu@akis:~$ netplan status bond0
Online state: online
DNS Addresses: 127.0.0.53 (stub)
DNS Search: maas

● 13: bond0 bond UP (networkd: bond0)
MAC Address: ce:03:e9:7f:f9:9d
Addresses: fe80::cc03:e9ff:fe7f:f99d/64 (link)
Routes: fe80::/64 metric 256

12 inactive interfaces hidden. Use "--all" to show all.

** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/1988018

Title:
[SRU][mlx5] Intermittent VF-LAG activation failure

Status in linux package in Ubuntu:
Fix Committed
Status in netplan.io package in Ubuntu:
Fix Released
Status in linux source package in Jammy:
Confirmed
Status in netplan.io source package in Jammy:
Fix Committed
Status in linux source package in Kinetic:
Won't Fix
Status in netplan.io source package in Kinetic:
Won't Fix
Status in linux source package in Mantic:
Won't Fix
Status in netplan.io source package in Mantic:
Won't Fix
Status in linux source package in Noble:
Fix Committed
Status in netplan.io source package in Noble:
Fix Released

Bug description:
[ Impact ]

Due to limitations in how Netplan handles SR-IOV devices, the VF-LAG
feature found on Mellanox NICs couldn't be used. Certain configuration steps
must happen in a very specific order and Netplan fails to perform the set up correctly.

Netplan must wait until the backend finishes adding interfaces to the Bond
and the Mellanox driver reports the VF-LAG feature as "active" before binding VFs to
the driver.

See also https://bugs.launchpad.net/netplan/+bug/2083008

This problem is fixed by introducing a proper ordering in the configuration process
and monitoring the driver state until it reports as ready (or times out).

This fix is available on Ubuntu 24.04.

[ Test Plan ]

To reproduce the problem addressed by this SRU one needs to
have access to specialized hardware (SR-IOV-capable Mellanox NICs).

The fix for the problem described above was already verified on Ubuntu 22.04 and
solved the problem (more details https://bugs.launchpad.net/netplan/+bug/2083008).

We will work with Canonical's Openstack team to do the fix
verification.

 * detailed instructions how to reproduce the bug

A configuration file that looks like the one below can be used
to test the fix.

After booting the system with this configuration, the Mellanox driver
should report the LAG state as "active" for all the devices.
It can be checked in the debugfs file: /sys/kernel/debug/mlx5/{pci_addr}/lag/state

network:
  version: 2
  ethernets:
    ens4f0np0:
      virtual-function-count: 16
      embedded-switch-mode: switchdev
      delay-virtual-functions-rebind: true

    ens4f1np1:
      virtual-function-count: 16
      embedded-switch-mode: switchdev
      delay-virtual-functions-rebind: true

  bonds:
    bond0:
      interfaces:
        - ens4f0np0
        - ens4f1np1
      parameters:
        mode: active-backup

[ Where problems could occur ]

These changes should affect only SR-IOV related scenarios.
Undetected problems could cause Netplan to fail to configure the device
and Virtual Functions wouldn't be created anymore.

[ Other Info ]

Related work:

https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1988018
https://github.com/canonical/netplan/pull/439

A PPA for Ubuntu 22.04 can be found here
https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan-sru

---- Original bug description ----

During system initialization there is a specific sequence that must be
followed to enable the use of hardware offload and VF-LAG.

Intermittently one may see that VF-LAG initialization fails:
[Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: lag map port 1:1 port 2:2 shared_fdb:1
[Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_cmd_check:782:(pid 9): CREATE_LAG(0x840) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x7d49cb)
[Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_create_lag:248:(pid 9): Failed to create LAG (-22)
[Thu Jul 21 10:54:58 2022] mlx5_core 0000:08:00.0: mlx5_activate_lag:288:(pid 9): Failed to activate VF LAG
                           Make sure all VFs are unbound prior to VF LAG activation or deactivation

This is caused by rebinding the driver prior to the VF lag being
ready.

A sysfs knob has recently been added to the driver [0] and we should
monitor it before attempting to rebind the driver:

    $ cat /sys/kernel/debug/mlx5/0000\:08\:00.0/lag/state

The kernel feature is available in the upcoming Kinetic 5.19 kernel
and we should probably backport it to the Jammy 5.15 kernel.

0:
https://github.com/torvalds/linux/commit/7f46a0b7327ae261f9981888708dbca22c283900

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1988018/+subscriptions