[Bug 2098961] Re: Severe network performance issues with Kernel 6.8.0-53 and 6.11.0-17 (Xubuntu) on Proxmox 8.3 (bnxt_en + VirtIO)

** Tags added: kernel-bug-reported-upstream

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2098961

Title:
Severe network performance issues with Kernel 6.8.0-53 and 6.11.0-17
(Xubuntu) on Proxmox 8.3 (bnxt_en + VirtIO)

Status in linux package in Ubuntu:
Triaged

Bug description:
Since upgrading to Kernel 6.8.0-53 and 6.11.0-17 on Xubuntu
22.04/24.04 VMs, we are experiencing extremely low network speeds
(100-200 KB/s) when running these VMs in a Proxmox 8.3.4 cluster with
AMD CPUs and Broadcom NetXtreme-E network adapters (bnxt_en driver).

With Kernel 6.8.0-31 inside the same Xubuntu VM, everything works as expected.
Other VMs (Debian, Ubuntu Server) running on the same Proxmox cluster are not affected.
The same Xubuntu VM, when moved to a different Proxmox cluster (Intel CPUs, Intel NICs, ZFS storage), runs fine with Kernel 6.8.0-53 and 6.11.0-17.
The issue only affects Xubuntu VMs running on a Proxmox 8.3.4 cluster with AMD CPUs, Broadcom NICs, and Ceph storage.

Expected behavior:

Network performance should be normal (10 Gbps or more) without issues.
Actual behavior:

Xubuntu VMs with Kernel 6.8.0-53 and 6.11.0-17 experience extremely slow network speeds (100-200 KB/s).
iperf3 tests show very low bandwidth (5-10 Mbps).
Disabling GSO, TSO, and GRO (ethtool -K ens18 tso off gso off gro off) did not improve performance.
Other VMs (Debian, Ubuntu Server) on the same cluster do not have this issue.
The issue disappears when booting Kernel 6.8.0-31 inside the same Xubuntu VM.

Steps to reproduce:

Proxmox environment:
Proxmox 8.3.4 (Kernel 6.8.12-8-pve)
AMD CPU + Broadcom NetXtreme-E BCM57504 (bnxt_en driver) + Ceph storage
VMs running Xubuntu 22.04 or 24.04 with VirtIO network (virtio-net)

Inside the Xubuntu VM:
Install and boot Kernel 6.8.0-53 or Kernel 6.11.0-17
Test network performance:

wget -O /dev/null http://speed.hetzner.de/1GB.bin
iperf3 -c <server-ip> -P 4

→ Speed is extremely slow (~200 KB/s - 5 Mbps).

Reboot the VM with Kernel 6.8.0-31

sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-31-generic"
sudo reboot

→ Network performance returns to normal (10 Gbps+).

Hardware & environment details:
Proxmox cluster (affected environment):

Proxmox 8.3.4 (Kernel 6.8.12-8-pve)
AMD CPUs
Broadcom NetXtreme-E BCM57504 25G SFP28 Ethernet (bnxt_en driver)
Ceph storage backend
VMs use VirtIO network adapter (virtio-net)
Bridging on Broadcom NICs

Working cluster (same Xubuntu VM, no issue):

Proxmox 8.3.4 (Kernel 6.8.12-8-pve)
Intel CPUs
Intel I350 NICs (igb driver)
ZFS storage backend
Same Xubuntu VM works fine with Kernel 6.8.0-53 and 6.11.0-17

Debugging information:

Xubuntu VM Kernel versions (affected):

uname -r
6.8.0-53-generic
6.11.0-17-generic

Broadcom firmware version (Proxmox host):

ethtool -i enp67s0f0np0 | grep firmware
firmware-version: 229.0.154.0/pkg 229.1.123.0

dmesg output related to bnxt_en NICs (Proxmox host):

dmesg | grep bnxt
[ 2.324347] bnxt_en 0000:c5:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 60 seconds, may trigger hung task watchdog
[ 2.364280] bnxt_en 0000:c5:00.0 eth0: Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem b7030000, node addr 14:23:f2:5d:18:14
...

→ No critical errors, but timeout warnings and performance degradation.
Possible causes:

🔹 Regression in virtio-net or bnxt_en driver in Kernel 6.8.0-53 and 6.11.0-17
🔹 Interaction issue between virtio-net and bnxt_en on AMD + Broadcom hardware
🔹 Proxmox bridge settings on Broadcom NICs interacting with Xubuntu's Kernel 6.8.0-53 and 6.11.0-17
Next Steps & Further Tests

Since this is a production system with 200+ running VMs, we will
conduct further tests once we schedule maintenance.

Planned additional tests:

Test a fresh Ubuntu Server VM with Kernel 6.8.0-53 and 6.11.0-17 (to rule out Xubuntu-specific issues).
Verify bnxt_en and virtio-net driver versions in Kernel 6.8.0-31 vs. 6.8.0-53 and 6.11.0-17.
Check IRQ assignments and CPU affinity for VirtIO network interfaces.

📌 If this issue is already reported, please link to the relevant bug report.
📌 Otherwise, we are happy to provide further debugging information.
Summary:

Xubuntu VMs running Kernel 6.8.0-53 and 6.11.0-17 on a Proxmox 8.3.4 cluster with AMD CPUs + Broadcom NICs suffer from extreme network slowdowns (~200 KB/s).
Booting the same VM with Kernel 6.8.0-31 restores full performance.
Same Kernels work fine on Intel-based Proxmox clusters.
Disabling TSO, GSO, GRO does NOT fix the issue.
Likely a regression in virtio-net or bnxt_en.

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-6.8.0-53-generic 6.8.0-53.55
ProcVersionSignature: Ubuntu 6.8.0-53.55-generic 6.8.12
Uname: Linux 6.8.0-53-generic x86_64
ApportVersion: 2.28.1-0ubuntu3.1
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/seq: sysadmin 1219 F.... pipewire
CRDA: N/A
CasperMD5CheckResult: pass
Date: Thu Feb 20 11:52:51 2025
InstallationDate: Installed on 2025-02-20 (0 days ago)
InstallationMedia: Xubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827)
IwConfig:
lo no wireless extensions.

ens18 no wireless extensions.
Lsusb:
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU Tablet
Lsusb-t:
/: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=uhci_hcd/2p, 12M
|__ Port 001: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 12M
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcFB: 0 bochs-drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.8.0-53-generic root=UUID=5f2acbcf-06d2-464a-83a0-12ef507709a5 ro quiet splash vt.handoff=7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.8.0-53-generic N/A
linux-backports-modules-6.8.0-53-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.9
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.release: 0.0
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-9.0
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org:bd04/01/2014:br0.0:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-9.0:cvnQEMU:ct1:cvrpc-i440fx-9.0:sku:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-9.0
dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098961/+subscriptions