[Bug 2051342] Re: Enable lowlatency settings in the generic kernel

This bug is awaiting verification that the linux-gcp/6.8.0-1008.9 kernel
in -proposed solves the problem. Please test the kernel and update this
bug with the results. If the problem is solved, change the tag
'verification-needed-noble-linux-gcp' to 'verification-done-noble-linux-
gcp'. If the problem still exists, change the tag 'verification-needed-
noble-linux-gcp' to 'verification-failed-noble-linux-gcp'.


If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.


See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!


** Tags added: kernel-spammed-noble-linux-gcp-v2 verification-needed-noble-linux-gcp

--
You received this bug notification because you are subscribed to linux
in Ubuntu.
Matching subscriptions: Bgg, Bmail, Nb
https://bugs.launchpad.net/bugs/2051342

Title:
Enable lowlatency settings in the generic kernel

Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Noble:
Fix Released

Bug description:
[Impact]

Ubuntu provides the "lowlatency" kernel: a kernel optimized for
applications that have special "low latency" requirements.

Currently, this kernel does not include any specific UBUNTU SAUCE
patches to improve the extra "low latency" requirements, but the only
difference is a small subset of .config options.

Almost all these options are now configurable either at boot-time or
even at run-time, with the only exception of CONFIG_HZ (250 in the
generic kernel vs 1000 in the lowlatency kernel).

Maintaining a separate kernel for a single config option seems a bit
overkill and it is a significant cost of engineering hours, build
time, regression testing time and resources. Not to mention the risk
of the low-latency kernel falling behind and not being perfectly in
sync with the latest generic kernel.

Enabling the low-latency settings in the generic kernel has been
evaluated before, but it has been never finalized due to the potential
risk of performance regressions in CPU-intensive applications
(increasing HZ from 250 to 1000 may introduce more kernel jitter in
number crunching workloads). The outcome of the original proposal
resulted in a re-classification of the lowlatency kernel as a desktop-
oriented kernel, enabling additional low latency features (LP:
#2023007).

As we are approaching the release of the new Ubuntu 24.04 we may want
to re-consider merging the low-latency settings in the generic kernel
again.

Following a detailed analysis of the specific low-latency features:

- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown
clock tick when possible across all the enabled CPUs if they are
either idle or running 1 task - reduce kernel jitter of running tasks
due to the periodic clock tick, must be enabled at boot time passing
`nohz_full=<cpu_list>`); this can actually help CPU-intensive
workloads and it could provide much more benefits than the CONFIG_HZ
difference (since it can potentially shutdown any kernel jitter on
specific CPUs), this one should really be enabled anyway, considering
that it is configurable at boot time

 - CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to
kthread context (reduce time spent in softirqs with preemption
disabled to improve the overall system responsiveness, at the cost of
introducing a potential performance penalty, because RCU callbacks are
not processed by kernel threads); this should be enabled as well,
since it is configurable at boot time (via the rcu_nocbs=<cpu_list>
parameter)

 - CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a
timed delay instead of executing them immediately (c'an provide 5~10%
power-savings for idle or lightly-loaded systems, this is extremely
useful for laptops / portable devices -
https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/);
this has the potential to introduce significant performance
regressions, but in the Noble kernel we already have a SAUCE patch
that allows to enable/disable this option at boot time (see LP:
#2045492), and by default it will be disabled
(CONFIG_RCU_LAZY_DEFAULT_OFF=y)

 - CONFIG_HZ=1000 last but not least, the only option that is *only*
tunable at compile time. As already mentioned there is a potential
risk of regressions for CPU-intensive applications, but they can be
mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On
the other hand, HZ=1000 can improve system responsiveness, that means
most of the desktop and server applications will benefit from this
(the largest part of the server workloads is I/O bound, more than CPU-
bound, so they can benefit from having a kernel that can react faster
at switching tasks), not to mention the benefit for the typical end
users applications (gaming, live conferencing, multimedia, etc.).

With all of that in place we can provide a kernel that has the
flexibility to be more responsive, more performant and more power
efficient (therefore more "generic"), simply by tuning run-time and
boot-time options.

Moreover, once these changes are applied we will be able to deprecate
the lowlatency kernel, saving engineering time and also reducing power
consumption (required to build the kernel and do all the testing).

Optionally, we can also provide optimal "lowlatency" settings as a
user-space package that would set the proper options in the kernel
boot command line (GRUB, or similar).

[Test case]

There are plenty of benchmarks that can prove the validity of each one
of the setting mentioned above, providing huge benefits in terms of
system responsive.

However, our main goal here is to mitigate as much as possible the
risk of regression for CPU-intensive applications, so the test case
should only be focused on this particular aspect, to evaluate the
impact of this change in the worst case scenario.

Test case (CPU-intensive stress test):

 - stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m
--metrics-brief

Metrics:

 - measure the bogo ops printed to stdout (not a great metric for
real-world applications, but in this case it can show the impact of
the additional kernel jitter introduced by the different CONFIG_HZ)

Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):

 - CONFIG_HZ=250 : 17415.60 bogo ops/s
 - CONFIG_HZ=1000 : 14866.05 bogo ops/s
 - CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s

Results confirm the theory about the performance drop of CPU-intensive
workloads (-~14%), but also confirms the benefit of NO_HZ_FULL (+~6%)
compared to the current HZ settings.

Let's also keep in mind that this is the worst case scenario and a
very specific one, where only HPC / scientific applications can be
affected, and even in this case we can always compensate and actually
get a better level performance exploiting the nohz_full capability.

[Fix]

Enable the .config options mentioned above in the generic kernel (only
on amd64 and arm64 for now).

[Regression potential]

As already covered we may experience performance regressions in CPU-
intensive (number crunching) applications (such as HPC for example),
but they can be compensated by the NO_HZ_FULL boot-time option.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2051342/+subscriptions