Tuning suggestions for high-core-count Linux servers

[Ray Bellis]

On 02/06/2017 08:12, Browne, Stuart wrote:

> Query rate thus far reached (on 24 cores, numa node restricted): 426k qps
> Query rate thus far reached (on 48 cores, numa nodes unrestricted): 321k qps

In our internal Performance Lab I've achieved nearly 900 kqps on small
authoritative zones when we had hyperthreading enabled, and 700 kqps
without.

The lab uses Dell R430s running Fedora Core 23 with Intel X710 10GB NICs
and each populated with a single Xeon E5-2680 v3 2.5 GHz 12-core CPU.

These systems have had *negligible* tuning applied - the vast majority
of the system settings changes I've made have been to improve the
repeatability of results, not the absolute performance.

The only major setting I've found which both helps performance and
improves consistency is to ensure that each NIC rx/tx queue IRQ is
assigned to a specific CPU core, with irqbalance disabled.

This is with a _single_ dnsperf client, too.  The settings I use are
-c24 -q82 -T6 -x2048.   However I do use a tweaked version of dnsperf
which assigns each thread pair (it uses separate threads for rx and tx)
to its own core.

You may find the presentation I made at the recent DNS-OARC workshop of
interest:

<https://indico.dns-oarc.net/event/26/session/3/contribution/18>

You didn't mention precisely which 9.10 series version you're running.
Note that versions prior to 9.10.4 defaulted to a -U value of ncores/2,
but investigation showed that on modern systems this was sub-optimal so
it was changed to ncores-1.  This makes a *very* big difference.

kind regards,

Ray Bellis
ISC Research Fellow