TCMalloc and Transparent Huge Pages

[max0x7ba]

https://google.github.io/tcmalloc/tuning.html#system-level-optimizations says:

TCMalloc heavily relies on Transparent Huge Pages (THP)

And requires that /sys/kernel/mm/transparent_hugepage/enabled be set to always.

This requirement is not ideal because:

1. The default for /sys/kernel/mm/transparent_hugepage/enabled is often madvise.
2. Some legacy applications exhibit performance degradation with always option.

I wonder why wouldn't TCMalloc explicitly request transparent huge pages using madvise(..., MADV_HUGEPAGE), to not have this requirement, please?

[SrivastavaAnubhav]

Are there any plans to take this up in the near future @ckennelly ? There appear to have been some changes in recent Linux kernels which cause thread stacks to be backed by hugepages (and therefore use a lot more memory) by default if transparent_hugepage/enabled is set to always.

[ckennelly]

This is not currently planned. There's no requirement that applications use transparent huge pages to use TCMalloc, it's just that we find it broadly improves application performance to do so.

Thread stacks are typically quite small and wouldn't be eligible for huge pages. Grabbing a data point from the machine (kernel ver 6.9.9) that I happen to be using to type this:

cat /proc/self/smaps | grep stack -A 1
7fff6ddce000-7fff6ddf0000 rw-p 00000000 00:00 0                          [stack]
Size:                136 kB

This wouldn't be huge page-eligible because it is neither aligned to a huge page boundary nor is it huge page-sized.

[SrivastavaAnubhav]

Ok. For what it's worth, I'm seeing stack sizes of 8 MiB (virtual memory) on Linux, which other people seem to agree with: https://unix.stackexchange.com/questions/473416/why-on-modern-linux-the-default-stack-size-is-so-huge-8mb-even-10-on-some-di?rq=1. I think there might have been a recent change (in the last year) that causes these stacks to be aligned properly to be backed by THP now because a couple people have noticed this now: https://bugs.openjdk.org/browse/JDK-8303215.

I'll do some profiling to see if there's a performance impact to our application with THP madvise instead of always though.

[akonradi]

I noticed that 08642c4 added calls to madvise with MADV_HUGEPAGE, but that it was reverted in 08642c4. @toddlipcon do you know why it was rolled back?

[toddlipcon]

Yea, the issue was that various tcmalloc use cases run on machines where the kernel setting for /sys/kernel/mm/transparent_hugepage/defrag is either defer+madvise or madvise -- in these cases, the MADV_HUGEPAGE was causing direct reclaim and synchronous hugepage allocation on first touch. It made their memory footprint increase in ways that broke various integration tests / load tests.

One might argue that this is what we want, but it caused us to roll this back. If you want to make an optional version of the feature, maybe we can reintroduce it (@ckennelly can probably opine if he disagrees)

[hsane-dev]

How about optional enabling based on size: PR