83 lines
2.9 KiB
Diff
83 lines
2.9 KiB
Diff
From 3202fa62fb43087387c65bfa9c100feffac74aa6 Mon Sep 17 00:00:00 2001
|
|
From: Kees Cook <keescook@chromium.org>
|
|
Date: Wed, 1 Apr 2020 21:04:27 -0700
|
|
Subject: [PATCH] slub: relocate freelist pointer to middle of object
|
|
|
|
In a recent discussion[1] with Vitaly Nikolenko and Silvio Cesare, it
|
|
became clear that moving the freelist pointer away from the edge of
|
|
allocations would likely improve the overall defensive posture of the
|
|
inline freelist pointer. My benchmarks show no meaningful change to
|
|
performance (they seem to show it being faster), so this looks like a
|
|
reasonable change to make.
|
|
|
|
Instead of having the freelist pointer at the very beginning of an
|
|
allocation (offset 0) or at the very end of an allocation (effectively
|
|
offset -sizeof(void *) from the next allocation), move it away from the
|
|
edges of the allocation and into the middle. This provides some
|
|
protection against small-sized neighboring overflows (or underflows), for
|
|
which the freelist pointer is commonly the target. (Large or well
|
|
controlled overwrites are much more likely to attack live object contents,
|
|
instead of attempting freelist corruption.)
|
|
|
|
The vaunted kernel build benchmark, across 5 runs. Before:
|
|
|
|
Mean: 250.05
|
|
Std Dev: 1.85
|
|
|
|
and after, which appears mysteriously faster:
|
|
|
|
Mean: 247.13
|
|
Std Dev: 0.76
|
|
|
|
Attempts at running "sysbench --test=memory" show the change to be well in
|
|
the noise (sysbench seems to be pretty unstable here -- it's not really
|
|
measuring allocation).
|
|
|
|
Hackbench is more allocation-heavy, and while the std dev is above the
|
|
difference, it looks like may manifest as an improvement as well:
|
|
|
|
20 runs of "hackbench -g 20 -l 1000", before:
|
|
|
|
Mean: 36.322
|
|
Std Dev: 0.577
|
|
|
|
and after:
|
|
|
|
Mean: 36.056
|
|
Std Dev: 0.598
|
|
|
|
[1] https://twitter.com/vnik5287/status/1235113523098685440
|
|
|
|
Signed-off-by: Kees Cook <keescook@chromium.org>
|
|
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Acked-by: Christoph Lameter <cl@linux.com>
|
|
Cc: Vitaly Nikolenko <vnik@duasynt.com>
|
|
Cc: Silvio Cesare <silvio.cesare@gmail.com>
|
|
Cc: Christoph Lameter <cl@linux.com>Cc: Pekka Enberg <penberg@kernel.org>
|
|
Cc: David Rientjes <rientjes@google.com>
|
|
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
|
|
Link: http://lkml.kernel.org/r/202003051624.AAAC9AECC@keescook
|
|
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
|
|
---
|
|
mm/slub.c | 7 +++++++
|
|
1 file changed, 7 insertions(+)
|
|
|
|
diff --git a/mm/slub.c b/mm/slub.c
|
|
index bc949e3428c97..3098e0cf28992 100644
|
|
--- a/mm/slub.c
|
|
+++ b/mm/slub.c
|
|
@@ -3581,6 +3581,13 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order)
|
|
*/
|
|
s->offset = size;
|
|
size += sizeof(void *);
|
|
+ } else if (size > sizeof(void *)) {
|
|
+ /*
|
|
+ * Store freelist pointer near middle of object to keep
|
|
+ * it away from the edges of the object to avoid small
|
|
+ * sized over/underflows from neighboring allocations.
|
|
+ */
|
|
+ s->offset = ALIGN(size / 2, sizeof(void *));
|
|
}
|
|
|
|
#ifdef CONFIG_SLUB_DEBUG
|