Bug#1008576: python3.10: Please backport patch to fix FTBFS on ia64
Source: python3.10
Version: 3.10.4-1
Severity: normal
Tags: patch upstream
User: debian-ia64@lists.debian.org
Usertags: ia64
X-Debbugs-Cc: debian-ia64@lists.debian.org
Hello!
The current FTBFS on ia64 is another variant of the segmentation
fault we already addressed in #995987 [1] with a work-around.
Upstream has decided now to fix the issue properly once and for
all [2]. I have cherry-picked the patch from [3] onto the 3.10
branch of cpython which fixed the issue for me.
I'm attaching the patch. The workaround in debian/rules from [1]
can be removed again.
Thanks,
Adrian
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995987
> [2] https://bugs.python.org/issue45526
> [3] https://github.com/python/cpython/commit/0224b7180b280794b9fba62057b278ffb536c86f
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
From 0224b7180b280794b9fba62057b278ffb536c86f Mon Sep 17 00:00:00 2001
From: Neil Schemenauer <nas-github@arctrix.com>
Date: Thu, 21 Oct 2021 14:05:46 -0700
Subject: [PATCH] bpo-45526: obmalloc radix use 64 addr bits (GH-29062)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
---
.../2021-10-19-10-29-47.bpo-45526.WQnvW9.rst | 3 +
Objects/obmalloc.c | 55 ++++++++++++-------
2 files changed, 38 insertions(+), 20 deletions(-)
create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst
diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst
new file mode 100644
index 0000000000..c35e5f58a4
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst
@@ -0,0 +1,3 @@
+In obmalloc, set ADDRESS_BITS to not ignore any bits (ignored 16 before). That is
+safer in the case that the kernel gives user-space virtual addresses that span
+a range greater than 48 bits.
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index 2eddb2cafd..4e17bf44b4 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1280,21 +1280,30 @@ _Py_GetAllocatedBlocks(void)
#if WITH_PYMALLOC_RADIX_TREE
/*==========================================================================*/
-/* radix tree for tracking arena usage
+/* radix tree for tracking arena usage. If enabled, used to implement
+ address_in_range().
- bit allocation for keys
+ memory address bit allocation for keys
- 64-bit pointers and 2^20 arena size:
- 16 -> ignored (POINTER_BITS - ADDRESS_BITS)
- 10 -> MAP_TOP
- 10 -> MAP_MID
- 8 -> MAP_BOT
+ 64-bit pointers, IGNORE_BITS=0 and 2^20 arena size:
+ 15 -> MAP_TOP_BITS
+ 15 -> MAP_MID_BITS
+ 14 -> MAP_BOT_BITS
+ 20 -> ideal aligned arena
+ ----
+ 64
+
+ 64-bit pointers, IGNORE_BITS=16, and 2^20 arena size:
+ 16 -> IGNORE_BITS
+ 10 -> MAP_TOP_BITS
+ 10 -> MAP_MID_BITS
+ 8 -> MAP_BOT_BITS
20 -> ideal aligned arena
----
64
32-bit pointers and 2^18 arena size:
- 14 -> MAP_BOT
+ 14 -> MAP_BOT_BITS
18 -> ideal aligned arena
----
32
@@ -1306,11 +1315,16 @@ _Py_GetAllocatedBlocks(void)
/* number of bits in a pointer */
#define POINTER_BITS 64
-/* Current 64-bit processors are limited to 48-bit physical addresses. For
- * now, the top 17 bits of addresses will all be equal to bit 2**47. If that
- * changes in the future, this must be adjusted upwards.
+/* High bits of memory addresses that will be ignored when indexing into the
+ * radix tree. Setting this to zero is the safe default. For most 64-bit
+ * machines, setting this to 16 would be safe. The kernel would not give
+ * user-space virtual memory addresses that have significant information in
+ * those high bits. The main advantage to setting IGNORE_BITS > 0 is that less
+ * virtual memory will be used for the top and middle radix tree arrays. Those
+ * arrays are allocated in the BSS segment and so will typically consume real
+ * memory only if actually accessed.
*/
-#define ADDRESS_BITS 48
+#define IGNORE_BITS 0
/* use the top and mid layers of the radix tree */
#define USE_INTERIOR_NODES
@@ -1318,7 +1332,7 @@ _Py_GetAllocatedBlocks(void)
#elif SIZEOF_VOID_P == 4
#define POINTER_BITS 32
-#define ADDRESS_BITS 32
+#define IGNORE_BITS 0
#else
@@ -1332,6 +1346,9 @@ _Py_GetAllocatedBlocks(void)
# error "arena size must be < 2^32"
#endif
+/* the lower bits of the address that are not ignored */
+#define ADDRESS_BITS (POINTER_BITS - IGNORE_BITS)
+
#ifdef USE_INTERIOR_NODES
/* number of bits used for MAP_TOP and MAP_MID nodes */
#define INTERIOR_BITS ((ADDRESS_BITS - ARENA_BITS + 2) / 3)
@@ -1360,11 +1377,9 @@ _Py_GetAllocatedBlocks(void)
#define MAP_MID_INDEX(p) ((AS_UINT(p) >> MAP_MID_SHIFT) & MAP_MID_MASK)
#define MAP_TOP_INDEX(p) ((AS_UINT(p) >> MAP_TOP_SHIFT) & MAP_TOP_MASK)
-#if ADDRESS_BITS > POINTER_BITS
-/* Return non-physical address bits of a pointer. Those bits should be same
- * for all valid pointers if ADDRESS_BITS set correctly. Linux has support for
- * 57-bit address space (Intel 5-level paging) but will not currently give
- * those addresses to user space.
+#if IGNORE_BITS > 0
+/* Return the ignored part of the pointer address. Those bits should be same
+ * for all valid pointers if IGNORE_BITS is set correctly.
*/
#define HIGH_BITS(p) (AS_UINT(p) >> ADDRESS_BITS)
#else
@@ -1416,7 +1431,7 @@ static arena_map_bot_t *
arena_map_get(block *p, int create)
{
#ifdef USE_INTERIOR_NODES
- /* sanity check that ADDRESS_BITS is correct */
+ /* sanity check that IGNORE_BITS is correct */
assert(HIGH_BITS(p) == HIGH_BITS(&arena_map_root));
int i1 = MAP_TOP_INDEX(p);
if (arena_map_root.ptrs[i1] == NULL) {
@@ -1476,7 +1491,7 @@ arena_map_get(block *p, int create)
static int
arena_map_mark_used(uintptr_t arena_base, int is_used)
{
- /* sanity check that ADDRESS_BITS is correct */
+ /* sanity check that IGNORE_BITS is correct */
assert(HIGH_BITS(arena_base) == HIGH_BITS(&arena_map_root));
arena_map_bot_t *n_hi = arena_map_get((block *)arena_base, is_used);
if (n_hi == NULL) {
--
2.34.1
Reply to: