X Tutup
Skip to content

Commit 34be807

Browse files
vstinnervstinner
committed
Add PYTHONMALLOC env var
Issue python#26516: * Add PYTHONMALLOC environment variable to set the Python memory allocators and/or install debug hooks. * PyMem_SetupDebugHooks() can now also be used on Python compiled in release mode. * The PYTHONMALLOCSTATS environment variable can now also be used on Python compiled in release mode. It now has no effect if set to an empty string. * In debug mode, debug hooks are now also installed on Python memory allocators when Python is configured without pymalloc.
1 parent c877658 commit 34be807

File tree

13 files changed

+383
-90
lines changed

13 files changed

+383
-90
lines changed

‎Doc/c-api/memory.rst‎

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -85,9 +85,12 @@ for the I/O buffer escapes completely the Python memory manager.
8585

8686
.. seealso::
8787

88+
The :envvar:`PYTHONMALLOC` environment variable can be used to configure
89+
the memory allocators used by Python.
90+
8891
The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print
89-
memory allocation statistics every time a new object arena is created, and
90-
on shutdown.
92+
statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
93+
new pymalloc object arena is created, and on shutdown.
9194

9295

9396
Raw Memory Interface
@@ -343,25 +346,36 @@ Customize Memory Allocators
343346
- detect write before the start of the buffer (buffer underflow)
344347
- detect write after the end of the buffer (buffer overflow)
345348
346-
The function does nothing if Python is not compiled is debug mode.
349+
These hooks are installed by default if Python is compiled in debug
350+
mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install
351+
debug hooks on a Python compiled in release mode.
352+
353+
.. versionchanged:: 3.6
354+
This function now also works on Python compiled in release mode.
355+
347356
357+
.. _pymalloc:
348358
349-
Customize PyObject Arena Allocator
350-
==================================
359+
The pymalloc allocator
360+
======================
351361
352-
Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
353-
allocator is optimized for small objects with a short lifetime. It uses memory
354-
mappings called "arenas" with a fixed size of 256 KB. It falls back to
355-
:c:func:`PyMem_RawMalloc` and :c:func:`PyMem_RawRealloc` for allocations larger
356-
than 512 bytes. *pymalloc* is the default allocator used by
357-
:c:func:`PyObject_Malloc`.
362+
Python has a *pymalloc* allocator optimized for small objects (smaller or equal
363+
to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
364+
with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
365+
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
358366
359-
The default arena allocator uses the following functions:
367+
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
368+
(:c:func:`PyObject_Malloc` & cie).
369+
370+
The arena allocator uses the following functions:
360371
361372
* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
362373
* :c:func:`mmap` and :c:func:`munmap` if available,
363374
* :c:func:`malloc` and :c:func:`free` otherwise.
364375
376+
Customize pymalloc Arena Allocator
377+
----------------------------------
378+
365379
.. versionadded:: 3.4
366380
367381
.. c:type:: PyObjectArenaAllocator

‎Doc/using/cmdline.rst‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -621,6 +621,51 @@ conflict.
621621
.. versionadded:: 3.4
622622

623623

624+
.. envvar:: PYTHONMALLOC
625+
626+
Set the Python memory allocators and/or install debug hooks.
627+
628+
Set the family of memory allocators used by Python:
629+
630+
* ``malloc``: use the :c:func:`malloc` function of the C library
631+
for all Python memory allocators (:c:func:`PyMem_RawMalloc`,
632+
:c:func:`PyMem_Malloc`, :c:func:`PyObject_Malloc` & cie).
633+
* ``pymalloc``: :c:func:`PyObject_Malloc`, :c:func:`PyObject_Calloc` and
634+
:c:func:`PyObject_Realloc` use the :ref:`pymalloc allocator <pymalloc>`.
635+
Other Python memory allocators (:c:func:`PyMem_RawMalloc`,
636+
:c:func:`PyMem_Malloc` & cie) use :c:func:`malloc`.
637+
638+
Install debug hooks:
639+
640+
* ``debug``: install debug hooks on top of the default memory allocator
641+
* ``malloc_debug``: same than ``malloc`` but also install debug hooks
642+
* ``pymalloc_debug``: same than ``malloc`` but also install debug hooks
643+
644+
See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python
645+
memory allocators.
646+
647+
.. note::
648+
``pymalloc`` and ``pymalloc_debug`` are not available if Python is
649+
configured without ``pymalloc`` support.
650+
651+
.. versionadded:: 3.6
652+
653+
654+
.. envvar:: PYTHONMALLOCSTATS
655+
656+
If set to a non-empty string, Python will print statistics of the
657+
:ref:`pymalloc memory allocator <pymalloc>` every time a new pymalloc object
658+
arena is created, and on shutdown.
659+
660+
This variable is ignored if the :envvar:`PYTHONMALLOC` environment variable
661+
is used to force the :c:func:`malloc` allocator of the C library, or if
662+
Python is configured without ``pymalloc`` support.
663+
664+
.. versionchanged:: 3.6
665+
This variable can now also be used on Python compiled in release mode.
666+
It now has no effect if set to an empty string.
667+
668+
624669
Debug-mode variables
625670
~~~~~~~~~~~~~~~~~~~~
626671

@@ -636,9 +681,3 @@ if Python was configured with the ``--with-pydebug`` build option.
636681

637682
If set, Python will dump objects and reference counts still alive after
638683
shutting down the interpreter.
639-
640-
641-
.. envvar:: PYTHONMALLOCSTATS
642-
643-
If set, Python will print memory allocation statistics every time a new
644-
object arena is created, and on shutdown.

‎Doc/whatsnew/3.6.rst‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,9 @@ Summary -- Release highlights
8080
PEP written by Carl Meyer
8181

8282

83+
New Features
84+
============
85+
8386
.. _whatsnew-fstrings:
8487

8588
PEP 498: Formatted string literals
@@ -98,6 +101,34 @@ evaluated at run time, and then formatted using the :func:`format` protocol.
98101
See :pep:`498` and the main documentation at :ref:`f-strings`.
99102

100103

104+
PYTHONMALLOC environment variable
105+
---------------------------------
106+
107+
The new :envvar:`PYTHONMALLOC` environment variable allows to set the Python
108+
memory allocators and/or install debug hooks.
109+
110+
It is now possible to install debug hooks on Python memory allocators on Python
111+
compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks:
112+
113+
* Newly allocated memory is filled with the byte ``0xCB``
114+
* Freed memory is filled with the byte ``0xDB``
115+
* Detect violations of Python memory allocator API. For example,
116+
:c:func:`PyObject_Free` called on a memory block allocated by
117+
:c:func:`PyMem_Malloc`.
118+
* Detect write before the start of the buffer (buffer underflow)
119+
* Detect write after the end of the buffer (buffer overflow)
120+
121+
See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python
122+
memory allocators.
123+
124+
It is now also possible to force the usage of the :c:func:`malloc` allocator of
125+
the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``.
126+
It helps to use external memory debuggers like Valgrind on a Python compiled in
127+
release mode.
128+
129+
(Contributed by Victor Stinner in :issue:`26516`.)
130+
131+
101132
Other Language Changes
102133
======================
103134

‎Include/pymem.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,17 @@ PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
1616
PyAPI_FUNC(void *) PyMem_RawCalloc(size_t nelem, size_t elsize);
1717
PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
1818
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
19+
20+
/* Configure the Python memory allocators. Pass NULL to use default
21+
allocators. */
22+
PyAPI_FUNC(int) _PyMem_SetupAllocators(const char *opt);
23+
24+
#ifdef WITH_PYMALLOC
25+
PyAPI_FUNC(int) _PyMem_PymallocEnabled(void);
1926
#endif
2027

28+
#endif /* !Py_LIMITED_API */
29+
2130

2231
/* BEWARE:
2332

‎Lib/test/test_capi.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
import random
77
import subprocess
88
import sys
9+
import sysconfig
910
import textwrap
1011
import time
1112
import unittest
@@ -521,6 +522,7 @@ def test_parse_tuple_and_keywords(self):
521522
self.assertRaises(ValueError, _testcapi.parse_tuple_and_keywords,
522523
(), {}, b'', [42])
523524

525+
524526
@unittest.skipUnless(threading, 'Threading required for this test.')
525527
class TestThreadState(unittest.TestCase):
526528

@@ -545,6 +547,7 @@ def callback():
545547
t.start()
546548
t.join()
547549

550+
548551
class Test_testcapi(unittest.TestCase):
549552
def test__testcapi(self):
550553
for name in dir(_testcapi):
@@ -553,5 +556,61 @@ def test__testcapi(self):
553556
test = getattr(_testcapi, name)
554557
test()
555558

559+
560+
class MallocTests(unittest.TestCase):
561+
ENV = 'debug'
562+
563+
def check(self, code):
564+
with support.SuppressCrashReport():
565+
out = assert_python_failure('-c', code, PYTHONMALLOC=self.ENV)
566+
stderr = out.err
567+
return stderr.decode('ascii', 'replace')
568+
569+
def test_buffer_overflow(self):
570+
out = self.check('import _testcapi; _testcapi.pymem_buffer_overflow()')
571+
regex = (r"Debug memory block at address p=0x[0-9a-f]+: API 'm'\n"
572+
r" 16 bytes originally requested\n"
573+
r" The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.\n"
574+
r" The 8 pad bytes at tail=0x[0-9a-f]+ are not all FORBIDDENBYTE \(0x[0-9a-f]{2}\):\n"
575+
r" at tail\+0: 0x78 \*\*\* OUCH\n"
576+
r" at tail\+1: 0xfb\n"
577+
r" at tail\+2: 0xfb\n"
578+
r" at tail\+3: 0xfb\n"
579+
r" at tail\+4: 0xfb\n"
580+
r" at tail\+5: 0xfb\n"
581+
r" at tail\+6: 0xfb\n"
582+
r" at tail\+7: 0xfb\n"
583+
r" The block was made by call #[0-9]+ to debug malloc/realloc.\n"
584+
r" Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb\n"
585+
r"Fatal Python error: bad trailing pad byte")
586+
self.assertRegex(out, regex)
587+
588+
def test_api_misuse(self):
589+
out = self.check('import _testcapi; _testcapi.pymem_api_misuse()')
590+
regex = (r"Debug memory block at address p=0x[0-9a-f]+: API 'm'\n"
591+
r" 16 bytes originally requested\n"
592+
r" The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.\n"
593+
r" The 8 pad bytes at tail=0x[0-9a-f]+ are FORBIDDENBYTE, as expected.\n"
594+
r" The block was made by call #[0-9]+ to debug malloc/realloc.\n"
595+
r" Data at p: .*\n"
596+
r"Fatal Python error: bad ID: Allocated using API 'm', verified using API 'r'\n")
597+
self.assertRegex(out, regex)
598+
599+
600+
class MallocDebugTests(MallocTests):
601+
ENV = 'malloc_debug'
602+
603+
604+
@unittest.skipUnless(sysconfig.get_config_var('WITH_PYMALLOC') == 1,
605+
'need pymalloc')
606+
class PymallocDebugTests(MallocTests):
607+
ENV = 'pymalloc_debug'
608+
609+
610+
@unittest.skipUnless(Py_DEBUG, 'need Py_DEBUG')
611+
class DefaultMallocDebugTests(MallocTests):
612+
ENV = ''
613+
614+
556615
if __name__ == "__main__":
557616
unittest.main()

‎Misc/NEWS‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,19 @@ Release date: tba
1010
Core and Builtins
1111
-----------------
1212

13+
- Issue #26516: Add :envvar`PYTHONMALLOC` environment variable to set the
14+
Python memory allocators and/or install debug hooks.
15+
16+
- Issue #26516: The :c:func`PyMem_SetupDebugHooks` function can now also be
17+
used on Python compiled in release mode.
18+
19+
- Issue #26516: The :envvar:`PYTHONMALLOCSTATS` environment variable can now
20+
also be used on Python compiled in release mode. It now has no effect if
21+
set to an empty string.
22+
23+
- Issue #26516: In debug mode, debug hooks are now also installed on Python
24+
memory allocators when Python is configured without pymalloc.
25+
1326
- Issue #26464: Fix str.translate() when string is ASCII and first replacements
1427
removes character, but next replacement uses a non-ASCII character or a
1528
string longer than 1 character. Regression introduced in Python 3.5.0.

‎Misc/README.valgrind‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@ This document describes some caveats about the use of Valgrind with
22
Python. Valgrind is used periodically by Python developers to try
33
to ensure there are no memory leaks or invalid memory reads/writes.
44

5+
UPDATE: Python 3.6 now supports PYTHONMALLOC=malloc environment variable which
6+
can be used to force the usage of the malloc() allocator of the C library.
7+
58
If you don't want to read about the details of using Valgrind, there
69
are still two things you must do to suppress the warnings. First,
710
you must use a suppressions file. One is supplied in

‎Modules/_testcapimodule.c‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3616,6 +3616,33 @@ get_recursion_depth(PyObject *self, PyObject *args)
36163616
return PyLong_FromLong(tstate->recursion_depth - 1);
36173617
}
36183618

3619+
static PyObject*
3620+
pymem_buffer_overflow(PyObject *self, PyObject *args)
3621+
{
3622+
char *buffer;
3623+
3624+
/* Deliberate buffer overflow to check that PyMem_Free() detects
3625+
the overflow when debug hooks are installed. */
3626+
buffer = PyMem_Malloc(16);
3627+
buffer[16] = 'x';
3628+
PyMem_Free(buffer);
3629+
3630+
Py_RETURN_NONE;
3631+
}
3632+
3633+
static PyObject*
3634+
pymem_api_misuse(PyObject *self, PyObject *args)
3635+
{
3636+
char *buffer;
3637+
3638+
/* Deliberate misusage of Python allocators:
3639+
allococate with PyMem but release with PyMem_Raw. */
3640+
buffer = PyMem_Malloc(16);
3641+
PyMem_RawFree(buffer);
3642+
3643+
Py_RETURN_NONE;
3644+
}
3645+
36193646

36203647
static PyMethodDef TestMethods[] = {
36213648
{"raise_exception", raise_exception, METH_VARARGS},
@@ -3798,6 +3825,8 @@ static PyMethodDef TestMethods[] = {
37983825
{"PyTime_AsMilliseconds", test_PyTime_AsMilliseconds, METH_VARARGS},
37993826
{"PyTime_AsMicroseconds", test_PyTime_AsMicroseconds, METH_VARARGS},
38003827
{"get_recursion_depth", get_recursion_depth, METH_NOARGS},
3828+
{"pymem_buffer_overflow", pymem_buffer_overflow, METH_NOARGS},
3829+
{"pymem_api_misuse", pymem_api_misuse, METH_NOARGS},
38013830
{NULL, NULL} /* sentinel */
38023831
};
38033832

‎Modules/main.c‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -93,14 +93,15 @@ static const char usage_5[] =
9393
" The default module search path uses %s.\n"
9494
"PYTHONCASEOK : ignore case in 'import' statements (Windows).\n"
9595
"PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n"
96-
"PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n\
97-
";
98-
static const char usage_6[] = "\
99-
PYTHONHASHSEED: if this variable is set to 'random', a random value is used\n\
100-
to seed the hashes of str, bytes and datetime objects. It can also be\n\
101-
set to an integer in the range [0,4294967295] to get hash values with a\n\
102-
predictable seed.\n\
103-
";
96+
"PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n";
97+
static const char usage_6[] =
98+
"PYTHONHASHSEED: if this variable is set to 'random', a random value is used\n"
99+
" to seed the hashes of str, bytes and datetime objects. It can also be\n"
100+
" set to an integer in the range [0,4294967295] to get hash values with a\n"
101+
" predictable seed.\n"
102+
"PYTHONMALLOC: set the Python memory allocators and/or install debug hooks\n"
103+
" on Python memory allocators. Use PYTHONMALLOC=debug to install debug\n"
104+
" hooks.\n";
104105

105106
static int
106107
usage(int exitcode, const wchar_t* program)
@@ -341,6 +342,7 @@ Py_Main(int argc, wchar_t **argv)
341342
int help = 0;
342343
int version = 0;
343344
int saw_unbuffered_flag = 0;
345+
char *opt;
344346
PyCompilerFlags cf;
345347
PyObject *warning_option = NULL;
346348
PyObject *warning_options = NULL;
@@ -365,6 +367,13 @@ Py_Main(int argc, wchar_t **argv)
365367
}
366368
}
367369

370+
opt = Py_GETENV("PYTHONMALLOC");
371+
if (_PyMem_SetupAllocators(opt) < 0) {
372+
fprintf(stderr,
373+
"Error in PYTHONMALLOC: unknown allocator \"%s\"!\n", opt);
374+
exit(1);
375+
}
376+
368377
Py_HashRandomizationFlag = 1;
369378
_PyRandom_Init();
370379

0 commit comments

Comments
 (0)
X Tutup