Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7839

Add evergreen task to run python tests with ASAN enabled

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: None
    • Labels:

      Description

      The WiredTiger python tests can be run with ASAN:

      • Build WT with ASAN enabled
      • Use the --asan option to run.py when running tests

      To improve the coverage of our automated tests, we should add an evergreen task that runs the python suite with ASAN.

      One tricky detail is that ASAN currently reports various memory leaks that appear to come from Python itself (or possibly our Python SWIG library?).  We will need to either eliminate these errors or recognize and ignore them.

      For reference, here is an example of a few of those memory leaks currently reported by ASAN:

      Direct leak of 1024 byte(s) in 1 object(s) allocated from:
          #0 0x7fb30a707f83 in malloc /data/mci/f6a3cdb8925fada697dec19875e037b4/toolchain-builder/tmp/build-llvm.sh-cpH/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
          #1 0x7fb309a31c35 in _PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:99:12
          #2 0x7fb309a31c35 in PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:572:12
          #3 0x7fb309a31c35 in _PyObject_Malloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:1645:11
          #4 0x7fb309a31c35 in PyMem_Malloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:605:12
          #5 0x7fb309a31c35 in set_table_resize /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/setobject.c:260:20
       
       
      Direct leak of 128 byte(s) in 4 object(s) allocated from:
          #0 0x7fb30a707f83 in malloc /data/mci/f6a3cdb8925fada697dec19875e037b4/toolchain-builder/tmp/build-llvm.sh-cpH/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
          #1 0x7fb309a11537 in _PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:99:12
          #2 0x7fb309a11537 in PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:572:12
          #3 0x7fb309a11537 in PyThread_allocate_lock.localalias.2 /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Python/thread_pthread.h:380:21
       
       
      Indirect leak of 199200 byte(s) in 209 object(s) allocated from:
          #0 0x7fb30a707f83 in malloc /data/mci/f6a3cdb8925fada697dec19875e037b4/toolchain-builder/tmp/build-llvm.sh-cpH/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
          #1 0x7fb309a22cf0 in _PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:99:12
          #2 0x7fb309a22cf0 in PyMem_RawMalloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:572:12
          #3 0x7fb309a22cf0 in _PyObject_Malloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:1645:11
          #4 0x7fb309a22cf0 in PyObject_Malloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/obmalloc.c:685:12
          #5 0x7fb309a22cf0 in _PyObject_GC_Alloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Modules/gcmodule.c:2225:26
          #6 0x7fb309a22cf0 in _PyObject_GC_Malloc /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Modules/gcmodule.c:2252:12
          #7 0x7fb309a22cf0 in PyType_GenericAlloc.localalias.10 /data/mci/78f810c8dfd4519e8edccff6788f14f0/toolchain-builder/tmp/build-python.sh-pQV/build-Python-3.9.2/../src/Python-3.9.2/Objects/typeobject.c:1047:15

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              backlog-server-storage-engines Backlog - Storage Engines Team
              Reporter:
              keith.smith Keith Smith
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated: