[SERVER-29688] ASAN failures should produce core dumps Created: 16/Jun/17  Updated: 30/Oct/23  Resolved: 13/Jul/17

Status: Closed
Project: Core Server
Component/s: Build, Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.11

Type: Bug Priority: Major - P3
Reporter: Spencer Jackson Assignee: Spencer Jackson
Resolution: Fixed Votes: 0
Labels: neweng, platforms-interns-2017
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Platforms 2017-07-10, Platforms 2017-07-31
Participants:

 Description   

This would help debug ASAN failures.

The following ASAN_OPTIONS are needed:
abort_on_error=1:disable_coredump=0

The following environment variable also useful to get addresses of leaked objects:
LSAN_OPTIONS=report_objects=1



 Comments   
Comment by Githook User [ 13/Jul/17 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-29688: Coredump on Sanizitizer failure
Branch: master
https://github.com/mongodb/mongo/commit/26a2c5ded2cd5adbd3f18e460dc61f7226a51573

Comment by Spencer Jackson [ 19/Jun/17 ]

The core also compresses down very nicely with bzip. Core + logs wound up being 38M.

Comment by Spencer Jackson [ 19/Jun/17 ]

Ran the following repro script:

ASAN_OPTIONS=detect_leaks=1:external_symbolizer_path=/opt/mongodbtoolchain/v2/bin/llvm-symbolizer:abort_on_error=1:disable_coredump=0 LSAN_OPTIONS=report_objects=1 python2 buildscripts/resmoke.py --executor sharding_auth_audit --storageEngine=wiredTiger jstests/sharding/commands_that_write_accept_wc_shards.js --repeat 10000 > log4

It produced the following ASAN log:

[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.085+0000 c20018| =================================================================                          
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.085+0000 c20018| ==29079==ERROR: LeakSanitizer: detected memory leaks    
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.085+0000 c20018|                                                         
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.085+0000 c20018| Direct leak of 144 byte(s) in 1 object(s) allocated from:                                  
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.086+0000 c20018|     #0 0x55976d60346b in operator new(unsigned long) /data/mci/be9af688d396a09d5ad1b3be40387f7c/toolchain-builder/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:78:35                       
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.086+0000 c20018|     #1 0x559770f20426 in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<mongo::executor::ThreadPoolTaskExecutor::CallbackState, std::allocator<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/ext/new_allocator.h:104:27
 
...
 
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.105+0000 c20018|     #32 0x559771cd88bf in execute_native_thread_routine /data/mci/be9af688d396a09d5ad1b3be40387f7c/toolchain-builder/build-gcc-v2.sh-aBl/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84                         
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.105+0000 c20018|                                                         
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.105+0000 c20018| Objects leaked above:                                   
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.105+0000 c20018| 0x60d000064760 (144 bytes)                              
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.106+0000 c20018|                                                         
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.106+0000 c20018| SUMMARY: AddressSanitizer: 144 byte(s) leaked in 1 allocation(s).                          
[js_test:commands_that_write_accept_wc_shards] 2017-06-17T02:46:12.106+0000 c20018| 2017-06-17T02:46:12.100+0000 F -        [signalProcessingThread] Got signal: 6 (Aborted).

This made the following core:

ubuntu@ip-10-137-149-240:/data/mci/artifacts-dbfe5d-ubuntu1604-debug-asan_compile$ ls -lh *.core                                            
-rw------- 1 ubuntu ubuntu 851M Jun 17 02:46 dump_signalP.gThread.29079.core

I had set the coredump_filter to 0x13, and core_pattern was the default.

Comment by Max Hirschhorn [ 16/Jun/17 ]

spencer.jackson, do you know if ASan will respect the core dump filter set on the system or how large (as well as naming convention) we should anticipate these core dumps to be?

if [ -f /proc/self/coredump_filter ]; then
  # Set the shell process (and its children processes) to dump ELF headers (bit 4),
  # anonymous shared mappings (bit 1), and anonymous private mappings (bit 0).
  echo 0x13 > /proc/self/coredump_filter
 
  if [ -f /sbin/sysctl ]; then
    # Check that the core pattern is set explicitly on our distro image instead
    # of being the OS's default value. This ensures that coredump names are consistent
    # across distros and can be picked up by Evergreen.
    core_pattern=$(/sbin/sysctl -n "kernel.core_pattern")
    if [ "$core_pattern" = "dump_%e.%p.core" ]; then
      echo "Enabling coredumps"
      ulimit -c unlimited
    fi
  fi
fi

Generated at Thu Feb 08 04:21:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.