ASan uses a very large amount of virtual memory to do its bookkeeping (20TB). When running the gcore command, GDB attempts to include the shadow memory of the process causing the core dump to reach the amount of available disk space before failing.

[2017/06/18 23:02:05.552] ubuntu   29054  1.9  0.6 21476345012 188920 ?  Sl   23:01   0:00 ./mongod --setParameter enableTestCommands=1 --setParameter enableLocalhostAuthBypass=false --dbpath /data/db/job0/resmoke --nopreallocj --auth --storageEngine mmapv1 --keyFile jstests/libs/authTestsKey --port 20000
...
[2017/06/19 01:16:11.226] ubuntu   29054  0.0  1.7 21476342668 533232 ?  tl   Jun18   0:06 ./mongod --setParameter enableTestCommands=1 --setParameter enableLocalhostAuthBypass=false --dbpath /data/db/job0/resmoke --nopreallocj --auth --storageEngine mmapv1 --keyFile jstests/libs/authTestsKey --port 20000

Based on my reading of https://sourceware.org/bugzilla/show_bug.cgi?id=16092, https://sourceware.org/ml/gdb-patches/2015-03/msg00144.html, and the changes in https://reviews.llvm.org/D7294?id=19035, I think GDB should already be respecting the madvise() settings when it goes to take a core dump.

However, until we are able to determine if there's different bits to set to modifies GDB's behavior, we should run hang_analyzer.py without specifying -c on the following build variants that compile with --sanitize=address:

• ubuntu1604-asan
• ubuntu1604-debug-asan
• ubuntu1604-debug-aubsan-lite

This can be accomplished by setting a new hang_analyzer_dump_core=false expansion on the aforementioned build variants and defaulting the expansion to true if it is unset.