As suggested by Thomas Schubert in #SERVER-24709 I have created a separate issue:

I am trying to create a complete dump of my database. This always fails after some hours because of a memory problem. The database to be dumped contains 3 large collections with documents of all size and grid.fs files of all size.

The crash does happen approximately after the same run-time - usually when the gzipped bson dump is between 600 and 650GiB.

The storage engine I use is MMAPv1.

db version v3.4.3
git version: f07437fb5a6cca07c10bafa78365456eb1d6d5e1
OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1604
distarch: x86_64
target_arch: x86_64

The VM has 3.9GB RAM. While dumping the database no other database operations were running. The system has 2GB (nearly unused) swap.

I a second try I changed the OOM settings of Ubuntu which ended up that I had to reboot the complete VM because Mongo did consume every single bit of memory - even a local logon session was immediately killed by the OOM. Even stopping the dump process did not recover the VM.

I have collected logs and diagnostic data with enabled heapProfilingEnabled=true. Where should I upload them?