Here is a sample of the messages:

[1441926873:563928][13434:0x7faad4c5b700], t, file:wt.wt, WT_SESSION.checkpoint: WT_PAGE.memory_footprint underflow: decrementing 89295

[1441926882:415399][13434:0x7faadaaf5740], t, file:wt.wt, WT_SESSION.verify: WT_CACHE.bytes_inmem underflow: decrementing 110488

[1441926917:912321][13434:0x7faadaaf5740], t, log-server: cache server: exiting with 21193 bytes in memory

Since this is "informational only" it does not exit and I have no idea what the CONFIG was for any of those runs. Tomorrow I'll restart so that it will fail and we can capture a CONFIG.

I added panic calls if those messages hit and hit one finally on the SSD box. Here is its CONFIG.

############################################

#  RUN PARAMETERS

############################################

abort=0

auto_throttle=1

firstfit=0

bitcnt=1

bloom=1

bloom_bit_count=46

bloom_hash_count=12

bloom_oldest=0

cache=71

checkpoints=1

checksum=uncompressed

chunk_size=3

compaction=0

compression=none

data_extend=0

data_source=table

delete_pct=32

dictionary=1

encryption=none

evict_max=0

file_type=variable-length column-store

backups=0

huffman_key=0

huffman_value=0

insert_pct=26

internal_key_truncation=1

internal_page_max=14

isolation=random

key_gap=8

key_max=78

key_min=25

leak_memory=0

leaf_page_max=12

logging=1

logging_compression=none

logging_archive=0

logging_prealloc=1

long_running_txn=0

lsm_worker_threads=3

merge_max=6

mmap=1

ops=1812711

prefix_compression=1

prefix_compression_min=6

repeat_data_pct=86

reverse=0

rows=253249

runs=100

salvage=1

split_pct=41

statistics=0

statistics_server=0

threads=32

timer=5

value_max=3038

value_min=12

verify=1

wiredtiger_config=

write_pct=27

############################################

This CONFIG has reproduced the messages in 3 out of 5 runs.

FTR, I added an abort when we print the underflow messages. (On my AWS HDD box it took about 15 iterations to fail - much slower than the SSD). This CONFIG failure gives this stack:

(gdb) bt

#0  0x00007f0a4469ebe9 in raise () from /lib64/libc.so.6

#1  0x00007f0a4469ffe8 in abort () from /lib64/libc.so.6

#2  0x00000000004b63dd in __wt_cache_decr_check_size (session=0x9ae2e0, 

    vp=0x2a5ce48, v=94400, fld=0x52a239 "WT_PAGE.memory_footprint")

    at ../src/include/btree.i:85

#3  0x00000000004b65a0 in __wt_cache_page_inmem_decr (session=0x9ae2e0, 

    page=0x2a5ce10, size=94400) at ../src/include/btree.i:185

#4  0x00000000004b7969 in __wt_update_obsolete_free (session=0x9ae2e0, 

    page=0x2a5ce10, upd=0x0) at ../src/btree/row_modify.c:352

#5  0x00000000004b17a1 in __wt_update_serial (session=0x9ae2e0, 

    page=0x2a5ce10, srch_upd=0x7f09dc04c340, updp=0x7f0a30ff0ba8, upd_size=64)

    at ../src/include/serial.i:325

#6  0x00000000004b1a32 in __wt_col_modify (session=0x9ae2e0, 

    cbt=0x7f09cc17af90, recno=38000, value=0x7f09cc17b090, upd_arg=0x0, 

    is_remove=false) at ../src/btree/col_modify.c:102

#7  0x00000000005054b9 in __cursor_col_modify (session=0x9ae2e0, 

    cbt=0x7f09cc17af90, is_remove=false) at ../src/btree/bt_cursor.c:254

#8  0x000000000050722e in __wt_btcur_update (cbt=0x7f09cc17af90)

    at ../src/btree/bt_cursor.c:785

#9  0x00000000004c9c43 in __curfile_update (cursor=0x7f09cc17af90)

    at ../src/cursor/cur_file.c:284

#10 0x00000000004073ff in col_update (tinfo=0x2b8dbf0, cursor=0x7f09cc17af90, 

    key=0x7f0a30ff0e70, value=0x7f0a30ff0e40, keyno=38000)

    at ../../../test/format/ops.c:888

#11 0x0000000000406396 in ops (arg=0x2b8dbf0) at ../../../test/format/ops.c:471

#12 0x00007f0a44e2df18 in start_thread () from /lib64/libpthread.so.0

#13 0x00007f0a4474db9d in clone () from /lib64/libc.so.6

Author:

Message: WT-2099 Account for overflow update structures that get appended to the
update list in page->memory_footprint.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/c9d932fe5b49c88b1b96c37c5008969fdbe1d72a

Author:

Message: Merge pull request #2382 from wiredtiger/memory-underflow

WT-2099 Account for overflow update structures
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/134a9dfa157f64fc58d140850e53d6505147866d

Author:

Message: Import wiredtiger-wiredtiger-2.7.0-269-g44463c5.tar.gz from wiredtiger branch mongodb-3.4

ref: 3c2ad56..44463c5

SERVER-21833 Compact does not release space to the system with WiredTiger
WT-2060 Simplify aggregation of statistics
WT-2099 Seeing memory underflow messages
WT-2113 truncate01 sometimes fails
WT-2177 Add a per-thread seed to random number generator
WT-2198 bulk load and column store appends
WT-2231 pinned page cursor searches could check parent keys
WT-2235 wt printlog option without unicode
WT-2245 WTPERF Truncate has no ability to catch up when it falls behind
WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations
WT-2256 WTPERFs throttle option fires in bursts
WT-2257 wtperf doesn't handle overriding workload config
WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error
WT-2260 Workloads evict internal pages unexpectedly
WT-2262 Random sampling is skewed by tree shape
WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h
WT-2266 Add wtperf config to set if perf thresholds are fatal
WT-2269 wtperf should dump its config everytime it runs
WT-2272 Stress test assertion in the sweep server
WT-2275 broken DB after application crash
WT-2276 tool to decode checkpoint addr
WT-2277 Remove WT check against big-endian systems
WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined
WT-2281 wtperf smoke.sh fails on ppc64le
WT-2282 error in wt_txn_update_oldest verbose message test
WT-2283 retry in txn_update_oldest results in a hang
WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux
WT-2289 failure in fast key check
WT-2290 WT_SESSION.compact could be more effective.
WT-2291 Random cursor walk inefficient in skip list only trees
WT-2297 Fix off-by-one error in Huffman config file parsing
WT-2299 upper-level WiredTiger code is reaching into the block manager
WT-2301 Add reading a range to wtperf
WT-2303 Build warning in wtperf
WT-2304 wtperf crash dumping config
WT-2307 Internal page splits can corrupt cursor iteration
WT-2311 Support Sparc
Branch: master
https://github.com/mongodb/mongo/commit/d845b75e5f0837f801bdf371babd985308a1ad80

Author:

Message: Import wiredtiger-wiredtiger-2.7.0-559-g07966a4.tar.gz from wiredtiger branch mongodb-3.2

ref: 3c2ad56..07966a4

WT-1517 schema format edge cases
WT-1801 Add a directory sync after rollback of a WT_SESSION::rename operation
WT-2060 Simplify aggregation of statistics
WT-2073 metadata cleanups
WT-2099 Seeing memory underflow messages
WT-2113 truncate01 sometimes fails
WT-2142 Connection cleanup in Python tests
WT-2177 Add an optional per-thread seed to random number generator
WT-2198 bulk load and column store appends
WT-2216 simplify row-store search loop slightly
WT-2225 New split code performance impact
WT-2231 pinned page cursor searches could check parent keys
WT-2235 wt printlog option without unicode
WT-2242 WiredTiger treats dead trees the same as other trees in eviction
WT-2244 Trigger in-memory splits sooner
WT-2245 WTPERF Truncate has no ability to catch up when it falls behind
WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations
WT-2247 variable-length column-store in-memory page splits
WT-2256 WTPERFs throttle option fires in bursts
WT-2257 wtperf doesn't handle overriding workload config
WT-2258 WiredTiger preloads pages even when direct-IO is configured.
WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error
WT-2260 Workloads evict internal pages unexpectedly
WT-2262 Random sampling is skewed by tree shape
WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h
WT-2266 Add wtperf config to set if perf thresholds are fatal
WT-2267 Improve wtperf throttling implementation to provide steady load
WT-2269 wtperf should dump its config everytime it runs
WT-2272 Stress test assertion in the sweep server
WT-2275 broken DB after application crash
WT-2276 tool to decode checkpoint addr
WT-2277 Remove WT check against big-endian systems
WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined
WT-2281 wtperf smoke.sh fails on ppc64le
WT-2282 error in wt_txn_update_oldest verbose message test
WT-2283 retry in txn_update_oldest results in a hang
WT-2284 Repeated macro definition
WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux
WT-2287 WT_SESSION.rebalance
WT-2289 failure in fast key check
WT-2290 WT_SESSION.compact could be more effective.
WT-2291 Random cursor walk inefficient in skip list only trees
WT-2295 WT_SESSION.create does a full-scan of the main table
WT-2296 New log algorithm needs improving for sync/flush settings
WT-2297 Fix off-by-one error in Huffman config file parsing
WT-2299 upper-level WiredTiger code is reaching into the block manager
WT-2301 Add reading a range to wtperf
WT-2303 Build warning in wtperf
WT-2304 wtperf crash dumping config
WT-2305 Fix coverity scan issues on 23/12/2015
WT-2307 Internal page splits can corrupt cursor iteration
WT-2308 custom extractor for ref_cursors in join cursor
WT-2311 Support Sparc
WT-2312 re-creating a deleted column-store page can corrupt the in-memory tree
WT-2313 sweep-server: conn_dhandle.c, 610: dhandle != conn->cache->evict_file_next
WT-2314 page-swap error handling is inconsistent
WT-2316 stress test failure: WT_CURSOR.prev out-of-order returns
WT-2320 Only check copyright when cutting releases
WT-2321 WT-2321: race between eviction and worker threads on the eviction queue
WT-2326 Change WTPERF to use new memory allocation functions instead of the standard
WT-2328 schema drop does direct unlink, it should use a block manager interface.
WT-2331 Checking of search() result for reference cursors before join()
WT-2332 Bug in logging write-no-sync mode
WT-2333 Add a flag so drop doesn't block
WT-2335 NULL pointer crash in config_check_search with invalid configuration string
WT-2338 Disable using pre-allocated log files when backup cursor is open
WT-2339 format post-rebalance verify failure (stress run #11586)
WT-2340 Add logging guarantee assertions, whitespace
WT-2342 Enhance wtperf to support background create and drop operations
WT-2344 OS X compiler warning
WT-2347 Java: schema format edge cases
WT-2348 xargs -P isn't portable
WT-2355 Fix minor scratch buffer usage in logging
SERVER-21833 Compact does not release space to the system with WiredTiger
SERVER-21887 $sample takes disproportionately long time on newly created collection
SERVER-22064 Coverity analysis defect 77699: Unchecked return value
SERVER-21944 WiredTiger changes for 3.2.2
Branch: v3.2
https://github.com/mongodb/mongo/commit/5d6532f3d5227ff76f62c4810c98a4ef4d0c8c56