Loading...

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: 3.0.2, 3.1.1
Affects Version/s: 3.0.0
Component/s: WiredTiger
Labels:
None

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Backport Completed:

3.0.2

Issue Status as of Apr 02, 2015

ISSUE SUMMARY
MongoDB running with the WiredTiger storage engine may hit a default limit and terminate with the following error message:

eviction-server: session 0x2b1a2c0: hazard pointer table full

The default limit for the number of hazard pointers is 1000, which may be hit if the database has a large number of collections and indexes.

USER IMPACT
The mongod process terminates and needs to be restarted.

WORKAROUNDS
Users affected by this issue can increase default limit for the size of the hazard pointer table to, for example, 10000, by using the following command line option:

--wiredTigerEngineConfigString="hazard_max=10000".

AFFECTED VERSIONS
MongoDB stable releases 3.0.0 and 3.0.1 are affected by this issue.

FIX VERSION
The fix is included in the 3.0.2 production release.

Original description

We're in the process of migrating our test environment to MongoDB 3.0. We spun up a new instance with 3.0, added it to the replica set and let it sync.
However, a few hours into the process it crashed with the following error:

2015-03-10T19:59:46.037+0000 I -        [repl writer worker 11]   Index Build: 196300/590825 33%
2015-03-10T19:59:46.272+0000 E STORAGE  WiredTiger (0) [1426017586:270981][3822:0x7f38f7a89700], file:index-3720--4215350639811228100.wt, eviction-server: session 0x2ffe2c0: hazard pointer table full
2015-03-10T19:59:46.274+0000 E STORAGE  WiredTiger (12) [1426017586:272380][3822:0x7f38f7a89700], eviction-server: cache eviction server error: Cannot allocate memory
2015-03-10T19:59:46.274+0000 E STORAGE  WiredTiger (-31804) [1426017586:274241][3822:0x7f38f7a89700], eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-03-10T19:59:46.274+0000 I -        Fatal Assertion 28558
2015-03-10T19:59:46.274+0000 I -        [repl writer worker 11] Fatal Assertion 28559
2015-03-10T19:59:46.288+0000 I NETWORK  [initandlisten] connection accepted from X.Y.Z.X:58398 #1900 (5 connections now open)
2015-03-10T19:59:46.308+0000 I NETWORK  [conn1897] end connection X.Y.Z.Y:58395 (4 connections now open)
2015-03-10T19:59:46.329+0000 I CONTROL
 0xf5b229 0xefba41 0xee0de1 0xd71e16 0x1382a50 0x1382d15 0x13831b1 0x133a99f 0x7f38f9abfe9a 0x7f38f85962ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5B229"},{"b":"400000","o":"AFBA41"},{"b":"400000","o":"AE0DE1"},{"b":"400000","o":"971E16"},{"b":"400000","o":"F82A50"},{"b":"400000","o":"F82D15"},{"b":"400000","o":"F831B1"},{"b":"400000","o":"F3A99F"},{"b":"7F38F9AB8000","o":"7E9A"},{"b":"7F38F84A2000","o":"F42ED"}],"processInfo":{ "mongodbVersion" : "3.0.0", "gitVersion" : "a841fd6394365954886924a35076691b4d149168", "uname" : { "sysname" : "Linux", "release" : "3.2.0-75-virtual", "version" : "#110-Ubuntu SMP Tue Dec 16 19:24:01 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5CFC121279316F2BC7FBB39F8268284EDF47E708" }, { "b" : "7FFF9A6FF000", "elfType" : 3, "buildId" : "975615E6DDB0F811039107F3F95DB4884097BE36" }, { "b" : "7F38F9AB8000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7F38F985A000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "E62302CCF011B0D1277B07E2B6170B678480F996" }, { "b" : "7F38F947F000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "DBF958ED97EEE7EAD42EC0F24EDB2E39ACFA2B83" }, { "b" : "7F38F9277000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7F38F9073000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7F38F8D73000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7F38F8A77000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7F38F8861000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7F38F84A2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "DDD4987072B438ECB112023E07E432844CF89076" }, { "b" : "7F38F9CD5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" }, { "b" : "7F38F828B000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5b229]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefba41]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xee0de1]
 mongod(+0x971E16) [0xd71e16]
 mongod(+0xF82A50) [0x1382a50]
 mongod(__wt_err+0x95) [0x1382d15]
 mongod(__wt_panic+0x21) [0x13831b1]
 mongod(+0xF3A99F) [0x133a99f]
 libpthread.so.0(+0x7E9A) [0x7f38f9abfe9a]
 libc.so.6(clone+0x6D) [0x7f38f85962ed]
-----  END BACKTRACE  -----
2015-03-10T19:59:46.329+0000 I -

***aborting after fassert() failure

The database was running on an AWS instance with 7.5GB RAM, using the default WiredTiger config. We attempted again, on a larger instance (16GB RAM), but it still fails:

2015-03-11T13:18:44.503+0000 E STORAGE  WiredTiger (0) [1426079924:501160][892:0x7f9799de0700], file:index-2117-6998163636744330624.wt, eviction-server: session 0x2b1a2c0: hazard pointer table full
2015-03-11T13:18:44.504+0000 E STORAGE  WiredTiger (12) [1426079924:503483][892:0x7f9799de0700], eviction-server: cache eviction server error: Cannot allocate memory
2015-03-11T13:18:44.504+0000 E STORAGE  WiredTiger (-31804) [1426079924:504371][892:0x7f9799de0700], eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-03-11T13:18:44.505+0000 I -        Fatal Assertion 28558
2015-03-11T13:18:44.506+0000 E STORAGE  WiredTiger (-31804) [1426079924:506395][892:0x7f9797ddc700], session.checkpoint: metadata unroll update file:collection-1552-6998163636744330624.wt to allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy,cache_resident=0,checkpoint=(WiredTigerCheckpoint.2=(addr="01c1e581e40bf68bf8c1e681e44df8a873c1e781e4c3e687b3808080e3226fc0e3217fc0",order=2,time=1426079809,size=2203648,write_gen=113)),checkpoint_lsn=(12784,6111872),checksum=uncompressed,collator=,columns=,dictionary=0,format=btree,huffman_key=,huffman_value=,id=14826,internal_item_max=0,internal_key_max=0,internal_key_truncate=,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=1MB,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=0,prefix_compression_min=4,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,value_format=u,version=(major=1,minor=1): WT_PANIC: WiredTiger library panic
2015-03-11T13:18:44.506+0000 I -        Fatal Assertion 28558
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 9] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 10] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 1] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 11] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 15] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 3] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 6] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 4] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 12] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 2] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 0] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 13] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 14] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 5] Fatal Assertion 28559
2015-03-11T13:18:44.506+0000 I -        [repl writer worker 8] Fatal Assertion 28559
2015-03-11T13:18:44.507+0000 I -        [repl writer worker 7] Fatal Assertion 28559
2015-03-11T13:18:44.591+0000 I CONTROL  [repl writer worker 0]
 0xf5b229 0xefba41 0xee0de1 0xd88fd9 0xd7d6b4 0xd7d70f 0xd7d775 0xd6c725 0xa9c8c5 0xa260c0 0xa522bc 0xbe50c4 0xbe5474 0xbe5ffd 0xb37531 0xc4e655 0xca4dfe 0xca78f8 0xef2a5b 0xfa7ce4 0x7f979be16e9a 0x7f979a8ed2ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5B229"},{"b":"400000","o":"AFBA41"},{"b":"400000","o":"AE0DE1"},{"b":"400000","o":"988FD9"},{"b":"400000","o":"97D6B4"},{"b":"400000","o":"97D70F"},{"b":"400000","o":"97D775"},{"b":"400000","o":"96C725"},{"b":"400000","o":"69C8C5"},{"b":"400000","o":"6260C0"},{"b":"400000","o":"6522BC"},{"b":"400000","o":"7E50C4"},{"b":"400000","o":"7E5474"},{"b":"400000","o":"7E5FFD"},{"b":"400000","o":"737531"},{"b":"400000","o":"84E655"},{"b":"400000","o":"8A4DFE"},{"b":"400000","o":"8A78F8"},{"b":"400000","o":"AF2A5B"},{"b":"400000","o":"BA7CE4"},{"b":"7F979BE0F000","o":"7E9A"},{"b":"7F979A7F9000","o":"F42ED"}],"processInfo":{ "mongodbVersion" : "3.0.0", "gitVersion" : "a841fd6394365954886924a35076691b4d149168", "uname" : { "sysname" : "Linux", "release" : "3.2.0-75-virtual", "version" : "#110-Ubuntu SMP Tue Dec 16 19:24:01 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5CFC121279316F2BC7FBB39F8268284EDF47E708" }, { "b" : "7FFFDEC64000", "elfType" : 3, "buildId" : "975615E6DDB0F811039107F3F95DB4884097BE36" }, { "b" : "7F979BE0F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7F979BBB1000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "E62302CCF011B0D1277B07E2B6170B678480F996" }, { "b" : "7F979B7D6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "DBF958ED97EEE7EAD42EC0F24EDB2E39ACFA2B83" }, { "b" : "7F979B5CE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7F979B3CA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7F979B0CA000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7F979ADCE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7F979ABB8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7F979A7F9000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "DDD4987072B438ECB112023E07E432844CF89076" }, { "b" : "7F979C02C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" }, { "b" : "7F979A5E2000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5b229]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefba41]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xee0de1]
 mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x309) [0xd88fd9]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit8_txnOpenEPNS_16OperationContextE+0x124) [0xd7d6b4]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit10getSessionEPNS_16OperationContextE+0x1F) [0xd7d70f]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x35) [0xd7d775]
 mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEi+0x55) [0xd6c725]
 mongod(_ZNK5mongo22BtreeBasedAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xa9c8c5]
 mongod(_ZN5mongo11IDHackStage4workEPm+0xD0) [0xa260c0]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x7C) [0xa522bc]
 mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbe50c4]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbe5474]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x3D) [0xbe5ffd]
 mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x111) [0xb37531]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbb+0x12B5) [0xc4e655]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0x2EE) [0xca4dfe]
 mongod(_ZN5mongo4repl21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x78) [0xca78f8]
 mongod(_ZN5mongo10threadpool6Worker4loopERKSs+0x2FB) [0xef2a5b]
 mongod(+0xBA7CE4) [0xfa7ce4]
 libpthread.so.0(+0x7E9A) [0x7f979be16e9a]
 libc.so.6(clone+0x6D) [0x7f979a8ed2ed]
-----  END BACKTRACE  -----
2015-03-11T13:18:44.591+0000 I CONTROL  [repl writer worker 6]
 0xf5b229 0xefba41 0xee0de1 0xd88fd9 0xd7d6b4 0xd7d70f 0xd7d775 0xd6c725 0xa9c8c5 0xa260c0 0xa522bc 0xbe50c4 0xbe5474 0xbe5ffd 0xb37531 0xc4e655 0xca4dfe 0xca78f8 0xef2a5b 0xfa7ce4 0x7f979be16e9a 0x7f979a8ed2ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5B229"},{"b":"400000","o":"AFBA41"},{"b":"400000","o":"AE0DE1"},{"b":"400000","o":"988FD9"},{"b":"400000","o":"97D6B4"},{"b":"400000","o":"97D70F"},{"b":"400000","o":"97D775"},{"b":"400000","o":"96C725"},{"b":"400000","o":"69C8C5"},{"b":"400000","o":"6260C0"},{"b":"400000","o":"6522BC"},{"b":"400000","o":"7E50C4"},{"b":"400000","o":"7E5474"},{"b":"400000","o":"7E5FFD"},{"b":"400000","o":"737531"},{"b":"400000","o":"84E655"},{"b":"400000","o":"8A4DFE"},{"b":"400000","o":"8A78F8"},{"b":"400000","o":"AF2A5B"},{"b":"400000","o":"BA7CE4"},{"b":"7F979BE0F000","o":"7E9A"},{"b":"7F979A7F9000","o":"F42ED"}],"processInfo":{ "mongodbVersion" : "3.0.0", "gitVersion" : "a841fd6394365954886924a35076691b4d149168", "uname" : { "sysname" : "Linux", "release" : "3.2.0-75-virtual", "version" : "#110-Ubuntu SMP Tue Dec 16 19:24:01 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5CFC121279316F2BC7FBB39F8268284EDF47E708" }, { "b" : "7FFFDEC64000", "elfType" : 3, "buildId" : "975615E6DDB0F811039107F3F95DB4884097BE36" }, { "b" : "7F979BE0F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7F979BBB1000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "E62302CCF011B0D1277B07E2B6170B678480F996" }, { "b" : "7F979B7D6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "DBF958ED97EEE7EAD42EC0F24EDB2E39ACFA2B83" }, { "b" : "7F979B5CE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7F979B3CA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7F979B0CA000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7F979ADCE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7F979ABB8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7F979A7F9000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "DDD4987072B438ECB112023E07E432844CF89076" }, { "b" : "7F979C02C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" }, { "b" : "7F979A5E2000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5b229]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefba41]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xee0de1]
 mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x309) [0xd88fd9]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit8_txnOpenEPNS_16OperationContextE+0x124) [0xd7d6b4]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit10getSessionEPNS_16OperationContextE+0x1F) [0xd7d70f]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x35) [0xd7d775]
 mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEi+0x55) [0xd6c725]
 mongod(_ZNK5mongo22BtreeBasedAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xa9c8c5]
 mongod(_ZN5mongo11IDHackStage4workEPm+0xD0) [0xa260c0]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x7C) [0xa522bc]
 mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbe50c4]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbe5474]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x3D) [0xbe5ffd]
 mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x111) [0xb37531]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbb+0x12B5) [0xc4e655]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0x2EE) [0xca4dfe]
 mongod(_ZN5mongo4repl21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x78) [0xca78f8]
 mongod(_ZN5mongo10threadpool6Worker4loopERKSs+0x2FB) [0xef2a5b]
 mongod(+0xBA7CE4) [0xfa7ce4]
 libpthread.so.0(+0x7E9A) [0x7f979be16e9a]
 libc.so.6(clone+0x6D) [0x7f979a8ed2ed]
-----  END BACKTRACE  -----
2015-03-11T13:18:44.591+0000 I -        [repl writer worker 0]

***aborting after fassert() failure


2015-03-11T13:18:44.591+0000 I -        [repl writer worker 6]

***aborting after fassert() failure

The database size on the 3.0.0 machine was ~50GB at this point. The master has ~220GB of data, but it's hard to know how far it had synced as it's using Snappy compression on the new node.

is depended on by

WT-1901 Issues resolved in WiredTiger 2.5.3

Closed

is duplicated by

SERVER-17614 WT invariant crash with "hazard pointer table full" error

Closed

SERVER-17814 eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic

Closed

Details

Description

Original description

Attachments

Issue Links

Activity

People

Dates