2015-07-30T19:55:50.488+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=30G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=10 0000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-07-30T19:55:50.540+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] MongoDB starting : pid=2537 port=27017 dbpath=/data/mongo 64-bit host=ip-172-30-1-173 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] db version v3.0.5 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] build info: Linux ip-10-230-137-3 3.4.43-43.43.amzn1.x86_64 #1 SMP Mon May 6 18:04:41 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] allocator: tcmalloc 2015-07-30T19:55:50.667+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", operationProfiling: { slowOpThresholdMs: 250 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "recomme ndations-production_uk" }, storage: { dbPath: "/data/mongo", engine: "wiredTiger" }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } } 2015-07-30T19:55:50.673+0000 I NETWORK [initandlisten] waiting for connections on port 27017 2015-07-30T19:55:50.679+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "recommendations-production_uk", version: 7, members: [ { _id: 2, host: "ip-172-30-3-187:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priorit y: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "ip-172-30-1-173:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 4, host: "ip-172-30-4-82:27017", arbiterOnly: false, build Indexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2015-07-30T19:55:50.679+0000 I REPL [ReplicationExecutor] This node is ip-172-30-1-173:27017 in the config 2015-07-30T19:55:50.679+0000 I REPL [ReplicationExecutor] transition to STARTUP2 2015-07-30T19:55:50.680+0000 I REPL [ReplicationExecutor] Starting replication applier threads 2015-07-30T19:55:50.680+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2015-07-30T19:55:50.681+0000 I REPL [ReplicationExecutor] Member ip-172-30-3-187:27017 is now in state ARBITER 2015-07-30T19:55:50.682+0000 I REPL [ReplicationExecutor] Member ip-172-30-4-82:27017 is now in state PRIMARY 2015-07-30T19:55:51.488+0000 I NETWORK [initandlisten] connection accepted from 172.30.3.187:57872 #1 (1 connection now open) 2015-07-30T19:55:51.497+0000 I NETWORK [initandlisten] connection accepted from 172.30.4.82:55893 #2 (2 connections now open) 2015-07-30T19:55:51.563+0000 I NETWORK [initandlisten] connection accepted from 172.30.3.199:60445 #3 (3 connections now open) 2015-07-30T19:55:51.904+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:56948 #4 (4 connections now open) 2015-07-30T19:55:51.904+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:56949 #5 (5 connections now open) 2015-07-30T19:55:52.014+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:56950 #6 (6 connections now open) 2015-07-30T19:55:52.015+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:56951 #7 (7 connections now open) 2015-07-30T19:55:53.128+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52586 #8 (8 connections now open) 2015-07-30T19:55:53.129+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52587 #9 (9 connections now open) 2015-07-30T19:55:53.129+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52588 #10 (10 connections now open) 2015-07-30T19:55:53.129+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52589 #11 (11 connections now open) 2015-07-30T19:55:53.681+0000 I REPL [ReplicationExecutor] syncing from: ip-172-30-4-82:27017 2015-07-30T19:55:53.887+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to ip-172-30-4-82:27017 2015-07-30T19:55:53.905+0000 I REPL [ReplicationExecutor] transition to SECONDARY 2015-07-30T19:55:53.952+0000 I REPL [ReplicationExecutor] syncing from: ip-172-30-4-82:27017 2015-07-30T19:55:53.989+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40390 #12 (12 connections now open) 2015-07-30T19:55:53.990+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40391 #13 (13 connections now open) 2015-07-30T19:55:53.990+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40392 #14 (14 connections now open) 2015-07-30T19:55:53.990+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40393 #15 (15 connections now open) 2015-07-30T19:55:54.418+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to ip-172-30-4-82:27017 2015-07-30T19:55:55.093+0000 I NETWORK [initandlisten] connection accepted from 172.30.0.204:48505 #16 (16 connections now open) 2015-07-30T19:55:56.701+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52776 #17 (17 connections now open) 2015-07-30T19:55:56.701+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52777 #18 (18 connections now open) 2015-07-30T19:55:56.702+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52778 #19 (19 connections now open) 2015-07-30T19:55:56.702+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52779 #20 (20 connections now open) 2015-07-30T19:55:58.151+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45756 #21 (21 connections now open) 2015-07-30T19:56:19.772+0000 I NETWORK [initandlisten] connection accepted from 10.196.164.37:42066 #22 (22 connections now open) 2015-07-30T19:56:21.501+0000 I NETWORK [conn1] end connection 172.30.3.187:57872 (21 connections now open) 2015-07-30T19:56:21.501+0000 I NETWORK [initandlisten] connection accepted from 172.30.3.187:57874 #23 (22 connections now open) 2015-07-30T19:56:21.513+0000 I NETWORK [conn2] end connection 172.30.4.82:55893 (21 connections now open) 2015-07-30T19:56:21.514+0000 I NETWORK [initandlisten] connection accepted from 172.30.4.82:55895 #24 (22 connections now open) 2015-07-30T19:56:28.653+0000 I NETWORK [initandlisten] connection accepted from 10.81.17.165:47698 #25 (23 connections now open) 2015-07-30T19:56:30.661+0000 I NETWORK [initandlisten] connection accepted from 10.99.132.140:45809 #26 (24 connections now open) 2015-07-30T19:56:36.491+0000 E STORAGE [rsSync] WiredTiger (12) [1438286196:491627][2537:0x7f07a2c1b700], session.commit_transaction: memory allocation: Cannot allocate memory 2015-07-30T19:56:36.526+0000 I - [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 305 2015-07-30T19:56:36.542+0000 I CONTROL [rsSync] 0xf77ca9 0xf16b61 0xefd29a 0xd9727c 0xd9738a 0x930c3c 0xc512b2 0xcbba40 0xcbcc68 0xcb53af 0xfc4e64 0x7f07b3f5cdf3 0x7f07b2a101bd ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B77CA9"},{"b":"400000","o":"B16B61"},{"b":"400000","o":"AFD29A"},{"b":"400000","o":"99727C"},{"b":"400000","o":"99738A"},{"b":"400000","o":"530C3C"},{"b":"400000","o":"8512B2"},{"b":"400000","o":"8BBA40"},{"b":"400000","o":"8BCC68"},{"b":"400000","o":"8B53AF"},{"b":"400000","o":"BC4E64"},{"b":"7F07B3F55000","o":"7DF3"},{"b":"7F07B291A000","o":"F61BD"}],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" : { "sysname" : "Linux", "release" : "3.14.44-32.39.amzn1.x86_64", "version" : "#1 SMP Thu Jun 11 20:33:38 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E8A046532FE2C5A8474250BA8E5B72C6C84B9D68" }, { "b" : "7FFC6EC17000", "elfType" : 3, "buildId" : "3C9325999495818299EC622C63D9330D234E02C9" }, { "b" : "7F07B3F55000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D48D3E6672A77B603B402F661BABF75E90AD570B" }, { "b" : "7F07B3CE8000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "22480480235F3B1C6C2E5E5953949728676D3796" }, { "b" : "7F07B3903000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F1C146B78505646930DD9003AA2B3622C5226D1B" }, { "b" : "7F07B36FB000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "E81013CBFA409053D58A65A0653271AB665A4619" }, { "b" : "7F07B34F7000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "62A8842157C62F95C3069CBF779AFCC26577A99A" }, { "b" : "7F07B31F3000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7F07B2EF1000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5F97F8F8E5024E29717CF35998681F84D4A22D45" }, { "b" : "7F07B2CDB000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7F07B291A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DF6DA145A649EA093507A635AF383F608E7CE3F2" }, { "b" : "7F07B4171000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F90843B9087FE91955FEB0355EB0858EF9E97B2" }, { "b" : "7F07B26D7000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DE5A9F7A11A0881CB64E375F4DDCA58028F0FAF8" }, { "b" : "7F07B23F2000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "A3E43FC66908AC8B00773707FECA3B1677AFF311" }, { "b" : "7F07B21EF000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "622F315EB5CB2F791E9B64020692EBA98195D06D" }, { "b" : "7F07B1FC4000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B10FBFEC246C4EAD1719D16090D0BE54904BBFC9" }, { "b" : "7F07B1DAE000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F07B1BA3000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "7292C0673D7C116E3389D3FFA67087A6B9287A71" }, { "b" : "7F07B19A0000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F07B1786000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6A7DA1CED90F65F27CB7B5BACDBB1C386C05F592" }, { "b" : "7F07B1565000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf77ca9] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf16b61] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xefd29a] mongod(_ZN5mongo22WiredTigerRecoveryUnit9_txnCloseEb+0x2EC) [0xd9727c] mongod(_ZN5mongo22WiredTigerRecoveryUnit7_commitEv+0x8A) [0xd9738a] mongod(_ZN5mongo15WriteUnitOfWork6commitEv+0x1C) [0x930c3c] mongod(_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt5dequeINS_7BSONObjESaIS4_EE+0x282) [0xc512b2] mongod(_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERSt5dequeINS_7BSONObjESaIS5_EE+0x460) [0xcbba40] mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEv+0x118) [0xcbcc68] mongod(_ZN5mongo4repl13runSyncThreadEv+0x1EF) [0xcb53af] mongod(+0xBC4E64) [0xfc4e64] libpthread.so.0(+0x7DF3) [0x7f07b3f5cdf3] libc.so.6(clone+0x6D) [0x7f07b2a101bd] ----- END BACKTRACE ----- 2015-07-30T19:56:36.542+0000 I - [rsSync] ***aborting after invariant() failure 2015-07-30T19:58:00.162+0000 I CONTROL ***** SERVER RESTARTED ***** 2015-07-30T19:58:00.195+0000 W - [initandlisten] Detected unclean shutdown - /data/mongo/mongod.lock is not empty. 2015-07-30T19:58:00.195+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2015-07-30T19:58:00.195+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=30G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-07-30T19:58:09.574+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] MongoDB starting : pid=2706 port=27017 dbpath=/data/mongo 64-bit host=ip-172-30-1-173 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] db version v3.0.5 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] build info: Linux ip-10-230-137-3 3.4.43-43.43.amzn1.x86_64 #1 SMP Mon May 6 18:04:41 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] allocator: tcmalloc 2015-07-30T19:58:09.671+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", operationProfiling: { slowOpThresholdMs: 250 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "recommendations-production_uk" }, storage: { dbPath: "/data/mongo", engine: "wiredTiger" }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } } 2015-07-30T19:58:09.673+0000 I NETWORK [initandlisten] waiting for connections on port 27017 2015-07-30T19:58:09.679+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "recommendations-production_uk", version: 7, members: [ { _id: 2, host: "ip-172-30-3-187:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "ip-172-30-1-173:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 4, host: "ip-172-30-4-82:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2015-07-30T19:58:09.679+0000 I REPL [ReplicationExecutor] This node is ip-172-30-1-173:27017 in the config 2015-07-30T19:58:09.679+0000 I REPL [ReplicationExecutor] transition to STARTUP2 2015-07-30T19:58:09.679+0000 I REPL [ReplicationExecutor] Starting replication applier threads 2015-07-30T19:58:09.679+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2015-07-30T19:58:09.680+0000 I REPL [ReplicationExecutor] Member ip-172-30-3-187:27017 is now in state ARBITER 2015-07-30T19:58:09.680+0000 I REPL [ReplicationExecutor] transition to SECONDARY 2015-07-30T19:58:09.681+0000 I REPL [ReplicationExecutor] Member ip-172-30-4-82:27017 is now in state PRIMARY 2015-07-30T19:58:10.161+0000 I NETWORK [initandlisten] connection accepted from 10.15.131.211:56078 #1 (1 connection now open) 2015-07-30T19:58:10.551+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52834 #2 (2 connections now open) 2015-07-30T19:58:10.551+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52833 #3 (3 connections now open) 2015-07-30T19:58:10.551+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52836 #4 (4 connections now open) 2015-07-30T19:58:10.551+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.139:52837 #5 (5 connections now open) 2015-07-30T19:58:10.670+0000 I NETWORK [initandlisten] connection accepted from 172.30.3.187:58015 #6 (6 connections now open) 2015-07-30T19:58:10.712+0000 I NETWORK [initandlisten] connection accepted from 172.30.4.82:56037 #7 (7 connections now open) 2015-07-30T19:58:11.929+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:57005 #8 (8 connections now open) 2015-07-30T19:58:11.929+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:57006 #9 (9 connections now open) 2015-07-30T19:58:12.035+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:57008 #10 (10 connections now open) 2015-07-30T19:58:12.035+0000 I NETWORK [initandlisten] connection accepted from 10.193.3.69:57009 #11 (11 connections now open) 2015-07-30T19:58:12.680+0000 I REPL [ReplicationExecutor] syncing from: ip-172-30-4-82:27017 2015-07-30T19:58:12.934+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to ip-172-30-4-82:27017 2015-07-30T19:58:13.147+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52647 #12 (12 connections now open) 2015-07-30T19:58:13.147+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52648 #13 (13 connections now open) 2015-07-30T19:58:13.147+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52649 #14 (14 connections now open) 2015-07-30T19:58:13.147+0000 I NETWORK [initandlisten] connection accepted from 10.101.19.41:52651 #15 (15 connections now open) 2015-07-30T19:58:14.015+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40452 #16 (16 connections now open) 2015-07-30T19:58:14.015+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40453 #17 (17 connections now open) 2015-07-30T19:58:14.015+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40455 #18 (18 connections now open) 2015-07-30T19:58:14.015+0000 I NETWORK [initandlisten] connection accepted from 10.193.22.229:40454 #19 (19 connections now open) 2015-07-30T19:58:29.406+0000 I NETWORK [initandlisten] connection accepted from 10.99.132.140:45903 #20 (20 connections now open) 2015-07-30T19:58:38.915+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45772 #21 (21 connections now open) 2015-07-30T19:58:40.086+0000 I NETWORK [initandlisten] connection accepted from 10.15.131.211:56092 #22 (22 connections now open) 2015-07-30T19:58:40.680+0000 I NETWORK [conn6] end connection 172.30.3.187:58015 (21 connections now open) 2015-07-30T19:58:40.681+0000 I NETWORK [initandlisten] connection accepted from 172.30.3.187:58017 #23 (22 connections now open) 2015-07-30T19:58:40.728+0000 I NETWORK [conn7] end connection 172.30.4.82:56037 (21 connections now open) 2015-07-30T19:58:40.729+0000 I NETWORK [initandlisten] connection accepted from 172.30.4.82:56040 #24 (22 connections now open) 2015-07-30T19:58:53.283+0000 E STORAGE [rsSync] WiredTiger (12) [1438286333:283781][2706:0x7f7a6bc63700], session.commit_transaction: memory allocation: Cannot allocate memory 2015-07-30T19:58:53.284+0000 I - [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 305 2015-07-30T19:58:53.298+0000 I CONTROL [rsSync] 0xf77ca9 0xf16b61 0xefd29a 0xd9727c 0xd9738a 0x930c3c 0xc512b2 0xcbba40 0xcbcc68 0xcb53af 0xfc4e64 0x7f7a7cfa4df3 0x7f7a7ba581bd ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B77CA9"},{"b":"400000","o":"B16B61"},{"b":"400000","o":"AFD29A"},{"b":"400000","o":"99727C"},{"b":"400000","o":"99738A"},{"b":"400000","o":"530C3C"},{"b":"400000","o":"8512B2"},{"b":"400000","o":"8BBA40"},{"b":"400000","o":"8BCC68"},{"b":"400000","o":"8B53AF"},{"b":"400000","o":"BC4E64"},{"b":"7F7A7CF9D000","o":"7DF3"},{"b":"7F7A7B962000","o":"F61BD"}],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" : { "sysname" : "Linux", "release" : "3.14.44-32.39.amzn1.x86_64", "version" : "#1 SMP Thu Jun 11 20:33:38 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E8A046532FE2C5A8474250BA8E5B72C6C84B9D68" }, { "b" : "7FFE78A96000", "elfType" : 3, "buildId" : "3C9325999495818299EC622C63D9330D234E02C9" }, { "b" : "7F7A7CF9D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D48D3E6672A77B603B402F661BABF75E90AD570B" }, { "b" : "7F7A7CD30000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "22480480235F3B1C6C2E5E5953949728676D3796" }, { "b" : "7F7A7C94B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F1C146B78505646930DD9003AA2B3622C5226D1B" }, { "b" : "7F7A7C743000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "E81013CBFA409053D58A65A0653271AB665A4619" }, { "b" : "7F7A7C53F000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "62A8842157C62F95C3069CBF779AFCC26577A99A" }, { "b" : "7F7A7C23B000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7F7A7BF39000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5F97F8F8E5024E29717CF35998681F84D4A22D45" }, { "b" : "7F7A7BD23000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7F7A7B962000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DF6DA145A649EA093507A635AF383F608E7CE3F2" }, { "b" : "7F7A7D1B9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F90843B9087FE91955FEB0355EB0858EF9E97B2" }, { "b" : "7F7A7B71F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DE5A9F7A11A0881CB64E375F4DDCA58028F0FAF8" }, { "b" : "7F7A7B43A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "A3E43FC66908AC8B00773707FECA3B1677AFF311" }, { "b" : "7F7A7B237000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "622F315EB5CB2F791E9B64020692EBA98195D06D" }, { "b" : "7F7A7B00C000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B10FBFEC246C4EAD1719D16090D0BE54904BBFC9" }, { "b" : "7F7A7ADF6000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F7A7ABEB000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "7292C0673D7C116E3389D3FFA67087A6B9287A71" }, { "b" : "7F7A7A9E8000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F7A7A7CE000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6A7DA1CED90F65F27CB7B5BACDBB1C386C05F592" }, { "b" : "7F7A7A5AD000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf77ca9] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf16b61] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xefd29a] mongod(_ZN5mongo22WiredTigerRecoveryUnit9_txnCloseEb+0x2EC) [0xd9727c] mongod(_ZN5mongo22WiredTigerRecoveryUnit7_commitEv+0x8A) [0xd9738a] mongod(_ZN5mongo15WriteUnitOfWork6commitEv+0x1C) [0x930c3c] mongod(_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt5dequeINS_7BSONObjESaIS4_EE+0x282) [0xc512b2] mongod(_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERSt5dequeINS_7BSONObjESaIS5_EE+0x460) [0xcbba40] mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEv+0x118) [0xcbcc68] mongod(_ZN5mongo4repl13runSyncThreadEv+0x1EF) [0xcb53af] mongod(+0xBC4E64) [0xfc4e64] libpthread.so.0(+0x7DF3) [0x7f7a7cfa4df3] libc.so.6(clone+0x6D) [0x7f7a7ba581bd] ----- END BACKTRACE ----- 2015-07-30T19:58:53.298+0000 I - [rsSync] ***aborting after invariant() failure 2015-07-30T20:00:10.580+0000 I CONTROL ***** SERVER RESTARTED *****