2014-10-30T05:33:34.748+0100 [conn721281] end connection 148.251.87.54:48582 (15 connections now open) 2014-10-30T05:33:34.764+0100 [initandlisten] connection accepted from 148.251.87.54:48639 #721311 (16 connections now open) 2014-10-30T05:33:35.382+0100 [repl index builder 83991] build index on: 542-110-6347.keyword_adgroup properties: { v: 1, key: { adgroup: 1 }, name : "adgroup_1", ns: "542-110-6347.keyword_adgroup", background: true } 2014-10-30T05:33:35.383+0100 [repl index builder 83991] building index in background 2014-10-30T05:33:35.453+0100 [repl writer worker 4] CMD: drop 542-110-6347.keyword_adgroup 2014-10-30T05:33:35.453+0100 [repl writer worker 4] halting index build: { campaign: 1 } 2014-10-30T05:33:35.453+0100 [repl writer worker 4] SEVERE: Invalid access at address: 0 2014-10-30T05:33:35.461+0100 [repl writer worker 4] SEVERE: Got signal: 11 (Segmentation fault). Backtrace:0x11e9b11 0x11e8eee 0x11e8fdf 0x7f41bf498340 0x8da668 0xb8aa95 0xa2f5a9 0xa31e80 0xa2939a 0xa2b151 0xa2c9a6 0xe547d3 0xeb8d9e 0xeb96b0 0 x117f0ee 0x122e4a9 0x7f41bf490182 0x7f41be79530d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod() [0x11e8eee] /usr/bin/mongod() [0x11e8fdf] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f41bf498340] /usr/bin/mongod(_ZN5mongo12IndexCatalog23killMatchingIndexBuildsERKNS0_17IndexKillCriteriaE+0x218) [0x8da668] /usr/bin/mongod(_ZN5mongo12IndexBuilder23killMatchingIndexBuildsEPNS_10CollectionERKNS_12IndexCatalog17IndexKillCriteriaE+0x15) [0xb8aa95] /usr/bin/mongod(_ZN5mongo7CmdDrop15stopIndexBuildsEPNS_8DatabaseERKNS_7BSONObjE+0xe9) [0xa2f5a9] /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1c0) [0xa31e80] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2939a] /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x19b1) [0xa2b151] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2c9a6] /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x973) [0xe547d3] /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x4fe) [0xeb8d9e] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb96b0] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117f0ee] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f41bf490182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f41be79530d] 2014-10-30T08:33:23.307+0100 ***** SERVER RESTARTED ***** 2014-10-30T08:33:23.310+0100 [initandlisten] MongoDB starting : pid=24801 port=27017 dbpath=/var/lib/mongodb 64-bit host=lohap2 2014-10-30T08:33:23.310+0100 [initandlisten] db version v2.6.5 2014-10-30T08:33:23.310+0100 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6 2014-10-30T08:33:23.310+0100 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49 2014-10-30T08:33:23.310+0100 [initandlisten] allocator: tcmalloc 2014-10-30T08:33:23.310+0100 [initandlisten] options: { config: "/etc/mongod.conf", replication: { oplogSizeMB: 20480, replSet: "rs0" }, storage: { dbPath: "/var/lib/mongodb", smallFiles: true }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2014-10-30T08:33:23.310+0100 [initandlisten] 2014-10-30T08:33:23.310+0100 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 2048KB 2014-10-30T08:33:23.310+0100 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less 2014-10-30T08:33:23.310+0100 [initandlisten] ** http://dochub.mongodb.org/core/readahead 2014-10-30T08:33:23.311+0100 [initandlisten] journal dir=/var/lib/mongodb/journal 2014-10-30T08:33:23.311+0100 [initandlisten] recover : no journal files present, no recovery needed 2014-10-30T08:33:23.335+0100 [initandlisten] allocating new ns file /var/lib/mongodb/local.ns, filling with zeroes... 2014-10-30T08:33:23.388+0100 [FileAllocator] allocating new datafile /var/lib/mongodb/local.0, filling with zeroes... 2014-10-30T08:33:23.388+0100 [FileAllocator] creating directory /var/lib/mongodb/_tmp 2014-10-30T08:33:23.389+0100 [FileAllocator] done allocating datafile /var/lib/mongodb/local.0, size: 16MB, took 0 secs 2014-10-30T08:33:23.393+0100 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" } 2014-10-30T08:33:23.393+0100 [initandlisten] added index to empty collection 2014-10-30T08:33:23.393+0100 [initandlisten] waiting for connections on port 27017 2014-10-30T08:33:23.393+0100 [initandlisten] connection accepted from 148.251.87.54:42223 #1 (1 connection now open) 2014-10-30T08:33:23.395+0100 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) 2014-10-30T08:33:23.395+0100 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done 2014-10-30T08:33:23.480+0100 [initandlisten] connection accepted from 192.168.0.1:49674 #2 (2 connections now open) 2014-10-30T08:33:24.430+0100 [rsStart] replSet got config version 7 from a remote, saving locally 2014-10-30T08:33:24.430+0100 [rsStart] replSet info saving a newer config version to local.system.replset: { _id: "rs0", version: 7, members: [ { _id: 0, host: "lohap1.apsl.net:27017" }, { _id: 1, host: "lohap2.apsl.net:27017" }, { _id: 2, host: "kiwi8.apsl.net:30000", arbiterOnly: true } ] } 2014-10-30T08:33:24.431+0100 [rsStart] build index on: local.system.replset properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.system.replset" } 2014-10-30T08:33:24.431+0100 [rsStart] added index to empty collection 2014-10-30T08:33:24.431+0100 [rsStart] replSet saveConfigLocally done 2014-10-30T08:33:24.433+0100 [rsStart] build index on: local.me properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.me" } 2014-10-30T08:33:24.433+0100 [rsStart] added index to empty collection 2014-10-30T08:33:24.433+0100 [rsStart] replSet STARTUP2 2014-10-30T08:33:24.433+0100 [rsSync] ****** 2014-10-30T08:33:24.433+0100 [rsSync] creating replication oplog of size: 20480MB... 2014-10-30T08:33:24.433+0100 [FileAllocator] allocating new datafile /var/lib/mongodb/local.1, filling with zeroes... 2014-10-30T08:33:24.434+0100 [FileAllocator] done allocating datafile /var/lib/mongodb/local.1, size: 511MB, took 0 secs 2014-10-30T08:33:24.436+0100 [FileAllocator] allocating new datafile /var/lib/mongodb/local.2, filling with zeroes...