2012-05-03_01:43:59.04702 Thu May 3 01:43:59 [journal] old journal file will be removed: ./data/journal/j._19 2012-05-03_01:44:26.18500 Thu May 3 01:44:26 [FileAllocator] allocating new datafile ./data/chalk-bump-f49/chalk-bump-f49.12, filling with zeroes... 2012-05-03_01:44:26.18930 Thu May 3 01:44:26 [FileAllocator] done allocating datafile ./data/chalk-bump-f49/chalk-bump-f49.12, size: 2047MB, took 0.004 secs 2012-05-03_01:44:30.06049 Thu May 3 01:44:30 [rsSync] clone chalk-bump-f49.b.note 1419519 2012-05-03_01:44:38.04906 Thu May 3 01:44:38 [rsSync] 1448945 objects cloned so far from collection chalk-bump-f49.b.note 2012-05-03_01:44:39.85270 Thu May 3 01:44:39 [journal] old journal file will be removed: ./data/journal/j._20 2012-05-03_01:44:49.88186 Thu May 3 01:44:49 [clientcursormon] mem (MB) res:12910 virt:96161 mapped:47818 2012-05-03_01:45:16.60239 Thu May 3 01:45:16 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 reslen:1791 192ms 2012-05-03_01:45:18.38364 Thu May 3 01:45:18 [journal] old journal file will be removed: ./data/journal/j._21 2012-05-03_01:45:18.38623 Thu May 3 01:45:18 [journal] old journal file will be removed: ./data/journal/j._22 2012-05-03_01:45:30.06846 Thu May 3 01:45:30 [rsSync] clone chalk-bump-f49.b.note 1706623 2012-05-03_01:45:33.46817 Thu May 3 01:45:33 [rsSync] chalk-bump-f49 Assertion failure approxSize < Extent::maxSize() db/pdfile.cpp 437 2012-05-03_01:45:33.46831 0x57a726 0x58520b 0x8ad13b 0x8a83c6 0x8a85c7 0x8b4069 0x8b576b 0x85aa85 0x5c183a 0x850885 0x852995 0x8542e9 0x82e76a 0x830323 0x828381 0x828438 0x8288d0 0xaabd90 0x7f492afa4d8c 0x7f492a54f04d 2012-05-03_01:45:33.46873 Thu May 3 01:45:33 [conn15] serverStatus was very slow: { after basic: 0, middle of mem: 2820, after mem: 2820, after connections: 2820, after extra info: 2820, after counters: 2820, after repl: 2820, after asserts: 2820, after dur: 2820, at end: 2820 } 2012-05-03_01:45:33.46876 Thu May 3 01:45:33 [FileAllocator] allocating new datafile ./data/chalk-bump-f49/chalk-bump-f49.13, filling with zeroes... 2012-05-03_01:45:33.46878 Thu May 3 01:45:33 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 reslen:1982 2849ms 2012-05-03_01:45:33.47216 bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a726] 2012-05-03_01:45:33.47218 bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x58520b] 2012-05-03_01:45:33.47219 bin/mongod(_ZN5mongo13MongoDataFile12createExtentEPKcibi+0x24b) [0x8ad13b] 2012-05-03_01:45:33.47220 bin/mongod(_ZN5mongo10outOfSpaceEPKcPNS_16NamespaceDetailsEibNS_7DiskLocE+0x116) [0x8a83c6] 2012-05-03_01:45:33.47221 bin/mongod(_ZN5mongo26allocateSpaceForANewRecordEPKcPNS_16NamespaceDetailsEib+0x77) [0x8a85c7] 2012-05-03_01:45:33.47222 bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x4a9) [0x8b4069] 2012-05-03_01:45:33.47223 bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x4b) [0x8b576b] 2012-05-03_01:45:33.47224 bin/mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x545) [0x85aa85] 2012-05-03_01:45:33.47225 bin/mongod(_ZN5mongo18DBClientConnection5queryEN5boost8functionIFvRNS_27DBClientCursorBatchIteratorEEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1aa) [0x5c183a] 2012-05-03_01:45:33.47227 bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbbbNS_5QueryE+0x3c5) [0x850885] 2012-05-03_01:45:33.47227 bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x1665) [0x852995] 2012-05-03_01:45:33.47228 bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x8542e9] 2012-05-03_01:45:33.47229 bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0xe6a) [0x82e76a] 2012-05-03_01:45:33.47235 bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x23) [0x830323] 2012-05-03_01:45:33.47236 bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x61) [0x828381] 2012-05-03_01:45:33.47237 bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x48) [0x828438] 2012-05-03_01:45:33.47238 bin/mongod(_ZN5mongo15startSyncThreadEv+0xa0) [0x8288d0] 2012-05-03_01:45:33.47239 bin/mongod(thread_proxy+0x80) [0xaabd90] 2012-05-03_01:45:33.47240 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x7f492afa4d8c] 2012-05-03_01:45:33.47241 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f492a54f04d] 2012-05-03_01:45:33.47250 Thu May 3 01:45:33 [FileAllocator] done allocating datafile ./data/chalk-bump-f49/chalk-bump-f49.13, size: 2047MB, took 0.003 secs 2012-05-03_01:45:33.55164 Thu May 3 01:45:33 [rsSync] Socket flush send() errno:9 Bad file descriptor 10.218.95.168:27018 2012-05-03_01:45:33.55175 Thu May 3 01:45:33 [rsSync] chalk-bump-f49 caught exception (socket exception) in destructor (~PiggyBackData) 2012-05-03_01:45:33.55199 Thu May 3 01:45:33 [rsSync] replSet initial sync exception 0 assertion db/pdfile.cpp:437 2012-05-03_01:46:03.55250 Thu May 3 01:46:03 [rsSync] replSet initial sync pending 2012-05-03_01:46:03.55256 Thu May 3 01:46:03 [rsSync] replSet syncing to: mo08.production.simperium.com:27018 2012-05-03_01:46:03.55513 Thu May 3 01:46:03 [rsSync] replSet initial sync drop all databases 2012-05-03_01:46:03.55516 Thu May 3 01:46:03 [rsSync] dropAllDatabasesExceptLocal 6 2012-05-03_01:46:03.57038 Thu May 3 01:46:03 [rsSync] removeJournalFiles 2012-05-03_01:46:03.70792 Thu May 3 01:46:03 [rsSync] removeJournalFiles 2012-05-03_01:46:03.70968 Thu May 3 01:46:03 [rsSync] removeJournalFiles 2012-05-03_01:46:03.76154 Thu May 3 01:46:03 [rsSync] removeJournalFiles 2012-05-03_01:46:03.76335 Thu May 3 01:46:03 [rsSync] removeJournalFiles 2012-05-03_01:46:05.54789 Thu May 3 01:46:05 [conn15] warning: virtual size (88994MB) - mapped size (82328MB) is large (6659MB). could indicate a memory leak 2012-05-03_01:46:06.00287 Thu May 3 01:46:05 [conn15] serverStatus was very slow: { after basic: 0, middle of mem: 620, after mem: 1050, after connections: 1050, after extra info: 1500, after counters: 1500, after repl: 1500, after asserts: 1500, after dur: 1500, at end: 1500 } 2012-05-03_01:46:06.00289 Thu May 3 01:46:05 [conn15] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 reslen:2052 1511ms 2012-05-03_01:46:10.39763 Thu May 3 01:46:10 [rsSync] replSet initial sync clone all databases ....