Details
-
Bug
-
Resolution: Duplicate
-
Major - P3
-
None
-
None
-
None
-
ALL
Description
I tried to shut down my 2.8.0rc0 (wiredtiger) secondary during an initial sync.
When I typed db.shutdownServer(), it hung. Eventually a CTRL-Ced that a few times and escaped the shell. Then, I tried a kill, that did not work.
The process is still running, but I cannot connect to it anymore in the shell. The log file says this:
2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] build index done. scanned 2 total records. 0 secs
|
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index on: mmsdbconfig.data.exposedHosts properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.data.exposedHosts" }
|
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index done. scanned 0 total records. 0 secs
|
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.replicaSetChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.replicaSetChartSelects" }
|
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index done. scanned 9 total records. 0 secs
|
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.registrationThrottles properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.registrationThrottles" }
|
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index done. scanned 2 total records. 0 secs
|
2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.automationVersions properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.automationVersions" }
|
2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.684+0000 I INDEXING [rsSync] build index done. scanned 28 total records. 0 secs
|
2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] build index on: mmsdbconfig.app.systemDbStats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.app.systemDbStats" }
|
2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.689+0000 I INDEXING [rsSync] build index done. scanned 16 total records. 0 secs
|
2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.alertConfigHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.alertConfigHistory" }
|
2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.696+0000 I INDEXING [rsSync] build index done. scanned 41 total records. 0 secs
|
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostChartSelects" }
|
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index done. scanned 15 total records. 0 secs
|
2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostClusterHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostClusterHistory" }
|
2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] building index using bulk method
|
2014-11-13T03:05:07.708+0000 I INDEXING [rsSync] build index done. scanned 14 total records. 0 secs
|
2014-11-13T03:05:07.708+0000 I REPLSETS [rsSync] initial sync cloning db: 540bb74fe4b039a8d31a04a7_A
|
2014-11-13T03:05:10.359+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45903 #348 (189 connections now open)
|
2014-11-13T03:05:15.341+0000 I COMMANDS [conn348] terminating, shutdown command received
|
2014-11-13T03:05:15.632+0000 I STORAGE [conn296] got request after shutdown()
|
2014-11-13T03:05:15.757+0000 I STORAGE [conn297] got request after shutdown()
|
2014-11-13T03:05:15.980+0000 I STORAGE [conn298] got request after shutdown()
|
2014-11-13T03:05:15.996+0000 I STORAGE [conn299] got request after shutdown()
|
2014-11-13T03:05:16.019+0000 I STORAGE [conn187] got request after shutdown()
|
2014-11-13T03:05:16.019+0000 I STORAGE [conn194] got request after shutdown()
|
2014-11-13T03:05:16.030+0000 I STORAGE [conn192] got request after shutdown()
|
2014-11-13T03:05:16.032+0000 I STORAGE [conn300] got request after shutdown()
|
2014-11-13T03:05:16.051+0000 I STORAGE [conn195] got request after shutdown()
|
2014-11-13T03:05:16.061+0000 I STORAGE [conn301] got request after shutdown()
|
2014-11-13T03:05:16.061+0000 I STORAGE [conn346] got request after shutdown()
|
2014-11-13T03:05:16.111+0000 I STORAGE [conn302] got request after shutdown()
|
2014-11-13T03:05:16.137+0000 I STORAGE [conn303] got request after shutdown()
|
2014-11-13T03:05:16.180+0000 I STORAGE [conn304] got request after shutdown()
|
2014-11-13T03:05:16.230+0000 I STORAGE [conn305] got request after shutdown()
|
2014-11-13T03:05:16.235+0000 I STORAGE [conn274] got request after shutdown()
|
2014-11-13T03:05:16.244+0000 I STORAGE [conn188] got request after shutdown()
|
2014-11-13T03:05:16.247+0000 I STORAGE [conn210] got request after shutdown()
|
2014-11-13T03:05:16.248+0000 I STORAGE [conn193] got request after shutdown()
|
2014-11-13T03:05:16.248+0000 I STORAGE [conn211] got request after shutdown()
|
2014-11-13T03:05:16.248+0000 I STORAGE [conn196] got request after shutdown()
|
2014-11-13T03:05:16.253+0000 I STORAGE [conn190] got request after shutdown()
|
2014-11-13T03:05:16.258+0000 I STORAGE [conn306] got request after shutdown()
|
2014-11-13T03:05:16.289+0000 I STORAGE [conn307] got request after shutdown()
|
2014-11-13T03:05:16.315+0000 I STORAGE [conn308] got request after shutdown()
|
2014-11-13T03:05:16.344+0000 I STORAGE [conn309] got request after shutdown()
|
2014-11-13T03:05:16.392+0000 I STORAGE [conn310] got request after shutdown()
|
2014-11-13T03:05:16.438+0000 I STORAGE [conn275] got request after shutdown()
|
2014-11-13T03:05:16.445+0000 I STORAGE [conn276] got request after shutdown()
|
2014-11-13T03:05:16.447+0000 I STORAGE [conn311] got request after shutdown()
|
2014-11-13T03:05:16.478+0000 I STORAGE [conn277] got request after shutdown()
|
2014-11-13T03:05:16.492+0000 I STORAGE [conn312] got request after shutdown()
|
2014-11-13T03:05:16.503+0000 I STORAGE [conn278] got request after shutdown()
|
2014-11-13T03:05:16.506+0000 I STORAGE [conn313] got request after shutdown()
|
2014-11-13T03:05:16.514+0000 I STORAGE [conn314] got request after shutdown()
|
2014-11-13T03:05:16.543+0000 I STORAGE [conn279] got request after shutdown()
|
2014-11-13T03:05:16.595+0000 I STORAGE [conn280] got request after shutdown()
|
2014-11-13T03:05:16.662+0000 I STORAGE [conn281] got request after shutdown()
|
2014-11-13T03:05:16.697+0000 I STORAGE [conn282] got request after shutdown()
|
2014-11-13T03:05:16.813+0000 I STORAGE [conn283] got request after shutdown()
|
2014-11-13T03:05:16.814+0000 I STORAGE [conn284] got request after shutdown()
|
2014-11-13T03:05:16.841+0000 I STORAGE [conn315] got request after shutdown()
|
2014-11-13T03:05:16.843+0000 I STORAGE [conn285] got request after shutdown()
|
2014-11-13T03:05:16.857+0000 I STORAGE [conn286] got request after shutdown()
|
2014-11-13T03:05:16.887+0000 I STORAGE [conn287] got request after shutdown()
|
2014-11-13T03:05:16.894+0000 I NETWORK [conn345] end connection 10.167.150.115:49672 (143 connections now open)
|
2014-11-13T03:05:16.914+0000 I STORAGE [conn288] got request after shutdown()
|
2014-11-13T03:05:16.938+0000 I STORAGE [conn347] got request after shutdown()
|
2014-11-13T03:05:16.951+0000 I STORAGE [conn289] got request after shutdown()
|
2014-11-13T03:05:16.976+0000 I STORAGE [conn290] got request after shutdown()
|
2014-11-13T03:05:17.019+0000 I STORAGE [conn291] got request after shutdown()
|
2014-11-13T03:05:17.197+0000 I STORAGE [conn317] got request after shutdown()
|
2014-11-13T03:05:17.222+0000 I STORAGE [conn318] got request after shutdown()
|
2014-11-13T03:05:17.233+0000 I STORAGE [conn319] got request after shutdown()
|
2014-11-13T03:05:17.268+0000 I STORAGE [conn320] got request after shutdown()
|
2014-11-13T03:05:17.314+0000 I STORAGE [conn321] got request after shutdown()
|
2014-11-13T03:05:17.326+0000 I STORAGE [conn95] got request after shutdown()
|
2014-11-13T03:05:17.342+0000 I STORAGE [conn55] got request after shutdown()
|
2014-11-13T03:05:17.362+0000 I STORAGE [conn322] got request after shutdown()
|
2014-11-13T03:05:17.370+0000 I STORAGE [conn96] got request after shutdown()
|
2014-11-13T03:05:17.389+0000 I STORAGE [conn98] got request after shutdown()
|
2014-11-13T03:05:17.389+0000 I STORAGE [conn97] got request after shutdown()
|
2014-11-13T03:05:17.395+0000 I STORAGE [conn323] got request after shutdown()
|
2014-11-13T03:05:17.413+0000 I STORAGE [conn99] got request after shutdown()
|
2014-11-13T03:05:17.424+0000 I STORAGE [conn324] got request after shutdown()
|
2014-11-13T03:05:17.425+0000 I STORAGE [conn203] got request after shutdown()
|
2014-11-13T03:05:17.426+0000 I STORAGE [conn183] got request after shutdown()
|
2014-11-13T03:05:17.427+0000 I NETWORK [conn180] end connection 10.218.181.240:36615 (121 connections now open)
|
2014-11-13T03:05:17.458+0000 I STORAGE [conn184] got request after shutdown()
|
2014-11-13T03:05:17.463+0000 I STORAGE [conn325] got request after shutdown()
|
2014-11-13T03:05:17.473+0000 I STORAGE [conn100] got request after shutdown()
|
2014-11-13T03:05:17.498+0000 I STORAGE [conn101] got request after shutdown()
|
2014-11-13T03:05:17.499+0000 I STORAGE [conn102] got request after shutdown()
|
2014-11-13T03:05:17.511+0000 I STORAGE [conn103] got request after shutdown()
|
2014-11-13T03:05:17.511+0000 I STORAGE [conn326] got request after shutdown()
|
2014-11-13T03:05:17.517+0000 I STORAGE [conn104] got request after shutdown()
|
2014-11-13T03:05:17.519+0000 I STORAGE [conn327] got request after shutdown()
|
2014-11-13T03:05:17.519+0000 I STORAGE [conn253] got request after shutdown()
|
2014-11-13T03:05:17.528+0000 I STORAGE [conn105] got request after shutdown()
|
2014-11-13T03:05:17.570+0000 I STORAGE [conn107] got request after shutdown()
|
2014-11-13T03:05:17.572+0000 I STORAGE [conn108] got request after shutdown()
|
2014-11-13T03:05:17.572+0000 I STORAGE [conn106] got request after shutdown()
|
2014-11-13T03:05:17.578+0000 I STORAGE [conn110] got request after shutdown()
|
2014-11-13T03:05:17.578+0000 I STORAGE [conn109] got request after shutdown()
|
2014-11-13T03:05:17.592+0000 I STORAGE [conn111] got request after shutdown()
|
2014-11-13T03:05:17.600+0000 I STORAGE [conn112] got request after shutdown()
|
2014-11-13T03:05:17.780+0000 I STORAGE [conn330] got request after shutdown()
|
2014-11-13T03:05:17.782+0000 I STORAGE [conn329] got request after shutdown()
|
2014-11-13T03:05:17.826+0000 I STORAGE [conn328] got request after shutdown()
|
2014-11-13T03:05:17.836+0000 I STORAGE [conn331] got request after shutdown()
|
2014-11-13T03:05:17.845+0000 I STORAGE [conn333] got request after shutdown()
|
2014-11-13T03:05:17.846+0000 I STORAGE [conn332] got request after shutdown()
|
2014-11-13T03:05:18.181+0000 I STORAGE [conn212] got request after shutdown()
|
2014-11-13T03:05:18.360+0000 I STORAGE [conn213] got request after shutdown()
|
2014-11-13T03:05:18.380+0000 I STORAGE [conn214] got request after shutdown()
|
2014-11-13T03:05:18.414+0000 I STORAGE [conn215] got request after shutdown()
|
2014-11-13T03:05:18.443+0000 I STORAGE [conn216] got request after shutdown()
|
2014-11-13T03:05:18.502+0000 I STORAGE [conn217] got request after shutdown()
|
2014-11-13T03:05:18.525+0000 I STORAGE [conn218] got request after shutdown()
|
2014-11-13T03:05:18.561+0000 I STORAGE [conn219] got request after shutdown()
|
2014-11-13T03:05:18.623+0000 I STORAGE [conn220] got request after shutdown()
|
2014-11-13T03:05:18.644+0000 I STORAGE [conn119] got request after shutdown()
|
2014-11-13T03:05:18.645+0000 I NETWORK [conn4] end connection 10.167.150.115:47633 (86 connections now open)
|
2014-11-13T03:05:18.670+0000 I STORAGE [conn221] got request after shutdown()
|
2014-11-13T03:05:18.686+0000 I STORAGE [conn222] got request after shutdown()
|
2014-11-13T03:05:18.728+0000 I STORAGE [conn223] got request after shutdown()
|
2014-11-13T03:05:18.752+0000 I STORAGE [conn224] got request after shutdown()
|
2014-11-13T03:05:18.775+0000 I STORAGE [conn225] got request after shutdown()
|
2014-11-13T03:05:18.808+0000 I STORAGE [conn226] got request after shutdown()
|
2014-11-13T03:05:18.823+0000 I STORAGE [conn227] got request after shutdown()
|
2014-11-13T03:05:18.857+0000 I STORAGE [conn228] got request after shutdown()
|
2014-11-13T03:05:18.876+0000 I STORAGE [conn229] got request after shutdown()
|
2014-11-13T03:05:19.024+0000 I STORAGE [conn73] got request after shutdown()
|
2014-11-13T03:05:19.075+0000 I STORAGE [conn230] got request after shutdown()
|
2014-11-13T03:05:19.144+0000 I STORAGE [conn74] got request after shutdown()
|
2014-11-13T03:05:19.145+0000 I STORAGE [conn115] got request after shutdown()
|
2014-11-13T03:05:19.168+0000 I STORAGE [conn75] got request after shutdown()
|
2014-11-13T03:05:19.182+0000 I STORAGE [conn76] got request after shutdown()
|
2014-11-13T03:05:19.221+0000 I STORAGE [conn77] got request after shutdown()
|
2014-11-13T03:05:19.221+0000 I STORAGE [conn78] got request after shutdown()
|
2014-11-13T03:05:19.242+0000 I STORAGE [conn79] got request after shutdown()
|
2014-11-13T03:05:19.261+0000 I STORAGE [conn231] got request after shutdown()
|
2014-11-13T03:05:19.280+0000 I STORAGE [conn80] got request after shutdown()
|
2014-11-13T03:05:19.290+0000 I STORAGE [conn81] got request after shutdown()
|
2014-11-13T03:05:19.295+0000 I STORAGE [conn232] got request after shutdown()
|
2014-11-13T03:05:19.303+0000 I STORAGE [conn233] got request after shutdown()
|
2014-11-13T03:05:19.311+0000 I STORAGE [conn82] got request after shutdown()
|
2014-11-13T03:05:19.329+0000 I STORAGE [conn234] got request after shutdown()
|
2014-11-13T03:05:19.369+0000 I STORAGE [conn85] got request after shutdown()
|
2014-11-13T03:05:19.369+0000 I STORAGE [conn84] got request after shutdown()
|
2014-11-13T03:05:19.373+0000 I STORAGE [conn83] got request after shutdown()
|
2014-11-13T03:05:19.378+0000 I STORAGE [conn235] got request after shutdown()
|
2014-11-13T03:05:19.391+0000 I STORAGE [conn86] got request after shutdown()
|
2014-11-13T03:05:19.404+0000 I STORAGE [conn236] got request after shutdown()
|
2014-11-13T03:05:19.420+0000 I STORAGE [conn256] got request after shutdown()
|
2014-11-13T03:05:19.430+0000 I STORAGE [conn87] got request after shutdown()
|
2014-11-13T03:05:19.439+0000 I STORAGE [conn89] got request after shutdown()
|
2014-11-13T03:05:19.439+0000 I STORAGE [conn90] got request after shutdown()
|
2014-11-13T03:05:19.444+0000 I STORAGE [conn88] got request after shutdown()
|
2014-11-13T03:05:19.450+0000 I STORAGE [conn237] got request after shutdown()
|
2014-11-13T03:05:19.532+0000 I STORAGE [conn238] got request after shutdown()
|
2014-11-13T03:05:19.556+0000 I STORAGE [conn239] got request after shutdown()
|
2014-11-13T03:05:19.612+0000 I STORAGE [conn240] got request after shutdown()
|
2014-11-13T03:05:19.624+0000 I STORAGE [conn241] got request after shutdown()
|
2014-11-13T03:05:19.642+0000 I STORAGE [conn242] got request after shutdown()
|
2014-11-13T03:05:19.667+0000 I STORAGE [conn243] got request after shutdown()
|
2014-11-13T03:05:19.687+0000 I STORAGE [conn50] got request after shutdown()
|
2014-11-13T03:05:19.713+0000 I STORAGE [conn244] got request after shutdown()
|
2014-11-13T03:05:19.723+0000 I STORAGE [conn257] got request after shutdown()
|
2014-11-13T03:05:19.732+0000 I STORAGE [conn245] got request after shutdown()
|
2014-11-13T03:05:19.752+0000 I STORAGE [conn258] got request after shutdown()
|
2014-11-13T03:05:19.756+0000 I STORAGE [conn246] got request after shutdown()
|
2014-11-13T03:05:19.791+0000 I STORAGE [conn247] got request after shutdown()
|
2014-11-13T03:05:19.793+0000 I STORAGE [conn259] got request after shutdown()
|
2014-11-13T03:05:19.802+0000 I STORAGE [conn52] got request after shutdown()
|
2014-11-13T03:05:19.803+0000 I STORAGE [conn51] got request after shutdown()
|
2014-11-13T03:05:19.805+0000 I STORAGE [conn53] got request after shutdown()
|
2014-11-13T03:05:19.816+0000 I STORAGE [conn260] got request after shutdown()
|
2014-11-13T03:05:19.846+0000 I STORAGE [conn54] got request after shutdown()
|
2014-11-13T03:05:19.850+0000 I STORAGE [conn261] got request after shutdown()
|
2014-11-13T03:05:19.891+0000 I STORAGE [conn56] got request after shutdown()
|
2014-11-13T03:05:19.910+0000 I STORAGE [conn262] got request after shutdown()
|
2014-11-13T03:05:19.910+0000 I STORAGE [conn58] got request after shutdown()
|
2014-11-13T03:05:19.923+0000 I STORAGE [conn59] got request after shutdown()
|
2014-11-13T03:05:19.931+0000 I STORAGE [conn57] got request after shutdown()
|
2014-11-13T03:05:19.955+0000 I STORAGE [conn263] got request after shutdown()
|
2014-11-13T03:05:19.966+0000 I STORAGE [conn60] got request after shutdown()
|
2014-11-13T03:05:19.980+0000 I STORAGE [conn61] got request after shutdown()
|
2014-11-13T03:05:19.985+0000 I STORAGE [conn264] got request after shutdown()
|
2014-11-13T03:05:20.020+0000 I STORAGE [conn91] got request after shutdown()
|
2014-11-13T03:05:20.020+0000 I STORAGE [conn63] got request after shutdown()
|
2014-11-13T03:05:20.021+0000 I STORAGE [conn62] got request after shutdown()
|
2014-11-13T03:05:20.039+0000 I STORAGE [conn64] got request after shutdown()
|
2014-11-13T03:05:20.048+0000 I STORAGE [conn65] got request after shutdown()
|
2014-11-13T03:05:20.050+0000 I STORAGE [conn265] got request after shutdown()
|
2014-11-13T03:05:20.069+0000 I STORAGE [conn66] got request after shutdown()
|
2014-11-13T03:05:20.072+0000 I STORAGE [conn266] got request after shutdown()
|
2014-11-13T03:05:20.100+0000 I STORAGE [conn267] got request after shutdown()
|
2014-11-13T03:05:20.114+0000 I STORAGE [conn67] got request after shutdown()
|
2014-11-13T03:05:20.189+0000 I STORAGE [conn268] got request after shutdown()
|
2014-11-13T03:05:20.209+0000 I STORAGE [conn269] got request after shutdown()
|
2014-11-13T03:05:20.227+0000 I STORAGE [conn270] got request after shutdown()
|
2014-11-13T03:05:20.230+0000 I STORAGE [conn271] got request after shutdown()
|
2014-11-13T03:05:20.265+0000 I STORAGE [conn272] got request after shutdown()
|
2014-11-13T03:05:20.309+0000 I STORAGE [conn273] got request after shutdown()
|
2014-11-13T03:05:21.032+0000 I NETWORK [conn186] end connection 10.167.150.115:48997 (3 connections now open)
|
2014-11-13T03:05:21.053+0000 I NETWORK [conn189] end connection 10.167.150.115:48999 (2 connections now open)
|
2014-11-13T03:05:22.468+0000 I NETWORK [conn181] end connection 10.218.181.240:36618 (1 connection now open)
|
2014-11-13T03:05:45.376+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:05:45.377+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:05:45.377+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:05:45.377+0000 I - [rsSync]
|
2014-11-13T03:05:45.388+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:06:15.424+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:06:15.424+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:06:15.424+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:06:15.424+0000 I - [rsSync]
|
2014-11-13T03:06:15.430+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:06:33.467+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
|
2014-11-13T03:06:45.466+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:06:45.466+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:06:45.466+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:06:45.466+0000 I - [rsSync]
|
2014-11-13T03:06:45.472+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:07:15.509+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:07:15.509+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:07:15.509+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:07:15.509+0000 I - [rsSync]
|
2014-11-13T03:07:15.516+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:07:45.551+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:07:45.551+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:07:45.551+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:07:45.551+0000 I - [rsSync]
|
2014-11-13T03:07:45.558+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:08:15.594+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:08:15.594+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:08:15.594+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:08:15.594+0000 I - [rsSync]
|
2014-11-13T03:08:15.600+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
2014-11-13T03:08:45.636+0000 I - [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
|
2014-11-13T03:08:45.636+0000 I - [rsSync] Dumping LockManager @ 0x1b5e000
|
2014-11-13T03:08:45.636+0000 I - [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
|
GRANTED:
|
LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE;
|
|
|
PENDING:
|
LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE;
|
2014-11-13T03:08:45.636+0000 I - [rsSync]
|
2014-11-13T03:08:45.642+0000 I - [rsSync]
|
0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
|
mongod(+0x5ADCCA) [0x9adcca]
|
mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
|
mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
|
mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
|
mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
|
mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
|
mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
|
mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
|
mongod(+0x8B8DF8) [0xcb8df8]
|
mongod(+0x8B9742) [0xcb9742]
|
mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
|
mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
|
mongod(+0xBBD5C4) [0xfbd5c4]
|
libpthread.so.0(+0x8182) [0x7fab5fcf7182]
|
libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
|
----- END BACKTRACE -----
|
A kill -9 was able to kill the process:
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
|
8065 ? Sl 6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
|
11956 pts/0 S+ 0:00 grep --color=auto mmsdev_4
|
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill 8065
|
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
|
8065 ? Sl 6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
|
11962 pts/0 R+ 0:00 grep --color=auto mmsdev_4
|
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill -9 8065
|
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
|
11974 pts/0 S+ 0:00 grep --color=auto mmsdev_4
|
Attachments
Issue Links
- duplicates
-
SERVER-16105 deadlock at shutdown
-
- Closed
-