Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Storage
Labels:
None

Operating System:
ALL

I tried to shut down my 2.8.0rc0 (wiredtiger) secondary during an initial sync.

When I typed db.shutdownServer(), it hung. Eventually a CTRL-Ced that a few times and escaped the shell. Then, I tried a kill, that did not work.

The process is still running, but I cannot connect to it anymore in the shell. The log file says this:

2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] build index done.  scanned 2 total records. 0 secs
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index on: mmsdbconfig.data.exposedHosts properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.data.exposedHosts" }
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index done.  scanned 0 total records. 0 secs
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.replicaSetChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.replicaSetChartSelects" }
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index done.  scanned 9 total records. 0 secs
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.registrationThrottles properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.registrationThrottles" }
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index done.  scanned 2 total records. 0 secs
2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.automationVersions properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.automationVersions" }
2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.684+0000 I INDEXING [rsSync] build index done.  scanned 28 total records. 0 secs
2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] build index on: mmsdbconfig.app.systemDbStats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.app.systemDbStats" }
2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.689+0000 I INDEXING [rsSync] build index done.  scanned 16 total records. 0 secs
2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.alertConfigHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.alertConfigHistory" }
2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.696+0000 I INDEXING [rsSync] build index done.  scanned 41 total records. 0 secs
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostChartSelects" }
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index done.  scanned 15 total records. 0 secs
2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostClusterHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostClusterHistory" }
2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T03:05:07.708+0000 I INDEXING [rsSync] build index done.  scanned 14 total records. 0 secs
2014-11-13T03:05:07.708+0000 I REPLSETS [rsSync] initial sync cloning db: 540bb74fe4b039a8d31a04a7_A
2014-11-13T03:05:10.359+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:45903 #348 (189 connections now open)
2014-11-13T03:05:15.341+0000 I COMMANDS [conn348] terminating, shutdown command received
2014-11-13T03:05:15.632+0000 I STORAGE  [conn296] got request after shutdown()
2014-11-13T03:05:15.757+0000 I STORAGE  [conn297] got request after shutdown()
2014-11-13T03:05:15.980+0000 I STORAGE  [conn298] got request after shutdown()
2014-11-13T03:05:15.996+0000 I STORAGE  [conn299] got request after shutdown()
2014-11-13T03:05:16.019+0000 I STORAGE  [conn187] got request after shutdown()
2014-11-13T03:05:16.019+0000 I STORAGE  [conn194] got request after shutdown()
2014-11-13T03:05:16.030+0000 I STORAGE  [conn192] got request after shutdown()
2014-11-13T03:05:16.032+0000 I STORAGE  [conn300] got request after shutdown()
2014-11-13T03:05:16.051+0000 I STORAGE  [conn195] got request after shutdown()
2014-11-13T03:05:16.061+0000 I STORAGE  [conn301] got request after shutdown()
2014-11-13T03:05:16.061+0000 I STORAGE  [conn346] got request after shutdown()
2014-11-13T03:05:16.111+0000 I STORAGE  [conn302] got request after shutdown()
2014-11-13T03:05:16.137+0000 I STORAGE  [conn303] got request after shutdown()
2014-11-13T03:05:16.180+0000 I STORAGE  [conn304] got request after shutdown()
2014-11-13T03:05:16.230+0000 I STORAGE  [conn305] got request after shutdown()
2014-11-13T03:05:16.235+0000 I STORAGE  [conn274] got request after shutdown()
2014-11-13T03:05:16.244+0000 I STORAGE  [conn188] got request after shutdown()
2014-11-13T03:05:16.247+0000 I STORAGE  [conn210] got request after shutdown()
2014-11-13T03:05:16.248+0000 I STORAGE  [conn193] got request after shutdown()
2014-11-13T03:05:16.248+0000 I STORAGE  [conn211] got request after shutdown()
2014-11-13T03:05:16.248+0000 I STORAGE  [conn196] got request after shutdown()
2014-11-13T03:05:16.253+0000 I STORAGE  [conn190] got request after shutdown()
2014-11-13T03:05:16.258+0000 I STORAGE  [conn306] got request after shutdown()
2014-11-13T03:05:16.289+0000 I STORAGE  [conn307] got request after shutdown()
2014-11-13T03:05:16.315+0000 I STORAGE  [conn308] got request after shutdown()
2014-11-13T03:05:16.344+0000 I STORAGE  [conn309] got request after shutdown()
2014-11-13T03:05:16.392+0000 I STORAGE  [conn310] got request after shutdown()
2014-11-13T03:05:16.438+0000 I STORAGE  [conn275] got request after shutdown()
2014-11-13T03:05:16.445+0000 I STORAGE  [conn276] got request after shutdown()
2014-11-13T03:05:16.447+0000 I STORAGE  [conn311] got request after shutdown()
2014-11-13T03:05:16.478+0000 I STORAGE  [conn277] got request after shutdown()
2014-11-13T03:05:16.492+0000 I STORAGE  [conn312] got request after shutdown()
2014-11-13T03:05:16.503+0000 I STORAGE  [conn278] got request after shutdown()
2014-11-13T03:05:16.506+0000 I STORAGE  [conn313] got request after shutdown()
2014-11-13T03:05:16.514+0000 I STORAGE  [conn314] got request after shutdown()
2014-11-13T03:05:16.543+0000 I STORAGE  [conn279] got request after shutdown()
2014-11-13T03:05:16.595+0000 I STORAGE  [conn280] got request after shutdown()
2014-11-13T03:05:16.662+0000 I STORAGE  [conn281] got request after shutdown()
2014-11-13T03:05:16.697+0000 I STORAGE  [conn282] got request after shutdown()
2014-11-13T03:05:16.813+0000 I STORAGE  [conn283] got request after shutdown()
2014-11-13T03:05:16.814+0000 I STORAGE  [conn284] got request after shutdown()
2014-11-13T03:05:16.841+0000 I STORAGE  [conn315] got request after shutdown()
2014-11-13T03:05:16.843+0000 I STORAGE  [conn285] got request after shutdown()
2014-11-13T03:05:16.857+0000 I STORAGE  [conn286] got request after shutdown()
2014-11-13T03:05:16.887+0000 I STORAGE  [conn287] got request after shutdown()
2014-11-13T03:05:16.894+0000 I NETWORK  [conn345] end connection 10.167.150.115:49672 (143 connections now open)
2014-11-13T03:05:16.914+0000 I STORAGE  [conn288] got request after shutdown()
2014-11-13T03:05:16.938+0000 I STORAGE  [conn347] got request after shutdown()
2014-11-13T03:05:16.951+0000 I STORAGE  [conn289] got request after shutdown()
2014-11-13T03:05:16.976+0000 I STORAGE  [conn290] got request after shutdown()
2014-11-13T03:05:17.019+0000 I STORAGE  [conn291] got request after shutdown()
2014-11-13T03:05:17.197+0000 I STORAGE  [conn317] got request after shutdown()
2014-11-13T03:05:17.222+0000 I STORAGE  [conn318] got request after shutdown()
2014-11-13T03:05:17.233+0000 I STORAGE  [conn319] got request after shutdown()
2014-11-13T03:05:17.268+0000 I STORAGE  [conn320] got request after shutdown()
2014-11-13T03:05:17.314+0000 I STORAGE  [conn321] got request after shutdown()
2014-11-13T03:05:17.326+0000 I STORAGE  [conn95] got request after shutdown()
2014-11-13T03:05:17.342+0000 I STORAGE  [conn55] got request after shutdown()
2014-11-13T03:05:17.362+0000 I STORAGE  [conn322] got request after shutdown()
2014-11-13T03:05:17.370+0000 I STORAGE  [conn96] got request after shutdown()
2014-11-13T03:05:17.389+0000 I STORAGE  [conn98] got request after shutdown()
2014-11-13T03:05:17.389+0000 I STORAGE  [conn97] got request after shutdown()
2014-11-13T03:05:17.395+0000 I STORAGE  [conn323] got request after shutdown()
2014-11-13T03:05:17.413+0000 I STORAGE  [conn99] got request after shutdown()
2014-11-13T03:05:17.424+0000 I STORAGE  [conn324] got request after shutdown()
2014-11-13T03:05:17.425+0000 I STORAGE  [conn203] got request after shutdown()
2014-11-13T03:05:17.426+0000 I STORAGE  [conn183] got request after shutdown()
2014-11-13T03:05:17.427+0000 I NETWORK  [conn180] end connection 10.218.181.240:36615 (121 connections now open)
2014-11-13T03:05:17.458+0000 I STORAGE  [conn184] got request after shutdown()
2014-11-13T03:05:17.463+0000 I STORAGE  [conn325] got request after shutdown()
2014-11-13T03:05:17.473+0000 I STORAGE  [conn100] got request after shutdown()
2014-11-13T03:05:17.498+0000 I STORAGE  [conn101] got request after shutdown()
2014-11-13T03:05:17.499+0000 I STORAGE  [conn102] got request after shutdown()
2014-11-13T03:05:17.511+0000 I STORAGE  [conn103] got request after shutdown()
2014-11-13T03:05:17.511+0000 I STORAGE  [conn326] got request after shutdown()
2014-11-13T03:05:17.517+0000 I STORAGE  [conn104] got request after shutdown()
2014-11-13T03:05:17.519+0000 I STORAGE  [conn327] got request after shutdown()
2014-11-13T03:05:17.519+0000 I STORAGE  [conn253] got request after shutdown()
2014-11-13T03:05:17.528+0000 I STORAGE  [conn105] got request after shutdown()
2014-11-13T03:05:17.570+0000 I STORAGE  [conn107] got request after shutdown()
2014-11-13T03:05:17.572+0000 I STORAGE  [conn108] got request after shutdown()
2014-11-13T03:05:17.572+0000 I STORAGE  [conn106] got request after shutdown()
2014-11-13T03:05:17.578+0000 I STORAGE  [conn110] got request after shutdown()
2014-11-13T03:05:17.578+0000 I STORAGE  [conn109] got request after shutdown()
2014-11-13T03:05:17.592+0000 I STORAGE  [conn111] got request after shutdown()
2014-11-13T03:05:17.600+0000 I STORAGE  [conn112] got request after shutdown()
2014-11-13T03:05:17.780+0000 I STORAGE  [conn330] got request after shutdown()
2014-11-13T03:05:17.782+0000 I STORAGE  [conn329] got request after shutdown()
2014-11-13T03:05:17.826+0000 I STORAGE  [conn328] got request after shutdown()
2014-11-13T03:05:17.836+0000 I STORAGE  [conn331] got request after shutdown()
2014-11-13T03:05:17.845+0000 I STORAGE  [conn333] got request after shutdown()
2014-11-13T03:05:17.846+0000 I STORAGE  [conn332] got request after shutdown()
2014-11-13T03:05:18.181+0000 I STORAGE  [conn212] got request after shutdown()
2014-11-13T03:05:18.360+0000 I STORAGE  [conn213] got request after shutdown()
2014-11-13T03:05:18.380+0000 I STORAGE  [conn214] got request after shutdown()
2014-11-13T03:05:18.414+0000 I STORAGE  [conn215] got request after shutdown()
2014-11-13T03:05:18.443+0000 I STORAGE  [conn216] got request after shutdown()
2014-11-13T03:05:18.502+0000 I STORAGE  [conn217] got request after shutdown()
2014-11-13T03:05:18.525+0000 I STORAGE  [conn218] got request after shutdown()
2014-11-13T03:05:18.561+0000 I STORAGE  [conn219] got request after shutdown()
2014-11-13T03:05:18.623+0000 I STORAGE  [conn220] got request after shutdown()
2014-11-13T03:05:18.644+0000 I STORAGE  [conn119] got request after shutdown()
2014-11-13T03:05:18.645+0000 I NETWORK  [conn4] end connection 10.167.150.115:47633 (86 connections now open)
2014-11-13T03:05:18.670+0000 I STORAGE  [conn221] got request after shutdown()
2014-11-13T03:05:18.686+0000 I STORAGE  [conn222] got request after shutdown()
2014-11-13T03:05:18.728+0000 I STORAGE  [conn223] got request after shutdown()
2014-11-13T03:05:18.752+0000 I STORAGE  [conn224] got request after shutdown()
2014-11-13T03:05:18.775+0000 I STORAGE  [conn225] got request after shutdown()
2014-11-13T03:05:18.808+0000 I STORAGE  [conn226] got request after shutdown()
2014-11-13T03:05:18.823+0000 I STORAGE  [conn227] got request after shutdown()
2014-11-13T03:05:18.857+0000 I STORAGE  [conn228] got request after shutdown()
2014-11-13T03:05:18.876+0000 I STORAGE  [conn229] got request after shutdown()
2014-11-13T03:05:19.024+0000 I STORAGE  [conn73] got request after shutdown()
2014-11-13T03:05:19.075+0000 I STORAGE  [conn230] got request after shutdown()
2014-11-13T03:05:19.144+0000 I STORAGE  [conn74] got request after shutdown()
2014-11-13T03:05:19.145+0000 I STORAGE  [conn115] got request after shutdown()
2014-11-13T03:05:19.168+0000 I STORAGE  [conn75] got request after shutdown()
2014-11-13T03:05:19.182+0000 I STORAGE  [conn76] got request after shutdown()
2014-11-13T03:05:19.221+0000 I STORAGE  [conn77] got request after shutdown()
2014-11-13T03:05:19.221+0000 I STORAGE  [conn78] got request after shutdown()
2014-11-13T03:05:19.242+0000 I STORAGE  [conn79] got request after shutdown()
2014-11-13T03:05:19.261+0000 I STORAGE  [conn231] got request after shutdown()
2014-11-13T03:05:19.280+0000 I STORAGE  [conn80] got request after shutdown()
2014-11-13T03:05:19.290+0000 I STORAGE  [conn81] got request after shutdown()
2014-11-13T03:05:19.295+0000 I STORAGE  [conn232] got request after shutdown()
2014-11-13T03:05:19.303+0000 I STORAGE  [conn233] got request after shutdown()
2014-11-13T03:05:19.311+0000 I STORAGE  [conn82] got request after shutdown()
2014-11-13T03:05:19.329+0000 I STORAGE  [conn234] got request after shutdown()
2014-11-13T03:05:19.369+0000 I STORAGE  [conn85] got request after shutdown()
2014-11-13T03:05:19.369+0000 I STORAGE  [conn84] got request after shutdown()
2014-11-13T03:05:19.373+0000 I STORAGE  [conn83] got request after shutdown()
2014-11-13T03:05:19.378+0000 I STORAGE  [conn235] got request after shutdown()
2014-11-13T03:05:19.391+0000 I STORAGE  [conn86] got request after shutdown()
2014-11-13T03:05:19.404+0000 I STORAGE  [conn236] got request after shutdown()
2014-11-13T03:05:19.420+0000 I STORAGE  [conn256] got request after shutdown()
2014-11-13T03:05:19.430+0000 I STORAGE  [conn87] got request after shutdown()
2014-11-13T03:05:19.439+0000 I STORAGE  [conn89] got request after shutdown()
2014-11-13T03:05:19.439+0000 I STORAGE  [conn90] got request after shutdown()
2014-11-13T03:05:19.444+0000 I STORAGE  [conn88] got request after shutdown()
2014-11-13T03:05:19.450+0000 I STORAGE  [conn237] got request after shutdown()
2014-11-13T03:05:19.532+0000 I STORAGE  [conn238] got request after shutdown()
2014-11-13T03:05:19.556+0000 I STORAGE  [conn239] got request after shutdown()
2014-11-13T03:05:19.612+0000 I STORAGE  [conn240] got request after shutdown()
2014-11-13T03:05:19.624+0000 I STORAGE  [conn241] got request after shutdown()
2014-11-13T03:05:19.642+0000 I STORAGE  [conn242] got request after shutdown()
2014-11-13T03:05:19.667+0000 I STORAGE  [conn243] got request after shutdown()
2014-11-13T03:05:19.687+0000 I STORAGE  [conn50] got request after shutdown()
2014-11-13T03:05:19.713+0000 I STORAGE  [conn244] got request after shutdown()
2014-11-13T03:05:19.723+0000 I STORAGE  [conn257] got request after shutdown()
2014-11-13T03:05:19.732+0000 I STORAGE  [conn245] got request after shutdown()
2014-11-13T03:05:19.752+0000 I STORAGE  [conn258] got request after shutdown()
2014-11-13T03:05:19.756+0000 I STORAGE  [conn246] got request after shutdown()
2014-11-13T03:05:19.791+0000 I STORAGE  [conn247] got request after shutdown()
2014-11-13T03:05:19.793+0000 I STORAGE  [conn259] got request after shutdown()
2014-11-13T03:05:19.802+0000 I STORAGE  [conn52] got request after shutdown()
2014-11-13T03:05:19.803+0000 I STORAGE  [conn51] got request after shutdown()
2014-11-13T03:05:19.805+0000 I STORAGE  [conn53] got request after shutdown()
2014-11-13T03:05:19.816+0000 I STORAGE  [conn260] got request after shutdown()
2014-11-13T03:05:19.846+0000 I STORAGE  [conn54] got request after shutdown()
2014-11-13T03:05:19.850+0000 I STORAGE  [conn261] got request after shutdown()
2014-11-13T03:05:19.891+0000 I STORAGE  [conn56] got request after shutdown()
2014-11-13T03:05:19.910+0000 I STORAGE  [conn262] got request after shutdown()
2014-11-13T03:05:19.910+0000 I STORAGE  [conn58] got request after shutdown()
2014-11-13T03:05:19.923+0000 I STORAGE  [conn59] got request after shutdown()
2014-11-13T03:05:19.931+0000 I STORAGE  [conn57] got request after shutdown()
2014-11-13T03:05:19.955+0000 I STORAGE  [conn263] got request after shutdown()
2014-11-13T03:05:19.966+0000 I STORAGE  [conn60] got request after shutdown()
2014-11-13T03:05:19.980+0000 I STORAGE  [conn61] got request after shutdown()
2014-11-13T03:05:19.985+0000 I STORAGE  [conn264] got request after shutdown()
2014-11-13T03:05:20.020+0000 I STORAGE  [conn91] got request after shutdown()
2014-11-13T03:05:20.020+0000 I STORAGE  [conn63] got request after shutdown()
2014-11-13T03:05:20.021+0000 I STORAGE  [conn62] got request after shutdown()
2014-11-13T03:05:20.039+0000 I STORAGE  [conn64] got request after shutdown()
2014-11-13T03:05:20.048+0000 I STORAGE  [conn65] got request after shutdown()
2014-11-13T03:05:20.050+0000 I STORAGE  [conn265] got request after shutdown()
2014-11-13T03:05:20.069+0000 I STORAGE  [conn66] got request after shutdown()
2014-11-13T03:05:20.072+0000 I STORAGE  [conn266] got request after shutdown()
2014-11-13T03:05:20.100+0000 I STORAGE  [conn267] got request after shutdown()
2014-11-13T03:05:20.114+0000 I STORAGE  [conn67] got request after shutdown()
2014-11-13T03:05:20.189+0000 I STORAGE  [conn268] got request after shutdown()
2014-11-13T03:05:20.209+0000 I STORAGE  [conn269] got request after shutdown()
2014-11-13T03:05:20.227+0000 I STORAGE  [conn270] got request after shutdown()
2014-11-13T03:05:20.230+0000 I STORAGE  [conn271] got request after shutdown()
2014-11-13T03:05:20.265+0000 I STORAGE  [conn272] got request after shutdown()
2014-11-13T03:05:20.309+0000 I STORAGE  [conn273] got request after shutdown()
2014-11-13T03:05:21.032+0000 I NETWORK  [conn186] end connection 10.167.150.115:48997 (3 connections now open)
2014-11-13T03:05:21.053+0000 I NETWORK  [conn189] end connection 10.167.150.115:48999 (2 connections now open)
2014-11-13T03:05:22.468+0000 I NETWORK  [conn181] end connection 10.218.181.240:36618 (1 connection now open)
2014-11-13T03:05:45.376+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:05:45.377+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:05:45.377+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:05:45.377+0000 I -        [rsSync] 
2014-11-13T03:05:45.388+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:06:15.424+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:06:15.424+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:06:15.424+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:06:15.424+0000 I -        [rsSync] 
2014-11-13T03:06:15.430+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:06:33.467+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2014-11-13T03:06:45.466+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:06:45.466+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:06:45.466+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:06:45.466+0000 I -        [rsSync] 
2014-11-13T03:06:45.472+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:07:15.509+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:07:15.509+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:07:15.509+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:07:15.509+0000 I -        [rsSync] 
2014-11-13T03:07:15.516+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:07:45.551+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:07:45.551+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:07:45.551+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:07:45.551+0000 I -        [rsSync] 
2014-11-13T03:07:45.558+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:08:15.594+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:08:15.594+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:08:15.594+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:08:15.594+0000 I -        [rsSync] 
2014-11-13T03:08:15.600+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----
2014-11-13T03:08:45.636+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
2014-11-13T03:08:45.636+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
2014-11-13T03:08:45.636+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
GRANTED:
	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 

PENDING:
	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
2014-11-13T03:08:45.636+0000 I -        [rsSync] 
2014-11-13T03:08:45.642+0000 I -        [rsSync] 
 0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0x5ADCCA) [0x9adcca]
 mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
 mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
 mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7fab5fcf7182]
 libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
-----  END BACKTRACE  -----

A kill -9 was able to kill the process:

ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
 8065 ?        Sl     6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
11956 pts/0    S+     0:00 grep --color=auto mmsdev_4
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill 8065
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
 8065 ?        Sl     6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
11962 pts/0    R+     0:00 grep --color=auto mmsdev_4
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill -9 8065 
ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
11974 pts/0    S+     0:00 grep --color=auto mmsdev_4

duplicates

SERVER-16105 deadlock at shutdown

Closed

Assignee:: Eric Milkie

Reporter:: Cailin Nelson

Participants:: Cailin Nelson, Eric Milkie

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: Nov 13 2014 03:11:59 AM UTC

Updated:: Dec 10 2014 11:06:10 PM UTC

Resolved:: Nov 13 2014 02:08:11 PM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates