Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16120

Failure to shut down 2.8.0rc0 during initial sync (wiredtiger)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Operating System:
      ALL

      Description

      I tried to shut down my 2.8.0rc0 (wiredtiger) secondary during an initial sync.

      When I typed db.shutdownServer(), it hung. Eventually a CTRL-Ced that a few times and escaped the shell. Then, I tried a kill, that did not work.

      The process is still running, but I cannot connect to it anymore in the shell. The log file says this:

      2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.663+0000 I INDEXING [rsSync] build index done.  scanned 2 total records. 0 secs
      2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index on: mmsdbconfig.data.exposedHosts properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.data.exposedHosts" }
      2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.668+0000 I INDEXING [rsSync] build index done.  scanned 0 total records. 0 secs
      2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.replicaSetChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.replicaSetChartSelects" }
      2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.673+0000 I INDEXING [rsSync] build index done.  scanned 9 total records. 0 secs
      2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.registrationThrottles properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.registrationThrottles" }
      2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.678+0000 I INDEXING [rsSync] build index done.  scanned 2 total records. 0 secs
      2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.automationVersions properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.automationVersions" }
      2014-11-13T03:05:07.683+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.684+0000 I INDEXING [rsSync] build index done.  scanned 28 total records. 0 secs
      2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] build index on: mmsdbconfig.app.systemDbStats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.app.systemDbStats" }
      2014-11-13T03:05:07.688+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.689+0000 I INDEXING [rsSync] build index done.  scanned 16 total records. 0 secs
      2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.alertConfigHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.alertConfigHistory" }
      2014-11-13T03:05:07.695+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.696+0000 I INDEXING [rsSync] build index done.  scanned 41 total records. 0 secs
      2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostChartSelects" }
      2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.702+0000 I INDEXING [rsSync] build index done.  scanned 15 total records. 0 secs
      2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostClusterHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostClusterHistory" }
      2014-11-13T03:05:07.707+0000 I INDEXING [rsSync] 	 building index using bulk method
      2014-11-13T03:05:07.708+0000 I INDEXING [rsSync] build index done.  scanned 14 total records. 0 secs
      2014-11-13T03:05:07.708+0000 I REPLSETS [rsSync] initial sync cloning db: 540bb74fe4b039a8d31a04a7_A
      2014-11-13T03:05:10.359+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:45903 #348 (189 connections now open)
      2014-11-13T03:05:15.341+0000 I COMMANDS [conn348] terminating, shutdown command received
      2014-11-13T03:05:15.632+0000 I STORAGE  [conn296] got request after shutdown()
      2014-11-13T03:05:15.757+0000 I STORAGE  [conn297] got request after shutdown()
      2014-11-13T03:05:15.980+0000 I STORAGE  [conn298] got request after shutdown()
      2014-11-13T03:05:15.996+0000 I STORAGE  [conn299] got request after shutdown()
      2014-11-13T03:05:16.019+0000 I STORAGE  [conn187] got request after shutdown()
      2014-11-13T03:05:16.019+0000 I STORAGE  [conn194] got request after shutdown()
      2014-11-13T03:05:16.030+0000 I STORAGE  [conn192] got request after shutdown()
      2014-11-13T03:05:16.032+0000 I STORAGE  [conn300] got request after shutdown()
      2014-11-13T03:05:16.051+0000 I STORAGE  [conn195] got request after shutdown()
      2014-11-13T03:05:16.061+0000 I STORAGE  [conn301] got request after shutdown()
      2014-11-13T03:05:16.061+0000 I STORAGE  [conn346] got request after shutdown()
      2014-11-13T03:05:16.111+0000 I STORAGE  [conn302] got request after shutdown()
      2014-11-13T03:05:16.137+0000 I STORAGE  [conn303] got request after shutdown()
      2014-11-13T03:05:16.180+0000 I STORAGE  [conn304] got request after shutdown()
      2014-11-13T03:05:16.230+0000 I STORAGE  [conn305] got request after shutdown()
      2014-11-13T03:05:16.235+0000 I STORAGE  [conn274] got request after shutdown()
      2014-11-13T03:05:16.244+0000 I STORAGE  [conn188] got request after shutdown()
      2014-11-13T03:05:16.247+0000 I STORAGE  [conn210] got request after shutdown()
      2014-11-13T03:05:16.248+0000 I STORAGE  [conn193] got request after shutdown()
      2014-11-13T03:05:16.248+0000 I STORAGE  [conn211] got request after shutdown()
      2014-11-13T03:05:16.248+0000 I STORAGE  [conn196] got request after shutdown()
      2014-11-13T03:05:16.253+0000 I STORAGE  [conn190] got request after shutdown()
      2014-11-13T03:05:16.258+0000 I STORAGE  [conn306] got request after shutdown()
      2014-11-13T03:05:16.289+0000 I STORAGE  [conn307] got request after shutdown()
      2014-11-13T03:05:16.315+0000 I STORAGE  [conn308] got request after shutdown()
      2014-11-13T03:05:16.344+0000 I STORAGE  [conn309] got request after shutdown()
      2014-11-13T03:05:16.392+0000 I STORAGE  [conn310] got request after shutdown()
      2014-11-13T03:05:16.438+0000 I STORAGE  [conn275] got request after shutdown()
      2014-11-13T03:05:16.445+0000 I STORAGE  [conn276] got request after shutdown()
      2014-11-13T03:05:16.447+0000 I STORAGE  [conn311] got request after shutdown()
      2014-11-13T03:05:16.478+0000 I STORAGE  [conn277] got request after shutdown()
      2014-11-13T03:05:16.492+0000 I STORAGE  [conn312] got request after shutdown()
      2014-11-13T03:05:16.503+0000 I STORAGE  [conn278] got request after shutdown()
      2014-11-13T03:05:16.506+0000 I STORAGE  [conn313] got request after shutdown()
      2014-11-13T03:05:16.514+0000 I STORAGE  [conn314] got request after shutdown()
      2014-11-13T03:05:16.543+0000 I STORAGE  [conn279] got request after shutdown()
      2014-11-13T03:05:16.595+0000 I STORAGE  [conn280] got request after shutdown()
      2014-11-13T03:05:16.662+0000 I STORAGE  [conn281] got request after shutdown()
      2014-11-13T03:05:16.697+0000 I STORAGE  [conn282] got request after shutdown()
      2014-11-13T03:05:16.813+0000 I STORAGE  [conn283] got request after shutdown()
      2014-11-13T03:05:16.814+0000 I STORAGE  [conn284] got request after shutdown()
      2014-11-13T03:05:16.841+0000 I STORAGE  [conn315] got request after shutdown()
      2014-11-13T03:05:16.843+0000 I STORAGE  [conn285] got request after shutdown()
      2014-11-13T03:05:16.857+0000 I STORAGE  [conn286] got request after shutdown()
      2014-11-13T03:05:16.887+0000 I STORAGE  [conn287] got request after shutdown()
      2014-11-13T03:05:16.894+0000 I NETWORK  [conn345] end connection 10.167.150.115:49672 (143 connections now open)
      2014-11-13T03:05:16.914+0000 I STORAGE  [conn288] got request after shutdown()
      2014-11-13T03:05:16.938+0000 I STORAGE  [conn347] got request after shutdown()
      2014-11-13T03:05:16.951+0000 I STORAGE  [conn289] got request after shutdown()
      2014-11-13T03:05:16.976+0000 I STORAGE  [conn290] got request after shutdown()
      2014-11-13T03:05:17.019+0000 I STORAGE  [conn291] got request after shutdown()
      2014-11-13T03:05:17.197+0000 I STORAGE  [conn317] got request after shutdown()
      2014-11-13T03:05:17.222+0000 I STORAGE  [conn318] got request after shutdown()
      2014-11-13T03:05:17.233+0000 I STORAGE  [conn319] got request after shutdown()
      2014-11-13T03:05:17.268+0000 I STORAGE  [conn320] got request after shutdown()
      2014-11-13T03:05:17.314+0000 I STORAGE  [conn321] got request after shutdown()
      2014-11-13T03:05:17.326+0000 I STORAGE  [conn95] got request after shutdown()
      2014-11-13T03:05:17.342+0000 I STORAGE  [conn55] got request after shutdown()
      2014-11-13T03:05:17.362+0000 I STORAGE  [conn322] got request after shutdown()
      2014-11-13T03:05:17.370+0000 I STORAGE  [conn96] got request after shutdown()
      2014-11-13T03:05:17.389+0000 I STORAGE  [conn98] got request after shutdown()
      2014-11-13T03:05:17.389+0000 I STORAGE  [conn97] got request after shutdown()
      2014-11-13T03:05:17.395+0000 I STORAGE  [conn323] got request after shutdown()
      2014-11-13T03:05:17.413+0000 I STORAGE  [conn99] got request after shutdown()
      2014-11-13T03:05:17.424+0000 I STORAGE  [conn324] got request after shutdown()
      2014-11-13T03:05:17.425+0000 I STORAGE  [conn203] got request after shutdown()
      2014-11-13T03:05:17.426+0000 I STORAGE  [conn183] got request after shutdown()
      2014-11-13T03:05:17.427+0000 I NETWORK  [conn180] end connection 10.218.181.240:36615 (121 connections now open)
      2014-11-13T03:05:17.458+0000 I STORAGE  [conn184] got request after shutdown()
      2014-11-13T03:05:17.463+0000 I STORAGE  [conn325] got request after shutdown()
      2014-11-13T03:05:17.473+0000 I STORAGE  [conn100] got request after shutdown()
      2014-11-13T03:05:17.498+0000 I STORAGE  [conn101] got request after shutdown()
      2014-11-13T03:05:17.499+0000 I STORAGE  [conn102] got request after shutdown()
      2014-11-13T03:05:17.511+0000 I STORAGE  [conn103] got request after shutdown()
      2014-11-13T03:05:17.511+0000 I STORAGE  [conn326] got request after shutdown()
      2014-11-13T03:05:17.517+0000 I STORAGE  [conn104] got request after shutdown()
      2014-11-13T03:05:17.519+0000 I STORAGE  [conn327] got request after shutdown()
      2014-11-13T03:05:17.519+0000 I STORAGE  [conn253] got request after shutdown()
      2014-11-13T03:05:17.528+0000 I STORAGE  [conn105] got request after shutdown()
      2014-11-13T03:05:17.570+0000 I STORAGE  [conn107] got request after shutdown()
      2014-11-13T03:05:17.572+0000 I STORAGE  [conn108] got request after shutdown()
      2014-11-13T03:05:17.572+0000 I STORAGE  [conn106] got request after shutdown()
      2014-11-13T03:05:17.578+0000 I STORAGE  [conn110] got request after shutdown()
      2014-11-13T03:05:17.578+0000 I STORAGE  [conn109] got request after shutdown()
      2014-11-13T03:05:17.592+0000 I STORAGE  [conn111] got request after shutdown()
      2014-11-13T03:05:17.600+0000 I STORAGE  [conn112] got request after shutdown()
      2014-11-13T03:05:17.780+0000 I STORAGE  [conn330] got request after shutdown()
      2014-11-13T03:05:17.782+0000 I STORAGE  [conn329] got request after shutdown()
      2014-11-13T03:05:17.826+0000 I STORAGE  [conn328] got request after shutdown()
      2014-11-13T03:05:17.836+0000 I STORAGE  [conn331] got request after shutdown()
      2014-11-13T03:05:17.845+0000 I STORAGE  [conn333] got request after shutdown()
      2014-11-13T03:05:17.846+0000 I STORAGE  [conn332] got request after shutdown()
      2014-11-13T03:05:18.181+0000 I STORAGE  [conn212] got request after shutdown()
      2014-11-13T03:05:18.360+0000 I STORAGE  [conn213] got request after shutdown()
      2014-11-13T03:05:18.380+0000 I STORAGE  [conn214] got request after shutdown()
      2014-11-13T03:05:18.414+0000 I STORAGE  [conn215] got request after shutdown()
      2014-11-13T03:05:18.443+0000 I STORAGE  [conn216] got request after shutdown()
      2014-11-13T03:05:18.502+0000 I STORAGE  [conn217] got request after shutdown()
      2014-11-13T03:05:18.525+0000 I STORAGE  [conn218] got request after shutdown()
      2014-11-13T03:05:18.561+0000 I STORAGE  [conn219] got request after shutdown()
      2014-11-13T03:05:18.623+0000 I STORAGE  [conn220] got request after shutdown()
      2014-11-13T03:05:18.644+0000 I STORAGE  [conn119] got request after shutdown()
      2014-11-13T03:05:18.645+0000 I NETWORK  [conn4] end connection 10.167.150.115:47633 (86 connections now open)
      2014-11-13T03:05:18.670+0000 I STORAGE  [conn221] got request after shutdown()
      2014-11-13T03:05:18.686+0000 I STORAGE  [conn222] got request after shutdown()
      2014-11-13T03:05:18.728+0000 I STORAGE  [conn223] got request after shutdown()
      2014-11-13T03:05:18.752+0000 I STORAGE  [conn224] got request after shutdown()
      2014-11-13T03:05:18.775+0000 I STORAGE  [conn225] got request after shutdown()
      2014-11-13T03:05:18.808+0000 I STORAGE  [conn226] got request after shutdown()
      2014-11-13T03:05:18.823+0000 I STORAGE  [conn227] got request after shutdown()
      2014-11-13T03:05:18.857+0000 I STORAGE  [conn228] got request after shutdown()
      2014-11-13T03:05:18.876+0000 I STORAGE  [conn229] got request after shutdown()
      2014-11-13T03:05:19.024+0000 I STORAGE  [conn73] got request after shutdown()
      2014-11-13T03:05:19.075+0000 I STORAGE  [conn230] got request after shutdown()
      2014-11-13T03:05:19.144+0000 I STORAGE  [conn74] got request after shutdown()
      2014-11-13T03:05:19.145+0000 I STORAGE  [conn115] got request after shutdown()
      2014-11-13T03:05:19.168+0000 I STORAGE  [conn75] got request after shutdown()
      2014-11-13T03:05:19.182+0000 I STORAGE  [conn76] got request after shutdown()
      2014-11-13T03:05:19.221+0000 I STORAGE  [conn77] got request after shutdown()
      2014-11-13T03:05:19.221+0000 I STORAGE  [conn78] got request after shutdown()
      2014-11-13T03:05:19.242+0000 I STORAGE  [conn79] got request after shutdown()
      2014-11-13T03:05:19.261+0000 I STORAGE  [conn231] got request after shutdown()
      2014-11-13T03:05:19.280+0000 I STORAGE  [conn80] got request after shutdown()
      2014-11-13T03:05:19.290+0000 I STORAGE  [conn81] got request after shutdown()
      2014-11-13T03:05:19.295+0000 I STORAGE  [conn232] got request after shutdown()
      2014-11-13T03:05:19.303+0000 I STORAGE  [conn233] got request after shutdown()
      2014-11-13T03:05:19.311+0000 I STORAGE  [conn82] got request after shutdown()
      2014-11-13T03:05:19.329+0000 I STORAGE  [conn234] got request after shutdown()
      2014-11-13T03:05:19.369+0000 I STORAGE  [conn85] got request after shutdown()
      2014-11-13T03:05:19.369+0000 I STORAGE  [conn84] got request after shutdown()
      2014-11-13T03:05:19.373+0000 I STORAGE  [conn83] got request after shutdown()
      2014-11-13T03:05:19.378+0000 I STORAGE  [conn235] got request after shutdown()
      2014-11-13T03:05:19.391+0000 I STORAGE  [conn86] got request after shutdown()
      2014-11-13T03:05:19.404+0000 I STORAGE  [conn236] got request after shutdown()
      2014-11-13T03:05:19.420+0000 I STORAGE  [conn256] got request after shutdown()
      2014-11-13T03:05:19.430+0000 I STORAGE  [conn87] got request after shutdown()
      2014-11-13T03:05:19.439+0000 I STORAGE  [conn89] got request after shutdown()
      2014-11-13T03:05:19.439+0000 I STORAGE  [conn90] got request after shutdown()
      2014-11-13T03:05:19.444+0000 I STORAGE  [conn88] got request after shutdown()
      2014-11-13T03:05:19.450+0000 I STORAGE  [conn237] got request after shutdown()
      2014-11-13T03:05:19.532+0000 I STORAGE  [conn238] got request after shutdown()
      2014-11-13T03:05:19.556+0000 I STORAGE  [conn239] got request after shutdown()
      2014-11-13T03:05:19.612+0000 I STORAGE  [conn240] got request after shutdown()
      2014-11-13T03:05:19.624+0000 I STORAGE  [conn241] got request after shutdown()
      2014-11-13T03:05:19.642+0000 I STORAGE  [conn242] got request after shutdown()
      2014-11-13T03:05:19.667+0000 I STORAGE  [conn243] got request after shutdown()
      2014-11-13T03:05:19.687+0000 I STORAGE  [conn50] got request after shutdown()
      2014-11-13T03:05:19.713+0000 I STORAGE  [conn244] got request after shutdown()
      2014-11-13T03:05:19.723+0000 I STORAGE  [conn257] got request after shutdown()
      2014-11-13T03:05:19.732+0000 I STORAGE  [conn245] got request after shutdown()
      2014-11-13T03:05:19.752+0000 I STORAGE  [conn258] got request after shutdown()
      2014-11-13T03:05:19.756+0000 I STORAGE  [conn246] got request after shutdown()
      2014-11-13T03:05:19.791+0000 I STORAGE  [conn247] got request after shutdown()
      2014-11-13T03:05:19.793+0000 I STORAGE  [conn259] got request after shutdown()
      2014-11-13T03:05:19.802+0000 I STORAGE  [conn52] got request after shutdown()
      2014-11-13T03:05:19.803+0000 I STORAGE  [conn51] got request after shutdown()
      2014-11-13T03:05:19.805+0000 I STORAGE  [conn53] got request after shutdown()
      2014-11-13T03:05:19.816+0000 I STORAGE  [conn260] got request after shutdown()
      2014-11-13T03:05:19.846+0000 I STORAGE  [conn54] got request after shutdown()
      2014-11-13T03:05:19.850+0000 I STORAGE  [conn261] got request after shutdown()
      2014-11-13T03:05:19.891+0000 I STORAGE  [conn56] got request after shutdown()
      2014-11-13T03:05:19.910+0000 I STORAGE  [conn262] got request after shutdown()
      2014-11-13T03:05:19.910+0000 I STORAGE  [conn58] got request after shutdown()
      2014-11-13T03:05:19.923+0000 I STORAGE  [conn59] got request after shutdown()
      2014-11-13T03:05:19.931+0000 I STORAGE  [conn57] got request after shutdown()
      2014-11-13T03:05:19.955+0000 I STORAGE  [conn263] got request after shutdown()
      2014-11-13T03:05:19.966+0000 I STORAGE  [conn60] got request after shutdown()
      2014-11-13T03:05:19.980+0000 I STORAGE  [conn61] got request after shutdown()
      2014-11-13T03:05:19.985+0000 I STORAGE  [conn264] got request after shutdown()
      2014-11-13T03:05:20.020+0000 I STORAGE  [conn91] got request after shutdown()
      2014-11-13T03:05:20.020+0000 I STORAGE  [conn63] got request after shutdown()
      2014-11-13T03:05:20.021+0000 I STORAGE  [conn62] got request after shutdown()
      2014-11-13T03:05:20.039+0000 I STORAGE  [conn64] got request after shutdown()
      2014-11-13T03:05:20.048+0000 I STORAGE  [conn65] got request after shutdown()
      2014-11-13T03:05:20.050+0000 I STORAGE  [conn265] got request after shutdown()
      2014-11-13T03:05:20.069+0000 I STORAGE  [conn66] got request after shutdown()
      2014-11-13T03:05:20.072+0000 I STORAGE  [conn266] got request after shutdown()
      2014-11-13T03:05:20.100+0000 I STORAGE  [conn267] got request after shutdown()
      2014-11-13T03:05:20.114+0000 I STORAGE  [conn67] got request after shutdown()
      2014-11-13T03:05:20.189+0000 I STORAGE  [conn268] got request after shutdown()
      2014-11-13T03:05:20.209+0000 I STORAGE  [conn269] got request after shutdown()
      2014-11-13T03:05:20.227+0000 I STORAGE  [conn270] got request after shutdown()
      2014-11-13T03:05:20.230+0000 I STORAGE  [conn271] got request after shutdown()
      2014-11-13T03:05:20.265+0000 I STORAGE  [conn272] got request after shutdown()
      2014-11-13T03:05:20.309+0000 I STORAGE  [conn273] got request after shutdown()
      2014-11-13T03:05:21.032+0000 I NETWORK  [conn186] end connection 10.167.150.115:48997 (3 connections now open)
      2014-11-13T03:05:21.053+0000 I NETWORK  [conn189] end connection 10.167.150.115:48999 (2 connections now open)
      2014-11-13T03:05:22.468+0000 I NETWORK  [conn181] end connection 10.218.181.240:36618 (1 connection now open)
      2014-11-13T03:05:45.376+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:05:45.377+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:05:45.377+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:05:45.377+0000 I -        [rsSync] 
      2014-11-13T03:05:45.388+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:06:15.424+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:06:15.424+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:06:15.424+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:06:15.424+0000 I -        [rsSync] 
      2014-11-13T03:06:15.430+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:06:33.467+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      2014-11-13T03:06:45.466+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:06:45.466+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:06:45.466+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:06:45.466+0000 I -        [rsSync] 
      2014-11-13T03:06:45.472+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:07:15.509+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:07:15.509+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:07:15.509+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:07:15.509+0000 I -        [rsSync] 
      2014-11-13T03:07:15.516+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:07:45.551+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:07:45.551+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:07:45.551+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:07:45.551+0000 I -        [rsSync] 
      2014-11-13T03:07:45.558+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:08:15.594+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:08:15.594+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:08:15.594+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:08:15.594+0000 I -        [rsSync] 
      2014-11-13T03:08:15.600+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----
      2014-11-13T03:08:45.636+0000 I -        [rsSync] LockerId 52886 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overloaded server.
      2014-11-13T03:08:45.636+0000 I -        [rsSync] Dumping LockManager @ 0x1b5e000
      2014-11-13T03:08:45.636+0000 I -        [rsSync] Lock @ 0xDFEC5A0: {9: Global, 1}
      GRANTED:
      	LockRequest 64561 @ 0x23629600: Mode = X; ConvertMode = NONE; 
       
      PENDING:
      	LockRequest 52886 @ 0xAE09E00: Mode = X; ConvertMode = NONE; 
      2014-11-13T03:08:45.636+0000 I -        [rsSync] 
      2014-11-13T03:08:45.642+0000 I -        [rsSync] 
       0xf6ffb9 0x9adcca 0x9b5c64 0x9af317 0x9a7292 0x923eef 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7fab5fcf7182 0x7fab5edf7fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"5ADCCA"},{"b":"400000","o":"5B5C64"},{"b":"400000","o":"5AF317"},{"b":"400000","o":"5A7292"},{"b":"400000","o":"523EEF"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7FAB5FCEF000","o":"8182"},{"b":"7FAB5ECFD000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE408000", "elfType" : 3 }, { "b" : "7FAB5FCEF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAB5FAE7000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FAB5F8E3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FAB5F5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAB5F2D9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FAB5F0C3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAB5ECFD000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FAB5FF0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
       mongod(+0x5ADCCA) [0x9adcca]
       mongod(_ZN5mongo10LockerImplILb0EE4lockERKNS_10ResourceIdENS_8LockModeEjb+0x1A4) [0x9b5c64]
       mongod(_ZN5mongo10LockerImplILb0EE10lockGlobalENS_8LockModeEj+0x1A7) [0x9af317]
       mongod(_ZN5mongo4Lock11GlobalWriteC1EPNS_6LockerEj+0x32) [0x9a7292]
       mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x8F) [0x923eef]
       mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
       mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
       mongod(+0x8B8DF8) [0xcb8df8]
       mongod(+0x8B9742) [0xcb9742]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
       mongod(+0xBBD5C4) [0xfbd5c4]
       libpthread.so.0(+0x8182) [0x7fab5fcf7182]
       libc.so.6(clone+0x6D) [0x7fab5edf7fbd]
      -----  END BACKTRACE  -----

      A kill -9 was able to kill the process:

      ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
       8065 ?        Sl     6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
      11956 pts/0    S+     0:00 grep --color=auto mmsdev_4
      ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill 8065
      ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
       8065 ?        Sl     6:53 /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod -f /data/mmsdev_4/automation-mongod.conf
      11962 pts/0    R+     0:00 grep --color=auto mmsdev_4
      ubuntu@ip-10-169-132-210:/data/mmsdev_4$ sudo kill -9 8065 
      ubuntu@ip-10-169-132-210:/data/mmsdev_4$ ps ax | grep mmsdev_4
      11974 pts/0    S+     0:00 grep --color=auto mmsdev_4

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              milkie Eric Milkie
              Reporter:
              cailin.nelson Cailin Nelson
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: