-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 3.0.0-rc7
-
Component/s: Replication
-
Labels:
-
Fully Compatible
-
ALL
-
Running a heavy insert workload on a three node replica set, the secondaries fell behind. One quit with Fatal Assertion 18750, the other stayed in RECOVERING.
It seems that after going into RECOVERING the secondary tried syncing from the other secondary, which had also gone into RECOVERING (EDIT: the secondary that asserted went into recovery before the other secondary) and then asserted:
Unable to find source-code formatter for language: log. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
2015-01-29T11:30:54.612+0000 I REPL [ReplicationExecutor] syncing from: shard2-01.testdomain.com:27017 2015-01-29T11:30:54.615+0000 W REPL [rsBackgroundSync] we are too stale to use shard2-01.testdomain.com:27017 as a sync source 2015-01-29T11:30:54.615+0000 I REPL [ReplicationExecutor] could not find member to sync from 2015-01-29T11:30:54.615+0000 I REPL [rsBackgroundSync] replSet error RS102 too stale to catch up 2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet our last optime : Jan 29 09:56:58 54ca03ea:1c1f 2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet oldest available is Jan 29 10:03:57 54ca058d:1803 2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember 2015-01-29T11:30:54.737+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2015-01-29T11:30:54.925+0000 I NETWORK [conn3286] end connection 10.93.30.140:59321 (5 connections now open) 2015-01-29T11:30:54.932+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.140:59323 #3288 (6 connections now open) 2015-01-29T11:31:17.471+0000 I QUERY [conn1] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{} 2015-01-29T11:31:24.407+0000 I NETWORK [conn3287] end connection 10.93.30.139:60915 (5 connections now open) 2015-01-29T11:31:24.408+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.139:60917 #3289 (6 connections now open) 2015-01-29T11:31:24.942+0000 I NETWORK [conn3288] end connection 10.93.30.140:59323 (5 connections now open) 2015-01-29T11:31:24.943+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.140:59325 #3290 (6 connections now open) 2015-01-29T11:31:35.744+0000 I REPL [ReplicationExecutor] syncing from: shard2-03.testdomain.com:27017 2015-01-29T11:31:36.077+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to shard2-03.testdomain.com:27017 2015-01-29T11:31:36.360+0000 I REPL [rsBackgroundSync] replSet our last op time fetched: Jan 29 09:56:58:1c1f 2015-01-29T11:31:36.360+0000 I REPL [rsBackgroundSync] replset source's GTE: Jan 29 09:57:02:5a0 2015-01-29T11:31:36.361+0000 F REPL [rsBackgroundSync] replSet need to rollback, but in inconsistent state 2015-01-29T11:31:36.361+0000 I REPL [rsBackgroundSync] minvalid: 54ca058d:1803 our last optime: 54ca03ea:1c1f 2015-01-29T11:31:36.362+0000 I - [rsBackgroundSync] Fatal Assertion 18750 2015-01-29T11:31:36.362+0000 I - [rsBackgroundSync]
- related to
-
-
- Closed
-