Details
-
Bug
-
Resolution: Done
-
Critical - P2
-
3.0.0-rc7
-
Fully Compatible
-
ALL
-
Description
Running a heavy insert workload on a three node replica set, the secondaries fell behind. One quit with Fatal Assertion 18750, the other stayed in RECOVERING.
It seems that after going into RECOVERING the secondary tried syncing from the other secondary, which had also gone into RECOVERING (EDIT: the secondary that asserted went into recovery before the other secondary) and then asserted:
2015-01-29T11:30:54.612+0000 I REPL [ReplicationExecutor] syncing from: shard2-01.testdomain.com:27017
|
2015-01-29T11:30:54.615+0000 W REPL [rsBackgroundSync] we are too stale to use shard2-01.testdomain.com:27017 as a sync source
|
2015-01-29T11:30:54.615+0000 I REPL [ReplicationExecutor] could not find member to sync from
|
2015-01-29T11:30:54.615+0000 I REPL [rsBackgroundSync] replSet error RS102 too stale to catch up
|
2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet our last optime : Jan 29 09:56:58 54ca03ea:1c1f
|
2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet oldest available is Jan 29 10:03:57 54ca058d:1803
|
2015-01-29T11:30:54.617+0000 I REPL [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember
|
2015-01-29T11:30:54.737+0000 I REPL [ReplicationExecutor] transition to RECOVERING
|
2015-01-29T11:30:54.925+0000 I NETWORK [conn3286] end connection 10.93.30.140:59321 (5 connections now open)
|
2015-01-29T11:30:54.932+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.140:59323 #3288 (6 connections now open)
|
2015-01-29T11:31:17.471+0000 I QUERY [conn1] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{}
|
2015-01-29T11:31:24.407+0000 I NETWORK [conn3287] end connection 10.93.30.139:60915 (5 connections now open)
|
2015-01-29T11:31:24.408+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.139:60917 #3289 (6 connections now open)
|
2015-01-29T11:31:24.942+0000 I NETWORK [conn3288] end connection 10.93.30.140:59323 (5 connections now open)
|
2015-01-29T11:31:24.943+0000 I NETWORK [initandlisten] connection accepted from 10.93.30.140:59325 #3290 (6 connections now open)
|
2015-01-29T11:31:35.744+0000 I REPL [ReplicationExecutor] syncing from: shard2-03.testdomain.com:27017
|
2015-01-29T11:31:36.077+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to shard2-03.testdomain.com:27017
|
2015-01-29T11:31:36.360+0000 I REPL [rsBackgroundSync] replSet our last op time fetched: Jan 29 09:56:58:1c1f
|
2015-01-29T11:31:36.360+0000 I REPL [rsBackgroundSync] replset source's GTE: Jan 29 09:57:02:5a0
|
2015-01-29T11:31:36.361+0000 F REPL [rsBackgroundSync] replSet need to rollback, but in inconsistent state
|
2015-01-29T11:31:36.361+0000 I REPL [rsBackgroundSync] minvalid: 54ca058d:1803 our last optime: 54ca03ea:1c1f
|
2015-01-29T11:31:36.362+0000 I - [rsBackgroundSync] Fatal Assertion 18750
|
2015-01-29T11:31:36.362+0000 I - [rsBackgroundSync]
|
Attachments
Issue Links
- related to
-
SERVER-17181 Correctly invalidate RocksDB cursor when restoring to a deleted doc
-
- Closed
-