[SERVER-17071] Resync crashes secondary Created: 27/Jan/15  Updated: 18/Sep/15  Resolved: 27/Jan/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.6.9, 3.0.0-rc6
Fix Version/s: 3.0.0-rc7

Type: Bug Priority: Major - P3
Reporter: Timothy Olsen (Inactive) Assignee: Scott Hernandez (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-17081 Resync command under write load can f... Closed
is related to SERVER-15704 Re-enable resync.js Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Set up a 3.0.0-rc6 mmapv1 replica set. Issue a resync command (db.runCommand("resync") on the admin database). Watch the secondary crash.

Participants:

 Description   

The resync command causes the secondary on which the command is issued to crash:

2015-01-27T10:53:29.877-0500 I REPL     [rsSync] initial sync pending
2015-01-27T10:53:29.877-0500 I -        [rsSync] Invariant failure isRollbackAllowed || mySlaveInfo->opTime <= ts src/mongo/db/repl/replication_coordinator_impl.cpp 693
2015-01-27T10:53:29.879-0500 I CONTROL  [rsSync] 
 0x10d64d1c9 0x10d601310 0x10d5ef173 0x10d369ff7 0x10d36cd07 0x10d3984a1 0x10d3981c9 0x10d3a55e7 0x10d6809c1 0x7fff8ae5f2fc 0x7fff8ae5f279 0x7fff8ae5d4b1
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"10CEF7000","o":"7561C9"},{"b":"10CEF7000","o":"70A310"},{"b":"10CEF7000","o":"6F8173"},{"b":"10CEF7000","o":"472FF7"},{"b":"10CEF7000","o":"475D07"},{"b":"10CEF7000","o":"4A14A1"},{"b":"10CEF7000","o":"4A11C9"},{"b":"10CEF7000","o":"4AE5E7"},{"b":"10CEF7000","o":"7899C1"},{"b":"7FFF8AE5C000","o":"32FC"},{"b":"7FFF8AE5C000","o":"3279"},{"b":"7FFF8AE5C000","o":"14B1"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc6", "gitVersion" : "8dd41ae58040dfa09c633c05ce5752d46402f9c5", "uname" : { "sysname" : "Darwin", "release" : "14.0.0", "version" : "Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/tmp/mms-automation/test/versions/mongodb-osx-x86_64-3.0.0-rc6/bin/mongod", "machType" : 2, "b" : "10CEF7000", "buildId" : "561EBEF000B63EABB1081C171AB7E6A5" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF9754E000", "buildId" : "DA954461EC6A3DF085516FC810627627" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8F22D000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF91F46000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8E89F000", "buildId" : "D381EBC669D831D380845A80A32CB748" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF957A9000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF8F2BF000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8B91E000", "buildId" : "E178980139853949B7366B3378873301" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8C522000", "buildId" : "502CF32B669B3709886208188225E4F0" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8F077000", "buildId" : "19FAF435C16533749DEFD7BBA7D61DB6" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8A4B2000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF91F35000", "buildId" : "8A9889248BE735FEBF7D322E90EFE49E" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF92658000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF89423000", "buildId" : "DC0416272D92361CBABFA869A5C72293" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8E748000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8C5C0000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF923A4000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8CDE6000", "buildId" : "C185E86274243210B5286B822577A4B8" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8B47F000", "buildId" : "9FBA1CE497D0347EA44393ED94512E92" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF92C02000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF89AAC000", "buildId" : "EBBF7EF680D83F8F825CB412BD6D22C0" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF93442000", "buildId" : "62B70ECAE40D3C63896E7F00EC386DDB" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8BB18000", "buildId" : "B85A85D585303A93B0C34DEC41F79478" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF92ADC000", "buildId" : "93E0E0A975B63904BB4E4BC7C05F4B6B" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF933AA000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF90D87000", "buildId" : "19BCC25757173502A71F95D65AFA861B" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF88891000", "buildId" : "C0B2313D47BE38A9BEE62634A4F5E14B" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF8F226000", "buildId" : "29AB225BD7FB30ED960065D44B9A9442" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8AA39000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF96E0C000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8AE5C000", "buildId" : "26B1897F0CD330F3B55A37CB45062D73" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF8A118000", "buildId" : "DB9962EF889831CC9B87E01F8CE74C9D" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF8AE5A000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF90237000", "buildId" : "1DB0443659743F1686CC5FF5F390339C" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF894A6000", "buildId" : "A9E6B7D8C3273742AC5486C94218B1DF" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF92A66000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF8B470000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8DC9B000", "buildId" : "9437C02EA07B38C891CB299FAA63083D" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8A7F6000", "buildId" : "3B60CD9074A23A5D9686B0772159792A" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF91F4D000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8EC02000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF93D1B000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10d64d1c9]
 mongod(_ZN5mongo10logContextEPKc+0x100) [0x10d601310]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x183) [0x10d5ef173]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl23_setMyLastOptime_inlockEPN5boost11unique_lockINS2_5mutexEEERKNS_6OpTimeEb+0x1A7) [0x10d369ff7]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl15setMyLastOptimeERKNS_6OpTimeE+0x77) [0x10d36cd07]
 mongod(_ZN5mongo4repl12_GLOBAL__N_112_initialSyncEv+0x151) [0x10d3984a1]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x1F9) [0x10d3981c9]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x297) [0x10d3a55e7]
 mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x10d6809c1]
 libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff8ae5f2fc]
 libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff8ae5f279]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ae5d4b1]
-----  END BACKTRACE  -----
2015-01-27T10:53:29.880-0500 I -        [rsSync] 
 
***aborting after invariant() failure



 Comments   
Comment by Githook User [ 28/Jan/15 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-15704, SERVER-17071: Use resetMyLastApplied and re-enable resync.js

(cherry picked from commit 0214774a1899a293efe0b3c682495886c24e9edd)
Branch: v3.0
https://github.com/mongodb/mongo/commit/9f3b84f9a1fa57d05725db1b6dc886bee43e8f80

Comment by Githook User [ 27/Jan/15 ]

Author:

{u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}

Message: SERVER-15704, SERVER-17071: Use resetMyLastApplied and re-enable resync.js
Branch: master
https://github.com/mongodb/mongo/commit/0214774a1899a293efe0b3c682495886c24e9edd

Comment by Scott Hernandez (Inactive) [ 27/Jan/15 ]

This is due to the checking we added to setMyLastOptime that the timestamp goes forward only, and resync setting it to 0. We also do this during initial sync when an error occurs, so there are two places which trip this validation.

Generated at Thu Feb 08 03:43:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.