-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Replication
-
Labels:None
-
Fully Compatible
-
ALL
-
v3.4
-
Repl 2017-01-23
-
0
When doing a resync in master-slave mode, the slave attempts to drop all the databases it knows about before resetting each database from the master.
Unfortunately, it assumes that the list of databases on the slave must match what's on the master even though the slave might not have replicated the database at the time the listDatabases command is run against the master. This might result in the following crash:
[js_test:repl2] 2016-09-02T20:21:33.809+0000 d20263| 2016-09-02T20:21:33.808+0000 I REPL [conn1] client forcing resync from 127.0.0.1:20262
[js_test:repl2] 2016-09-02T20:21:33.810+0000 d20263| 2016-09-02T20:21:33.808+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG
[js_test:repl2] 2016-09-02T20:21:33.810+0000 d20262| 2016-09-02T20:21:33.809+0000 I NETWORK [thread1] connection accepted from 127.0.0.1:53696 #8 (3 connections now open)
[js_test:repl2] 2016-09-02T20:21:33.810+0000 d20263| 2016-09-02T20:21:33.809+0000 D NETWORK [conn1] connected to server 127.0.0.1:20262 (127.0.0.1)
[js_test:repl2] 2016-09-02T20:21:33.811+0000 d20262| 2016-09-02T20:21:33.809+0000 I NETWORK [conn8] received client metadata from 127.0.0.1:53696 conn8: { driver: { name: "MongoDB Internal Client", version: "3.3.12-64-gf288ea8" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server release 6.2 (Santiago)", architecture: "x86_64", version: "Kernel 2.6.32-220.el6.x86_64" } }
[js_test:repl2] 2016-09-02T20:21:33.823+0000 d20262| 2016-09-02T20:21:33.823+0000 I ACCESS [conn8] Successfully authenticated as principal __system on local
[js_test:repl2] 2016-09-02T20:21:33.824+0000 d20263| 2016-09-02T20:21:33.823+0000 I REPL [conn1] resync: dropping database admin
[js_test:repl2] 2016-09-02T20:21:33.824+0000 d20263| 2016-09-02T20:21:33.823+0000 D STORAGE [conn1] dropDatabase admin
[js_test:repl2] 2016-09-02T20:21:33.824+0000 d20263| 2016-09-02T20:21:33.824+0000 D STORAGE [conn1] deleting metadata for admin.system.version @ RecordId(5)
[js_test:repl2] 2016-09-02T20:21:33.825+0000 d20263| 2016-09-02T20:21:33.824+0000 D STORAGE [conn1] WT drop of table:index-7--2768966931139727106 res 16
[js_test:repl2] 2016-09-02T20:21:33.825+0000 d20263| 2016-09-02T20:21:33.824+0000 D STORAGE [conn1] ~WiredTigerRecordStore for: admin.system.version
[js_test:repl2] 2016-09-02T20:21:33.825+0000 d20263| 2016-09-02T20:21:33.824+0000 D STORAGE [conn1] WT drop of table:collection-6--2768966931139727106 res 16
[js_test:repl2] 2016-09-02T20:21:33.826+0000 d20263| 2016-09-02T20:21:33.824+0000 I REPL [conn1] resync: dropping database foo
[js_test:repl2] 2016-09-02T20:21:33.826+0000 d20263| 2016-09-02T20:21:33.824+0000 I - [conn1] Invariant failure db src/mongo/db/catalog/database.cpp 631
[js_test:repl2] 2016-09-02T20:21:33.826+0000 d20263| 2016-09-02T20:21:33.824+0000 I - [conn1]
[js_test:repl2] 2016-09-02T20:21:33.826+0000 d20263|
[js_test:repl2] 2016-09-02T20:21:33.827+0000 d20263| ***aborting after invariant() failure
[js_test:repl2] 2016-09-02T20:21:33.827+0000 d20263|
[js_test:repl2] 2016-09-02T20:21:33.827+0000 d20263|
[js_test:repl2] 2016-09-02T20:21:33.837+0000 d20263| 2016-09-02T20:21:33.837+0000 F - [conn1] Got signal: 6 (Aborted).
[js_test:repl2] 2016-09-02T20:21:33.837+0000 d20263|
[js_test:repl2] 2016-09-02T20:21:33.838+0000 d20263| 0x7ff6ef2b3071 0x7ff6ef2b2169 0x7ff6ef2b264d 0x7ff6eb2487e0 0x7ff6eaed75e5 0x7ff6eaed8dc5 0x7ff6ee4dbf9c 0x7ff6ee6c5a83 0x7ff6eecbac8a 0x7ff6eecbe625 0x7ff6eecc14bf 0x7ff6eed81a21 0x7ff6ee7296a7 0x7ff6ee72a8c3 0x7ff6eedd759d 0x7ff6ee941fb9 0x7ff6ee550aad 0x7ff6ee551410 0x7ff6ef21ae8b 0x7ff6eb240aa1 0x7ff6eaf8daad
[js_test:repl2] 2016-09-02T20:21:33.838+0000 d20263| ----- BEGIN BACKTRACE -----
[js_test:repl2] 2016-09-02T20:21:33.853+0000 d20263| {"backtrace":[{"b":"7FF6EDC97000","o":"161C071","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FF6EDC97000","o":"161B169"},{"b":"7FF6EDC97000","o":"161B64D"},{"b":"7FF6EB239000","o":"F7E0"},{"b":"7FF6EAEA5000","o":"325E5","s":"gsignal"},{"b":"7FF6EAEA5000","o":"33DC5","s":"abort"},{"b":"7FF6EDC97000","o":"844F9C","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7FF6EDC97000","o":"A2EA83","s":"_ZN5mongo8Database12dropDatabaseEPNS_16OperationContextEPS0_"},{"b":"7FF6EDC97000","o":"1023C8A","s":"_ZN5mongo4repl10ReplSource10resyncDropEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"7FF6EDC97000","o":"1027625","s":"_ZN5mongo4repl10ReplSource11forceResyncEPNS_16OperationContextEPKc"},{"b":"7FF6EDC97000","o":"102A4BF","s":"_ZN5mongo4repl10ReplSource15forceResyncDeadEPNS_16OperationContextEPKc"},{"b":"7FF6EDC97000","o":"10EAA21","s":"_ZN5mongo4repl9CmdResync3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE"},{"b":"7FF6EDC97000","o":"A926A7","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"7FF6EDC97000","o":"A938C3","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"7FF6EDC97000","o":"114059D","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"7FF6EDC97000","o":"CAAFB9","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"7FF6EDC97000","o":"8B9AAD","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopEPNS_9transport7SessionE"},{"b":"7FF6EDC97000","o":"8BA410"},{"b":"7FF6EDC97000","o":"1583E8B"},{"b":"7FF6EB239000","o":"7AA1"},{"b":"7FF6EAEA5000","o":"E8AAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.3.12-64-gf288ea8", "gitVersion" : "f288ea86db58c6aefe4807ed7ac1815577da2752", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.el6.x86_64", "version" : "#1 SMP Wed Nov 9 08:03:13 EST 2011", "machine" : "x86_64" }, "somap" : [ { "b" : "7FF6EDC97000", "elfType" : 3, "buildId" : "C313CEAEF4B5AD6FD4320703DBE4955F73C03431" }, { "b" : "7FFF6B4FF000", "elfType" : 3, "buildId" : "08F634A1D22DEFF00461D50A7699DACDC97657BF" }, { "b" : "7FC65825A000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "E0AEE889D5BF1373F2F9EE0D448DBF3F5B5113F0" }, { "b" : "7FC658816000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "b" : "7FF6ED3CA000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "1270BB069D761BD79C79F8986BB3ED5DCAA7D06D" }, { "b" : "7FF6ED1A4000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "3FA4F246A7DF00EC1355C5226C9308DC7B4AB5CD" }, { "b" : "7FF6ECCDC000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "A12008903CCB8F814439276CDF5238BB9C289D4C" }, { "b" : "7FF6ECA01000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "0FB0EADB5887E03256EF6C7EEA8B3388E39E4BB0" }, { "b" : "7FF6EC7AC000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "F8B2645ECACB90EC35AB098EDFBC5540BAF37CE1" }, { "b" : "7FC65635B000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "1FA3BC4E18EEEB915FDD4E9BE33D0542C3FB2804" }, { "b" : "7FC65554C000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "244D2593BDE4FE657BC88572DB5DA88FA274B7F3" }, { "b" : "7FF6EC0E0000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "b" : "7FF6EBCFC000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2CF03CE94B9388A10544E4EF073450851A4D6AEB" }, { "b" : "7FC65C0F4000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "b" : "7FC65CEF0000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "b" : "7FC65B86C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "b" : "7FC65AA56000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "b" : "7FC65C039000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE4607C86FEEC14F47B46D0DCC" }, { "b" : "7FC65C0A5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "24D3AB3DB0F38C7515FEADF82191651DA4117A18" }, { "b" : "7FC65F474000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F3EEBD18E66EB139EA4D76CDFA86D643ABCF0070" }, { "b" : "7FC659E8B000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EEE9AC218BFC460" }, { "b" : "7FC659454000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "B21E32412356755F1851BAE219A0C8EFDAEEC686" }, { "b" : "7FC65556D000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "b" : "7FC656741000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "b" : "7FC65993D000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "b" : "7FC655B32000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "b" : "7FC65552F000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7FF6E9D24000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "083332F88CF3C61AB0184D8F397FC8BFF4548D8E" }, { "b" : "7FC65A3B9000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "545478030DF991A635CC5E3258A3F5D8A7E94561" }, { "b" : "7FC657DA0000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "BC86E56751E93653BD1C92975968937148A407CD" }, { "b" : "7FC659B9D000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "82DEB5906312B8D8F888D206DE11BC6B6FDF57D8" }, { "b" : "7FC654532000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "C65174824A80EDE5374CFF6143C808807160CA63" }, { "b" : "7FC654703000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "F858A331FA080C7E82549BE3191EB4BADE02A5C0" }, { "b" : "7FC6580FA000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "E7B49911F1136073DD7DC58E8118CD9A4FBE2A19" }, { "b" : "7FC6596E4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7FF6E8AD4000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "6855E5BF5B3634C15F01B1043BD892D727EE3C08" }, { "b" : "7FC652EA2000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "5659EB985475B586E3BBCB95BA21F4A30BE5EBF4" }, { "b" : "7FC65545F000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "9080D18543F337F6F6B5C5265B1A3D2073A0FFBF" }, { "b" : "7FC654E32000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "DE75A3731E7ABC427888BA8D38E96606264FBEBB" }, { "b" : "7FC655AF2000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "0375F2A6DA6EDCF870C52584B71798AC9003CFF2" }, { "b" : "7FC6550C6000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "F3A25CFCCA8191255ECFCFCD62248E393AFF3D01" }, { "b" : "7FC656AC2000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "1D3CD12F36DFB9E232953D3B73C34F8C0EF1004D" }, { "b" : "7FC6550BD000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "535FB904872A936ECC2E926C612B1B2BFD0FB722" }, { "b" : "7FC655E7F000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "29B15E2260EA9A50E0993DEEF7ABD8334F37E6B9" }, { "b" : "7FF6E7657000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "8727EC925D6D91DAC74A99BDE8B3C6EE96AF13EA" }, { "b" : "7FC655254000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, { "b" : "7FC657035000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, { "b" : "7FC655A24000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, { "b" : "7FC654C0E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "50517407A07B8D6C9A55A392E99246B52E8BFEEA" }, { "b" : "7FC6535ED000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "6FF9BAEEEE9DDEEF2DFA5CBD36147A75891C0AD4" }, { "b" : "7FC6523C0000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "6BDB4E1990D6EBA12A5C8D39A7650DB8798BF568" }, { "b" : "7FC6525BC000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "A436538388F1F25113FDA834CA2EED524EFA17D6" }, { "b" : "7FC6527B4000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "26CC708AC7C0FC1797A2340C024F0ADD0CE054D8" }, { "b" : "7FC653040000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "54DB4E3C4EC743FE95DD31C9D312E2898724577E" }, { "b" : "7FC65423B000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" } ] }}
[js_test:repl2] 2016-09-02T20:21:33.853+0000 d20263| mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7ff6ef2b3071]
[js_test:repl2] 2016-09-02T20:21:33.853+0000 d20263| mongod(+0x161B169) [0x7ff6ef2b2169]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| mongod(+0x161B64D) [0x7ff6ef2b264d]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| libpthread.so.0(+0xF7E0) [0x7ff6eb2487e0]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| libc.so.6(gsignal+0x35) [0x7ff6eaed75e5]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| libc.so.6(abort+0x175) [0x7ff6eaed8dc5]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7ff6ee4dbf9c]
[js_test:repl2] 2016-09-02T20:21:33.854+0000 d20263| mongod(_ZN5mongo8Database12dropDatabaseEPNS_16OperationContextEPS0_+0x353) [0x7ff6ee6c5a83]
[js_test:repl2] 2016-09-02T20:21:33.855+0000 d20263| mongod(_ZN5mongo4repl10ReplSource10resyncDropEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xAA) [0x7ff6eecbac8a]
[js_test:repl2] 2016-09-02T20:21:33.855+0000 d20263| mongod(_ZN5mongo4repl10ReplSource11forceResyncEPNS_16OperationContextEPKc+0x495) [0x7ff6eecbe625]
[js_test:repl2] 2016-09-02T20:21:33.855+0000 d20263| mongod(_ZN5mongo4repl10ReplSource15forceResyncDeadEPNS_16OperationContextEPKc+0xDF) [0x7ff6eecc14bf]
[js_test:repl2] 2016-09-02T20:21:33.856+0000 d20263| mongod(_ZN5mongo4repl9CmdResync3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE+0x1D1) [0x7ff6eed81a21]
[js_test:repl2] 2016-09-02T20:21:33.856+0000 d20263| mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x617) [0x7ff6ee7296a7]
[js_test:repl2] 2016-09-02T20:21:33.856+0000 d20263| mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x9A3) [0x7ff6ee72a8c3]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x26D) [0x7ff6eedd759d]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xD29) [0x7ff6ee941fb9]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopEPNS_9transport7SessionE+0x1AD) [0x7ff6ee550aad]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| mongod(+0x8BA410) [0x7ff6ee551410]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| mongod(+0x1583E8B) [0x7ff6ef21ae8b]
[js_test:repl2] 2016-09-02T20:21:33.857+0000 d20263| libpthread.so.0(+0x7AA1) [0x7ff6eb240aa1]
[js_test:repl2] 2016-09-02T20:21:33.858+0000 d20263| libc.so.6(clone+0x6D) [0x7ff6eaf8daad]
[js_test:repl2] 2016-09-02T20:21:33.858+0000 d20263| ----- END BACKTRACE -----