[SERVER-42946] Setting 4.2 FCV in standalone with existing transaction table crashes MongoDB Created: 21/Aug/19  Updated: 29/Oct/23  Resolved: 02/Oct/19

Status: Closed
Project: Core Server
Component/s: Upgrade/Downgrade
Affects Version/s: 4.2.0
Fix Version/s: 4.2.1

Type: Bug Priority: Critical - P2
Reporter: Bjorn Harvold Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod_log.txt     Text File mongod_log.txt    
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:
  • Running Ubuntu 18.04.3
  • Upgraded from MongoDb 4.0 to 4.2. (Feature compatibility is 4.0.)
  • Run apt-get commands as outlined in MongoDb instructions
  • Restart mongodb 
  • Verify that Mongo binaries are 4.2
  • Open up a shell and and run db.adminCommand*(* { setFeatureCompatibilityVersion*:* "4.2" } ) on admin db
  • MongoDb crashes
Sprint: Repl 2019-09-09, Repl 2019-09-23, Repl 2019-10-07
Participants:

 Description   

Here's the mongo log upon failure:

2019-08-21T19:40:51.308+0700 I SHARDING [conn5] Marking collection traveliko-analytics.bookingAnalyticsYearByCountryView as collection version: <unsharded>
2019-08-21T19:40:51.308+0700 I SHARDING [conn5] Marking collection traveliko-analytics.roomCalendarRateRequestView as collection version: <unsharded>
2019-08-21T19:40:51.308+0700 I SHARDING [conn5] Marking collection traveliko-analytics.hotelAnalyticsDayByCountryAndCityView as collection version: <unsharded>
2019-08-21T19:40:51.311+0700 I SHARDING [conn5] Marking collection traveliko_eventstore.snapshotevents as collection version: <unsharded>
2019-08-21T19:40:51.311+0700 I SHARDING [conn5] Marking collection traveliko_eventstore.domainevents as collection version: <unsharded>
2019-08-21T19:40:51.312+0700 I COMMAND [conn5] Finished updating version of unique indexes for upgrade, waiting for all index updates to be committed at optime { ts: Timestamp(0, 0), t: -1 }
2019-08-21T19:40:51.312+0700 F - [conn5] Invariant failure oplog src/mongo/db/transaction_history_iterator.cpp 91
2019-08-21T19:40:51.312+0700 F - [conn5]

***aborting after invariant() failure

2019-08-21T19:40:51.330+0700 F - [conn5] Got signal: 6 (Aborted).
0x555f7237db31 0x555f7237d32e 0x555f7237d3c6 0x7f06af44d890 0x7f06af088e97 0x7f06af08a801 0x555f708b6350 0x555f70726a4d 0x555f711b8e1e 0x555f711c3ff2 0x555f711afbaf 0x555f70df4846 0x555f70df6dd9 0x555f71d8d254 0x555f70cf7f09 0x555f70cf95c3 0x555f70cfa47e 0x555f70cfad50 0x555f70ce8d4c 0x555f70cf493c 0x555f70cf028f 0x555f70cf350c 0x555f71ad55e2 0x555f70cedcad 0x555f70cf0f43 0x555f70cef377 0x555f70cf01eb 0x555f70cf350c 0x555f71ad5a4b 0x555f7210fc64 0x7f06af4426db 0x7f06af16b88f
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"555F6FBFD000","o":"2780B31","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"555F6FBFD000","o":"278032E"},{"b":"555F6FBFD000","o":"27803C6"},{"b":"7F06AF43B000","o":"12890"},{"b":"7F06AF04A000","o":"3EE97","s":"gsignal"},{"b":"7F06AF04A000","o":"40801","s":"abort"},{"b":"555F6FBFD000","o":"CB9350","s":"ZN5mongo22invariantFailedWithMsgEPKcRKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b":"555F6FBFD000","o":"B29A4D"},{"b":"555F6FBFD000","o":"15BBE1E","s":"_ZN5mongo26TransactionHistoryIterator4nextEPNS_16OperationContextE"},{"b":"555F6FBFD000","o":"15C6FF2","s":"_ZN5mongo22TransactionParticipant11Participant26refreshFromStorageIfNeededEPNS_16OperationContextE"},{"b":"555F6FBFD000","o":"15B2BAF","s":"_ZN5mongo29MongoDOperationContextSessionC2EPNS_16OperationContextE"},{"b":"555F6FBFD000","o":"11F7846"},{"b":"555F6FBFD000","o":"11F9DD9"},{"b":"555F6FBFD000","o":"2190254","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"555F6FBFD000","o":"10FAF09"},{"b":"555F6FBFD000","o":"10FC5C3"},{"b":"555F6FBFD000","o":"10FD47E"},{"b":"555F6FBFD000","o":"10FDD50","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"555F6FBFD000","o":"10EBD4C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"555F6FBFD000","o":"10F793C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"555F6FBFD000","o":"10F328F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"555F6FBFD000","o":"10F650C"},{"b":"555F6FBFD000","o":"1ED85E2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"555F6FBFD000","o":"10F0CAD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"555F6FBFD000","o":"10F3F43","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"555F6FBFD000","o":"10F2377","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"555F6FBFD000","o":"10F31EB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"555F6FBFD000","o":"10F650C"},{"b":"555F6FBFD000","o":"1ED8A4B"},{"b":"555F6FBFD000","o":"2512C64"},{"b":"7F06AF43B000","o":"76DB"},{"b":"7F06AF04A000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.0", "gitVersion" : "a4b751dcf51dd249c5865812b390cfd1c0129c30", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.15.0-58-generic", "version" : "#64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019", "machine" : "x86_64" }

, "somap" : [ { "b" : "555F6FBFD000", "elfType" : 3, "buildId" : "D849CC14858F969B5888F892C9E16777E4C6FF32" }, { "b" : "7FFD9E2FC000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "D523EBC9181DE86DAB768A6167F8B401D3054550" }, { "b" : "7F06B098F000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "20310A72634F3373FA7BCAB64DFB64CC204554B2" }, { "b" : "7F06B0774000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F06B02A9000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "CB6876717C83B0CC01C3C919B9B6E86D8554F546" }, { "b" : "7F06B001C000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "439A262CC0127BA401707DEC7A28884D617550E0" }, { "b" : "7F06AFE18000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F06AFC10000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F06AF872000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F06AF65A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F06AF43B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F06AF04A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F06B0C0E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F06AEE25000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F06AEC08000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F06AE9EC000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F06AE7DE000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F06AE593000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F06AE341000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "EC29F33C48F75AA2C48BA08E0F74D9546622314D" }, { "b" : "7F06AE133000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "25604CADD65A7057CB774139D0CBDF14460C1C78" }, { "b" : "7F06ADF16000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F06ADB98000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F06AD833000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "E5AE5C31F804BE96532D0DB2091F19E472F2D4A0" }, { "b" : "7F06AD5FF000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F06AD3C9000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F06AD148000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F06ACE72000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F06ACC40000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F06ACA3C000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "C0CB7E35A4566A443F99DFBC1A54D3A0677C8A10" }, { "b" : "7F06AC831000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F06AC616000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F06AC3D5000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F06AC0A6000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F06ABE93000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F06ABC8F000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F06ABA86000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F06AB7F9000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F06AB557000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F06AB321000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F06AB10B000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F06AAF03000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F06AACDA000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F06AAACB000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F06AA881000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F06AA578000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "BCE351987CF42B3D258B09F0CAC867758D935086" }, { "b" : "7F06AA340000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x555f7237db31]
mongod(+0x278032E) [0x555f7237d32e]
mongod(+0x27803C6) [0x555f7237d3c6]
libpthread.so.0(+0x12890) [0x7f06af44d890]
libc.so.6(gsignal+0xC7) [0x7f06af088e97]
libc.so.6(abort+0x141) [0x7f06af08a801]
mongod(ZN5mongo22invariantFailedWithMsgEPKcRKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x555f708b6350]
mongod(+0xB29A4D) [0x555f70726a4d]
mongod(_ZN5mongo26TransactionHistoryIterator4nextEPNS_16OperationContextE+0x4E) [0x555f711b8e1e]
mongod(_ZN5mongo22TransactionParticipant11Participant26refreshFromStorageIfNeededEPNS_16OperationContextE+0x802) [0x555f711c3ff2]
mongod(_ZN5mongo29MongoDOperationContextSessionC2EPNS_16OperationContextE+0x4F) [0x555f711afbaf]
mongod(+0x11F7846) [0x555f70df4846]
mongod(+0x11F9DD9) [0x555f70df6dd9]
mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE+0x74) [0x555f71d8d254]
mongod(+0x10FAF09) [0x555f70cf7f09]
mongod(+0x10FC5C3) [0x555f70cf95c3]
mongod(+0x10FD47E) [0x555f70cfa47e]
mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x540) [0x555f70cfad50]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x555f70ce8d4c]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x555f70cf493c]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x555f70cf028f]
mongod(+0x10F650C) [0x555f70cf350c]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x555f71ad55e2]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x555f70cedcad]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x843) [0x555f70cf0f43]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x2E7) [0x555f70cef377]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x555f70cf01eb]
mongod(+0x10F650C) [0x555f70cf350c]
mongod(+0x1ED8A4B) [0x555f71ad5a4b]
mongod(+0x2512C64) [0x555f7210fc64]
libpthread.so.0(+0x76DB) [0x7f06af4426db]
libc.so.6(clone+0x3F) [0x7f06af16b88f]
----- END BACKTRACE -----



 Comments   
Comment by Githook User [ 02/Oct/19 ]

Author:

{'name': 'Jason Chan', 'username': 'jasonjhchan', 'email': 'jason.chan@mongodb.com'}

Message: SERVER-42946 Prevent FCV upgrade/downgrade when in standalone mode with a non-empty config.transactions table.
Branch: v4.2
https://github.com/mongodb/mongo/commit/292c390a47413e90623314fb777b05edc36750c6

Comment by Bjorn Harvold [ 31/Aug/19 ]

Hi @jason.chan

Removing all documents in db.system.replset and dropping config.transactions collections did the trick.

Regarding the dump. I don't know about that option in MongoDb Atlas. I restored from a snapshot and the ONLY way I could get it to work was to replace my entire data directory with the snapshot and restart MongoDb.

At least now I know what to do for next time. However, as was also mentioned above, distributing a production dump to developers to be able to reproduce bugs more easily or do a deep dive into performance issues etc, I would think the copying of a replica set db into a standalone developer box will be fairly common. However, don't know what else I can do with a snapshot to make this process easier. Will include these steps into our process for now.

Thank you for your help!
Bjorn

Comment by Jason Chan [ 30/Aug/19 ]

bjorn.harvold@traveliko.com
I just want to confirm whether you have also dropped the config.transactions table and are still seeing the crash. If you intend to continue upgrading as a standalone, you must drop the config.transactions table in addition to the local replica set collections.

We recommend that when making a new standalone node from replica set data, you dump and restore only the collections outside of the local and config db.

Comment by Bjorn Harvold [ 30/Aug/19 ]

Hi @milkie

I removed the record and tried again. Mongo goes BOOM again. mongod_log.txt

Comment by Eric Milkie [ 29/Aug/19 ]

Hi bjorn.harvold@traveliko.com – sorry for the slow response.
I think you can just delete the one record in the local.system.replset collection and it should have the effect we want:

MongoDB Enterprise > db.system.replset.remove({})
WriteResult({ "nRemoved" : 1 })

The other collections that start with "replset." I tried dropping and they were successful. Let us know if there are other undroppable collections you find.

Comment by Judah Schvimer [ 29/Aug/19 ]

jason.chan, one last clarification. Will this be on downgrading FCV on a 4.2 binary as well? I suspect we'd hit the same invariant, and it seems easiest and clearest to be consistent.

Comment by Bjorn Harvold [ 24/Aug/19 ]

One last question here: How do I remove the local.system.replset and other replica set collections from local? MongoDb won't let me just drop the collection even though I haven't started mongod as a replica set.

Comment by Jason Chan [ 23/Aug/19 ]

Thanks bjorn.harvold@traveliko.com, sounds like we found the issue. A workaround would be to do the upgrade as a replica set. You can follow the link posted by Siyuan above to find the documentation on how to do so.

 

We will look to work on a fix for this when we can.

Comment by Bjorn Harvold [ 23/Aug/19 ]

I think I know what this is based on this text:

2019-08-23T10:58:15.663+0700 I  STORAGE  [initandlisten] ** WARNING: mongod started without --replSet yet document(s) are present in local.system.replset.

The data is a dump from MongoDb Atlas that does contain replicasets. I unzipped the snapshot into the mongodb data directory and started Mongo last time.

Comment by Bjorn Harvold [ 23/Aug/19 ]

ok got it. Here is the log from restart. See attachment.

Comment by Siyuan Zhou [ 22/Aug/19 ]

If you run mongod with --replSet on command line or replSetName in the config file, you are running a replica set, perhaps a single-node replica set. Here is more information about running a replica set.

The tail above was taken from right before the feature compat upgrade was executed. There was no more.

We need the whole log from the restart of the server, not just from the start of the setFeatureCompatibilityVersion command. It's likely to be the whole file you are tailing from. Right after the restart, the server will log whether it's a standalone or a replica set and the configuration of the replica set. There are also logs about the startup recovery which might be helpful. They are all before the moment you ran the setFeatureCompatibilityVersion command and can help save the guesswork. You can attach the file to this ticket in the "Attachments" section above.

Comment by Bjorn Harvold [ 22/Aug/19 ]

Hi Jason,

I am running a standalone. I downloaded the latest binaries through apt-get, according to your installation instructions, and started MongoDb. Not sure why my computer would be configured as a single node replica set unless that is part of the default install.

How can I check if MongoDb is running as a replica set and how can I disable that?

Comment by Jason Chan [ 22/Aug/19 ]

Hi bjorn.harvold@traveliko.com,

I was able to reproduce the problem on my end following these steps:

  1. Starting a 4.2 mongod as a single node replica set 
  2. Set FCV to 4.0
  3. Do an insert with retryable write that inserts a record into the transaction table.
  4. Shutdown node
  5. Restart node as a standalone and try to upgrade FCV to 4.2

Does this sound like it mirrors what you were doing? The issue is due to trying to do an FCV upgrade while the node is a standalone.

Comment by Bjorn Harvold [ 22/Aug/19 ]

Hi Erik. The tail above was taken from right before the feature compat upgrade was executed. There was no more.

Comment by Eric Milkie [ 22/Aug/19 ]

bjorn.harvold@traveliko.com could you please provide the full log since the start of the failed 4.2 node?

Comment by Bjorn Harvold [ 22/Aug/19 ]

Can reproduce every time:

> db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } )
2019-08-22T12:35:23.935+0700 I NETWORK [js] DBClientConnection failed to receive message from 127.0.0.1:27017 - HostUnreachable: Connection closed by peer
2019-08-22T12:35:23.935+0700 E QUERY [js] uncaught exception: Error: error doing query: failed: network error while attempting to run command 'setFeatureCompatibilityVersion' on host '127.0.0.1:27017' :
DB.prototype.runCommand@src/mongo/shell/db.js:169:19
DB.prototype.adminCommand@src/mongo/shell/db.js:187:12
@(shell):1:1
2019-08-22T12:35:23.937+0700 I NETWORK [js] trying reconnect to 127.0.0.1:27017 failed
2019-08-22T12:35:23.937+0700 I NETWORK [js] reconnect 127.0.0.1:27017 failed failed

tail -f

2019-08-22T12:35:17.976+0700 I  NETWORK  [listener] connection accepted from 127.0.0.1:47906 #12 (12 connections now open)
2019-08-22T12:35:17.976+0700 I  NETWORK  [conn12] received client metadata from 127.0.0.1:47906 conn12: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 12
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 11
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 10
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 9
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 8
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 7
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 6
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 5
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 4
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 3
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 2
2019-08-22T12:35:23.397+0700 I  NETWORK  [conn12] Skip closing connection for connection # 1
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.system.rollback.id as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.replset.oplogTruncateAfterPoint as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.oplog.rs as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.clustermanager as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.replset.election as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection local.replset.minvalid as collection version: <unsharded>
2019-08-22T12:35:23.397+0700 I  SHARDING [conn12] Marking collection admin.system.keys as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsDayByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsYearByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsDayView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsDayView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsYearByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsYearView as collection version: <unsharded>
2019-08-22T12:35:23.399+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsHourByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsDayByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsYearView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsYearByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsMonthByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsHourByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsHourByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsMonthView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.chatAnalyticsDayView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsDetailsView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsHourView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.rateRequestDetailsView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.chatAnalyticsYearView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsMonthByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.chatAnalyticsMonthView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsDayByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsHourByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.400+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.chatAnalyticsHourView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsMonthByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsMonthByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsMonthView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsHourView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.bookingAnalyticsYearByCountryView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.roomCalendarRateRequestView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko-analytics.hotelAnalyticsDayByCountryAndCityView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.reviewQuestionView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.recognitionView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.membershipView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.hotelView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.policyView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.customerView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.countryView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.lookupView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.subCountryView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.geoIPView as collection version: <unsharded>
2019-08-22T12:35:23.401+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedMultimediaView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.subSubCountryView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.parityPlusSettingsView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedMeetingRoomView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.channelConfigurationItemView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.perkView as collection version: <unsharded>
2019-08-22T12:35:23.402+0700 I  SHARDING [conn12] Marking collection traveliko_cms.announcementView as collection version: <unsharded>
2019-08-22T12:35:23.403+0700 I  SHARDING [conn12] Marking collection traveliko_cms.greendexAnswerView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.bucketListEntryView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.cityView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.attractionView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.hotelInternalMessageView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.parityPlusView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.orderMarketplaceItemView as collection version: <unsharded>
2019-08-22T12:35:23.404+0700 I  SHARDING [conn12] Marking collection traveliko_cms.channelConfigurationView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.lifestyleView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedRestaurantView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.reviewRewardView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.organizationView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.cancellationPolicyView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.geoNameView as collection version: <unsharded>
2019-08-22T12:35:23.408+0700 I  SHARDING [conn12] Marking collection traveliko_cms.principal as collection version: <unsharded>
2019-08-22T12:35:23.409+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedSpaView as collection version: <unsharded>
2019-08-22T12:35:23.409+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedGuestRoomMultimediaView as collection version: <unsharded>
2019-08-22T12:35:23.409+0700 I  SHARDING [conn12] Marking collection traveliko_cms.recreationView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.channelInventoryView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.pointOfInterestView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.scheduleItemView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.subscriptionView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.profileView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.hotelCodeGenView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedBookingView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.improvedGuestRoomView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.priceGainView as collection version: <unsharded>
2019-08-22T12:35:23.410+0700 I  SHARDING [conn12] Marking collection traveliko_cms.internalMessageView as collection version: <unsharded>
2019-08-22T12:35:23.411+0700 I  SHARDING [conn12] Marking collection traveliko_cms.refPointView as collection version: <unsharded>
2019-08-22T12:35:23.411+0700 I  SHARDING [conn12] Marking collection traveliko_cms.reviewView as collection version: <unsharded>
2019-08-22T12:35:23.411+0700 I  SHARDING [conn12] Marking collection traveliko_cms.greendexQuestionView as collection version: <unsharded>
2019-08-22T12:35:23.412+0700 I  SHARDING [conn12] Marking collection traveliko_cms.roleView as collection version: <unsharded>
2019-08-22T12:35:23.412+0700 I  SHARDING [conn12] Marking collection traveliko_cms.hotelierView as collection version: <unsharded>
2019-08-22T12:35:23.413+0700 I  SHARDING [conn12] Marking collection traveliko_eventstore.snapshotevents as collection version: <unsharded>
2019-08-22T12:35:23.413+0700 I  SHARDING [conn12] Marking collection traveliko_eventstore.domainevents as collection version: <unsharded>
2019-08-22T12:35:23.414+0700 I  COMMAND  [conn12] Finished updating version of unique indexes for upgrade, waiting for all index updates to be committed at optime { ts: Timestamp(0, 0), t: -1 }
2019-08-22T12:35:23.414+0700 F  -        [conn12] Invariant failure oplog src/mongo/db/transaction_history_iterator.cpp 91
2019-08-22T12:35:23.414+0700 F  -        [conn12] 
 
***aborting after invariant() failure
 
 
2019-08-22T12:35:23.432+0700 F  -        [conn12] Got signal: 6 (Aborted).
 0x5637a4799b31 0x5637a479932e 0x5637a47993c6 0x7f48e56df890 0x7f48e531ae97 0x7f48e531c801 0x5637a2cd2350 0x5637a2b42a4d 0x5637a35d4e1e 0x5637a35dfff2 0x5637a35cbbaf 0x5637a3210846 0x5637a3212dd9 0x5637a41a9254 0x5637a3113f09 0x5637a31155c3 0x5637a311647e 0x5637a3116d50 0x5637a3104d4c 0x5637a311093c 0x5637a310c28f 0x5637a310f50c 0x5637a3ef15e2 0x5637a3109cad 0x5637a310cf43 0x5637a310b377 0x5637a310c1eb 0x5637a310f50c 0x5637a3ef1a4b 0x5637a452bc64 0x7f48e56d46db 0x7f48e53fd88f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5637A2019000","o":"2780B31","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5637A2019000","o":"278032E"},{"b":"5637A2019000","o":"27803C6"},{"b":"7F48E56CD000","o":"12890"},{"b":"7F48E52DC000","o":"3EE97","s":"gsignal"},{"b":"7F48E52DC000","o":"40801","s":"abort"},{"b":"5637A2019000","o":"CB9350","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b":"5637A2019000","o":"B29A4D"},{"b":"5637A2019000","o":"15BBE1E","s":"_ZN5mongo26TransactionHistoryIterator4nextEPNS_16OperationContextE"},{"b":"5637A2019000","o":"15C6FF2","s":"_ZN5mongo22TransactionParticipant11Participant26refreshFromStorageIfNeededEPNS_16OperationContextE"},{"b":"5637A2019000","o":"15B2BAF","s":"_ZN5mongo29MongoDOperationContextSessionC2EPNS_16OperationContextE"},{"b":"5637A2019000","o":"11F7846"},{"b":"5637A2019000","o":"11F9DD9"},{"b":"5637A2019000","o":"2190254","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"5637A2019000","o":"10FAF09"},{"b":"5637A2019000","o":"10FC5C3"},{"b":"5637A2019000","o":"10FD47E"},{"b":"5637A2019000","o":"10FDD50","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"5637A2019000","o":"10EBD4C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"5637A2019000","o":"10F793C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"5637A2019000","o":"10F328F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"5637A2019000","o":"10F650C"},{"b":"5637A2019000","o":"1ED85E2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"5637A2019000","o":"10F0CAD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"5637A2019000","o":"10F3F43","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"5637A2019000","o":"10F2377","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"5637A2019000","o":"10F31EB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"5637A2019000","o":"10F650C"},{"b":"5637A2019000","o":"1ED8A4B"},{"b":"5637A2019000","o":"2512C64"},{"b":"7F48E56CD000","o":"76DB"},{"b":"7F48E52DC000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.0", "gitVersion" : "a4b751dcf51dd249c5865812b390cfd1c0129c30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-58-generic", "version" : "#64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5637A2019000", "elfType" : 3, "buildId" : "D849CC14858F969B5888F892C9E16777E4C6FF32" }, { "b" : "7FFD04382000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "D523EBC9181DE86DAB768A6167F8B401D3054550" }, { "b" : "7F48E6C21000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "20310A72634F3373FA7BCAB64DFB64CC204554B2" }, { "b" : "7F48E6A06000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F48E653B000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "CB6876717C83B0CC01C3C919B9B6E86D8554F546" }, { "b" : "7F48E62AE000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "439A262CC0127BA401707DEC7A28884D617550E0" }, { "b" : "7F48E60AA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F48E5EA2000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F48E5B04000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F48E58EC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F48E56CD000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F48E52DC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F48E6EA0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F48E50B7000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F48E4E9A000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F48E4C7E000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F48E4A70000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F48E4825000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F48E45D3000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "EC29F33C48F75AA2C48BA08E0F74D9546622314D" }, { "b" : "7F48E43C5000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "25604CADD65A7057CB774139D0CBDF14460C1C78" }, { "b" : "7F48E41A8000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F48E3E2A000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F48E3AC5000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "E5AE5C31F804BE96532D0DB2091F19E472F2D4A0" }, { "b" : "7F48E3891000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F48E365B000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F48E33DA000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F48E3104000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F48E2ED2000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F48E2CCE000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "C0CB7E35A4566A443F99DFBC1A54D3A0677C8A10" }, { "b" : "7F48E2AC3000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F48E28A8000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F48E2667000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F48E2338000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F48E2125000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F48E1F21000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F48E1D18000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F48E1A8B000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F48E17E9000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F48E15B3000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F48E139D000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F48E1195000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F48E0F6C000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F48E0D5D000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F48E0B13000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F48E080A000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "BCE351987CF42B3D258B09F0CAC867758D935086" }, { "b" : "7F48E05D2000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5637a4799b31]
 mongod(+0x278032E) [0x5637a479932e]
 mongod(+0x27803C6) [0x5637a47993c6]
 libpthread.so.0(+0x12890) [0x7f48e56df890]
 libc.so.6(gsignal+0xC7) [0x7f48e531ae97]
 libc.so.6(abort+0x141) [0x7f48e531c801]
 mongod(_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x5637a2cd2350]
 mongod(+0xB29A4D) [0x5637a2b42a4d]
 mongod(_ZN5mongo26TransactionHistoryIterator4nextEPNS_16OperationContextE+0x4E) [0x5637a35d4e1e]
 mongod(_ZN5mongo22TransactionParticipant11Participant26refreshFromStorageIfNeededEPNS_16OperationContextE+0x802) [0x5637a35dfff2]
 mongod(_ZN5mongo29MongoDOperationContextSessionC2EPNS_16OperationContextE+0x4F) [0x5637a35cbbaf]
 mongod(+0x11F7846) [0x5637a3210846]
 mongod(+0x11F9DD9) [0x5637a3212dd9]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE+0x74) [0x5637a41a9254]
 mongod(+0x10FAF09) [0x5637a3113f09]
 mongod(+0x10FC5C3) [0x5637a31155c3]
 mongod(+0x10FD47E) [0x5637a311647e]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x540) [0x5637a3116d50]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x5637a3104d4c]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x5637a311093c]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x5637a310c28f]
 mongod(+0x10F650C) [0x5637a310f50c]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x5637a3ef15e2]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x5637a3109cad]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x843) [0x5637a310cf43]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x2E7) [0x5637a310b377]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x5637a310c1eb]
 mongod(+0x10F650C) [0x5637a310f50c]
 mongod(+0x1ED8A4B) [0x5637a3ef1a4b]
 mongod(+0x2512C64) [0x5637a452bc64]
 libpthread.so.0(+0x76DB) [0x7f48e56d46db]
 libc.so.6(clone+0x3F) [0x7f48e53fd88f]
-----  END BACKTRACE  -----

Comment by Siyuan Zhou [ 22/Aug/19 ]

bjorn.harvold@traveliko.com, sorry to hear the issue and thanks for filing the ticket! While we are investigating the root cause, could you please provide the full log since the start of the failed 4.2 node? We'd love to learn more about the status of the failed node. Besides, could you reproduce this issue on Ubuntu 18.04.3? A repro will certainly speed up the investigation.

Comment by Bjorn Harvold [ 22/Aug/19 ]

Just as a note. I did the upgrade flawlessly on Mac OS X Mojave using the same database in case you were pondering whether my collections were the culprit.

 

I noticed MongoDb 4.2 got stricter on indexes and actually found a duplicate index on one of my collections which 4.0 had not complained about. I had to remove the duplicate before I could successfully do the feature compatibility upgrade on Mojave.

Comment by Kelsey Schubert [ 21/Aug/19 ]

Thanks for the report, bjorn.harvold@traveliko.com. We're investigating and will update you when we know more.

Generated at Thu Feb 08 05:01:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.