Loading...

Type: Bug
Resolution: Works as Designed
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None
Environment:

Hide
OS: Ubuntu 22.04.5 LTS
node.js / npm versions: v20.18.0 / 10.8.2
Additional info:
- db version v8.0.5
- Build Info: {
    "version": "8.0.5",
    "gitVersion": "cb9e2e5e552ee39dea1e39d7859336456d0c9820",
    "openSSLVersion": "OpenSSL 3.0.2 15 Mar 2022",
    "modules": [],
    "allocator": "tcmalloc-google",
    "environment": {
        "distmod": "ubuntu2204",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

Show
OS: Ubuntu 22.04.5 LTS node.js / npm versions: v20.18.0 / 10.8.2 Additional info: - db version v8.0.5 - Build Info: {     "version": "8.0.5",     "gitVersion": "cb9e2e5e552ee39dea1e39d7859336456d0c9820",     "openSSLVersion": "OpenSSL 3.0.2 15 Mar 2022",     "modules": [],     "allocator": "tcmalloc-google",     "environment": {         "distmod": "ubuntu2204",         "distarch": "x86_64",         "target_arch": "x86_64"     } }

Assigned Teams:

Replication
Sprint:
Repl 2025-03-17
Confidence Status:
None
Work Order:
3

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name:
None
Goal Link:
None

Problem Statement/Rationale

I removed certain secondary members of a stable working replicaSet, and updated their replicaSet to another new one, but ended up with `(code=dumped, signal=SEGV)` for secondaries after adding them to replicaset from primary using rs.add("vps-2:27017")

Please be sure to attach relevant logs with any sensitive data redacted.

{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-18092751186542437838","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-1923058991556977842","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-3740509501332648949","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-5533034944742718843","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-5561148320984306233","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":22251,   "ctx":"initandlisten","msg":"Dropping unknown ident","attr":{"ident":"index-9-7092794553971574351","ts":{"$timestamp":{"t":1740484647,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.026+00:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
{"t":{"$date":"2025-02-25T15:05:07.042+00:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2025-02-25T15:05:07.042+00:00"},"s":"I",  "c":"STORAGE",  "id":5380103, "ctx":"initandlisten","msg":"Unpin oldest timestamp request","attr":{"service":"_wt_startup","requestedTs":{"$timestamp":{"t":1740484347,"i":1}}}}
{"t":{"$date":"2025-02-25T15:05:07.042+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/var/lib/mongodb/diagnostic.data"}}
{"t":{"$date":"2025-02-25T15:05:07.054+00:00"},"s":"I",  "c":"REPL",     "id":40440,   "ctx":"initandlisten","msg":"Starting the TopologyVersionObserver"}
{"t":{"$date":"2025-02-25T15:05:07.054+00:00"},"s":"I",  "c":"REPL",     "id":40445,   "ctx":"TopologyVersionObserver","msg":"Started TopologyVersionObserver"}
{"t":{"$date":"2025-02-25T15:05:07.055+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigStartingUp","oldState":"ConfigPreStart"}}
{"t":{"$date":"2025-02-25T15:05:07.055+00:00"},"s":"I",  "c":"REPL",     "id":6005300, "ctx":"initandlisten","msg":"Starting up replica set aware services"}
{"t":{"$date":"2025-02-25T15:05:07.055+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":200}}
{"t":{"$date":"2025-02-25T15:05:07.056+00:00"},"s":"I",  "c":"REPL",     "id":4280500, "ctx":"initandlisten","msg":"Attempting to create internal replication collections"}
{"t":{"$date":"2025-02-25T15:05:07.056+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.oplogTruncateAfterPoint","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"7c8c99d9-a04c-465a-bd94-236625a73d8d"}},"options":{}}}
{"t":{"$date":"2025-02-25T15:05:07.063+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"7c8c99d9-a04c-465a-bd94-236625a73d8d"}},"namespace":"local.replset.oplogTruncateAfterPoint","index":"id","ident":"index-1-14942985220493008659","collectionIdent":"collection-0-14942985220493008659","commitTimestamp":null}}
{"t":{"$date":"2025-02-25T15:05:07.063+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.minvalid","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"1bc00966-9cb2-429e-9761-6bbfb4bab23e"}},"options":{}}}
{"t":{"$date":"2025-02-25T15:05:07.070+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"1bc00966-9cb2-429e-9761-6bbfb4bab23e"}},"namespace":"local.replset.minvalid","index":"id","ident":"index-3-14942985220493008659","collectionIdent":"collection-2-14942985220493008659","commitTimestamp":null}}
{"t":{"$date":"2025-02-25T15:05:07.071+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.election","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"24abd2e8-c60c-41eb-9499-6e5a1abcdf2b"}},"options":{}}}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"24abd2e8-c60c-41eb-9499-6e5a1abcdf2b"}},"namespace":"local.replset.election","index":"id","ident":"index-5-14942985220493008659","collectionIdent":"collection-4-14942985220493008659","commitTimestamp":null}}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"REPL",     "id":4280501, "ctx":"initandlisten","msg":"Attempting to load local voted for document"}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"REPL",     "id":21311,   "ctx":"initandlisten","msg":"Did not find local initialized voted for document at startup"}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"REPL",     "id":4280502, "ctx":"initandlisten","msg":"Searching for local Rollback ID document"}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"REPL",     "id":21312,   "ctx":"initandlisten","msg":"Did not find local Rollback ID document at startup. Creating one"}
{"t":{"$date":"2025-02-25T15:05:07.078+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.system.rollback.id","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"e10550ce-846c-4d63-a2f3-59cd7d26aff2"}},"options":{}}}
{"t":{"$date":"2025-02-25T15:05:07.085+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"e10550ce-846c-4d63-a2f3-59cd7d26aff2"}},"namespace":"local.system.rollback.id","index":"id","ident":"index-7-14942985220493008659","collectionIdent":"collection-6-14942985220493008659","commitTimestamp":null}}
{"t":{"$date":"2025-02-25T15:05:07.085+00:00"},"s":"I",  "c":"REPL",     "id":21531,   "ctx":"initandlisten","msg":"Initialized the rollback ID","attr":{"rbid":1}}
{"t":{"$date":"2025-02-25T15:05:07.085+00:00"},"s":"I",  "c":"REPL",     "id":21313,   "ctx":"initandlisten","msg":"Did not find local replica set configuration document at startup","attr":{"error":{"code":47,"codeName":"NoMatchingDocument","errmsg":"Did not find replica set configuration document in local.system.replset"}}}
{"t":{"$date":"2025-02-25T15:05:07.085+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigUninitialized","oldState":"ConfigStartingUp"}}
{"t":{"$date":"2025-02-25T15:05:07.085+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.system.views","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"c2bcbfa4-9671-4951-961b-427085889da8"}},"options":{}}}
{"t":{"$date":"2025-02-25T15:05:07.092+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"c2bcbfa4-9671-4951-961b-427085889da8"}},"namespace":"local.system.views","index":"id","ident":"index-9-14942985220493008659","collectionIdent":"collection-8-14942985220493008659","commitTimestamp":null}}
{"t":{"$date":"2025-02-25T15:05:07.093+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2025-02-25T15:05:07.093+00:00"},"s":"I",  "c":"STORAGE",  "id":7333401, "ctx":"initandlisten","msg":"Starting the DiskSpaceMonitor"}
{"t":{"$date":"2025-02-25T15:05:07.093+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Invalid access at address: 0n"}}
{"t":{"$date":"2025-02-25T15:05:07.093+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Dumping siginfo (si_code=1): 0b 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0>
{"t":{"$date":"2025-02-25T15:05:07.093+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).n"}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"E",  "c":"CONTROL",  "id":31430,   "ctx":"initandlisten","msg":"Error collecting stack trace","attr":{"error":"unw_get_proc_name(7FB64AC30520): no unwind info foundnunw_get_proc_name(7FB64AC17D90): no unwind info foundn"}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"initandlisten","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"564D0937ABC8","b":"564D00497000","o":"8EE3BC8","s":"_ZN5mongo15printStackTrace	Ev","C":"mongo::printStackTrace()","s+":"38"},{"a":"564D0937504B","b":"564D00497000","o":"8EDE04B","s":"abruptQuitWithAddrSignal","s+":"12B"},{"a":"7FB64AC30520","b":"7FB64ABEE000","o":"42520"},{"a":"564D04E1D849","b":"564D00497000","o":"4986849",>
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D0937ABC8","b":"564D00497000","o":"8EE3BC8","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"38"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D0937504B","b":"564D00497000","o":"8EDE04B","s":"abruptQuitWithAddrSignal","s+":"12B"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7FB64AC30520","b":"7FB64ABEE000","o":"42520"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04E1D849","b":"564D00497000","o":"4986849","s":"_ZN5mongo4repl20StorageInterfaceImpl25getEarliestOplogTimestampEPNS_16OperationContextE","C":"mongo::repl::StorageInterfaceImpl::getEarliestOplogTimestamp(mongo::OperationContext*)","s+":"59"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04BCA91B","b":"564D00497000","o":"473391B","s":"ZZN5mongo12_GLOBAL_N_146cleanupPreImagesCollectionAfterUncleanShutdownEPNS_16OperationContextEN5boost8optionalINS_8TenantIdEEEENKUlvE_clEv","C":"mongo::(anonymous namespace)::cleanupPreImagesCollectionAfterUncleanShutdown(mongo::OperationContext*, boost::optional<mongo::TenantId>)::{la>
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04BCB0D5","b":"564D00497000","o":"47340D5","s":"_ZN5mongo12_GLOBAL__N_146cleanupPreImagesCollectionAfterUncleanShutdownEPNS_16OperationContextEN5boost8optionalINS_8TenantIdEEE","C":"mongo::(anonymous namespace)::cleanupPreImagesCollectionAfterUncleanShutdown(mongo::OperationContext*, boost::optional<mongo::TenantId>)","s+":"135"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04BD13EE","b":"564D00497000","o":"473A3EE","s":"_ZN5mongo16startup_recovery30recoverChangeStreamCollectionsEPNS_16OperationContextEbNS_13StorageEngine17LastShutdownStateE","C":"mongo::startup_recovery::recoverChangeStreamCollections(mongo::OperationContext*, bool, mongo::StorageEngine::LastShutdownState)","s+":"6EE"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04B66763","b":"564D00497000","o":"46CF763","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"1923"}}}
{"t":{"$date":"2025-02-25T15:05:07.156+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04B5CBC9","b":"564D00497000","o":"46C5BC9","s":"main","s+":"9"}}}
{"t":{"$date":"2025-02-25T15:05:07.157+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7FB64AC17D90","b":"7FB64ABEE000","o":"29D90"}}}
{"t":{"$date":"2025-02-25T15:05:07.157+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7FB64AC17E40","b":"7FB64ABEE000","o":"29E40","s":"__libc_start_main","s+":"80"}}}
{"t":{"$date":"2025-02-25T15:05:07.157+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"564D04B5CAA5","b":"564D00497000","o":"46C5AA5","s":"_start","s+":"25"}}}

Steps to Reproduce

I removed old stable working replication by following these steps:

1- clear `replication` in /etc/mongod.conf

2- sudo systemctl restart mongod

3- connect as admin to mongosh

4- use local

5- db.dropDatabase()

6- sudo systemctl restart mongod

And then I changed to another replSetName, after restarting mongod coredump error happened

Expected Results

For the replication to be successfull since it worked fine before

Actual Results

Core Dump error

Additional Notes

I have 4 VPS with replication working, vps-1 was primary, made vps-4 primary. Before clearing replication data from vps-1, vps-2 and vps-3. After update to a new replicaSet name, I started with vps-1, which was primary, worked fine, but for vps-2 and vps-3 it did not go well, since after adding vps-2 and vps-3 to replication set from primary (vps-1) CoreDump error happened on both of them.

related to

SERVER-101656 Consider preventing dropping the local database

Investigating

Details

Description

Problem Statement/Rationale

Steps to Reproduce

Expected Results

Actual Results

Additional Notes

Attachments

Issue Links

Activity

People

Dates