[SERVER-55177] My Mongo server stopped unexpectedly. i am not able to restart it Created: 12/Mar/21  Updated: 29/Mar/21  Resolved: 29/Mar/21

Status: Closed
Project: Core Server
Component/s: Admin, Diagnostics, Querying, WiredTiger
Affects Version/s: 3.6.3
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: k srinivasan Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive datadiagnostic_1.zip     Zip Archive datadiagnostic_2.zip     Zip Archive datadiagnostic_3.zip     Zip Archive datadiagnostic_4.zip     File mongolog.tar.gz    
Participants:

 Description   

My server got shutdown unexpectedly. And I am not able to restart my mongo service. its showing WiredtTiger error. can some one help in starting my server. i dont have a backup of the database also. i am listing the log file here 

 

 

2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] MongoDB starting : pid=20420 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-5-216
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] db version v3.6.3
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] allocator: tcmalloc
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] modules: none
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] build environment:
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] distarch: x86_64
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] target_arch: x86_64
2021-03-12T08:31:05.266+0000 I CONTROL [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", unixDomainSocket:

{ pathPrefix: "/run/mongodb" }

}, storage: { dbPath: "/var/lib/mongodb", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2021-03-12T08:31:05.268+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-03-12T08:31:05.268+0000 I STORAGE [initandlisten]
2021-03-12T08:31:05.268+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-03-12T08:31:05.268+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-03-12T08:31:05.268+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1430M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-03-12T08:31:05.813+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1615537865:813422][20420:0x7f3c644860c0], txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2021-03-12T08:31:05.813+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615537865:813475][20420:0x7f3c644860c0], txn-recover: WiredTiger is unable to read the recovery log.
2021-03-12T08:31:05.813+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615537865:813482][20420:0x7f3c644860c0], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2021-03-12T08:31:05.813+0000 E STORAGE [initandlisten] WiredTiger error (0) [1615537865:813489][20420:0x7f3c644860c0], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2021-03-12T08:31:05.813+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1615537865:813497][20420:0x7f3c644860c0], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2021-03-12T08:31:05.816+0000 E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 413
2021-03-12T08:31:05.816+0000 I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2021-03-12T08:31:05.816+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2021-03-12T08:31:05.816+0000 I NETWORK [initandlisten] removing socket file: /run/mongodb/mongodb-27017.sock
2021-03-12T08:31:05.816+0000 I CONTROL [initandlisten] now exiting
2021-03-12T08:31:05.816+0000 I CONTROL [initandlisten] shutting down with code:100



 Comments   
Comment by k srinivasan [ 28/Mar/21 ]

Eric, 

  Thanks, i have updated the database its working fine now.. thanks for your support.. 

 

Comment by Eric Sedor [ 25/Mar/21 ]

Thanks ksrinivasan@xperiolabs.com,

The diagnostic data spans a time range between 2020-12-22T02:04:37.534Z and 2021-02-09T16:04:41.697Z. Unfortunately this does not cover the range of restarts reported in the logs (10 between 2021-02-10T02:15:49.919Z and 2021-03-19T06:47:33.643Z).

It's not immediately clear to me that the logs and diagnostic data are from the same server. But I can say the diagnostic data reports the server version as version 4.2.5 for the entire time range (mid-December to early February).

Starting version 3.6.3 server against a dbpath created by MongoDB 4.2 would explain the issue you are seeing. My suggestion would be to double check that these files are from the same server. If they are, make a copy of the dbpath (to be safe), then try to start using version 4.2.5.

Does this help?
Eric

Comment by k srinivasan [ 25/Mar/21 ]

Eric, 

 I have uploaded the datadiagnostic file as 4 zip files. i couldn't upload as one zip file. 

Comment by k srinivasan [ 25/Mar/21 ]

Eric , i have uploaded the files. pleases check it. Thanx a ton in advance.

Comment by Eric Sedor [ 24/Mar/21 ]

Would you please archive (tar or zip) and upload to this ticket the mongod logs and the $dbpath/diagnostic.data directory (the contents are described here)?

Comment by k srinivasan [ 17/Mar/21 ]

Hi Eric , The expected mongo DB version is 3.6, the Database is a standalone server in an aws instance. i have installed this DB few months ago for mongo version 3.6 and i dint make any update to it. 

 

Comment by Eric Sedor [ 16/Mar/21 ]

Hi ksrinivasan@xperiolabs.com, can you clarify the expected MongoDB version for this server? Is it possible that on restart the mongod binaries were of a different version prior to this restart attempt?

Generated at Thu Feb 08 05:35:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.