[SERVER-27376] repair mongodb fail and not start up mongodbdaba Created: 12/Dec/16  Updated: 06/Jan/17  Resolved: 06/Jan/17

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Do Thanh Tung Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Issue Links:
Duplicate
is duplicated by SERVER-27374 repair mongodb fail and not start up ... Closed
is duplicated by SERVER-27369 repair mongod db fail Closed
is duplicated by TOOLS-1574 repair mongod db fail Closed
is duplicated by TOOLS-1575 repair mongod db fail Closed
Operating System: ALL
Steps To Reproduce:

mongod --repair --dbpath /var/lib/mongo/ --storageEngine wiredTiger

Participants:

 Description   

2016-12-12T09:11:48.303+0700 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] MongoDB starting : pid=18768 port=27017 dbpath=/var/lib/mongo 64-bit host=localhost.localdomain
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] db version v3.2.9
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] allocator: tcmalloc
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] modules: none
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] build environment:
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten]     distmod: rhel62
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten]     distarch: x86_64
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten]     target_arch: x86_64
2016-12-12T09:11:48.318+0700 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-12-12T09:11:48.360+0700 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-12-12T09:11:48.360+0700 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2016-12-12T09:11:48.360+0700 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-12-12T09:11:48.360+0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=75G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-12-12T09:12:07.099+0700 I STORAGE  [initandlisten] dropping unused ident: index-1996--4952926493820669755
2016-12-12T09:12:07.100+0700 I STORAGE  [initandlisten] dropping unused ident: index-1997--4952926493820669755
2016-12-12T09:12:07.100+0700 I STORAGE  [initandlisten] dropping unused ident: index-1998--4952926493820669755
2016-12-12T09:12:07.100+0700 I STORAGE  [initandlisten] dropping unused ident: index-1999--4952926493820669755
2016-12-12T09:12:07.100+0700 I STORAGE  [initandlisten] dropping unused ident: index-2000--4952926493820669755
2016-12-12T09:12:07.100+0700 I STORAGE  [initandlisten] dropping unused ident: index-2001--4952926493820669755
2016-12-12T09:12:07.101+0700 I STORAGE  [initandlisten] dropping unused ident: index-2002--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2003--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2004--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2005--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2006--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2007--4952926493820669755
2016-12-12T09:12:07.102+0700 I STORAGE  [initandlisten] dropping unused ident: index-2008--4952926493820669755
2016-12-12T09:12:07.103+0700 I STORAGE  [initandlisten] dropping unused ident: index-2009--4952926493820669755
2016-12-12T09:12:07.103+0700 I STORAGE  [initandlisten] dropping unused ident: index-2010--4952926493820669755
2016-12-12T09:12:07.103+0700 I STORAGE  [initandlisten] dropping unused ident: index-2011--4952926493820669755
2016-12-12T09:12:07.103+0700 I STORAGE  [initandlisten] dropping unused ident: index-2012--4952926493820669755
2016-12-12T09:12:07.104+0700 I STORAGE  [initandlisten] dropping unused ident: index-2013--4952926493820669755
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] 
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] 
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-12-12T09:12:07.131+0700 I CONTROL  [initandlisten] 
2016-12-12T09:12:07.453+0700 I INDEX    [initandlisten] found 2 index(es) that wasn't finished before shutdown
2016-12-12T09:12:07.453+0700 I -        [initandlisten] Fatal assertion 28579 UnsupportedFormat: Unable to find metadata for table:index-1998--5723870115755711383 Index: {name: _id_, ns: ungdichvu.s0996} - version too new for this mongod. See http://dochub.mongodb.org/core/3.4-index-downgrade for detailed instructions on how to handle this error.
2016-12-12T09:12:07.453+0700 I -        [initandlisten] 
 
***aborting after fassert() failure



 Comments   
Comment by Kelsey Schubert [ 06/Jan/17 ]

Hi dotung,

Since we haven't heard back from you for some time and the issue has been resolved, I'm going to close this ticket. If you encounter this issue again please let us know and provide the information I requested.

Thank you,
Thomas

Comment by Kelsey Schubert [ 13/Dec/16 ]

Hi dotung,

Thanks for the additional information. I'm glad you were able to restore your database. It appears that the error message you are received was caused by data corruption. In cases like this, it is difficult to determine the root cause. However, I have a few additional questions that will help us get a better understanding of what happened here.

  1. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  2. Are you using journaling?
  3. Preceding the corruption, were there any other server errors logged?
  4. Preceding the corruption, were there any system errord logged?
  5. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  6. Have you run out of disk space recently?
  7. Would you please check the integrity of your disks?

Thank you,
Thomas

Comment by Do Thanh Tung [ 13/Dec/16 ]

Dear Thomas

issue have been solved, after i stop all service connect to mongod and run command:
mongod -vvvvvv -dbpath /var/lib/mongo/
and then remove datafile corupt: rm f index-1862-726338552627266085.wt
And run command mongod --repair --dbpath /var/lib/mongo/ --storageEngine wiredTiger
and repair susccessfull. Please tell me explain, i not downgrad version and repair suscess?

Comment by Kelsey Schubert [ 12/Dec/16 ]

Hi dotung,

I see you've opened TOOLS-1574, TOOLS-1575, SERVER-27369, and SERVER-27374 in the past day as well. My understanding is that all of these tickets relate to the same issue. Therefore, I've closed them as duplicates of this most recent ticket.

It appears that you have have upgraded to MongoDB 3.4.0, and are encountering issues after downgrading. Would you please follow the steps here to remove incompatible features and confirm the issue is resolved?

Thank you,
Thomas

Generated at Thu Feb 08 04:14:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.