[SERVER-41403] mongo无法启动 Created: 30/May/19  Updated: 11/Jul/19  Resolved: 11/Jul/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.9
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: xu wang Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS_7.2 vm


Participants:

 Description   

启动时报以下错误,有没有遇到类似问题,谢谢!
2019-05-31T02:00:02.027+0800 I CONTROL [initandlisten] MongoDB starting : pid=100849 port=27017 dbpath=/data01/shardsvr/data 64-bit host=dggphicpra01821
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] db version v3.4.9
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] modules: none
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] build environment:
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] distmod: rhel70
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] distarch: x86_64
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-05-31T02:00:02.028+0800 I CONTROL [initandlisten] options: { net:

{ port: 27017 }

, processManagement:

{ fork: true }

, replication:

{ replSet: "shardsvrReplC" }

, security:

{ keyFile: "/data01/mongo.key" }

, sharding:

{ clusterRole: "shardsvr" }

, storage:

{ dbPath: "/data01/shardsvr/data" }

, systemLog:

{ destination: "file", path: "/data01/shardsvr/log/mongod.log" }

}
2019-05-31T02:00:02.028+0800 W - [initandlisten] Detected unclean shutdown - /data01/shardsvr/data/mongod.lock is not empty.
2019-05-31T02:00:02.058+0800 I - [initandlisten] Detected data files in /data01/shardsvr/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-05-31T02:00:02.059+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-05-31T02:00:02.059+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7419M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2019-05-31T02:00:11.469+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559239211:469944][100849:0x7f8d5a0e6e40], file:index-70441-3718103833483666497.wt, WT_CURSOR.insert: read checksum error for 4096B block at offset 389120: block header checksum of 2995986505 doesn't match expected checksum of 3353248455
2019-05-31T02:00:11.470+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559239211:470134][100849:0x7f8d5a0e6e40], file:index-70441-3718103833483666497.wt, WT_CURSOR.insert: index-70441-3718103833483666497.wt: encountered an illegal file format or internal value
2019-05-31T02:00:11.470+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1559239211:470183][100849:0x7f8d5a0e6e40], file:index-70441-3718103833483666497.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-05-31T02:00:11.470+0800 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-05-31T02:00:11.470+0800 I - [initandlisten]

***aborting after fassert() failure

2019-05-31T02:00:11.491+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x5604d5a695e1 0x5604d5a687f9 0x5604d5a68cdd 0x7f8d58d57370 0x7f8d589bc1d7 0x7f8d589bd8c8 0x5604d4d0fa33 0x5604d577baf6 0x5604d4d19d42 0x5604d4d19f5e 0x5604d4d1a1b6 0x5604d6373be5 0x5604d638d96b 0x5604d6394ac5 0x5604d63b59b0 0x5604d63838d2 0x5604d63d27d4 0x5604d646324e 0x5604d63f9020 0x5604d6463f72 0x5604d63c8f07 0x5604d63c1122 0x5604d57608ba 0x5604d5759105 0x5604d564b627 0x5604d4cfbc0c 0x5604d4d1b56b 0x7f8d589a8b35 0x5604d4d7a4ff
----- BEGIN BACKTRACE -----



 Comments   
Comment by Eric Sedor [ 11/Jul/19 ]

Hi,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Be well!

Comment by Eric Sedor [ 24/Jun/19 ]

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please consider my above questions?

Thanks,
Eric

Comment by Eric Sedor [ 03/Jun/19 ]

wangxu1234, this error message leads us to suspect some form of physical corruption. Please make a complete copy of the database's $dbpath directory to work off of and safeguard the current $dbpath.

The ideal resolution is to perform a clean resync from an unaffected node.

You can try mongod --repair], but this may be unlikely to work as the error message suggests physical corruption within an index file ( file:index-70441-3718103833483666497.wt).

Our ability to determine the source of this corruption depends greatly on your ability to provide:

  1. The logs for the affected node, including before, leading up to, and after the first sign of corruption.
  2. A description of the underlying storage mechanism in use, including details like:
    1. What file system and/or volume management system is in use?
    2. Is data storage locally attached or network-attached?
    3. Are disks RAIDed and if so how?
    4. Are disks SSDs or HDDs?
  3. A description of your backup method, if any.
  4. A description of your disks have been recently checked for integrity?
  5. A history of the deployment, including:
    1. a timeline of version changes
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes
    3. a timeline of disaster recovery or backup restoration activities
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation.
Generated at Thu Feb 08 04:57:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.