-
Type:
Question
-
Resolution: Incomplete
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.4.24
-
Component/s: Index Maintenance, WiredTiger
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
MongoDB Version: 3.4.24
MongoDB hosted on Linux server was abruptly shut down due to the over-utilization of memory.
Initiated the mongodb repair using: sudo mongod -f /etc/mongodrepair.conf --repair
Repair Logs
2020-07-04T17:17:07.441+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] build index on: ZionsBank.summary properties: { v: 1, key:
{ totalVolume: -1 }
, name: "totalV$
lume_-1", ns: "ZionsBank.summary", background: true }
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] build index on: ZionsBank.summary properties: { v: 1, key:
{ ts: -1 }
, name: "ts_-1", ns: "Zi$
nsBank.summary", background: true }
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:08.673+0000 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195
2020-07-04T17:17:08.673+0000 I - [initandlisten]
***aborting after invariant() failure
2020-07-04T17:17:08.717+0000 F - [initandlisten] Got signal: 6 (Aborted).
Restart Logs
2020-07-04T17:39:14.476+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] MongoDB starting : pid=20485 port=27017 dbpath=/home/db324 64-bit host=ip-*--**-*
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] db version v3.4.24
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] modules: none
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] build environment:
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] distarch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
{ bindIp: "-.-.-.-", port: 27017 }
, replication: $
oplogSizeMB: 10240, replSetName: "rs1" }, storage: { dbPath: "/home/db324", directoryPerDB: true, engine: "wiredTiger", journal:
{ enabled$ true }
, wiredTiger: { engineConfig:
{ cacheSizeGB: 108.0 }
} }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.lo$
" } }
2020-07-04T17:39:14.480+0000 W - [initandlisten] Detected unclean shutdown - /home/db324/mongod.lock is not empty.
2020-07-04T17:39:14.499+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten]
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] ** WARNING: The configured WiredTiger cache size is more than 80% of available RAM.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=110592M,session_max=20000,eviction=(threads_min=4,t$
reads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000)$
checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667272][20485:0x7fdbcb287580], txn-recover: Main recovery loop$
starting at 73368/128
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667951][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
8 through 73369
2020-07-04T17:39:14.733+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:733044][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
9 through 73369
2020-07-04T17:39:15.164+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1593884355:164908][20485:0x7fdbcb287580], [file:ZionsBank/collectio$|file://zionsbank/collectio$]
-56-3854974571131417844.wt, WT_SESSION.open_cursor: /home/db324/ZionsBank/collection-56-3854974571131417844.wt: handle-read: pread: failed $
to read 4096 bytes at offset 28672: WT_ERROR: non-specific WiredTiger error
2020-07-04T17:39:15.164+0000 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredT$
ger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 113
2020-07-04T17:39:15.165+0000 I - [initandlisten]
***aborting after invariant() failure
{{}}
Is there a way to repair/recover the last part of the DB? or
Is there a way to ignore the broken DB? or
Is it possible to cut out the whole 2.4TB data sans the last error db and create a new MongoDB instance with 2.4TB?
{{}}
I would greatly appreciate the help.
Thanks in Advance
{{}}