[SERVER-38780] Mongodb restarts with the Got signal: 6 Aborted (after suddenly loses power) Created: 27/Dec/18  Updated: 27/Dec/18  Resolved: 27/Dec/18

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.18
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Liu Pengshi Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File data.rar     File repair_attempt.tar.gz    
Issue Links:
Duplicate
duplicates SERVER-38763 Mongodb restarts with the Got signal:... Closed
Participants:

 Description   

After the mongodb server suddenly loses power, it restarts with the Got signal: 6 (Aborted).

The mongodb engineer helped me fix the WiredTiger.wt and WiredTiger.turtle files, but the problem still exists.

What else can I do to save my data? Thank goodness.

Most recent stacktrace is here:


2018-12-27T20:32:33.069+0800 I CONTROL [initandlisten] options: { config: "/usr/local/iGET/SRE/software/mongo/bin/../conf/mongodb_single.conf", cpu: false, net: { bindIp: "127.0.0.1,3.1.204.206", http:

{ RESTInterfaceEnabled: false }

, maxIncomingConnections: 2048, port: 8888, ssl: { CAFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/root-ca.pem", PEMKeyFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", PEMKeyPassword: "<password>", allowInvalidCertificates: true, allowInvalidHostnames: true, clusterFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", clusterPassword: "<password>", disabledProtocols: "TLS1_0,TLS1_1", mode: "requireSSL", sslCipherConfig: "TLSv1.2+AES", weakCertificateValidation: false }, unixDomainSocket: { enabled: false }, wireObjectCheck: true }, notablescan: false, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/sre_storage/data1/mongo/mongo.pid" }, repair: true, replication: { oplogSizeMB: 81920 }, security: { authorization: "enabled", clusterAuthMode: "x509", javascriptEnabled: false, keyFile: "/usr/local/iGET/SRE/software/mongo/bin/key" }, setParameter: { enableLocalhostAuthBypass: "0", enableTestCommands: "0", logUserIds: "1" }, storage: { dbPath: "/sre_storage/data1/mongo/data/", directoryPerDB: true, engine: "wiredTiger", mmapv1: { nsSize: 64, preallocDataFiles: true, quota:

{ enforced: false }

, smallFiles: false }, syncPeriodSecs: 5.0, wiredTiger: { collectionConfig:

{ blockCompressor: "snappy" }

, engineConfig: { cacheSizeGB: 4, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/sre_storage/data1/mongo/logs/mongostatus.log", verbosity: 0 } }
2018-12-27T20:32:33.070+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-12-27T20:32:33.070+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=5,log_size=2GB),statistics_log=(wait=0),
2018-12-27T20:32:33.083+0800 E STORAGE [initandlisten] WiredTiger (0) [1545913953:83030][5671:0x7fcf046ccdc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-12-27T20:32:33.083+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1545913953:83063][5671:0x7fcf046ccdc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-12-27T20:32:33.083+0800 I - [initandlisten] Fatal Assertion 28558
2018-12-27T20:32:33.083+0800 I - [initandlisten]

***aborting after fassert() failure

2018-12-27T20:32:33.099+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x130ab72 0x1309cc9 0x130a4d2 0x7fcf02fb5100 0x7fcf02c1a597 0x7fcf02c1bc88 0x1291192 0x108b693 0x94dc9e 0x94dd73 0x94df37 0x1a3d606 0x1a3bf9a 0x19f30e1 0x1a79dfe 0x1a7a485 0x1a7a8fb 0x1a01d06 0x1a7001d 0x1a3b78f 0x1a3b89e 0x19ef62e 0x1072ab1 0x106ee93 0xf98788 0x99613e 0x9986f0 0x94e79d 0x7fcf02c06bb5 0x992aad
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"400000","o":"F0AB72","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"400000","o":"F09CC9"},{"b":"400000","o":"F0A4D2"},{"b":"7FCF02FA6000","o":"F100"},{"b":"7FCF02BE5000","o":"35597","s":"gsignal"},{"b":"7FCF02BE5000","o":"36C88","s":"abort"},{"b":"400000","o":"E91192","s":"ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B693"},{"b":"400000","o":"54DC9E"},{"b":"400000","o":"54DD73","s":"wt_err"},{"b":"400000","o":"54DF37","s":"wt_panic"},{"b":"400000","o":"163D606"},{"b":"400000","o":"163BF9A"},{"b":"400000","o":"15F30E1"},{"b":"400000","o":"1679DFE"},{"b":"400000","o":"167A485"},{"b":"400000","o":"167A8FB"},{"b":"400000","o":"1601D06"},{"b":"400000","o":"167001D"},{"b":"400000","o":"163B78F"},{"b":"400000","o":"163B89E"},{"b":"400000","o":"15EF62E","s":"wiredtiger_open"},{"b":"400000","o":"C72AB1","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6EE93"},{"b":"400000","o":"B98788","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"59613E"},{"b":"400000","o":"5986F0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54E79D","s":"main"},{"b":"7FCF02BE5000","o":"21BB5","s":"_libc_start_main"},{"b":"400000","o":"592AAD"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.10.0-327.59.59.46.h24.x86_64", "version" : "#1 SMP Wed Oct 11 06:59:50 UTC 2017", "machine" : "x86_64" }

, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "ADE07DD69BB7FD3B7000A0A0FE3C8178E7C83809" }, { "b" : "7FFD1FEA3000", "elfType" : 3, "buildId" : "385332720BD3BAAB7719A3D57F3FA52B9E70D20E" }, { "b" : "7FCF04252000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AAB14F6D12976911FDFDA468E4F6213595F05798" }, { "b" : "7FCF03E68000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8B2012EB69FF14394DDC8321F468022AD82B8482" }, { "b" : "7FCF03C60000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2D227847AB521134592AEC7BA0F89025137449ED" }, { "b" : "7FCF03A5C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "0D169E3DB55D4C66617903F940E412803298DA80" }, { "b" : "7FCF036DA000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "3BCAF1CF6BBCC1C79A0A64B3E996BD365F11C71A" }, { "b" : "7FCF033D8000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0AFFB8028C0403515173F63B973AFA45D8A6E7DD" }, { "b" : "7FCF031C2000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "802D73D3C6B7203CF2CACBA3AF94A463E536FA57" }, { "b" : "7FCF02FA6000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2DB99C0B19EB7547970E44D450444332D06EBAA8" }, { "b" : "7FCF02BE5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "28D908BB0F165C9EA3B2501734B6E6CA619BDA7F" }, { "b" : "7FCF044C0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5460CA1B85820630FE5171A049178C9774B1BB12" }, { "b" : "7FCF02997000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A875D496A04DEECBE1B99645B2AB3AF9D57EAFDC" }, { "b" : "7FCF026B0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "83AC4B0EA72D0C262414C60A23841CA30FD4FAEB" }, { "b" : "7FCF024AC000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "F91C59010F3EADFB0F37BD4A3D9F9DA7F2A5449E" }, { "b" : "7FCF0227A000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5A49BC797F1927AB1091AB74D2FCD5394DFEC678" }, { "b" : "7FCF02064000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "AAC85EE763F961681951A9D68D9F8A608178FEF2" }, { "b" : "7FCF01E55000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "55AAF7A0C4220AD1E5D4A73D2AC33BB78279539F" }, { "b" : "7FCF01C51000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E7FD292753065CB083D21C743F998F9ECC824B1C" }, { "b" : "7FCF01A37000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "55A18352E2DB168A3EE70CFB1C553C138B8F7F5A" }, { "b" : "7FCF01812000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "DA49E0D5C8E9150C3860617509A0CD6106E487A0" }, { "b" : "7FCF015B1000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "7D76D546405803D934E009061E6DECD4874EA63E" }, { "b" : "7FCF0138C000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "0815E89D27D798AB2F45CA887DF9B3085E3BAB52" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ab72]
mongod(+0xF09CC9) [0x1309cc9]
mongod(+0xF0A4D2) [0x130a4d2]
libpthread.so.0(+0xF100) [0x7fcf02fb5100]
libc.so.6(gsignal+0x37) [0x7fcf02c1a597]
libc.so.6(abort+0x148) [0x7fcf02c1bc88]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1291192]
mongod(+0xC8B693) [0x108b693]
mongod(+0x54DC9E) [0x94dc9e]
mongod(__wt_err+0x8D) [0x94dd73]
mongod(__wt_panic+0x24) [0x94df37]
mongod(+0x163D606) [0x1a3d606]
mongod(+0x163BF9A) [0x1a3bf9a]
mongod(+0x15F30E1) [0x19f30e1]
mongod(+0x1679DFE) [0x1a79dfe]
mongod(+0x167A485) [0x1a7a485]
mongod(+0x167A8FB) [0x1a7a8fb]
mongod(+0x1601D06) [0x1a01d06]
mongod(+0x167001D) [0x1a7001d]
mongod(+0x163B78F) [0x1a3b78f]
mongod(+0x163B89E) [0x1a3b89e]
mongod(wiredtiger_open+0x148E) [0x19ef62e]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x681) [0x1072ab1]
mongod(+0xC6EE93) [0x106ee93]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf98788]
mongod(+0x59613E) [0x99613e]
mongod(_ZN5mongo13initAndListenEi+0x10) [0x9986f0]
mongod(main+0x15D) [0x94e79d]
libc.so.6(__libc_start_main+0xF5) [0x7fcf02c06bb5]
mongod(+0x592AAD) [0x992aad]
----- END BACKTRACE -----


 

Attachment data.rar is all files in the dbpath, except the original data file.

Attachment repair_attempt.tar.gz is the data that mongo engineers helped me to fix last time.



 Comments   
Comment by Danny Hatcher (Inactive) [ 27/Dec/18 ]

Hello,

Unfortunately, this error after the repair attempt indicates that there was unrecoverable corruption on the disk. In this situation, our best recommendation would be to [resync the affected node|https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/] or [restore from a backup|https://docs.mongodb.com/manual/core/backups/] if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures. 

Thank you,

Danny

Generated at Thu Feb 08 04:50:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.