[SERVER-38780] Mongodb restarts with the Got signal: 6 Aborted (after suddenly loses power) Created: 27/Dec/18 Updated: 27/Dec/18 Resolved: 27/Dec/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 3.2.18 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Liu Pengshi | Assignee: | Danny Hatcher (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
After the mongodb server suddenly loses power, it restarts with the Got signal: 6 (Aborted). The mongodb engineer helped me fix the WiredTiger.wt and WiredTiger.turtle files, but the problem still exists. What else can I do to save my data? Thank goodness. Most recent stacktrace is here: 2018-12-27T20:32:33.069+0800 I CONTROL [initandlisten] options: { config: "/usr/local/iGET/SRE/software/mongo/bin/../conf/mongodb_single.conf", cpu: false, net: { bindIp: "127.0.0.1,3.1.204.206", http: { RESTInterfaceEnabled: false }, maxIncomingConnections: 2048, port: 8888, ssl: { CAFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/root-ca.pem", PEMKeyFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", PEMKeyPassword: "<password>", allowInvalidCertificates: true, allowInvalidHostnames: true, clusterFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", clusterPassword: "<password>", disabledProtocols: "TLS1_0,TLS1_1", mode: "requireSSL", sslCipherConfig: "TLSv1.2+AES", weakCertificateValidation: false }, unixDomainSocket: { enabled: false }, wireObjectCheck: true }, notablescan: false, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/sre_storage/data1/mongo/mongo.pid" }, repair: true, replication: { oplogSizeMB: 81920 }, security: { authorization: "enabled", clusterAuthMode: "x509", javascriptEnabled: false, keyFile: "/usr/local/iGET/SRE/software/mongo/bin/key" }, setParameter: { enableLocalhostAuthBypass: "0", enableTestCommands: "0", logUserIds: "1" }, storage: { dbPath: "/sre_storage/data1/mongo/data/", directoryPerDB: true, engine: "wiredTiger", mmapv1: { nsSize: 64, preallocDataFiles: true, quota: { enforced: false }, smallFiles: false }, syncPeriodSecs: 5.0, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { cacheSizeGB: 4, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/sre_storage/data1/mongo/logs/mongostatus.log", verbosity: 0 } } ***aborting after fassert() failure 2018-12-27T20:32:33.099+0800 F - [initandlisten] Got signal: 6 (Aborted). 0x130ab72 0x1309cc9 0x130a4d2 0x7fcf02fb5100 0x7fcf02c1a597 0x7fcf02c1bc88 0x1291192 0x108b693 0x94dc9e 0x94dd73 0x94df37 0x1a3d606 0x1a3bf9a 0x19f30e1 0x1a79dfe 0x1a7a485 0x1a7a8fb 0x1a01d06 0x1a7001d 0x1a3b78f 0x1a3b89e 0x19ef62e 0x1072ab1 0x106ee93 0xf98788 0x99613e 0x9986f0 0x94e79d 0x7fcf02c06bb5 0x992aad ,{"b":"400000","o":"F09CC9"},{"b":"400000","o":"F0A4D2"},{"b":"7FCF02FA6000","o":"F100"},{"b":"7FCF02BE5000","o":"35597","s":"gsignal"},{"b":"7FCF02BE5000","o":"36C88","s":"abort"},{"b":"400000","o":"E91192","s":"ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B693"},{"b":"400000","o":"54DC9E"},{"b":"400000","o":"54DD73","s":"wt_err"},{"b":"400000","o":"54DF37","s":"wt_panic"},{"b":"400000","o":"163D606"},{"b":"400000","o":"163BF9A"},{"b":"400000","o":"15F30E1"},{"b":"400000","o":"1679DFE"},{"b":"400000","o":"167A485"},{"b":"400000","o":"167A8FB"},{"b":"400000","o":"1601D06"},{"b":"400000","o":"167001D"},{"b":"400000","o":"163B78F"},{"b":"400000","o":"163B89E"},{"b":"400000","o":"15EF62E","s":"wiredtiger_open"},{"b":"400000","o":"C72AB1","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6EE93"},{"b":"400000","o":"B98788","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"59613E"},{"b":"400000","o":"5986F0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54E79D","s":"main"},{"b":"7FCF02BE5000","o":"21BB5","s":"_libc_start_main"},{"b":"400000","o":"592AAD"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.59.59.46.h24.x86_64", "version" : "#1 SMP Wed Oct 11 06:59:50 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "ADE07DD69BB7FD3B7000A0A0FE3C8178E7C83809" }, { "b" : "7FFD1FEA3000", "elfType" : 3, "buildId" : "385332720BD3BAAB7719A3D57F3FA52B9E70D20E" }, { "b" : "7FCF04252000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AAB14F6D12976911FDFDA468E4F6213595F05798" }, { "b" : "7FCF03E68000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8B2012EB69FF14394DDC8321F468022AD82B8482" }, { "b" : "7FCF03C60000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2D227847AB521134592AEC7BA0F89025137449ED" }, { "b" : "7FCF03A5C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "0D169E3DB55D4C66617903F940E412803298DA80" }, { "b" : "7FCF036DA000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "3BCAF1CF6BBCC1C79A0A64B3E996BD365F11C71A" }, { "b" : "7FCF033D8000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0AFFB8028C0403515173F63B973AFA45D8A6E7DD" }, { "b" : "7FCF031C2000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "802D73D3C6B7203CF2CACBA3AF94A463E536FA57" }, { "b" : "7FCF02FA6000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2DB99C0B19EB7547970E44D450444332D06EBAA8" }, { "b" : "7FCF02BE5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "28D908BB0F165C9EA3B2501734B6E6CA619BDA7F" }, { "b" : "7FCF044C0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5460CA1B85820630FE5171A049178C9774B1BB12" }, { "b" : "7FCF02997000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A875D496A04DEECBE1B99645B2AB3AF9D57EAFDC" }, { "b" : "7FCF026B0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "83AC4B0EA72D0C262414C60A23841CA30FD4FAEB" }, { "b" : "7FCF024AC000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "F91C59010F3EADFB0F37BD4A3D9F9DA7F2A5449E" }, { "b" : "7FCF0227A000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5A49BC797F1927AB1091AB74D2FCD5394DFEC678" }, { "b" : "7FCF02064000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "AAC85EE763F961681951A9D68D9F8A608178FEF2" }, { "b" : "7FCF01E55000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "55AAF7A0C4220AD1E5D4A73D2AC33BB78279539F" }, { "b" : "7FCF01C51000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E7FD292753065CB083D21C743F998F9ECC824B1C" }, { "b" : "7FCF01A37000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "55A18352E2DB168A3EE70CFB1C553C138B8F7F5A" }, { "b" : "7FCF01812000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "DA49E0D5C8E9150C3860617509A0CD6106E487A0" }, { "b" : "7FCF015B1000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "7D76D546405803D934E009061E6DECD4874EA63E" }, { "b" : "7FCF0138C000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "0815E89D27D798AB2F45CA887DF9B3085E3BAB52" } ] }}
Attachment data.rar is all files in the dbpath, except the original data file. Attachment repair_attempt.tar.gz is the data that mongo engineers helped me to fix last time. |
| Comments |
| Comment by Danny Hatcher (Inactive) [ 27/Dec/18 ] |
|
Hello, Unfortunately, this error after the repair attempt indicates that there was unrecoverable corruption on the disk. In this situation, our best recommendation would be to [resync the affected node|https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/] or [restore from a backup|https://docs.mongodb.com/manual/core/backups/] if possible. To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.
Thank you, Danny |