[SERVER-26890] Mongodb 3.0.13 crashed by Segmentation fault Created: 03/Nov/16  Updated: 03/Nov/16  Resolved: 03/Nov/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Feng Yu Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-26889 Incorrect memory access on 3.0.13 tri... Closed
Operating System: ALL
Participants:

 Description   

I don't know how to reproduce this issue, but here is the log:

2016-11-03T06:57:38.202+0800 I WRITE    [conn5] insert ksRealtimeDB.equipment.rtdata query: { _id: ObjectId('581a6f5f99695e4a7203f030'), 1: 66.0, 10: 479.0, 11: 380.0, 12: 816.0, 13: 20528.0, 2: 107.0, 20: 1.0, 
21: 1.0, 22: 0.0, 23: 0.0, 24: 0.0, 25: 0.0, 26: 0.0, 27: 0.0, 28: 0.0, 29: 0.0, 3: 3719.0, 30: 0.0, 31: 0.0, 32: 0.0, 33: 0.0, 34: 0.0, 35: 0.0, 36: 0.0, 37: 0.0, 38: 0.0, 39: 0.0, 4: 3717.0, 40: 0.0, 41: 0.0, 
5: 1191.0, 6: 479.0, 65531: 460.0, 65532: 0.0, 65533: 22567.0, 65534: 36295.0, 7: 479.0, 8: 479.0, 9: 479.0, isLoad: 1.0, isRun: 1.0, power: 697.64306640625, src: 2493, timestamp: 1478127453863 } ninserted:1 key
Updates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 2808ms
2016-11-03T06:57:38.228+0800 I COMMAND  [conn5] command ksRealtimeDB.$cmd command: insert { insert: "equipment.rtdata", ordered: true, documents: 150 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:80 locks:{
 Global: { acquireCount: { r: 153, w: 153 } }, Database: { acquireCount: { w: 153 } }, Collection: { acquireCount: { w: 3 } }, oplog: { acquireCount: { w: 150 } } } 2834ms
2016-11-03T06:57:42.440+0800 F -        Invalid access at address: 0x5
2016-11-03T06:57:42.678+0800 F -        Got signal: 11 (Segmentation fault).
 
 0xfa9262 0xfa8b13 0xfa8e74 0x7f573a965330 0x7f57396c4ee0 0x13e70e0 0x13e8210 0x1377f19 0x140a5e2 0x1409408 0x140b095 0x13fdd7c 0x1391d0c 0x7f573a95d184 0x7f573972737d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BA9262","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"BA8B13"},{"b":"400000","o":"BA8E74"},{"b":"7F573A955000","o":"10330"},{"b":"7F573962D000","o":"97EE0"},{"b":"4000
00","o":"FE70E0"},{"b":"400000","o":"FE8210","s":"__wt_reconcile"},{"b":"400000","o":"F77F19","s":"__wt_cache_op"},{"b":"400000","o":"100A5E2","s":"__wt_checkpoint"},{"b":"400000","o":"1009408"},{"b":"400000","o
":"100B095","s":"__wt_txn_checkpoint"},{"b":"400000","o":"FFDD7C"},{"b":"400000","o":"F91D0C"},{"b":"7F573A955000","o":"8184"},{"b":"7F573962D000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.
0.13", "gitVersion" : "44ae815066cdee7127ddeff34d3a04d75378fd61", "uname" : { "sysname" : "Linux", "release" : "3.13.0-91-generic", "version" : "#138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016", "machine" : "x86_64
" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "CC7EA50DEE323F333F8D66EA6795090E87BB9426" }, { "b" : "7FFC8C041000", "elfType" : 3, "buildId" : "4EAD28F888EF3520872B73C6F2FE7306C41AF22F" }, { "b" 
: "7F573A955000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F573A6F6000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.
0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F573A31A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "607C4A6F3428B6F065D55691AFDC4
E343786A8A5" }, { "b" : "7F573A112000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F5739F0E000", "path" : "/lib/x86_64-linux-g
nu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F5739C08000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2
283A878670D5BC" }, { "b" : "7F57399F2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F573962D000", "path" : "/lib/x86_64-l
inux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F573AB73000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A3734
1F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xfa9262]
 mongod(+0xBA8B13) [0xfa8b13]
 mongod(+0xBA8E74) [0xfa8e74]
 libpthread.so.0(+0x10330) [0x7f573a965330]
 libc.so.6(+0x97EE0) [0x7f57396c4ee0]
 mongod(+0xFE70E0) [0x13e70e0]
 mongod(__wt_reconcile+0x6B0) [0x13e8210]
 mongod(__wt_cache_op+0x4E9) [0x1377f19]
 mongod(__wt_checkpoint+0x992) [0x140a5e2]
 mongod(+0x1009408) [0x1409408]
 mongod(__wt_txn_checkpoint+0x8B5) [0x140b095]
 mongod(+0xFFDD7C) [0x13fdd7c]
 mongod(+0xF91D0C) [0x1391d0c]
 libpthread.so.0(+0x8184) [0x7f573a95d184]
 libc.so.6(clone+0x6D) [0x7f573972737d]
-----  END BACKTRACE  -----

My OS is Ubunut Server 14.04.5.
I suggest add "respawn" to "/etc/init/mongod.conf" to let mongodb auto restart if crashed.



 Comments   
Comment by Ramon Fernandez Marina [ 03/Nov/16 ]

abcfy2, our investigation shows that this issue is the same as the one described in SERVER-26889, so I'm going to close this ticket as a duplicate.

We have a fix that we're planning on releasing very soon. If this is bug is critical to your deployment you can consider downgrading to 3.0.12 or upgrading to 3.2.10, where this bug is not present.

My apologies that you were hit by this bug, and thanks for reporting it promptly.

Regards,
Ramón.

Comment by Keith Bostic (Inactive) [ 03/Nov/16 ]

abcfy2, I believe we understand the problem and won't need your database files. Thank you!

Comment by Feng Yu [ 03/Nov/16 ]

It's too large:

du -sh /mnt/mongodb_wt/
567G	/mnt/mongodb_wt/

How could I upload this so large directory ?

Comment by Ramon Fernandez Marina [ 03/Nov/16 ]

Sorry to hear you've hit this issue. This looks the same as SERVER-26889, which is being investigated right now.

Would it be possible for you to upload the contents of your dbpath? I've created a [private, secure upload portal|https://10gen-httpsupload.s3.amazonaws.com/upload_forms/61d177d1-a524-444e-b36b-30842f974f6e.html
] for you to upload those files to us, since they may be needed to investigate this bug.

Thanks,
Ramón.

Generated at Thu Feb 08 04:13:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.