[SERVER-32127] segmentation Fault Created: 30/Nov/17  Updated: 05/Feb/18  Resolved: 09/Jan/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Maria Malik Assignee: Mark Agarunov
Resolution: Incomplete Votes: 0
Labels: RF
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongo.log     Text File mongo.log     Text File ycsb-output.txt    
Operating System: ALL
Participants:

 Description   

Running MongoDB Enterprise (3.4.10) on suse 12 with YCSB workload.
Every time I try to run the MongoDB with multiple threads, I get the segmentation fault.
Command:

./bin/mongod --storageEngine wiredTiger --wiredTigerEngineConfigString "cache_size=200GB,eviction=(threads_min=4,threads_max=16)"  --dbpath  /mnt/nvme0n1/db --logpath /mnt/nvme0n1/logs/mongo.log

Your help is appreciated.



 Comments   
Comment by Mark Agarunov [ 09/Jan/18 ]

Hello ayam2011,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 27/Dec/17 ]

Hello ayam2011,

Is this still an issue for you? If so, please let us know the outcome of testing on another machine.

Thanks,
Mark

Comment by Maria Malik [ 05/Dec/17 ]

Thank you Mark,

Let me check on another machine. I will try to get back to you with the response asap.

Regards,

Comment by Mark Agarunov [ 30/Nov/17 ]

Hello ayam2011,

Thank you for the information. Looking over this, I believe this error may be caused by corruption on the disk or in memory which is indicated by the __wt_row_search+0x3F. Additionally, from the logs, it appears this is a startup which goes into recovery due to an unclean shutdown. However, if this starts on some attempts but not others, this could suggest memory corruption. To verify this is not due to memory corruption, please copy the dataset to another machine and attempt to run this if possible.

Thanks,
Mark

Comment by Maria Malik [ 30/Nov/17 ]

Thank you for your response.

Does this crash happen at the same point every time?
I do not believe so, as in 1 out of 10 tries I was able to run the workload successfully but I could not figure out the problem.
Which filesystem is being used?
I am using PCI SSD that I created based on the recommend file system "XFS" with the WiredTiger storage engine
How is the underlying storage layer set up (RAID, nfs, physical disk, etc)?
*Physical Memory: 385GB
Hard Disk: 2TB*
Does this behavior happen when running mongodb with an empty dataset?
No
Could you confirm which version of ycsb you are using?
I am using ycsb version 0.12.0
Additionally, please provide the output generated from ycsb. This may have some information that could help diagnose this issue.
Please refer to the attachment.
mongo.log ycsb-output.txt

Appreciate your help.

Comment by Mark Agarunov [ 30/Nov/17 ]

Hello ayam2011,

Thank you for the report. Unfortunately I have not yet been able to reproduce this behavior. To get a better idea of what may be causing this, I'd like to ask for some additional information:

  • Does this crash happen at the same point every time?
  • Which filesystem is being used?
  • How is the underlying storage layer set up (RAID, nfs, physical disk, etc)?
  • Does this behavior happen when running mongodb with an empty dataset?
  • Could you confirm which version of ycsb you are using?
  • Additionally, please provide the output generated from ycsb. This may have some information that could help diagnose this issue.

Thanks,
Mark

Comment by Maria Malik [ 30/Nov/17 ]

Sorry forgot to mention the ycsb command:
./bin/ycsb run mongodb -s -P workloads/workloadb -p operationcount=10000000 -threads 16

Generated at Thu Feb 08 04:29:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.