[SERVER-27400] RocksDB insertion count failure in concurrency test Created: 13/Dec/16 Updated: 06/Dec/22 Resolved: 03/Jul/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Trivial - P5 |
| Reporter: | Eric Milkie | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Won't Fix | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Assigned Teams: |
Storage Execution
|
||||
| Operating System: | ALL | ||||
| Participants: | |||||
| Linked BF Score: | 65 | ||||
| Description |
concurrency_simultaneous failed on ubuntu1404-rocksdbProject: mongodb-mongo-master fsm_all_simultaneous.js - Logs | History
|
| Comments |
| Comment by Ian Whalen (Inactive) [ 19/Jun/17 ] | ||||||||||||||||||||||||
|
Moving to backlog to keep a clean view for myself. PMs will own pinging the rocks team occasionally to see if they've fixed. | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 16/Jun/17 ] | ||||||||||||||||||||||||
|
Filed https://github.com/mongodb-partners/mongo-rocks/issues/80 for tracking | ||||||||||||||||||||||||
| Comment by Ian Whalen (Inactive) [ 04/Apr/17 ] | ||||||||||||||||||||||||
|
igor - have you had a chance to repro this yet? | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 06/Mar/17 ] | ||||||||||||||||||||||||
|
Thanks Eric! I'll try reproducing as soon as I can built it | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 03/Mar/17 ] | ||||||||||||||||||||||||
|
The EC2 instance type is: c3.4xlarge | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 03/Mar/17 ] | ||||||||||||||||||||||||
|
Someone pointed out that we have had success running the gcc version of ASAN (rather than the Clang version), so I may try going down that route now. | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 03/Mar/17 ] | ||||||||||||||||||||||||
|
Can you share EC2 instance on which you're running this test? Sometimes the test will only repro on the select hardware. I'll setup the test to run many times in the loop on the exact same EC2 instance type. Once we have a repro it'll be much easier to debug. | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 02/Mar/17 ] | ||||||||||||||||||||||||
|
I tried running a patch on ASAN, but I failed to get the server to start. It was a challenge just to get Clang to build it, and then after that I think there are binary activation problems that our build system isn't picking up. | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 28/Feb/17 ] | ||||||||||||||||||||||||
|
Hi Eric, I think I fixed the compile with https://github.com/mongodb-partners/mongo-rocks/commit/f692561c99563f8bc1e3a5b086070d9bfeab7515. I'm afraid I still got nothing on the source of this bug. Would it make sense to run MongoRocks tests under ASAN build? ASAN sometimes just magically finds the root cause of weird issues | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 27/Feb/17 ] | ||||||||||||||||||||||||
|
Hi igor | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 15/Feb/17 ] | ||||||||||||||||||||||||
|
Here's another failure: concurrency_replication failed on Ubuntu 14.04 (RocksDB)Project: MongoDB (3.4) | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 24/Jan/17 ] | ||||||||||||||||||||||||
|
Ah, I though I was lucky! I'll try reproing again I guess. Thanks Eric! | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 18/Jan/17 ] | ||||||||||||||||||||||||
|
It turns out I was looking at Jira incorrectly and was mistaken – this type of failure continues to happen in our test suite. Here's one of the latest failures: concurrency failed on Ubuntu 14.04 (RocksDB)December 2 is the earliest incidence of failure that I can find. | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 18/Jan/17 ] | ||||||||||||||||||||||||
|
Hi Eric, I haven't been able to reproduce unfortunately (I ran the test in a loop for a day). I also looked deep through the code and haven't detected anything that might have caused this. It might be that there was an underlying bug in RocksDB, since the tests in Evergreen run on RocksDB's master branch. Would it make sense to close it for now and reopen if we see the issue happening again? | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 18/Jan/17 ] | ||||||||||||||||||||||||
|
Hi igor, | ||||||||||||||||||||||||
| Comment by Igor Canadi [ 14/Dec/16 ] | ||||||||||||||||||||||||
|
Thanks Eric, taking a look. | ||||||||||||||||||||||||
| Comment by Eric Milkie [ 13/Dec/16 ] | ||||||||||||||||||||||||
|
Finally, here is a similar failure but in a different way: concurrency failed on ubuntu1404-rocksdbProject: mongodb-mongo-master
| ||||||||||||||||||||||||
| Comment by Eric Milkie [ 13/Dec/16 ] | ||||||||||||||||||||||||
|
Here is another instance of the failure: concurrency_sharded failed on ubuntu1404-rocksdbProject: mongodb-mongo-master fsm_all_sharded_replication.js - Logs | History
| ||||||||||||||||||||||||
| Comment by Eric Milkie [ 13/Dec/16 ] | ||||||||||||||||||||||||
|
Hi igor, |