[SERVER-29561] dbtest failed when dropCollection Created: 12/Jun/17  Updated: 15/Nov/21  Resolved: 22/Jun/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.4.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zhang Youdong Assignee: Kelsey Schubert
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

run ./dbtest failed, error message

 
2017-06-12T17:57:45.332+0800 I -        [testsuite] going to run suite: validate_tests
2017-06-12T17:57:45.332+0800 I -        [testsuite] 	 going to run test: ValidateTests::ValidateIdIndexCount<true>
2017-06-12T17:57:45.344+0800 I INDEX    [testsuite] validating index unittests.validate_tests.$_id_
2017-06-12T17:57:45.356+0800 I INDEX    [testsuite] validating index unittests.validate_tests.$_id_
2017-06-12T17:57:45.367+0800 I INDEX    [testsuite] validating index unittests.validate_tests.$_id_
2017-06-12T17:57:45.369+0800 I -        [testsuite] 	 going to run test: ValidateTests::ValidateIdIndexCount<false>
2017-06-12T17:57:45.369+0800 I -        [testsuite] Invariant failure txn->lockState()->isDbLockedForMode(name(), MODE_X) src/mongo/db/catalog/database.cpp 372
2017-06-12T17:57:45.369+0800 I -        [testsuite]
 
***aborting after invariant() failure
 
 
2017-06-12T17:57:45.375+0800 F -        [testsuite] Got signal: 6 (Aborted).
 
 0x7f71119df291 0x7f71119de219 0x7f71119de6fd 0x7f710efe0500 0x7f710ec718a5 0x7f710ec73085 0x7f71109949b6 0x7f7110e6e100 0x7f7110e6e533 0x7f7110ced83f 0x7f7110cee63b 0x7f7111962565 0x7f71119639bd 0x7f711170bbf8 0x7f71109cdddd 0x7f71109cf1b9 0x7f710ec5dcdd 0x7f711099ecf9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F71100A7000","o":"1938291","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F71100A7000","o":"1937219"},{"b":"7F71100A7000","o":"19376FD"},{"b":"7F710EFD1000","o":"F500"},{"b":"7F710EC3F000","o":"328A5","s":"gsignal"},{"b":"7F710EC3F000","o":"34085","s":"abort"},{"b":"7F71100A7000","o":"8ED9B6","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F71100A7000","o":"DC7100","s":"_ZN5mongo8Database26dropCollectionEvenIfSystemEPNS_16OperationContextERKNS_15NamespaceStringE"},{"b":"7F71100A7000","o":"DC7533","s":"_ZN5mongo8Database14dropCollectionEPNS_16OperationContextENS_10StringDataE"},{"b":"7F71100A7000","o":"C4683F","s":"_ZN13ValidateTests20ValidateIdIndexCountILb0EE3runEv"},{"b":"7F71100A7000","o":"C4763B","s":"_ZN5mongo8unittest5Suite13runTestObjectIN13ValidateTests20ValidateIdIndexCountILb0EEEEEvv"},{"b":"7F71100A7000","o":"18BB565","s":"_ZN5mongo8unittest5Suite3runERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi"},{"b":"7F71100A7000","o":"18BC9BD","s":"_ZN5mongo8unittest5Suite3runERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EERKS8_i"},{"b":"7F71100A7000","o":"1664BF8","s":"_ZN5mongo7dbtests10runDbTestsEiPPc"},{"b":"7F71100A7000","o":"926DDD","s":"_Z11dbtestsMainiPPcS0_"},{"b":"7F71100A7000","o":"9281B9","s":"main"},{"b":"7F710EC3F000","o":"1ECDD","s":"__libc_start_main"},{"b":"7F71100A7000","o":"8F7CF9"}],"processInfo":{ "mongodbVersion" : "3.4.5-rc4-1-g7af9a88", "gitVersion" : "7af9a88227ee3d59ec32878253cda784cbf6b6a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.23.2.ali878.el6.x86_64", "version" : "#1 SMP Mon Jan 28 17:12:52 CST 2013", "machine" : "x86_64" }, "somap" : [ { "b" : "7F71100A7000", "elfType" : 3, "buildId" : "AAE895545CEEB6E011ACB2B845389C41C41BBA8F" }, { "b" : "7FFFA3EFF000", "elfType" : 3, "buildId" : "505D1DB16903CBB2ECAC8DD8137C641A63080C97" }, { "b" : "7F710FC83000", "path" : "/opt/alisentry/lib64/alisentry_connect.so", "elfType" : 3, "buildId" : "47FE7B5536DEF6EAB336914A3B472BFA7376EFDA" }, { "b" : "7F710FA80000", "path" : "/opt/alisentry/lib64/alisentry_exec.so", "elfType" : 3, "buildId" : "E5F829C9E29CD713CC37127751CB44182E2AF5E8" }, { "b" : "7F710F87E000", "path" : "/opt/alisentry/lib64/alisentry_kill.so", "elfType" : 3, "buildId" : "E40870CE7652BECE4399D99D766275FC225F677A" }, { "b" : "7F710F676000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "1574F9C2DDAECEE537C45143BB79E8D61BED98FE" }, { "b" : "7F3D0BA72000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "0B4FE52FE93C0B9894775AFDD53E2DF9D3C2839A" }, { "b" : "7F710F1EE000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "BF14593D7915402AA62C2573FCCDB252AEEBF754" }, { "b" : "7F710EFD1000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "48A9F8600F0A15F6418EDE25846C324EC8891DD4" }, { "b" : "7F3D0AE3F000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "647BA5BBE0CB2DBDB1C7AFBC9C4E74DA5DB5F29E" }, { "b" : "7F710FE85000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "42AEFAFC23375DC250C49C420C37EDC4515B9C02" } ] }}
 dbtest(_ZN5mongo15printStackTraceERSo+0x41) [0x7f71119df291]
 dbtest(+0x1937219) [0x7f71119de219]
 dbtest(+0x19376FD) [0x7f71119de6fd]
 libpthread.so.0(+0xF500) [0x7f710efe0500]
 libc.so.6(gsignal+0x35) [0x7f710ec718a5]
 libc.so.6(abort+0x175) [0x7f710ec73085]
 dbtest(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f71109949b6]
 dbtest(_ZN5mongo8Database26dropCollectionEvenIfSystemEPNS_16OperationContextERKNS_15NamespaceStringE+0x450) [0x7f7110e6e100]
 dbtest(_ZN5mongo8Database14dropCollectionEPNS_16OperationContextENS_10StringDataE+0x1C3) [0x7f7110e6e533]
 dbtest(_ZN13ValidateTests20ValidateIdIndexCountILb0EE3runEv+0x7F) [0x7f7110ced83f]
 dbtest(_ZN5mongo8unittest5Suite13runTestObjectIN13ValidateTests20ValidateIdIndexCountILb0EEEEEvv+0x10B) [0x7f7110cee63b]
 dbtest(_ZN5mongo8unittest5Suite3runERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x685) [0x7f7111962565]
 dbtest(_ZN5mongo8unittest5Suite3runERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EERKS8_i+0x4AD) [0x7f71119639bd]
 dbtest(_ZN5mongo7dbtests10runDbTestsEiPPc+0x158) [0x7f711170bbf8]
 dbtest(_Z11dbtestsMainiPPcS0_+0x17D) [0x7f71109cdddd]
 dbtest(main+0x9) [0x7f71109cf1b9]
 libc.so.6(__libc_start_main+0xFD) [0x7f710ec5dcdd]
 dbtest(+0x8F7CF9) [0x7f711099ecf9]
-----  END BACKTRACE  -----
 



 Comments   
Comment by Jianghua Yang [ 16/Jun/17 ]

Hi, Thomas Schubert, Youdong Zhang,
I spent some time to investigate this case. The deep reason of core dump is that _client.dropCollection(_ns) got failed and add print got message that below.

{ ok: 0.0, errmsg: "not master", code: 10107, codeName: "NotMaster" }.

class ValidateBase {
public:
    explicit ValidateBase(bool full) : _ctx(&_txn, _ns), _client(&_txn), _full(full) {
        _client.createCollection(_ns);
    }
    ~ValidateBase() {
        _client.dropCollection(_ns);
        getGlobalServiceContext()->unsetKillAllOperations();
    }
    Collection* collection() {
        return _ctx.getCollection();
    }

I investigate the unit tests framework, found we only have setup tests interface but no teardown tests. So when run ./dbtest, other tests affect the environment of validate_tests suite.

Comment by Kelsey Schubert [ 12/Jun/17 ]

Hi zyd_com,

We're unable to reproduced this behavior. Please take a look at our latest tests on the 3.4 branch:

https://evergreen.mongodb.com/task/mongodb_mongo_v3.4_enterprise_rhel_62_64_bit_dbtest_7af9a88227ee3d59ec32878253cda784cbf6b6a5_17_06_09_15_58_47

https://evergreen.mongodb.com/task/mongodb_mongo_v3.4_enterprise_rhel_62_64_bit_dbtest_WT_7af9a88227ee3d59ec32878253cda784cbf6b6a5_17_06_09_15_58_47

On SERVER-29217, I can see that you are running a custom build. Have local modifications been made prior to running this tests? If so, would you please confirm that the standard binaries produced by these tests on evergreen runs do not produce the error locally?

Thank you,
Thomas

Generated at Thu Feb 08 04:21:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.