Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19309

DBtest fails ThreadedTests::Slack<mongo::SimpleRWLock, mongo::SimpleRWLock::Exclusive>

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      It is build by:

      scons install \
              %{?_smp_mflags} \
              --use-system-all \
              --variant-dir=build%{?dist} \
              --nostrip \
              --ssl \
              --disable-warnings-as-errors \
              --prefix=%{buildroot}%{_prefix} \
              --wiredtiger=off \
              CCFLAGS="%{?optflags}" LINKFLAGS="%{?__global_ldflags}"
      

      Where %{?optflags} = "-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic" and %{?__global_ldflags} = "-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld".

      Because it is not failing in all cases, it is necessary to run dbtest more times...

      Script to reproduce:

      ret=0
      i=0
      while [ $ret -eq 0 -a $i -le 100 ]; do
      	./dbtest threading
      	ret=$?
      	i=$(($i+1))
      	echo "Program $i ended with error code $ret"
      done
      

      Show
      It is build by: scons install \ %{?_smp_mflags} \ --use-system-all \ --variant-dir=build%{?dist} \ --nostrip \ --ssl \ --disable-warnings-as-errors \ --prefix=%{buildroot}%{_prefix} \ --wiredtiger=off \ CCFLAGS="%{?optflags}" LINKFLAGS="%{?__global_ldflags}" Where %{?optflags} = "-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic" and %{?__global_ldflags} = "-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld". Because it is not failing in all cases, it is necessary to run dbtest more times... Script to reproduce: ret=0 i=0 while [ $ret -eq 0 -a $i -le 100 ]; do ./dbtest threading ret=$? i=$(($i+1)) echo "Program $i ended with error code $ret" done

      Description

      When I run "./dbtest --dbpath ./var/dbtest" it fails with

      "dbtest: /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:86: boost::condition_variable::~condition_variable(): Assertion `!ret' failed."

      2015-06-09T14:22:54.176+0000 I STORAGE  [testsuite] CMD fsync: sync:1 lock:0
      2015-06-09T14:22:54.181+0000 I NETWORK  [testsuite] replacing connection to $hostFooBar:27017 with localhost
      2015-06-09T14:22:54.183+0000 I SHARDING [testsuite] ChunkManager: time to load chunks for foo.bar: 1ms sequenceNumber: 24 version: 1|92||5576f6bdcbe8e366972d28f5 based on: (empty)
      2015-06-09T14:22:54.183+0000 I NETWORK  [testsuite] replacing connection to $hostFooBar:27017 with localhost
      2015-06-09T14:22:54.183+0000 I SHARDING [testsuite] ChunkManager: time to load chunks for foo.bar: 0ms sequenceNumber: 25 version: 2|1||5576f6bdcbe8e366972d28f5 based on: 1|92||5576f6bdcbe8e366972d28f5
      2015-06-09T14:22:54.184+0000 I -        [testsuite] 	 going to run test: ShardingTests::ChunkDiffUnitTestNormal
      2015-06-09T14:23:04.020+0000 I -        [testsuite] 	 going to run test: ShardingTests::ChunkDiffUnitTestInverse
      2015-06-09T14:23:15.637+0000 I -        [testsuite] 	 DONE running tests
      2015-06-09T14:23:15.637+0000 I -        [testsuite] going to run suite: sock
      2015-06-09T14:23:15.637+0000 I -        [testsuite] 	 going to run test: SockTests::HostByName
      2015-06-09T14:23:16.200+0000 W NETWORK  [testsuite] getaddrinfo("asdfasdfasdf_no_such_host") failed: No address associated with hostname
      2015-06-09T14:23:16.201+0000 I -        [testsuite] 	 DONE running tests
      2015-06-09T14:23:16.201+0000 I -        [testsuite] going to run suite: threading
      2015-06-09T14:23:16.201+0000 I -        [testsuite] 	 going to run test: ThreadedTests::WriteLocksAreGreedy
      2015-06-09T14:23:16.202+0000 I COMMAND  [utest] 6202 1
      2015-06-09T14:23:16.302+0000 I COMMAND  [utest] 6302 2
      2015-06-09T14:23:16.402+0000 I COMMAND  [utest] 6402 3
      2015-06-09T14:23:16.602+0000 I COMMAND  [utest] 6602 1x
      2015-06-09T14:23:16.602+0000 I COMMAND  [utest] 6602 2x
      2015-06-09T14:23:16.602+0000 I COMMAND  [utest] 6602 3x
      2015-06-09T14:23:16.602+0000 I COMMAND  [utest] 200
      2015-06-09T14:23:16.602+0000 I -        [testsuite] 	 going to run test: ThreadedTests::Slack<mongo::mutex, mongo::mutex::scoped_lock>
      Boost version : 105700
      N5mongo5mutexE Slack useful work fraction: 0.845048 locks:125431
      2015-06-09T14:23:18.104+0000 I -        [testsuite] 	 going to run test: ThreadedTests::Slack<mongo::SimpleMutex, mongo::SimpleMutex::scoped_lock>
      N5mongo11SimpleMutexE Slack useful work fraction: 0.83404 locks:123297
      2015-06-09T14:23:19.606+0000 I -        [testsuite] 	 going to run test: ThreadedTests::Slack<mongo::SimpleRWLock, mongo::SimpleRWLock::Exclusive>
      N5mongo12SimpleRWLockE Slack useful work fraction: 0.726603 locks:105394
      dbtest: /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:86: boost::condition_variable::~condition_variable(): Assertion `!ret' failed.
      2015-06-09T14:23:21.117+0000 F -        [testsuite] Got signal: 6 (Aborted).
       0xf72fc1a9 0xf72fba21 0xf72fbedb 0xf5fd4be0 0xf5fd4c10 0xf520dcd6 0xf520f567 0xf52066c7 0xf520674b 0xf66b9b17 0xf6875026 0xf641c2e9 0xf72548b8 0xf7257321 0xf7054d08 0xf648a25d 0xf63df13a 0xf51f7925 0xf641ba6b
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"F5FFB000","o":"13011A9"},{"b":"F5FFB000","o":"1300A21"},{"b":"F5FFB000","o":"1300EDB"},{"b":"F5FD4000","o":"BE0"},{"b":"F5FD4000","o":"C10"},{"b":"F51DF000","o":"2ECD6"},{"b":"F51DF000","o":"30567"},{"b":"F51DF000","o":"276C7"},{"b":"F51DF000","o":"2774B"},{"b":"F5FFB000","o":"6BEB17"},{"b":"F5FFB000","o":"87A026"},{"b":"F5FFB000","o":"4212E9"},{"b":"F5FFB000","o":"12598B8"},{"b":"F5FFB000","o":"125C321"},{"b":"F5FFB000","o":"1059D08"},{"b":"F5FFB000","o":"48F25D"},{"b":"F5FFB000","o":"3E413A"},{"b":"F51DF000","o":"18925"},{"b":"F5FFB000","o":"420A6B"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "3.19.5-200.fc21.x86_64", "version" : "#1 SMP Mon Apr 20 19:51:56 UTC 2015", "machine" : "i686" }, "somap" : [ { "b" : "F5FFB000", "elfType" : 3, "buildId" : "932FBCAE7965F7AEF51E8515791D7E396371AEE3" }, { "b" : "F5FD4000", "path" : "linux-gate.so.1", "elfType" : 3, "buildId" : "CFE4EF1700503FD86C66A3B5D0354E569C965202" }, { "b" : "F5F8C000", "path" : "/lib/libstemmer.so.0", "elfType" : 3, "buildId" : "3BFA005A37EEF8E38EFF29FEBD2AA7E59077B55A" }, { "b" : "F5F85000", "path" : "/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "E24EE9C67C5043538F067ACD5668F294BAD7A3E6" }, { "b" : "F5F13000", "path" : "/lib/libpcre.so.1", "elfType" : 3, "buildId" : "C476DFFD40599C9FEFEB173A067A2A564437BB2B" }, { "b" : "F5F08000", "path" : "/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "FDE36F1AD7ACAF669F58BDD8992449D567D6B71E" }, { "b" : "F5E99000", "path" : "/lib/libboost_program_options.so.1.57.0", "elfType" : 3, "buildId" : "EFBCCE1DB7A34A855E5CDBCDE3FFE71A2E798FE5" }, { "b" : "F5E83000", "path" : "/lib/libboost_filesystem.so.1.57.0", "elfType" : 3, "buildId" : "8688B5714D25DB33212EDD6B45FD9B58ACFF8963" }, { "b" : "F5E5E000", "path" : "/lib/libboost_thread.so.1.57.0", "elfType" : 3, "buildId" : "72940075269BF89764D84AC1CFB7224FB07D38F6" }, { "b" : "F5E59000", "path" : "/lib/libboost_system.so.1.57.0", "elfType" : 3, "buildId" : "9DC72F96CD87E40E62E59F462A933E17C657C240" }, { "b" : "F5DF2000", "path" : "/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "EF452B401365E0CA4EB830EB84ACD45356084319" }, { "b" : "F58AB000", "path" : "/lib/libv8.so.3", "elfType" : 3, "buildId" : "39E3336133B7A3ACA8BE1294954BAEE0E566441B" }, { "b" : "F5829000", "path" : "/lib/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "24CB4550A6796BADCFB18336D5489073DE22763A" }, { "b" : "F57B6000", "path" : "/lib/libssl.so.10", "elfType" : 3, "buildId" : "6562F5E4E4BC47DFAD9AD9E6BA93BAA277FF4351" }, { "b" : "F55C8000", "path" : "/lib/libcrypto.so.10", "elfType" : 3, "buildId" : "EBC1D7DCE3F241BA28732EEC92F1D9DE7561E4E3" }, { "b" : "F55BF000", "path" : "/lib/librt.so.1", "elfType" : 3, "buildId" : "90FF057B09FD4C9C39AD56F82D914A97565EDD3A" }, { "b" : "F55BA000", "path" : "/lib/libdl.so.2", "elfType" : 3, "buildId" : "EC8287BAB9B738E6A2C9100563C7EF7A65A4D8A0" }, { "b" : "F543F000", "path" : "/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "6312822B99518F61DC3C97703C6B9144AEB72670" }, { "b" : "F53F1000", "path" : "/lib/libm.so.6", "elfType" : 3, "buildId" : "369850833603EE6844E9D18537EBED321B3EF6D6" }, { "b" : "F53D3000", "path" : "/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "7A0239F9F4C535715C5D0A06D0D6E1BD3949C535" }, { "b" : "F53B6000", "path" : "/lib/libpthread.so.0", "elfType" : 3, "buildId" : "682912D2E1D0E7DBAB002EFFE63DDD755130ECEA" }, { "b" : "F51DF000", "path" : "/lib/libc.so.6", "elfType" : 3, "buildId" : "E80463C24626C2FD20830E386FF758562468D739" }, { "b" : "F5FD5000", "path" : "/lib/ld-linux.so.2", "elfType" : 3, "buildId" : "B6282FD64643B1B3ECF5D5033588E3BA5AE2B0E0" }, { "b" : "F51C9000", "path" : "/lib/libunwind.so.8", "elfType" : 3, "buildId" : "19F22D6C9CC1EA966D664D11DE79F13D52209A5F" }, { "b" : "F4F64000", "path" : "/lib/libicui18n.so.54", "elfType" : 3, "buildId" : "260EA5D4FCC662C8F8ACBC05FD3243BA35C39788" }, { "b" : "F4DD3000", "path" : "/lib/libicuuc.so.54", "elfType" : 3, "buildId" : "471457014879AA7B03E97906E0D4C8A1DF487D75" }, { "b" : "F35A7000", "path" : "/lib/libicudata.so.54", "elfType" : 3, "buildId" : "A420902092223A1853BD1BB2B79659FD0371308D" }, { "b" : "F355A000", "path" : "/lib/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "8E3C811ACF248F91C4F653827AF1A692EDEFC2F2" }, { "b" : "F3483000", "path" : "/lib/libkrb5.so.3", "elfType" : 3, "buildId" : "2631054252FEFC7B3D35B212BDA779849F959B35" }, { "b" : "F347E000", "path" : "/lib/libcom_err.so.2", "elfType" : 3, "buildId" : "E9BDE6F6CF577212CD60C605A31D79D925CCA8A8" }, { "b" : "F344B000", "path" : "/lib/libk5crypto.so.3", "elfType" : 3, "buildId" : "1674B297F79DB6A20479AEBF8952359898B3BBFB" }, { "b" : "F342E000", "path" : "/lib/libresolv.so.2", "elfType" : 3, "buildId" : "5AB9E6B9A7473B060A1A0AB60DDF8F8F7DA729FB" }, { "b" : "F3416000", "path" : "/lib/libz.so.1", "elfType" : 3, "buildId" : "61BD9738D3A26D072C40A50439A08A4B7DE9FCB1" }, { "b" : "F3406000", "path" : "/lib/libkrb5support.so.0", "elfType" : 3, "buildId" : "D43AE6AEE2DB2EDD10A957D478CC3A0DC952D5A6" }, { "b" : "F3401000", "path" : "/lib/libkeyutils.so.1", "elfType" : 3, "buildId" : "D72A0BB73DDF9B5A7B661A457A99FFED4C6E83D2" }, { "b" : "F33DC000", "path" : "/lib/libselinux.so.1", "elfType" : 3, "buildId" : "F24BBD1B967EAA1A7DC29EA0FEB0D8DC1DEC5439" } ] }}
       dbtest(_ZN5mongo15printStackTraceERSo+0x39) [0xf72fc1a9]
       dbtest(+0x1300A21) [0xf72fba21]
       dbtest(+0x1300EDB) [0xf72fbedb]
       linux-gate.so.1(__kernel_sigreturn+0x0) [0xf5fd4be0]
       linux-gate.so.1(__kernel_vsyscall+0x10) [0xf5fd4c10]
       libc.so.6(gsignal+0x46) [0xf520dcd6]
       libc.so.6(abort+0x157) [0xf520f567]
       libc.so.6(+0x276C7) [0xf52066c7]
       libc.so.6(+0x2774B) [0xf520674b]
       dbtest(_ZN5boost18condition_variableD1Ev+0x67) [0xf66b9b17]
       dbtest(_ZN5mongo8unittest5Suite13runTestObjectIN13ThreadedTests5SlackINS_12SimpleRWLockENS5_9ExclusiveEEEEEvv+0xE6) [0xf6875026]
       dbtest(_ZN5boost6detail8function22void_function_invoker0IPFvvEvE6invokeERNS1_15function_bufferE+0x9) [0xf641c2e9]
       dbtest(_ZN5mongo8unittest5Suite3runERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x848) [0xf72548b8]
       dbtest(_ZN5mongo8unittest5Suite3runERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EERKS8_i+0x861) [0xf7257321]
       dbtest(_ZN5mongo7dbtests10runDbTestsEiPPc+0xB8) [0xf7054d08]
       dbtest(_Z11dbtestsMainiPPcS0_+0x18D) [0xf648a25d]
       dbtest(main+0x2A) [0xf63df13a]
       libc.so.6(__libc_start_main+0xF5) [0xf51f7925]
       dbtest(+0x420A6B) [0xf641ba6b]
      -----  END BACKTRACE  -----
      

      See this bug - https://jira.mongodb.org/browse/SERVER-18906

      I have a wrong assumption that this is caused by --dbpath handling.

      It could be some race condition problem. It is not failing in every run. All tests in threading suite pass fine, only in some cases it fails.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: