Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26749

RocksDB CounterManager can load negative counter value

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Hide

      This may be reproducible by often running jstests/sharding/shard_aware_init.js. If it is known that without storage.rocksdb.crashSafeCounters set, counters might become negative, the invariant should be loosened.

      Show
      This may be reproducible by often running jstests/sharding/shard_aware_init.js . If it is known that without storage.rocksdb.crashSafeCounters set, counters might become negative, the invariant should be loosened.
    • 0

      See the log file, look for invariant:

       _dataSize.load() >= 0 src/mongo/db/modules/rocksdb/src/rocks_record_store.cpp 258
      
      [js_test:shard_aware_init] 2016-08-10T16:26:12.434+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] Block Cache Size GB: 1
      [js_test:shard_aware_init] 2016-08-10T16:26:12.434+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] Compression: snappy
      [js_test:shard_aware_init] 2016-08-10T16:26:12.434+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] MaxWriteMBPerSec: 1024
      [js_test:shard_aware_init] 2016-08-10T16:26:12.434+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] Crash safe counters: 0
      [js_test:shard_aware_init] 2016-08-10T16:26:12.435+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] Counters: 0
      [js_test:shard_aware_init] 2016-08-10T16:26:12.435+0000 d20265| 2016-08-10T16:26:12.434+0000 I STORAGE  [main] Use SingleDelete in index: 0
      [js_test:shard_aware_init] 2016-08-10T16:26:12.451+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] MongoDB starting : pid=11903 port=20265 dbpath=/data/db/job1/mongorunner/mongod-20265 64-bit host=ip-10-179-146-107
      [js_test:shard_aware_init] 2016-08-10T16:26:12.452+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] db version v3.3.10-348-g043b3b8
      [js_test:shard_aware_init] 2016-08-10T16:26:12.452+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] git version: 043b3b87e29d7891e3e8893aca13854fcd7fc5d6
      [js_test:shard_aware_init] 2016-08-10T16:26:12.452+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      [js_test:shard_aware_init] 2016-08-10T16:26:12.452+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] modules: rocksdb
      [js_test:shard_aware_init] 2016-08-10T16:26:12.452+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] build environment:
      [js_test:shard_aware_init] 2016-08-10T16:26:12.453+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten]     distarch: x86_64
      [js_test:shard_aware_init] 2016-08-10T16:26:12.453+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      [js_test:shard_aware_init] 2016-08-10T16:26:12.453+0000 d20265| 2016-08-10T16:26:12.451+0000 I CONTROL  [initandlisten] options: { net: { port: 20265 }, nopreallocj: true, security: { keyFile: "jstests/libs/authTestsKey" }, setParameter: { enableLocalhostAuthBypass: "false", enableTestCommands: "1" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db/job1/mongorunner/mongod-20265", engine: "rocksdb", rocksdb: { cacheSizeGB: 1 } } }
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265| 2016-08-10T16:26:12.498+0000 I -        [initandlisten] Invariant failure _dataSize.load() >= 0 src/mongo/db/modules/rocksdb/src/rocks_record_store.cpp 258
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265| 2016-08-10T16:26:12.498+0000 I -        [initandlisten]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265|
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265| ***aborting after invariant() failure
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265|
      [js_test:shard_aware_init] 2016-08-10T16:26:12.499+0000 d20265|
      [js_test:shard_aware_init] 2016-08-10T16:26:12.509+0000 d20265| 2016-08-10T16:26:12.509+0000 F -        [initandlisten] Got signal: 6 (Aborted).
      [js_test:shard_aware_init] 2016-08-10T16:26:12.509+0000 d20265|
      [js_test:shard_aware_init] 2016-08-10T16:26:12.510+0000 d20265|  0x1791ea8 0x1790ead 0x17912b4 0x7f947ad86330 0x7f947a9e7c37 0x7f947a9eb028 0xaa9336 0xf75efc 0xf5e02f 0x13fdf98 0x14029e4 0xf6de99 0x13b132c 0xb55341 0xb58013 0xab415a 0x7f947a9d2f45 0xb50d1f
      [js_test:shard_aware_init] 2016-08-10T16:26:12.510+0000 d20265| ----- BEGIN BACKTRACE -----
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265| {"backtrace":[{"b":"400000","o":"1391EA8","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"1390EAD"},{"b":"400000","o":"13912B4"},{"b":"7F947AD76000","o":"10330"},{"b":"7F947A9B1000","o":"36C37","s":"gsignal"},{"b":"7F947A9B1000","o":"3A028","s":"abort"},{"b":"400000","o":"6A9336","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"B75EFC","s":"_ZN5mongo16RocksRecordStoreC1ENS_10StringDataES1_PN7rocksdb2DBEPNS_19RocksCounterManagerENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbllPNS_14CappedCallbackE"},{"b":"400000","o":"B5E02F","s":"_ZN5mongo11RocksEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"FFDF98","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},{"b":"400000","o":"10029E4","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"B6DE99"},{"b":"400000","o":"FB132C","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"755341"},{"b":"400000","o":"758013","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"6B415A","s":"main"},{"b":"7F947A9B1000","o":"21F45","s":"__libc_start_main"},{"b":"400000","o":"750D1F"}],"processInfo":{ "mongodbVersion" : "3.3.10-348-g043b3b8", "gitVersion" : "043b3b87e29d7891e3e8893aca13854fcd7fc5d6", "compiledModules" : [ "rocksdb" ], "uname" : { "sysname" : "Linux", "release" : "3.13.0-24-generic", "version" : "#46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "3F57F300555B6D9900B91132C2DC247D9F7C0FF3" }, { "b" : "7FFFF4CFE000", "elfType" : 3, "buildId" : "6755FAD2CADACDF1667E5B57FF1EDFC28DD1C976" }, { "b" : "7F947BACC000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "7EE9BC42787A0F9A793BDBFCB0671FD12C52BC2A" }, { "b" : "7F947B8BC000", "path" : "/lib/x86_64-linux-gnu/libbz2.so.1.0", "elfType" : 3, "buildId" : "E1031DDBFFE20367E874B7093EEC0C8D9F3B43F6" }, { "b" : "7F947B6B4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F947B4B0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F947B1AA000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F947AF94000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F947AD76000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F947A9B1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F947BCE5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265|  mongod(_ZN5mongo15printStackTraceERSo+0x28) [0x1791ea8]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265|  mongod(+0x1390EAD) [0x1790ead]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265|  mongod(+0x13912B4) [0x17912b4]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265|  libpthread.so.0(+0x10330) [0x7f947ad86330]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.512+0000 d20265|  libc.so.6(gsignal+0x37) [0x7f947a9e7c37]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  libc.so.6(abort+0x148) [0x7f947a9eb028]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0xaa9336]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  mongod(_ZN5mongo16RocksRecordStoreC1ENS_10StringDataES1_PN7rocksdb2DBEPNS_19RocksCounterManagerENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbllPNS_14CappedCallbackE+0xC1C) [0xf75efc]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  mongod(_ZN5mongo11RocksEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x73F) [0xf5e02f]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x268) [0x13fdf98]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.513+0000 d20265|  mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0xA44) [0x14029e4]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(+0xB6DE99) [0xf6de99]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x51C) [0x13b132c]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(+0x755341) [0xb55341]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(_ZN5mongo13initAndListenEi+0x13) [0xb58013]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(main+0x98A) [0xab415a]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  libc.so.6(__libc_start_main+0xF5) [0x7f947a9d2f45]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265|  mongod(+0x750D1F) [0xb50d1f]
      [js_test:shard_aware_init] 2016-08-10T16:26:12.514+0000 d20265| -----  END BACKTRACE  -----
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            geert.bosch@mongodb.com Geert Bosch
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: