Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21201

Increase logging severity for Assertion: 28556:Didn't find RecordId in WiredTigerRecordStore

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Duplicate
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 3.0.5
    • Component/s: Logging
    • Labels:
    • Storage Execution

      We are experiencing an error where we can find some records with a query on the secondary but not on the primary. We are still working on a reproducible test case for this and will file a bug once we have once.

      However.

      When a record can't be found the only indication we have in mongodb.log is an assertion log with severity I(nformational):

      2015-10-29T01:21:45.869Z I - [conn225391] Assertion: 28556:Didn't find RecordId in WiredTigerRecordStore

with an associated stack trace:

2015-10-29T01:21:45.877Z I CONTROL [conn225391] 
 0xf77ca9 0xf16b61 0xefbdbf 0xd8ff0c 0x92cb70 0xa1cb0b 0xa2dcd4 0xa2e5c7 0xbeaae2 0xbeb14c 0xbeb47f 0xbbccff 0xbb8736 0xaceb60 0x829dcd 0xf2ac4b 0x7fde2d045df5 0x7fde2baf8bfd 
----- BEGIN BACKTRACE ----- 
{"backtrace":[{"b":"400000","o":"B77CA9"},{"b":"400000","o":"B16B61"},{"b":"400000","o":"AFBDBF"},{"b":"400000","o":"98FF0C"},{"b":"400000","o":"52CB70"},{"b":"400000","o":"61CB0B"},{"b":"400000","o":"62D 
CD4"},{"b":"400000","o":"62E5C7"},{"b":"400000","o":"7EAAE2"},{"b":"400000","o":"7EB14C"},{"b":"400000","o":"7EB47F"},{"b":"400000","o":"7BCCFF"},{"b":"400000","o":"7B8736"},{"b":"400000","o":"6CEB60"},{" 
b":"400000","o":"429DCD"},{"b":"400000","o":"B2AC4B"},{"b":"7FDE2D03E000","o":"7DF5"},{"b":"7FDE2BA02000","o":"F6BFD"}],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338 
d9e7be6698e4a3a3", "uname" : { "sysname" : "Linux", "release" : "4.1.7-15.23.amzn1.x86_64", "version" : "#1 SMP Mon Sep 14 23:20:33 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "4 
00000", "buildId" : "E8A046532FE2C5A8474250BA8E5B72C6C84B9D68" }, { "b" : "7FFF5D5FC000", "elfType" : 3, "buildId" : "AF79256455DAB4D8F13725CDA5E71975F69FED3F" }, { "b" : "7FDE2D03E000", "path" : "/lib64/ 
libpthread.so.0", "elfType" : 3, "buildId" : "21054340E69FD5F7DDC9DECEF7951CF3F3D2E78E" }, { "b" : "7FDE2CDD1000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "22480480235F3B1C6C2E5E595 
3949728676D3796" }, { "b" : "7FDE2C9EC000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F1C146B78505646930DD9003AA2B3622C5226D1B" }, { "b" : "7FDE2C7E4000", "path" : "/lib64/librt.so.1" 
, "elfType" : 3, "buildId" : "8A6EB7521ABE0518C2803504B52F18412D69C713" }, { "b" : "7FDE2C5E0000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7030B651A4F5D6F32A8D83F3541990E342C50E6C" }, { 
"b" : "7FDE2C2DC000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7FDE2BFDA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "bu
ildId" : "EB09196C8A859427FD964381CDAC80EC0C4C4296" }, { "b" : "7FDE2BDC4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7FDE2BA020 
00", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "0EB7F7FA93C9C3E017F7257AE97415730345767B" }, { "b" : "7FDE2D25A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F451C 
8F279B6860B00F835764FFBA0D54879E1C6" }, { "b" : "7FDE2B7B6000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "FF843C37C38E5BFFD57F7BCCAE05FDADC6390C8F" }, { "b" : "7FDE2B4D3000", "pat 
h" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "0BB150CC29DB5B0E039879EFC00152A75E3B00B9" }, { "b" : "7FDE2B2D0000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1 
B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7FDE2B09E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "1485992B0E5CDBA0A34817FC8C6A4C45E82CD1A9" }, { "b" : "7FDE2AE88000", "path" : "/lib 
64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7FDE2AC79000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A75A81EC50E9E0164A12B59D9 
987AD61AC7576C8" }, { "b" : "7FDE2AA76000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7FDE2A85C000", "path" : "/lib64/libresolv. 
so.2", "elfType" : 3, "buildId" : "F586C9A0950DFD60125B2F56B0EC6D690128B1FC" }, { "b" : "7FDE2A63B000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "C16D759FEE22609796EA66A648C297C1E 
B9191D1" } ] }} 
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf77ca9] 
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf16b61] 
 mongod(_ZN5mongo11msgassertedEiPKc+0xAF) [0xefbdbf] 
 mongod(_ZNK5mongo21WiredTigerRecordStore7dataForEPNS_16OperationContextERKNS_8RecordIdE+0xDC) [0xd8ff0c] 
 mongod(_ZNK5mongo10Collection6docForEPNS_16OperationContextERKNS_8RecordIdE+0x20) [0x92cb70] 
 mongod(_ZN5mongo10FetchStage4workEPm+0x2BB) [0xa1cb0b] 
 mongod(_ZN5mongo14MultiPlanStage12workAllPlansEmPNS_15PlanYieldPolicyE+0xE4) [0xa2dcd4] 
 mongod(_ZN5mongo14MultiPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE+0xC7) [0xa2e5c7] 
 mongod(_ZN5mongo12PlanExecutor12pickBestPlanENS0_11YieldPolicyE+0x72) [0xbeaae2] 
 mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextEPNS_10WorkingSetEPNS_9PlanStageEPNS_13QuerySolutionEPNS_14CanonicalQueryEPKNS_10CollectionERKSsNS0_11YieldPolicyEPPS0_+0x7C) [0xbeb14c] 
 mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextEPNS_10WorkingSetEPNS_9PlanStageEPNS_13QuerySolutionEPNS_14CanonicalQueryEPKNS_10CollectionENS0_11YieldPolicyEPPS0_+0x7F) [0xbeb47f] 
 mongod(_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionEPNS_14CanonicalQueryENS_12PlanExecutor11YieldPolicyEPPS6_m+0xCF) [0xbbccff] 
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_+0x666) [0xbb8736] 
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB10) [0xaceb60] 
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x829dcd] 
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf2ac4b] 
 libpthread.so.0(+0x7DF5) [0x7fde2d045df5] 
 libc.so.6(clone+0x6D) [0x7fde2baf8bfd] 
----- END BACKTRACE -----

      There were no messages with Warning or Error severity to report the failure to find the record. This failure can lead to loss of data, and warrants higher severity, and probably a dedicated error message.

        1. SERVER-21201.log
          29.32 MB
          Mike Zraly

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            mzraly Mike Zraly
            Votes:
            2 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: