Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16325

HTTP console UI asserts if clock jumps backwards

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: HTTP Console
    • None
    • ALL
    • Hide
      • Use the date command to set the clock back in time (1 minute should be enough)
      • Load the HTTP console at http://localhost:28017/
      • Issue should resolve after a few minutes, when the snapshots from before the clock change have rolled off the list.
      Show
      Use the date command to set the clock back in time (1 minute should be enough) Load the HTTP console at http://localhost:28017/ Issue should resolve after a few minutes, when the snapshots from before the clock change have rolled off the list.

      The HTTP console UI hits an assert in db/stats/snapshots.cpp if the system clock has been set back in time recently. Including when the clock has been set automatically by ntpd. This results in the below messages, and an HTTP 500 error from the HTTP console:

      mongodb 2.0.4

      Tue Nov 25 14:44:17 [websvr]   Assertion failure _newer._created > _older._created db/stats/snapshots.cpp 39

      mongodb 2.4.9

      Tue Nov 25 13:18:04.743 [websvr]   Assertion failure _newer._created > _older._created src/mongo/db/stats/snapshots.cpp 40
0x981b16 0x945a30 0x886673 0x886c53 0x888010 0x6ce7d9 0x6d367d 0x96eed5 0x96ab25 0x6ce924 0x7f049c6d0a4a 0x7f049cf1b182 0x7f049b2e6fbd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x981b16]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xc0) [0x945a30]
 /usr/bin/mongod() [0x886673]
 /usr/bin/mongod(_ZN5mongo9Snapshots18outputLockInfoHTMLERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x133) [0x886c53]
 /usr/bin/mongod(_ZN5mongo15WriteLockStatus3runERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x20) [0x888010]
 /usr/bin/mongod(_ZN5mongo15WebStatusPlugin6runAllERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0xd9) [0x6ce7d9]
 /usr/bin/mongod(_ZN5mongo11DbWebServer9doRequestEPKcSsRSsRiRSt6vectorISsSaISsEERKNS_8SockAddrE+0x86d) [0x6d367d]
 /usr/bin/mongod(_ZN5mongo13MiniWebServer8acceptedEN5boost10shared_ptrINS_6SocketEEEx+0x165) [0x96eed5]
 /usr/bin/mongod(_ZN5mongo8Listener13initAndListenEv+0x585) [0x96ab25]
 /usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x94) [0x6ce924]
 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0(+0xba4a) [0x7f049c6d0a4a]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f049cf1b182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f049b2e6fbd]

      This is probably the same underlying issue as SERVER-7788. However, I think it should get a second look. This makes the console not reliable for automated monitoring. Also, regular users will see the assertion failure in their browser and get no indication why.

      My plan to avoid hitting this again is to enable the REST API with --rest, and have our monitoring tool use that instead.

      My thought is that since this only seems to affect the DbTopStatus and WriteLockStatus objects, can the failure be caught in there, and not prevent the entire HTTP console from loading?

        1. console_error.png
          console_error.png
          33 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            tblamer Tom Blamer
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: