[SERVER-16325] HTTP console UI asserts if clock jumps backwards Created: 25/Nov/14  Updated: 15/Jan/15  Resolved: 15/Jan/15

Status: Closed
Project: Core Server
Component/s: HTTP Console
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Tom Blamer Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File console_error.png    
Operating System: ALL
Steps To Reproduce:
  • Use the date command to set the clock back in time (1 minute should be enough)
  • Load the HTTP console at http://localhost:28017/
  • Issue should resolve after a few minutes, when the snapshots from before the clock change have rolled off the list.
Participants:

 Description   

The HTTP console UI hits an assert in db/stats/snapshots.cpp if the system clock has been set back in time recently. Including when the clock has been set automatically by ntpd. This results in the below messages, and an HTTP 500 error from the HTTP console:

mongodb 2.0.4

Tue Nov 25 14:44:17 [websvr]   Assertion failure _newer._created > _older._created db/stats/snapshots.cpp 39

mongodb 2.4.9

Tue Nov 25 13:18:04.743 [websvr]   Assertion failure _newer._created > _older._created src/mongo/db/stats/snapshots.cpp 40
0x981b16 0x945a30 0x886673 0x886c53 0x888010 0x6ce7d9 0x6d367d 0x96eed5 0x96ab25 0x6ce924 0x7f049c6d0a4a 0x7f049cf1b182 0x7f049b2e6fbd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x981b16]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xc0) [0x945a30]
 /usr/bin/mongod() [0x886673]
 /usr/bin/mongod(_ZN5mongo9Snapshots18outputLockInfoHTMLERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x133) [0x886c53]
 /usr/bin/mongod(_ZN5mongo15WriteLockStatus3runERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x20) [0x888010]
 /usr/bin/mongod(_ZN5mongo15WebStatusPlugin6runAllERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0xd9) [0x6ce7d9]
 /usr/bin/mongod(_ZN5mongo11DbWebServer9doRequestEPKcSsRSsRiRSt6vectorISsSaISsEERKNS_8SockAddrE+0x86d) [0x6d367d]
 /usr/bin/mongod(_ZN5mongo13MiniWebServer8acceptedEN5boost10shared_ptrINS_6SocketEEEx+0x165) [0x96eed5]
 /usr/bin/mongod(_ZN5mongo8Listener13initAndListenEv+0x585) [0x96ab25]
 /usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x94) [0x6ce924]
 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0(+0xba4a) [0x7f049c6d0a4a]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f049cf1b182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f049b2e6fbd]

This is probably the same underlying issue as SERVER-7788. However, I think it should get a second look. This makes the console not reliable for automated monitoring. Also, regular users will see the assertion failure in their browser and get no indication why.

My plan to avoid hitting this again is to enable the REST API with --rest, and have our monitoring tool use that instead.

My thought is that since this only seems to affect the DbTopStatus and WriteLockStatus objects, can the failure be caught in there, and not prevent the entire HTTP console from loading?



 Comments   
Comment by Ramon Fernandez Marina [ 15/Jan/15 ]

While showing the current error message may not be very user-friendly, I think it's hard to say what should the HTTP console do and display if the clock jumps back significantly. What I would recommend is the use of the shell to gather whatever metrics are needed, as this method is not affected if the clock jumps backwards.

Given that, as of right now, the HTTP console is not a top priority, I'm going to resolve this issue. This decision can be revisited at a later time depending on interest, priorities, and resource availability.

Regards,
Ramón.

Generated at Thu Feb 08 03:40:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.