[SERVER-6627] top command should return cursor Created: 27/Jul/12  Updated: 07/Feb/24

Status: Backlog
Project: Core Server
Component/s: Diagnostics
Affects Version/s: 2.0.1
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: charity majors Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 17
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ubuntu lucid, ec2, mongodb-10gen 2.0.1


Issue Links:
Depends
is depended on by COMPASS-1658 Trim database/collections to a reason... Closed
Duplicate
is duplicated by SERVER-21081 mongotop dies with BSONObj size on se... Closed
is duplicated by SERVER-25437 db.runCommand("top") fails when there... Closed
Related
related to SERVER-46196 Failed collection creation leaves an ... Closed
is related to SERVER-78999 Top still keeps state about operation... Open
is related to SERVER-21167 Top keeps state about operations agai... Closed
is related to SERVER-21168 Dropping database does not clear stat... Closed
is related to SERVER-27602 High CPU usage and Error running mong... Closed
Assigned Teams:
Query Execution
Participants:
Case:
Linked BF Score: 26

 Description   

Mongotop dies with the following error on the primary node:

ubuntu@db04:~$ mongotop
connected to: 127.0.0.1
assertion: 13106 nextSafe():

{ $err: "BufBuilder grow() > 64MB", code: 13548 }

 Comments   
Comment by Melvin George [ 08/Jul/22 ]

Reference to my post in Developer Community forum: https://www.mongodb.com/community/forums/t/mongotop-fails-after-sometime-with-error-failed-bsonobj-size-59797193-0x3906ec9-is-invalid-size-must-be-between-0-and-16793600-16mb/172572

 

When server is started, mongotop works. After a few hours/days, mongotop fails. The error shown is of the form:
2022-07-01T01:38:14.747+0000 Failed: BSONObj size: 59797193 (0x3906EC9) is invalid. Size must be between 0 and 16793600(16MB) First element: note: “all times in microseconds”

If the mongod service is restarted, mongotop works again. But, again after a few hours/days, mongotop fails again with the same error above.

Another case is, when a script is run to check document object size across all documents in all collections in all databases, script fails after a while. It is Never always the same database/collection/document.

 

  • What version of MongoDB you are using?
    MongoDB 4.4.14 (same error happened with 4.4.6, so we upgraded to 4.4.14 and the error still exists)
  • How many active collections do you have?
    Around 80 databases, with mostly 1300 collections each.
  • What information you are looking for in mongotop?
    Identify the database.collection having most activity during peak load.

 

Please make mongotop work for large deployments also.

 

Thanking you,

Melvin

Comment by Arie Grapa [ 08/Sep/16 ]

you could make it the default behavior with a flag to return everything if the user prefers that.

Comment by Kyle Erf [ 04/Aug/16 ]

If we believe this will require too much time to implement as a cursor, we could potentially approach this problem from another angle. Generally, mongotop only returns the N most high-traffic collections, so we could also address the root issue here by adding a parameter to the top command to sort and return those N results for us. It's not a full solution for users manually calling the command, but it would make the tool functional for users with many collections.

Comment by J Rassi [ 27/Oct/15 ]

I can reproduce this issue on 3.2.0-rc0 after creating 6,000 collections.

Changed title of issue to "top command should return cursor". Moving back into "Needs Triage".

Comment by Gregor Macadam [ 09/Aug/12 ]

I can repro this is in 2.2.0rc0

> db.stats()
{
	"db" : "test",
	"collections" : 134033,
	"objects" : 532481,
	"avgObjSize" : 43.916706887194096,
	"dataSize" : 23384812,
	"storageSize" : 586797056,
	"numExtents" : 134047,
	"indexes" : 133120,
	"indexSize" : 1088389120,
	"fileSize" : 8519680000,
	"nsSizeMB" : 160,
	"ok" : 1
}
> use admin
switched to db admin
> db.runCommand({top:1})
Thu Aug  9 15:05:22 uncaught exception: error {
	"$err" : "BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.",
	"code" : 13548
}
>

Comment by charity majors [ 01/Aug/12 ]

We have around 170k-200k collections.

Server logs:

Mon Jul 30 17:40:34 [initandlisten] connection accepted from 127.0.0.1:59497 #3205323
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x689f68 0x68b259 0x68b6ca 0x68d023 0x976b51 0x9783b5 0x961745 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZNK5mongo3Top17_appendStatsEntryERNS_14BSONObjBuilderEPKcRKNS0_9UsageDataE+0x538) [0x689f68]
 /usr/bin/mongod(_ZNK5mongo3Top17_appendToUsageMapERNS_14BSONObjBuilderERKSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS4_EEE+0x159) [0x68b259]
 /usr/bin/mongod(_ZN5mongo3Top6appendERNS_14BSONObjBuilderE+0x3a) [0x68b6ca]
 /usr/bin/mongod(_ZN5mongo6TopCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x183) [0x68d023]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xaa1) [0x976b51]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x705) [0x9783b5]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x961745]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x507c8d 0x68a104 0x68b259 0x68b6ca 0x68d023 0x976b51 0x9783b5 0x961745 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZN5mongo14BSONObjBuilder5_doneEv+0x21d) [0x507c8d]
 /usr/bin/mongod(_ZNK5mongo3Top17_appendStatsEntryERNS_14BSONObjBuilderEPKcRKNS0_9UsageDataE+0x6d4) [0x68a104]
 /usr/bin/mongod(_ZNK5mongo3Top17_appendToUsageMapERNS_14BSONObjBuilderERKSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS4_EEE+0x159) [0x68b259]
 /usr/bin/mongod(_ZN5mongo3Top6appendERNS_14BSONObjBuilderE+0x3a) [0x68b6ca]
 /usr/bin/mongod(_ZN5mongo6TopCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x183) [0x68d023]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xaa1) [0x976b51]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x705) [0x9783b5]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x961745]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x507c8d 0x68b655 0x68b6ca 0x68d023 0x976b51 0x9783b5 0x961745 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZN5mongo14BSONObjBuilder5_doneEv+0x21d) [0x507c8d]
 /usr/bin/mongod(_ZNK5mongo3Top17_appendToUsageMapERNS_14BSONObjBuilderERKSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS4_EEE+0x555) [0x68b655]
 /usr/bin/mongod(_ZN5mongo3Top6appendERNS_14BSONObjBuilderE+0x3a) [0x68b6ca]
 /usr/bin/mongod(_ZN5mongo6TopCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x183) [0x68d023]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xaa1) [0x976b51]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x705) [0x9783b5]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x961745]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x507c8d 0x68d1d7 0x976b51 0x9783b5 0x961745 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZN5mongo14BSONObjBuilder5_doneEv+0x21d) [0x507c8d]
 /usr/bin/mongod(_ZN5mongo6TopCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x337) [0x68d1d7]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xaa1) [0x976b51]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x705) [0x9783b5]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x961745]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x97740d 0x9783b5 0x961745 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x135d) [0x97740d]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x705) [0x9783b5]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x961745]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] Assertion: 13548:BufBuilder grow() > 64MB
0x588cb2 0x504259 0x5891bd 0x961797 0x964ad1 0x882407 0x888c2c 0xa9c576 0x638937 0x7fbbfa8929ca 0x7fbbf9e4170d 
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x588cb2]
 /usr/bin/mongod() [0x504259]
 /usr/bin/mongod(_ZNK5mongo13ExceptionInfo6appendERNS_14BSONObjBuilderEPKcS4_+0x2bd) [0x5891bd]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x87) [0x961797]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x11e1) [0x964ad1]
 /usr/bin/mongod() [0x882407]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x888c2c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa9c576]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638937]
 /lib/libpthread.so.0(+0x69ca) [0x7fbbfa8929ca]
 /lib/libc.so.6(clone+0x6d) [0x7fbbf9e4170d]
Mon Jul 30 17:40:34 [conn3205323] end connection 127.0.0.1:59497

Comment by Daniel Pasette (Inactive) [ 29/Jul/12 ]

Can you post the server log messages as well?
Also, how many collections do you have on this server?

Generated at Thu Feb 08 03:12:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.