[SERVER-7872] big_object1.js is failing Created: 07/Dec/12  Updated: 08/Mar/13  Resolved: 28/Jan/13

Status: Closed
Project: Core Server
Component/s: JavaScript, Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ben Becker Assignee: Mathias Stearn
Resolution: Won't Fix Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

http://buildlogs.mongodb.org/Linux%2064-bit%20Legacy/builds/5257/test/js/big_object1.js

Participants:

 Comments   
Comment by Mathias Stearn [ 07/Jan/13 ]

The legacy-static build is being removed

Comment by auto [ 16/Dec/12 ]

Author:

{u'date': u'2012-12-16T18:43:40Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7872: check retval of backtrace_symbols
Branch: master
https://github.com/mongodb/mongo/commit/e21d8122f4613756c018b0140f191bf0c6ab3c34

Comment by Ian Whalen (Inactive) [ 14/Dec/12 ]

http://buildlogs.mongodb.org/Linux%2064-bit%20Legacy/builds/5348/test/js/big_object1.js

Fri Dec 14 14:06:00.182 Assertion: 10334:BSONObj size: 17408175 (0xAFA00901) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."

Comment by auto [ 10/Dec/12 ]

Author:

{u'date': u'2012-12-10T18:24:27Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7872: allow v8 gc to account for external bson size
Branch: master
https://github.com/mongodb/mongo/commit/6c5f4f918ac7edc814ef7432180d2698d5894477

Comment by Ben Becker [ 07/Dec/12 ]

Further testing seems to indicate the issue is more likely with getLastError(). Reverting changes until this is sorted.

Comment by auto [ 07/Dec/12 ]

Author:

{u'date': u'2012-12-07T21:10:15Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7872: revert to db.getLastError()
Branch: master
https://github.com/mongodb/mongo/commit/6776b2556a07d72a8300ee5215f3eaabec5ba236

Comment by Ben Becker [ 07/Dec/12 ]

yep, my bad. GLE was supposed to be on the db, not the collection.

Comment by Ian Whalen (Inactive) [ 07/Dec/12 ]

Also maybe worth checking this out: http://buildlogs.mongodb.org/Linux%2064-bit%20multiVersion/builds/829/test/multiVersion/multi_version_sharding_passthrough.js

 *******************************************
         Test : jstests/big_object1.js ...
 m30999| Fri Dec  7 20:02:08.289 [conn1] DROP: test.big_object1
 m30001| Fri Dec  7 20:02:08 [conn4] CMD: drop test.big_object1
Fri Dec  7 20:02:08.404 exec error: jstests/big_object1.js:25 TypeError: Property 'getLastError' of object test.big_object1 is not a function
        if ( t.getLastError() != null )
               ^
Caught error : 
"error loading file: jstests/big_object1.js"

Comment by Ian Whalen (Inactive) [ 07/Dec/12 ]

possible issue with this commit at http://buildlogs.mongodb.org/Linux%2064-bit%20DUR%20OFF/builds/1895/test/recent%20failures/sharding_passthrough.js

 *******************************************
         Test : jstests/big_object1.js ...
 m30999| Fri Dec  7 14:57:15.858 [conn1] DROP: test.big_object1
 m30001| Fri Dec  7 14:57:15.858 [conn69] CMD: drop test.big_object1
Fri Dec  7 14:57:15.958 exec error: jstests/big_object1.js:25 TypeError: Property 'getLastError' of object test.big_object1 is not a function
        if ( t.getLastError() != null )
               ^
Fri Dec  7 14:57:15.958 exec error: /home/yellow/buildslave/Linux_64bit_DUR_OFF/mongo/jstests/slowNightly/sharding_passthrough.js:84 error loading file: jstests/big_object1.js
		      load(x.name);
        ^
failed to load: /home/yellow/buildslave/Linux_64bit_DUR_OFF/mongo/jstests/slowNightly/sharding_passthrough.js

Comment by Ben Becker [ 07/Dec/12 ]

getLastError() issue was due to being called on the wrong db.

https://github.com/mongodb/mongo/commit/ae915f73283a3f96f468fa384fe1e43fe54d7efe

Comment by Ben Becker [ 07/Dec/12 ]

Also note that in ~500 runs, only 2 displayed db.stats() output with less than 6 documents. One with 3 (pasted in the previous comment), and one with 5.

Comment by Ben Becker [ 07/Dec/12 ]

With extra debug output, it appears this test isn't actually blocking on getLastError():

...
Fri Dec  7 01:54:21.278 Assertion: 10334:BSONObj size: 17408175 (0xAFA00901) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."
0x1000ec2bb 0x1000c88ce 0x1000c898d 0x100069b01 0x100069bf6 0x100034a33 0x1000aed90 0x1000ae682 0x1000aed61 0x1000c1d1a 0x1000b10d1 0x14f1c1363059 
 0   mongo                               0x00000001000ec2bb _ZN5mongo15printStackTraceERSo + 43
 1   mongo                               0x00000001000c88ce _ZN5mongo11msgassertedEiPKc + 174
 2   mongo                               0x00000001000c898d _ZN5mongo11msgassertedEiRKSs + 29
 3   mongo                               0x0000000100069b01 _ZNK5mongo7BSONObj14_assertInvalidEv + 1193
 4   mongo                               0x0000000100069bf6 _ZN5mongo7BSONObj4initEPNS0_6HolderE + 102
 5   mongo                               0x0000000100034a33 _ZN5mongo14BSONObjBuilder3objEv + 67
 6   mongo                               0x00000001000aed90 _ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi + 918
 7   mongo                               0x00000001000ae682 _ZN5mongo7V8Scope16v8ToMongoElementERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE + 386
 8   mongo                               0x00000001000aed61 _ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi + 871
 9   mongo                               0x00000001000c1d1a _ZN5mongo11mongoInsertEPNS_7V8ScopeERKN2v89ArgumentsE + 1578
 10  mongo                               0x00000001000b10d1 _ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE + 105
 11  ???                                 0x000014f1c1363059 0x0 + 23028561227865
 
DEBUG: caught exception:  BSONObj size: 17408175 (0xAFA00901) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."
DEBUG: x6
{
	"ns" : "test.big_object1",
	"count" : 3,
	"size" : 42205136,
	"avgObjSize" : 14068378.666666666,
	"storageSize" : 194973696,
	"numExtents" : 1,
	"nindexes" : 1,
	"lastExtentSize" : 194973696,
	"paddingFactor" : 1,
	"systemFlags" : 1,
	"userFlags" : 0,
	"totalIndexSize" : 8176,
	"indexSizes" : {
		"_id_" : 8176
	},
	"ok" : 1
}
16537787
assert: [6] != [5] are not equal : A3
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:7)
    at doassert (src/mongo/shell/utils.js:58:1)
    at Function.assert.eq (src/mongo/shell/utils.js:88:1)
    at ./jstests/big_object1.js:45:12
Fri Dec  7 01:54:28.473 exec error: src/mongo/shell/utils.js:59 [6] != [5] are not equal : A3
throw msg;
      ^
 
failed to load: ./jstests/big_object1.js
 
Program exited with code 0375.
(gdb) 

Note that when db.stat() is run, only 3 documents exist. When the assertion check calls count(), only 5 docs existed. When I queried db.stat() after the test, all 6 documents existed.

This is a bit tangential to the buildbot issue, but appears related. This test also exposes a memory leak where the mongo client process consumes ~700mb of resident space, and 18GB of virtual space in one minute. It appears our weak gc callback isn't called every time. Adding a Context scope appears to have resolved the missed GC callbacks, but introduced new issues. WIP.

Generated at Thu Feb 08 03:15:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.