[SERVER-10496] Unit test ExpressionTests::Set failure on Windows and C++11 builds Created: 13/Aug/13  Updated: 11/Jul/16  Resolved: 13/Aug/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 2.5.2

Type: Bug Priority: Critical - P2
Reporter: Andrew Morrow (Inactive) Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: C++11, buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows, OS X C+11 builds with libc+


Operating System: ALL
Participants:

 Description   

Windows and OSX C++11 builders have been failing unit tests recently.

Example 1: Windows 64-bit Build #5644 (Aug 9, rev cd81198d)

http://buildlogs.mongodb.org/Windows%2064-bit/builds/5644/test/core/test.exe

Fri Aug 09 19:25:37.429 [testsuite] 	 going to run test: class ExpressionTests::Set::NoOverlap
Fri Aug 09 19:25:37.429 [testsuite]  Assertion failure 0 D:\slave\Windows_64bit\mongo\src\mongo/bson/bsontypes.h 155
Fri Aug 09 19:25:38.272 [testsuite] FAIL: class ExpressionTests::Set::NoOverlap	 std::exception: assertion D:\slave\Windows_64bit\mongo\src\mongo/bson/bsontypes.h:155 in test class ExpressionTests::Set::NoOverlap
Fri Aug 09 19:25:38.272 [testsuite] 	 going to run test: class ExpressionTests::Set::Overlap
Fri Aug 09 19:25:38.272 [testsuite] *** unhandled exception (access violation) at 0x000000013FC40176, terminating
Fri Aug 09 19:25:38.272 [testsuite] *** access violation was a read from 0xFFFFFFFFFFFFFFFF

Example 2: OS X C++11: http://buildlogs.mongodb.org/mci_0.9_osx_108_cxx11_debug/builds/198/test/core_0/test

Mon Aug 12 17:24:44.241 [testsuite] expression           | tests:  298 | fails:    1 | assert calls:    680
	ExpressionTests::Set::NoOverlap	for expression $setIntersection with argument [[1, 2], [8, 4]] full tree: {$setIntersection: [{$const: [1, 2]}, {$const: [8, 4]}]} expected: [] but got: [1] @src/mongo/dbtests/expressiontests.cpp:2930

Windows: http://buildlogs.mongodb.org/mci_0.9_windows_64/builds/481/test/core_0/test.exe

The breakage was bisected to: https://github.com/mongodb/mongo/commit/af6828e05aaea88e181cad598cc1274e2838d25a

Running the test under valgrind on OS X shows a use after free:

Mon Aug 12 19:28:18.169 [testsuite] 	 going to run test: ExpressionTests::Set::NoOverlap
==14518== Invalid read of size 8
==14518==    at 0x100DABD21: mongo::ExpressionSetIntersection::evaluateInternal(mongo::Variables const&) const (__hash_table:196)
==14518==    by 0x1001A00A5: mongo::Expression::evaluate(mongo::Variables const&) const (expression.h:127)
==14518==    by 0x10019FE16: mongo::Expression::evaluate(mongo::Document const&) const (expression.h:123)
==14518==    by 0x1001A38A5: ExpressionTests::Set::ExpectedResultBase::run() (expressiontests.cpp:2919)
==14518==    by 0x1001AA810: void mongo::unittest::Suite::runTestObject<ExpressionTests::Set::NoOverlap>() (unittest.h:296)
==14518==    by 0x10000AF89: boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) (function_template.hpp:112)
==14518==    by 0x1011D915A: boost::function0<void>::operator()() const (function_template.hpp:759)
==14518==    by 0x1011D398B: mongo::unittest::TestHolder::run() const (unittest.h:194)
==14518==    by 0x1011D04D5: mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (unittest.cpp:164)
==14518==    by 0x1011D1F77: mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (unittest.cpp:228)
==14518==    by 0x100F94D97: mongo::dbtests::runDbTests(int, char**, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) (framework.cpp:270)
==14518==    by 0x100123355: dbtestsMain(int, char**, char**) (dbtests.cpp:40)
==14518==  Address 0x10681c460 is 0 bytes inside a block of size 32 free'd
==14518==    at 0x4D0B: free (vg_replace_malloc.c:450)
==14518==    by 0x100DB519A: std::__1::__hash_table<mongo::Value, mongo::Value::Hash, std::__1::equal_to<mongo::Value>, std::__1::allocator<mongo::Value> >::erase(std::__1::__hash_const_iterator<std::__1::__hash_node<mongo::Value, void*> const*>) (memory:1672)
==14518==    by 0x100DB4B97: unsigned long std::__1::__hash_table<mongo::Value, mongo::Value::Hash, std::__1::equal_to<mongo::Value>, std::__1::allocator<mongo::Value> >::__erase_unique<mongo::Value>(mongo::Value const&) (__hash_table:1731)
==14518==    by 0x100DABCCD: mongo::ExpressionSetIntersection::evaluateInternal(mongo::Variables const&) const (__hash_table:189)
==14518==    by 0x1001A00A5: mongo::Expression::evaluate(mongo::Variables const&) const (expression.h:127)
==14518==    by 0x10019FE16: mongo::Expression::evaluate(mongo::Document const&) const (expression.h:123)
==14518==    by 0x1001A38A5: ExpressionTests::Set::ExpectedResultBase::run() (expressiontests.cpp:2919)
==14518==    by 0x1001AA810: void mongo::unittest::Suite::runTestObject<ExpressionTests::Set::NoOverlap>() (unittest.h:296)
==14518==    by 0x10000AF89: boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) (function_template.hpp:112)
==14518==    by 0x1011D915A: boost::function0<void>::operator()() const (function_template.hpp:759)
==14518==    by 0x1011D398B: mongo::unittest::TestHolder::run() const (unittest.h:194)
==14518==    by 0x1011D04D5: mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (unittest.cpp:164)
==14518==
Mon Aug 12 19:28:18.278 [testsuite] FAIL: ExpressionTests::Set::NoOverlap	for expression $setIntersection with argument [[1, 2], [8, 4]] full tree: {$setIntersection: [{$const: [1, 2]}, {$const: [8, 4]}]} expected: [] but got: [1] @src/mongo/dbtests/expressiontests.cpp:2930



 Comments   
Comment by Matt Dannenberg [ 13/Aug/13 ]

This fixed it

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-10086 fix improper removing from a container
also may fix SERVER 10496
Branch: master
https://github.com/mongodb/mongo/commit/072e12a5e5b853435e47046d979b307fa2676951

Comment by Matt Kangas [ 13/Aug/13 ]

As you might expect, all of the master Windows builders are failing on the same. But I just noticed that the DEBUG builds cough out a significantly more helpful message. (As they should!)

Debug message: Windows 64-bit DEBUG Build #957 - buildlogger

Tue Aug 13 11:25:20.231 [testsuite] 	 going to run test: class ExpressionTests::Set::NoOverlap
Tue Aug 13 11:25:20.231 [testsuite] *** C runtime error: C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\list(227) : Assertion failed: list iterator not incrementable, terminating
Tue Aug 13 11:25:20.231 [testsuite]  Fatal Assertion 17006
2013-08-13 11:25:21 EDT	
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\util\stacktrace.cpp(169)                                    mongo::printStackTrace+0x5b
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\util\log.cpp(131)                                           mongo::logContext+0xd8
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\util\assert_util.cpp(130)                                   mongo::fassertFailed+0x94
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\util\stacktrace.cpp(285)                                    mongo::crtDebugCallback+0x107
Tue Aug 13 11:25:21.619 [testsuite] test.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\dbgrptt.c(605)                _VCrtDbgReportW+0x812
Tue Aug 13 11:25:21.619 [testsuite] test.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\dbgrpt.c(242)                 _CrtDbgReportWV+0x43
Tue Aug 13 11:25:21.619 [testsuite] test.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\dbgrpt.c(258)                 _CrtDbgReportW+0x4d
Tue Aug 13 11:25:21.619 [testsuite] test.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\stdthrow.cpp(13)              std::_Debug_message+0x34
Tue Aug 13 11:25:21.619 [testsuite] test.exe  c:\program files (x86)\microsoft visual studio 10.0\vc\include\list(228)  std::_List_const_iterator<std::_List_val<mongo::Value,std::allocator<mongo::Value> > >::operator+++0x6f
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\db\pipeline\expression.cpp(2514)                            mongo::ExpressionSetIntersection::evaluateInternal+0x3ad
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\db\pipeline\expression.h(127)                               mongo::Expression::evaluate+0x4a
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\db\pipeline\expression.h(123)                               mongo::Expression::evaluate+0x77
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\dbtests\expressiontests.cpp(2919)                           ExpressionTests::Set::ExpectedResultBase::run+0x384
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\unittest\unittest.h(296)                                    mongo::unittest::Suite::runTestObject<ExpressionTests::Set::NoOverlap>+0x33
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\third_party\boost\boost\function\function_template.hpp(113)       boost::detail::function::void_function_invoker0<void (__cdecl*)(void),void>::invoke+0x2f
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\third_party\boost\boost\function\function_template.hpp(761)       boost::function0<void>::operator()+0x87
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\unittest\unittest.h(194)                                    mongo::unittest::TestHolder::run+0x2f
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\unittest\unittest.cpp(164)                                  mongo::unittest::Suite::run+0x74b
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\unittest\unittest.cpp(228)                                  mongo::unittest::Suite::run+0x498
Tue Aug 13 11:25:21.619 [testsuite] test.exe  ...\src\mongo\dbtests\framework.cpp(276)                                  mongo::dbtests::runDbTests+0x1390

Non-debug message: Windows 64-bit 2008R2+ Build #155 - buildlogger

Mon Aug 12 14:51:01.549 [testsuite] 	 going to run test: class ExpressionTests::Set::NoOverlap
Mon Aug 12 14:51:01.549 [testsuite] *** unhandled exception (access violation) at 0x00000001403A1C58, terminating
Mon Aug 12 14:51:01.549 [testsuite] *** access violation was a read from 0xFFFFFFFFFFFFFFFF
Mon Aug 12 14:51:01.549 [testsuite] *** stack trace for unhandled exception:
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\db\pipeline\value.cpp(622)                                   mongo::Value::hash_combine+0x18
Mon Aug 12 14:51:02.391 [testsuite] test.exe      c:\program files (x86)\microsoft visual studio 10.0\vc\include\xhash(666)  std::_Hash<std::tr1::_Uset_traits<mongo::Value,std::_Hash_compare<mongo::Value,mongo::Value::Hash,std::equal_to<mongo::Value> >,std::allocator<mongo::Value>,0> >::equal_range+0x33
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\db\pipeline\expression.cpp(2515)                             mongo::ExpressionSetIntersection::evaluateInternal+0x2f1
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\db\pipeline\expression.h(127)                                mongo::Expression::evaluate+0x17
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\db\pipeline\expression.h(123)                                mongo::Expression::evaluate+0x84
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\dbtests\expressiontests.cpp(2919)                            ExpressionTests::Set::ExpectedResultBase::run+0x41e
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\unittest\unittest.h(296)                                     mongo::unittest::Suite::runTestObject<ExpressionTests::Set::NoOverlap>+0x23
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\unittest\unittest.cpp(163)                                   mongo::unittest::Suite::run+0x628
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\unittest\unittest.cpp(228)                                   mongo::unittest::Suite::run+0x7b7
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\dbtests\framework.cpp(276)                                   mongo::dbtests::runDbTests+0x18d3
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\dbtests\dbtests.cpp(41)                                      dbtestsMain+0x107
Mon Aug 12 14:51:02.391 [testsuite] test.exe      ...\src\mongo\dbtests\dbtests.cpp(52)                                      wmain+0x35
Mon Aug 12 14:51:02.391 [testsuite] test.exe      f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c(278)                    __tmainCRTStartup+0x13b
Mon Aug 12 14:51:02.391 [testsuite] kernel32.dll                                                                             BaseThreadInitThunk+0xd
Mon Aug 12 14:51:02.391 [testsuite] 
Mon Aug 12 14:51:02.391 [testsuite] writing minidump diagnostic file mongo.dmp
Mon Aug 12 14:51:02.407 [testsuite] *** immediate exit due to unhandled exception

Comment by Matt Kangas [ 13/Aug/13 ]

All of the nightly Windows failures are due to this.

Nightly Windows 64-bit Build #1369 Mon Aug 12
http://buildlogs.mongodb.org/Nightly%20Windows%2064-bit/builds/1369/test/recent%20failures/test.exe

Nightly Windows 64-bit DEBUG Build #433 Tue Aug 13
http://buildlogs.mongodb.org/Nightly%20Windows%2064-bit%20DEBUG/builds/433/test/recent%20failures/test.exe

Nightly Windows 64-bit 2008R2+ Build #131 Tue Aug 13
http://buildlogs.mongodb.org/Nightly%20Windows%2064-bit%202008R2%2B/builds/131/test/recent%20failures/test.exe

Nightly Windows 64-bit 2008R2+ DEBUG Build #123 Tue Aug 13
http://buildlogs.mongodb.org/Nightly%20Windows%2064-bit%202008R2%2B%20DEBUG/builds/123/test/recent%20failures/test.exe

Comment by Andrew Morrow (Inactive) [ 13/Aug/13 ]

Info on building with libc++ on OS X since this may be easier than building on Windows. This assumes you have the XCode command line tools installed, and valgrind via macports or homebrew in your path.

scons --d --c++11 --libc++ --osx-version-min=10.8 --allocator=system -j 10 --cc=/usr/bin/clang --cxx=/usr/bin/clang++ --mute ./test && valgrind --dsymutil=yes --track-origins=yes ./test expression

Generated at Thu Feb 08 03:23:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.