[SERVER-22756] collection_cloner_test failure Created: 19/Feb/16  Updated: 30/Mar/16  Resolved: 17/Mar/16

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 3.2.1, 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Marek Skalický Assignee: Jonathan Reams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Fedora OS


Backwards Compatibility: Fully Compatible
Operating System: Linux
Steps To Reproduce:

scons ./build/opt/mongo/db/repl/collection_cloner_test -j2 --use-system-tcmalloc --use-system-pcre --use-system-boost --use-system-snappy --use-system-valgrind --use-system-zlib --use-system-stemmer --use-system-yaml --use-system-mozjs --nostrip --ssl --disable-warnings-as-errors --wiredtiger=on --experimental-decimal-support=off --variables-files=variables.list

variables.list:
CCFLAGS="-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic"
LINKFLAGS="-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld"
CPPDEFINES="BOOST_OPTIONAL_USE_SINGLETON_DEFINITION_OF_NONE"

Sprint: Platforms 12 (04/01/16)
Participants:

 Description   

Hello,
I am compiling MongoDB 3.2.3 and collection_cloner_test is failing (also failing in 3.2.1).

The versions of installed packages could be found there - https://kojipkgs.fedoraproject.org//work/tasks/821/12840821/root.log

It seems to me that this failure could appear with upgrade to gcc6 (I know that MongoDB does not support it yet ), but I don't know...

Do you have any idea where could be a problem?

+ ./build/opt/mongo/db/repl/collection_cloner_test
2016-02-04T09:33:40.293+0000 I -        [main] going to run suite: CollectionClonerTest
2016-02-04T09:33:40.293+0000 I -        [main] 	 going to run test: InvalidConstruction
2016-02-04T09:33:40.294+0000 I -        [main] 	 going to run test: ClonerLifeCycle
2016-02-04T09:33:40.296+0000 I -        [main] 	 going to run test: FirstRemoteCommand
2016-02-04T09:33:40.296+0000 I -        [main] 	 going to run test: RemoteCollectionMissing
2016-02-04T09:33:40.296+0000 I -        [main] 	 going to run test: ListIndexesReturnedNoIndexes
2016-02-04T09:33:40.297+0000 W REPL     [ReplicationExecutor] No indexes found for collection db.coll while cloning from localhost:27017
2016-02-04T09:33:40.297+0000 I -        [main] 	 going to run test: BeginCollectionScheduleDbWorkFailed
2016-02-04T09:33:40.297+0000 I -        [main] 	 going to run test: BeginCollectionCallbackCanceled
2016-02-04T09:33:40.297+0000 I -        [main] 	 going to run test: BeginCollectionFailed
2016-02-04T09:33:40.298+0000 I -        [main] 	 going to run test: BeginCollection
2016-02-04T09:33:40.298+0000 I -        [main] 	 going to run test: FindFetcherScheduleFailed
2016-02-04T09:33:40.298+0000 I -        [main] 	 going to run test: FindCommandAfterBeginCollection
2016-02-04T09:33:40.299+0000 I -        [main] 	 going to run test: FindCommandFailed
2016-02-04T09:33:40.299+0000 I -        [main] 	 going to run test: FindCommandCanceled
2016-02-04T09:33:40.299+0000 I -        [main] 	 going to run test: InsertDocumentsScheduleDbWorkFailed
2016-02-04T09:33:40.300+0000 I -        [main] 	 going to run test: InsertDocumentsCallbackCanceled
2016-02-04T09:33:40.300+0000 I -        [main] 	 going to run test: InsertDocumentsFailed
2016-02-04T09:33:40.300+0000 I -        [main] 	 going to run test: InsertDocumentsSingleBatch
2016-02-04T09:33:40.301+0000 I -        [main] 	 going to run test: InsertDocumentsMultipleBatches
2016-02-04T09:33:40.301+0000 I -        [main] FAIL: InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-02-04T09:33:40.301+0000 I -        [main] 	 DONE running tests
2016-02-04T09:33:40.301+0000 I -        [main] **************************************************
2016-02-04T09:33:40.301+0000 I -        [main] CollectionClonerTest           | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.008
	InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-02-04T09:33:40.301+0000 I -        [main] TOTALS                         | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.008
2016-02-04T09:33:40.301+0000 I -        [main] Failing tests:
2016-02-04T09:33:40.301+0000 I -        [main] 	 CollectionClonerTest/InsertDocumentsMultipleBatches Failed
2016-02-04T09:33:40.301+0000 I -        [main] FAILURE - 1 tests in 1 suites failed



 Comments   
Comment by Jonathan Reams [ 17/Mar/16 ]

Since this was because of a bug in unreleased upstream gcc, I'm resolving this. We can re-open if there are any further issues.

Comment by Marek Skalický [ 17/Mar/16 ]

Thanks for help with solving this bug.

Fortunately I report gcc bug very occasionally so I am not registered in gcc bugzilla to be able to add JIRA link. I hope it is not a problem

I hope you can close this bug. In case of any other problem I will be glad to "consult" it with you in JIRA.

Comment by Andrew Morrow (Inactive) [ 17/Mar/16 ]

Thanks mskalick, sounds like the GCC issue got promptly fixed upstream so hopefully this will no longer be a problem for you, and also not a perplexing surprise for us when we upgrade to GCC 6 in the future. We definitely appreciate the early testing. It might be nice to add a link back to this JIRA ticket to the GCC bug though. Please let us know if there is anything more we can do for you, otherwise we will close this ticket in a day or so.

Comment by Marek Skalický [ 17/Mar/16 ]

Upstream bug - https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70259

Comment by Marek Skalický [ 15/Mar/16 ]

Of course I will post link there.

I have no update from gcc Fedora maintainers, so I am sorry that I have nothing to update here.

I don't know gcc, so I hoped that people packaging it would know more and would report upstream bug if it should be necessary. I asked them if I should report bug, no response yet.

Comment by Jonathan Reams [ 14/Mar/16 ]

mskalick, any word on this from your end?

Comment by Andrew Morrow (Inactive) [ 08/Mar/16 ]

mskalick - If this results in you filing an upstream GCC bug, could you please post the link to the bug in this ticket?

Comment by Jonathan Reams [ 07/Mar/16 ]

Hi mskalick, so I've actually narrowed this down to the gcc optimizer. I've written the following minimal test case to demonstrate:

#include <iostream>
#include <functional>
 
struct testStruct {
void runme() {
        bool second = true;
        auto c = std::bind(&testStruct::testFn, this, std::placeholders::_1, second);
        std::cout << c("foo") << std::endl;
}
 
bool testFn(std::string blah, bool second) {
        std::cout << second << std::endl;
        return (blah=="foo" && second);
}
};
 
int main() {
        testStruct s;
        s.runme();
}

When you run it with default optimization, everything works and the bool parameter to the testFn is printed to std::cout as "1":

[fedora@ip-10-10-1-124 mongo]$ g++ -o testcase -std=c++11 testcase.cpp
[fedora@ip-10-10-1-124 mongo]$ ./testcase
1
1

When you use -O2, it gets passed as 168, and the function "fails"

[fedora@ip-10-10-1-124 mongo]$ g++ -o testcase -std=c++11 -O2 testcase.cpp
[fedora@ip-10-10-1-124 mongo]$ ./testcase
168
0

I've tested that the unittests/integration/jsCore test suites built with scons --opt=off all pass with fedora's gcc 6.

Comment by Marek Skalický [ 07/Mar/16 ]

Jonathan, I am not able to reproduce this "bug" in libstdc++ in different code. Do you have any idea?

Comment by Marek Skalický [ 07/Mar/16 ]

Jonathan Reams, could you help me please to separate this bug in libstdc++ to be able to pass it to gcc maintainers.

A swap of position of lastBatch does not work for me. I am not able to complile it.

Comment by Marek Skalický [ 03/Mar/16 ]

I reported a bug to gcc in Fedora - https://bugzilla.redhat.com/show_bug.cgi?id=1314174

What about other tests? Do you have any idea?
Is it possible that the failures are also caused by this bug?

Comment by Jonathan Reams [ 02/Mar/16 ]

mskalick, I've reproduced this issue. I narrowed the failure down to lastBatch not being correctly passed to _insertDocumentsCallback here: https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/collection_cloner.cpp#L236. When we replaced std::bind with a lambda, this test passes.

Interestingly, if I swap the position of lastBatch, so that it's

    auto&& scheduleResult = _scheduleDbWorkFn(stdx::bind(
        &CollectionCloner::_insertDocumentsCallback, this, lastBatch, stdx::placeholders::_1));

instead of

    auto&& scheduleResult = _scheduleDbWorkFn(stdx::bind(
        &CollectionCloner::_insertDocumentsCallback, this, stdx::placeholders::_1, lastBatch));

this test also passes. This looks to me like a bug in libstdc++ in gcc 6.

Comment by Marek Skalický [ 02/Mar/16 ]

I changed collection_cloner.cpp:

  • added "#include <iostream>
  • added to the beginning of CollectionCloner::_insertDocumentsCallback:

    void CollectionCloner::_insertDocumentsCallback(const ReplicationExecutor::CallbackArgs& cbd,
                                                    bool lastBatch) {
        std::cerr << std::endl << "CollectionCloner::_insertDocumentsCallback lastBatch - " << lastBatch << std::endl << std::endl;
    

and I am getting this weird output:

[mskalick@localhost mongodb-src-r3.2.3]$ build/opt/mongo/db/repl/collection_cloner_test 
2016-03-02T12:25:23.554+0100 I -        [main] going to run suite: CollectionClonerTest
2016-03-02T12:25:23.554+0100 I -        [main] 	 going to run test: InvalidConstruction
2016-03-02T12:25:23.555+0100 I -        [main] 	 going to run test: ClonerLifeCycle
2016-03-02T12:25:23.558+0100 I -        [main] 	 going to run test: FirstRemoteCommand
2016-03-02T12:25:23.558+0100 I -        [main] 	 going to run test: RemoteCollectionMissing
2016-03-02T12:25:23.558+0100 I -        [main] 	 going to run test: ListIndexesReturnedNoIndexes
2016-03-02T12:25:23.558+0100 W REPL     [ReplicationExecutor] No indexes found for collection db.coll while cloning from localhost:27017
2016-03-02T12:25:23.558+0100 I -        [main] 	 going to run test: BeginCollectionScheduleDbWorkFailed
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: BeginCollectionCallbackCanceled
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: BeginCollectionFailed
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: BeginCollection
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: FindFetcherScheduleFailed
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: FindCommandAfterBeginCollection
2016-03-02T12:25:23.559+0100 I -        [main] 	 going to run test: FindCommandFailed
2016-03-02T12:25:23.560+0100 I -        [main] 	 going to run test: FindCommandCanceled
2016-03-02T12:25:23.560+0100 I -        [main] 	 going to run test: InsertDocumentsScheduleDbWorkFailed
2016-03-02T12:25:23.560+0100 I -        [main] 	 going to run test: InsertDocumentsCallbackCanceled
 
CollectionCloner::_insertDocumentsCallback lastBatch - 0
 
2016-03-02T12:25:23.560+0100 I -        [main] 	 going to run test: InsertDocumentsFailed
 
CollectionCloner::_insertDocumentsCallback lastBatch - 105
 
2016-03-02T12:25:23.561+0100 I -        [main] 	 going to run test: InsertDocumentsSingleBatch
 
CollectionCloner::_insertDocumentsCallback lastBatch - 45
 
2016-03-02T12:25:23.561+0100 I -        [main] 	 going to run test: InsertDocumentsMultipleBatches
 
CollectionCloner::_insertDocumentsCallback lastBatch - 111
 
2016-03-02T12:25:23.561+0100 I -        [main] FAIL: InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-03-02T12:25:23.561+0100 I -        [main] 	 DONE running tests
2016-03-02T12:25:23.561+0100 I -        [main] **************************************************
2016-03-02T12:25:23.561+0100 I -        [main] CollectionClonerTest           | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.007
	InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-03-02T12:25:23.561+0100 I -        [main] TOTALS                         | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.007
2016-03-02T12:25:23.561+0100 I -        [main] Failing tests:
2016-03-02T12:25:23.561+0100 I -        [main] 	 CollectionClonerTest/InsertDocumentsMultipleBatches Failed
2016-03-02T12:25:23.561+0100 I -        [main] FAILURE - 1 tests in 1 suites failed
[mskalick@localhost mongodb-src-r3.2.3]$ 

Comment by Marek Skalický [ 02/Mar/16 ]

Do you have a toolchain we could try building with to reproduce this error?

Do you think gcc and binutils? Something else?

It can be downloaded from http://koji.fedoraproject.org/koji/buildinfo?buildID=739365 and http://koji.fedoraproject.org/koji/buildinfo?buildID=739153
This are rpm packages, so you can install it by rpm or extract them to the disk.

If you have any questions to Fedora build environment feel free to contact me...

Comment by Marek Skalický [ 02/Mar/16 ]

Hi, also other unit tests are failing:

[mskalick@localhost mongodb-src-r3.2.3]$ build/opt/mongo/db/repl/collection_cloner_test
2016-03-02T11:00:29.300+0100 I -        [main] going to run suite: CollectionClonerTest
2016-03-02T11:00:29.300+0100 I -        [main] 	 going to run test: InvalidConstruction
2016-03-02T11:00:29.301+0100 I -        [main] 	 going to run test: ClonerLifeCycle
2016-03-02T11:00:29.302+0100 I -        [main] 	 going to run test: FirstRemoteCommand
2016-03-02T11:00:29.302+0100 I -        [main] 	 going to run test: RemoteCollectionMissing
2016-03-02T11:00:29.302+0100 I -        [main] 	 going to run test: ListIndexesReturnedNoIndexes
2016-03-02T11:00:29.302+0100 W REPL     [ReplicationExecutor] No indexes found for collection db.coll while cloning from localhost:27017
2016-03-02T11:00:29.302+0100 I -        [main] 	 going to run test: BeginCollectionScheduleDbWorkFailed
2016-03-02T11:00:29.303+0100 I -        [main] 	 going to run test: BeginCollectionCallbackCanceled
2016-03-02T11:00:29.303+0100 I -        [main] 	 going to run test: BeginCollectionFailed
2016-03-02T11:00:29.303+0100 I -        [main] 	 going to run test: BeginCollection
2016-03-02T11:00:29.303+0100 I -        [main] 	 going to run test: FindFetcherScheduleFailed
2016-03-02T11:00:29.303+0100 I -        [main] 	 going to run test: FindCommandAfterBeginCollection
2016-03-02T11:00:29.304+0100 I -        [main] 	 going to run test: FindCommandFailed
2016-03-02T11:00:29.304+0100 I -        [main] 	 going to run test: FindCommandCanceled
2016-03-02T11:00:29.304+0100 I -        [main] 	 going to run test: InsertDocumentsScheduleDbWorkFailed
2016-03-02T11:00:29.304+0100 I -        [main] 	 going to run test: InsertDocumentsCallbackCanceled
2016-03-02T11:00:29.304+0100 I -        [main] 	 going to run test: InsertDocumentsFailed
2016-03-02T11:00:29.305+0100 I -        [main] 	 going to run test: InsertDocumentsSingleBatch
2016-03-02T11:00:29.305+0100 I -        [main] 	 going to run test: InsertDocumentsMultipleBatches
2016-03-02T11:00:29.305+0100 I -        [main] FAIL: InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-03-02T11:00:29.305+0100 I -        [main] 	 DONE running tests
2016-03-02T11:00:29.305+0100 I -        [main] **************************************************
2016-03-02T11:00:29.305+0100 I -        [main] CollectionClonerTest           | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.005
	InsertDocumentsMultipleBatches	Expected getDetectableErrorStatus() == getStatus() (InternalError Not mutated == OK ) @src/mongo/db/repl/collection_cloner_test.cpp:478
2016-03-02T11:00:29.305+0100 I -        [main] TOTALS                         | tests:   18 | fails:    1 | assert calls:          0 | time secs:  0.005
2016-03-02T11:00:29.305+0100 I -        [main] Failing tests:
2016-03-02T11:00:29.305+0100 I -        [main] 	 CollectionClonerTest/InsertDocumentsMultipleBatches Failed
2016-03-02T11:00:29.305+0100 I -        [main] FAILURE - 1 tests in 1 suites failed
[mskalick@localhost mongodb-src-r3.2.3]$ 
[mskalick@localhost mongodb-src-r3.2.3]$ build/opt/mongo/executor/connection_pool_asio_integration_test
2016-03-02T11:00:37.240+0100 I -        [main] Using test fixture with connection string = localhost:27017
2016-03-02T11:00:37.242+0100 I -        [main] going to run suite: ConnectionPoolASIO
2016-03-02T11:00:37.242+0100 I -        [main] 	 going to run test: TestPing
2016-03-02T11:00:37.244+0100 F -        [thread1] terminate() called. An exception is active; attempting to gather more information
2016-03-02T11:00:37.245+0100 F -        [thread1] A non-standard exception type was thrown
 
 0x703a6b 0x702d9e 0x7fca154671f6 0x7fca15467241 0x7fca15492e94 0x7fca14ca258a 0x7fca149d86ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"303A6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"302D9E"},{"b":"7FCA153D8000","o":"8F1F6"},{"b":"7FCA153D8000","o":"8F241"},{"b":"7FCA153D8000","o":"BAE94"},{"b":"7FCA14C9B000","o":"758A"},{"b":"7FCA148CD000","o":"10B6ED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.5.0-0.rc5.git0.2.fc25.x86_64", "version" : "#1 SMP Thu Feb 25 10:53:02 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "86493CE67F52BB47C7689F2AFA17E2E878CB04FC" }, { "b" : "7FFD1BDB8000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "22F44E8910F384F8C9BBE30FBBEE1930B3ADB4D5" }, { "b" : "7FCA15FCC000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "4DA8A842D7266295105157B0882465D99484BA01" }, { "b" : "7FCA15B6E000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "B25C88DDAAB758FF6176A662BBD96F9876F7AC76" }, { "b" : "7FCA15966000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "AB8B384CC31F9352B8C305129D6CC190DD972648" }, { "b" : "7FCA15762000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7D87A6E478575D06DF58FE4C522BD9AA0388B9AB" }, { "b" : "7FCA153D8000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "059D484908949C4B8289AF390853A5031B23F5C8" }, { "b" : "7FCA150CE000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1FBC11E41AA10EC3316C907E2836B6932BEF5C67" }, { "b" : "7FCA14EB7000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9D431E9E14F2909997BAD221DCCBD3C034DAE6BF" }, { "b" : "7FCA14C9B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "898C5DA8332C99A305102AB69CF090DA716601A6" }, { "b" : "7FCA148CD000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "51BBD61C36C32725119DBD79E946F83FC436ACD4" }, { "b" : "7FCA16246000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "135761436E392CABDFD4732541101CCDE19EA78A" }, { "b" : "7FCA14680000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "93148DEBDA085FB3B03546ACF2760C55A8CAC977" }, { "b" : "7FCA1439A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "C44A1DE9A61C1E54E8C68E5D5A97E46B5708C973" }, { "b" : "7FCA14196000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2A733856F3E29CEF497D5563C59BB1B346E58F48" }, { "b" : "7FCA13F64000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D0DD0953C2A0F9E4665257184A95C8A0B6B60663" }, { "b" : "7FCA13D4E000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "919D7128D3D20B8904E76D7F7A35A4788EEA5E0B" }, { "b" : "7FCA13B3F000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "0DEA73B02E4E59A795472EDE2FCCE04F0F79F19E" }, { "b" : "7FCA1393B000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "83B5DC698C4FBF60FEBE682891F6F423FCB91479" }, { "b" : "7FCA13720000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C6B2F732E6728D65958A4775AB7C95E6995E3895" }, { "b" : "7FCA134F9000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "0E678866DBD0C5C186295B2CFF9230B8DD30E2B2" }, { "b" : "7FCA13284000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "32025BCC44E124E551DE47E31DB5BACBC6775906" } ] }}
 connection_pool_asio_integration_test(_ZN5mongo15printStackTraceERSo+0x2B) [0x703a6b]
 connection_pool_asio_integration_test(+0x302D9E) [0x702d9e]
 libstdc++.so.6(+0x8F1F6) [0x7fca154671f6]
 libstdc++.so.6(+0x8F241) [0x7fca15467241]
 libstdc++.so.6(+0xBAE94) [0x7fca15492e94]
 libpthread.so.0(+0x758A) [0x7fca14ca258a]
 libc.so.6(clone+0x6D) [0x7fca149d86ed]
-----  END BACKTRACE  -----
Neúspěšně ukončen (SIGABRT) (core dumped [obraz paměti uložen])
[mskalick@localhost mongodb-src-r3.2.3]$ 
[mskalick@localhost mongodb-src-r3.2.3]$ build/opt/mongo/executor/network_interface_perf_test
2016-03-02T11:00:49.527+0100 I -        [main] Using test fixture with connection string = localhost:27017
2016-03-02T11:00:49.529+0100 I -        [main] going to run suite: NetworkInterfaceASIO
2016-03-02T11:00:49.529+0100 I -        [main] 	 going to run test: SerialPerf
2016-03-02T11:00:49.530+0100 F ASIO     [NetworkInterfaceASIO-0] Uncaught exception in NetworkInterfaceASIO IO worker thread of type: HostUnreachable: Connection refused
2016-03-02T11:00:49.530+0100 I -        [NetworkInterfaceASIO-0] Fatal Assertion 28820
2016-03-02T11:00:49.530+0100 I -        [NetworkInterfaceASIO-0] 
 
***aborting after fassert() failure
 
 
2016-03-02T11:00:49.532+0100 F -        [NetworkInterfaceASIO-0] Got signal: 6 (Aborted).
 
 0x90c29b 0x90b269 0x90bad0 0x7ff8cae4bc30 0x7ff8caaa30f8 0x7ff8caaa4d0a 0x89ab93 0x84bf8c 0x7ff8cb632e62 0x7ff8cae4258a 0x7ff8cab786ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"50C29B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"50B269"},{"b":"400000","o":"50BAD0"},{"b":"7FF8CAE3B000","o":"10C30"},{"b":"7FF8CAA6D000","o":"360F8","s":"gsignal"},{"b":"7FF8CAA6D000","o":"37D0A","s":"abort"},{"b":"400000","o":"49AB93","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"44BF8C"},{"b":"7FF8CB578000","o":"BAE62"},{"b":"7FF8CAE3B000","o":"758A"},{"b":"7FF8CAA6D000","o":"10B6ED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.5.0-0.rc5.git0.2.fc25.x86_64", "version" : "#1 SMP Thu Feb 25 10:53:02 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0BE77E98872F757B52D785FD47FD00BE565B8A9F" }, { "b" : "7FFD1B7FE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "22F44E8910F384F8C9BBE30FBBEE1930B3ADB4D5" }, { "b" : "7FF8CC16C000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "4DA8A842D7266295105157B0882465D99484BA01" }, { "b" : "7FF8CBD0E000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "B25C88DDAAB758FF6176A662BBD96F9876F7AC76" }, { "b" : "7FF8CBB06000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "AB8B384CC31F9352B8C305129D6CC190DD972648" }, { "b" : "7FF8CB902000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7D87A6E478575D06DF58FE4C522BD9AA0388B9AB" }, { "b" : "7FF8CB578000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "059D484908949C4B8289AF390853A5031B23F5C8" }, { "b" : "7FF8CB26E000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1FBC11E41AA10EC3316C907E2836B6932BEF5C67" }, { "b" : "7FF8CB057000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9D431E9E14F2909997BAD221DCCBD3C034DAE6BF" }, { "b" : "7FF8CAE3B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "898C5DA8332C99A305102AB69CF090DA716601A6" }, { "b" : "7FF8CAA6D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "51BBD61C36C32725119DBD79E946F83FC436ACD4" }, { "b" : "7FF8CC3E6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "135761436E392CABDFD4732541101CCDE19EA78A" }, { "b" : "7FF8CA820000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "93148DEBDA085FB3B03546ACF2760C55A8CAC977" }, { "b" : "7FF8CA53A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "C44A1DE9A61C1E54E8C68E5D5A97E46B5708C973" }, { "b" : "7FF8CA336000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2A733856F3E29CEF497D5563C59BB1B346E58F48" }, { "b" : "7FF8CA104000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D0DD0953C2A0F9E4665257184A95C8A0B6B60663" }, { "b" : "7FF8C9EEE000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "919D7128D3D20B8904E76D7F7A35A4788EEA5E0B" }, { "b" : "7FF8C9CDF000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "0DEA73B02E4E59A795472EDE2FCCE04F0F79F19E" }, { "b" : "7FF8C9ADB000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "83B5DC698C4FBF60FEBE682891F6F423FCB91479" }, { "b" : "7FF8C98C0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C6B2F732E6728D65958A4775AB7C95E6995E3895" }, { "b" : "7FF8C9699000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "0E678866DBD0C5C186295B2CFF9230B8DD30E2B2" }, { "b" : "7FF8C9424000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "32025BCC44E124E551DE47E31DB5BACBC6775906" } ] }}
 network_interface_perf_test(_ZN5mongo15printStackTraceERSo+0x2B) [0x90c29b]
 network_interface_perf_test(+0x50B269) [0x90b269]
 network_interface_perf_test(+0x50BAD0) [0x90bad0]
 libpthread.so.0(+0x10C30) [0x7ff8cae4bc30]
 libc.so.6(gsignal+0x38) [0x7ff8caaa30f8]
 libc.so.6(abort+0x16A) [0x7ff8caaa4d0a]
 network_interface_perf_test(_ZN5mongo13fassertFailedEi+0x93) [0x89ab93]
 network_interface_perf_test(+0x44BF8C) [0x84bf8c]
 libstdc++.so.6(+0xBAE62) [0x7ff8cb632e62]
 libpthread.so.0(+0x758A) [0x7ff8cae4258a]
 libc.so.6(clone+0x6D) [0x7ff8cab786ed]
-----  END BACKTRACE  -----
Neúspěšně ukončen (SIGABRT) (core dumped [obraz paměti uložen])
[mskalick@localhost mongodb-src-r3.2.3]$ 
[mskalick@localhost mongodb-src-r3.2.3]$ build/opt/mongo/executor/network_interface_asio_integration_test
2016-03-02T11:01:43.867+0100 I -        [main] Using test fixture with connection string = localhost:27017
2016-03-02T11:01:43.868+0100 I -        [main] going to run suite: NetworkInterfaceASIOIntegrationTest
2016-03-02T11:01:43.868+0100 I -        [main] 	 going to run test: Ping
2016-03-02T11:01:43.869+0100 I ASIO     [main] command failed: HostUnreachable: Connection refused
2016-03-02T11:01:43.870+0100 I -        [main] FAIL: Ping	Expected ::mongo::Status::OK() == (swt.getStatus()) (OK  == HostUnreachable Connection refused) @src/mongo/unittest/unittest.h:538
2016-03-02T11:01:43.870+0100 I -        [main] 	 going to run test: Timeouts
2016-03-02T11:01:43.870+0100 I ASIO     [main] command failed: HostUnreachable: Connection refused
2016-03-02T11:01:43.870+0100 I -        [main] FAIL: Timeouts	Expected: clientStatus == reason @src/mongo/executor/network_interface_asio_integration_test.cpp:126
2016-03-02T11:01:43.870+0100 I -        [main] 	 going to run test: StressTest
2016-03-02T11:01:43.871+0100 I ASIO     [main] Random seed is -3075814368889834087
2016-03-02T11:01:43.871+0100 I ASIO     [main] Starting stress test...
2016-03-02T11:02:44.546+0100 I ASIO     [main] running ops
2016-03-02T11:02:44.566+0100 I -        [main] FAIL: StressTest	Expected ::mongo::Status::OK() == (res) (OK  == BadValue Expected OK but got HostUnreachable: Connection refused) @src/mongo/executor/network_interface_asio_integration_test.cpp:301
2016-03-02T11:02:44.566+0100 I -        [main] 	 going to run test: HookHangs
2016-03-02T11:02:44.567+0100 I ASIO     [main] command failed: HostUnreachable: Connection refused
2016-03-02T11:02:44.567+0100 I -        [main] FAIL: HookHangs	Expected: clientStatus == reason @src/mongo/executor/network_interface_asio_integration_test.cpp:126
2016-03-02T11:02:44.567+0100 I -        [main] 	 DONE running tests
2016-03-02T11:02:44.567+0100 I -        [main] **************************************************
2016-03-02T11:02:44.567+0100 I -        [main] NetworkInterfaceASIOIntegrationTest | tests:    4 | fails:    4 | assert calls:          0 | time secs: 60.698
	Ping	Expected ::mongo::Status::OK() == (swt.getStatus()) (OK  == HostUnreachable Connection refused) @src/mongo/unittest/unittest.h:538
	Timeouts	Expected: clientStatus == reason @src/mongo/executor/network_interface_asio_integration_test.cpp:126
	StressTest	Expected ::mongo::Status::OK() == (res) (OK  == BadValue Expected OK but got HostUnreachable: Connection refused) @src/mongo/executor/network_interface_asio_integration_test.cpp:301
	HookHangs	Expected: clientStatus == reason @src/mongo/executor/network_interface_asio_integration_test.cpp:126
2016-03-02T11:02:44.567+0100 I -        [main] TOTALS                         | tests:    4 | fails:    4 | assert calls:          0 | time secs: 60.698
2016-03-02T11:02:44.567+0100 I -        [main] Failing tests:
2016-03-02T11:02:44.567+0100 I -        [main] 	 NetworkInterfaceASIOIntegrationTest/Ping Failed
2016-03-02T11:02:44.567+0100 I -        [main] 	 NetworkInterfaceASIOIntegrationTest/Timeouts Failed
2016-03-02T11:02:44.567+0100 I -        [main] 	 NetworkInterfaceASIOIntegrationTest/StressTest Failed
2016-03-02T11:02:44.567+0100 I -        [main] 	 NetworkInterfaceASIOIntegrationTest/HookHangs Failed
2016-03-02T11:02:44.567+0100 I -        [main] FAILURE - 4 tests in 1 suites failed
[mskalick@localhost mongodb-src-r3.2.3]$

Build by: scons all -j2 --nostrip --ssl --disable-warnings-as-errors --wiredtiger=on --experimental-decimal-support=off

Comment by Benety Goh [ 29/Feb/16 ]

mskalick, if this is happening consistently for you, can you try putting in some debug output in CollectionCloner::_insertDocumentsCallback?

I'm interested to see how many times this function is called and with what value for the lastBatch argument before the test hits the assertion in line 478.

Comment by Jonathan Reams [ 29/Feb/16 ]

Hi Marek, we're trying to reproduce this on our end. It is very possible this failure is due to building with gcc 6. Do you have a toolchain we could try building with to reproduce this error? Also, are any other unit tests failing or is it just this one?

Comment by Marek Skalický [ 29/Feb/16 ]

Even with bundled versions of libraries this test is failing for me.

Comment by Marek Skalický [ 25/Feb/16 ]

Any progress.

Why should the inserting result in detectableError?

ASSERT_EQUALS(getDetectableErrorStatus(), getStatus());

There - https://github.com/mongodb/mongo/blob/v3.2/src/mongo/db/repl/collection_cloner_test.cpp#L478

Is there any documentation for this please?

Generated at Thu Feb 08 04:01:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.