[CDRIVER-2828] Failed tests Created: 18/Sep/18  Updated: 25/Oct/18  Resolved: 25/Oct/18

Status: Closed
Project: C Driver
Component/s: libmongoc
Affects Version/s: 1.13.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Remi Collet Assignee: A. Jesse Jiryu Davis
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Fedora with mongodb 4.0.1



 Description   

See https://apps.fedoraproject.org/koschei/package/mongo-c-driver

 

error: Failed to receive length header from server.
FAIL

Assert Failure: 3 == 2
/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/test-mongoc-async.c:73 test_ismaster_helper()
{ "status": "FAIL", "test_file": "/Async/ismaster_ssl", "seed": "4075364019", "start": 1927.665743, "end": 1927.717326, "elapsed": 0.051583 },

 

/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/mock_server/mock-server.c:1918 _mock_server_reply_with_stream(): precondition failed: n_written == expected
{ "status": "FAIL", "test_file": "/Client/ssl_opts/copies_single", "seed": "1348030844", "start": 1969.900057, "end": 1969.932482, "elapsed": 0.032425 },

 

/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/mock_server/mock-server.c:1918 _mock_server_reply_with_stream(): precondition failed: n_written == expected
{ "status": "FAIL", "test_file": "/Client/ssl_opts/copies_pooled", "seed": "3111671023", "start": 1969.932542, "end": 1970.476470, "elapsed": 0.543928 },

 

/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/test-mongoc-client.c:2793 _cmd(): precondition failed: request
{ "status": "FAIL", "test_file": "/Client/ssl/reconnect/single", "seed": "4131135309", "start": 1970.476539, "end": 1980.511182, "elapsed": 10.034643 },

 

/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/mock_server/mock-server.c:1918 _mock_server_reply_with_stream(): precondition failed: n_written == expected
{ "status": "FAIL", "test_file": "/Client/ssl/reconnect/pooled", "seed": "2546724188", "start": 1980.511246, "end": 1980.550429, "elapsed": 0.039183 },

 

scanner error: Failed to receive length header from server.
{ "status": "FAIL", "test_file": "/TOPOLOGY/scanner_ssl", "seed": "3927626548", "start": 2120.362617, "end": 2120.406044, "elapsed": 0.043427 },

 

 



 Comments   
Comment by Kevin Albertson [ 25/Oct/18 ]

Hi Remi - I'm closing this for now. Feel free to re-open if you need help investigating.

Comment by A. Jesse Jiryu Davis [ 02/Oct/18 ]

OK, I hope that someone who does have access to the ARM builders and the server logs can assist with debugging. It seems clear to me that MongoDB is crashing or shutting down on Intel, and you'll need to read the logs to understand why.

On ARM, I don't have a good hypothesis for the error "Storage engine does not support read concern". You could try:

  • Make sure MongoDB is really version 4.0.1 there
  • Make sure MongoDB isn't started with a --storageEngine command line option. If it's started with "–storageEngine mmapv1" or "–storageEngine inMemory", that will produce the "does not support read concern" error.
Comment by Remi Collet [ 02/Oct/18 ]

Sorry, but I don't have direct access to an ARM builder... I can only run build in Fedora infrastructure, without any access to the server, only to the build.log

Comment by A. Jesse Jiryu Davis [ 01/Oct/18 ]

Hi Remi - let me know when you have the opportunity to investigate this. The next two steps are:

1. Could you please try starting mongod on ARM and running the mongo shell command I shared, above?
2. Could you look at the mongod.log files and determine whether MongoDB crashed during the tests on Intel?

Comment by A. Jesse Jiryu Davis [ 19/Sep/18 ]

These tests pass for me with a standalone MongoDB 4.0.1, I wonder what's different about Fedora's x86 and ARM builder environment?

I can't reproduce the "/change_stream/live/single_server" failure. I start a MongoDB 4.0.1 server with the following options:

mongod --journal --ipv6 --unixSocketPrefix /tmp --logpath ~/data/server.log --pidfilepath ~/data/server.pid --dbpath ~/data --fork

Then I run the following command in the "mongo" shell and it fails with the expected error:

> db.runCommand({aggregate: 'collection', pipeline:[{$changeStream: {}}], cursor: {}})
{
	"ok" : 0,
	"errmsg" : "The $changeStream stage is only supported on replica sets",
	"code" : 40573,
	"codeName" : "Location40573"
}

That's the error that the "/change_stream/live/single_server" test expects, so that test passes when I run it.

Could you please try starting mongod on ARM and running that mongo shell command and see if it returns the correct error or not? I'm concerned that on ARM either mongod is out of date or it uses a different storage engine.

For the rest of the errors, it certainly appears the MongoDB crashed, the cause should be in server.log.

Comment by Remi Collet [ 19/Sep/18 ]

> What version of MongoDB please?

=> https://apps.fedoraproject.org/packages/mongodb

 

So version 4.0.1

 

Comment by A. Jesse Jiryu Davis [ 19/Sep/18 ]

Great. What version of MongoDB please? Are the MongoDB versions the same on all architectures?

On the i686 build, the tests start failing here:

1:     { "status": "PASS", "test_file": "/BulkOperation/OP_MSG/max_batch_size", "seed": "3285696675", "start": 2129.907142, "end": 2132.309746, "elapsed": 2.402604  },
1: FAIL:/builddir/build/BUILD/mongo-c-driver-1.13.0/src/libmongoc/tests/test-mongoc-bulk.c:3599  test_bulk_max_msg_size()
1:   retval
1:   Failed to send "insert" command with database "test": Failed to read 4 bytes: socket error or timeout
1: 
1:     { "status": "FAIL", "test_file": "/BulkOperation/OP_MSG/max_msg_size", "seed": "4177024666", "start": 2132.309794, "end": 2135.222353, "elapsed": 2.912559  },

It appears to me that the server crashed or shut down between those two tests. It's available for the max_batch_size test, but it's down when the max_msg_size test starts. Do you have access to /builddir/build/BUILD/mongo-c-driver-1.13.0/server.log?

On the ARM test, two things go wrong. First, you need to pass --enableMajorityReadConcern to MongoDB. I think that will fix this error in /change_stream/live/single_server:

"Storage engine does not support read concern: { readConcern: { level: \"majority\" } }

Second, it appears that this MongoDB server also crashes, perhaps as a result of the driver trying to run /BulkOperation/upsert_huge. The server log might provide answers here, too.

Comment by Remi Collet [ 19/Sep/18 ]

Yes, a server run for the test suite

 

From the log:

 

+ : Run a server
+ mkdir dbtest
+ mongod --journal --ipv6 --unixSocketPrefix /tmp --logpath /builddir/build/BUILD/mongo-c-driver-1.13.0/server.log --pidfilepath /builddir/build/BUILD/mongo-c-driver-1.13.0/server.pid --dbpath /builddir/build/BUILD/mongo-c-driver-1.13.0/dbtest --fork
2018-09-19T06:56:54.169+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
about to fork child process, waiting until server is ready for connections.
forked process: 4137
child process started successfully, parent exiting

 

Comment by A. Jesse Jiryu Davis [ 19/Sep/18 ]

Thanks. Is a MongoDB server running on localhost:27017 for all these tests? What version is MongoDB, and is it running standalone or a replica set?

Comment by Remi Collet [ 19/Sep/18 ]

Second scratch build, test suite enabled on all arches with MONGOC_TEST_SKIP_MOCK=on

 

https://koji.fedoraproject.org/koji/taskinfo?taskID=29760389

 

  • passes on all 64-bit arches
  • fails on 32-bit (i686 and armv7hl)

 

Comment by Remi Collet [ 19/Sep/18 ]

First scratch build, full test suite enabled on all arches

 

https://koji.fedoraproject.org/koji/taskinfo?taskID=29760303

 

  • fails everywhere
Comment by Remi Collet [ 19/Sep/18 ]

> Can you help me understand why the package is built with "make check" for x86_64 and with "make" for other platforms?

Because at some time in the past, I have strange failure on other arch, else tests were enabled on all 64-bit arch

> The tests that fail use our mock server.

They pass in other version, seems something specific to Fedora 29+.

I will run a scratch build.

 

Comment by A. Jesse Jiryu Davis [ 18/Sep/18 ]

Thanks Remi. Can you help me understand why the package is built with "make check" for x86_64 and with "make" for other platforms?

The tests that fail use our mock server. The mock server tries to open a port on localhost so that the driver can connect to it; the mock server and the driver communicate as part of these tests. Is there a reason that these tests wouldn't work in the Fedora test system? You can disable mock server tests by setting "MONGOC_TEST_SKIP_MOCK=on" in the environment.

Generated at Wed Feb 07 21:16:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.