[SERVER-7028] ReplSetConfig constructor using DBDirectClient can trigger page faults Created: 13/Sep/12  Updated: 11/Jul/16  Resolved: 21/Sep/12

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.3.0

Type: Bug Priority: Major - P3
Reporter: Tad Marshall Assignee: Aaron Staple
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

The ReplSetConfig constructor can perform database reads using a DBDirectClient:

                // on startup, socket is not listening yet
                DBDirectClient cli;
                cfg = cli.findOne( rsConfigNs, Query() ).getOwned();
                count = cli.count(rsConfigNs);

This can trigger a PageFaultException (potentially from a simulated page fault in a debug build), which can trigger an abort in a debug build where constructor stack auditing on page faults is enabled.

– Aaron

------------------------------------------------------------

Buildbot test on Linux 64-bit debug dur off failed in jstests/replsets/fastsync.js:

http://buildlogs.mongodb.org/Linux%2064-bit%20debug%20dur%20off/builds/1616/test/fastsync.js
http://buildbot.mongodb.org/builders/Linux%2064-bit%20debug%20dur%20off/builds/1616/steps/test_8/logs/stdio

Wed Sep 12 16:54:58 shell: started program /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod --port 31002 --dbpath /data/db/jstests_fastsync-s2 --nohttpinterface --noprealloc --smallfiles --replSet jstests_fastsync --fastsync --oplogSize 2
m31002| note: noprealloc may hurt performance in many applications
m31002| Wed Sep 12 16:54:58 [initandlisten] MongoDB starting : pid=29021 port=31002 dbpath=/data/db/jstests_fastsync-s2 64-bit host=tp1.10gen.cc
m31002| Wed Sep 12 16:54:58 [initandlisten] _DEBUG build (which is slower)
m31002| Wed Sep 12 16:54:58 [initandlisten] 
m31002| Wed Sep 12 16:54:58 [initandlisten] ** NOTE: This is a development version (2.3.0-pre-) of MongoDB.
m31002| Wed Sep 12 16:54:58 [initandlisten] ** Not recommended for production.
m31002| Wed Sep 12 16:54:58 [initandlisten] 
m31002| Wed Sep 12 16:54:58 [initandlisten] db version v2.3.0-pre-, pdfile version 4.5
m31002| Wed Sep 12 16:54:58 [initandlisten] git version: 291b2e263b2ea3d578f183da7a9187c88d498d98
m31002| Wed Sep 12 16:54:58 [initandlisten] build info: Linux tp1.10gen.cc 2.6.31-14-server #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 x86_64 BOOST_LIB_VERSION=1_49
m31002| Wed Sep 12 16:54:58 [initandlisten] options: { dbpath: "/data/db/jstests_fastsync-s2", fastsync: true, nohttpinterface: true, noprealloc: true, oplogSize: 2, port: 31002, replSet: "jstests_fastsync", smallfiles: true }
m31002| Wed Sep 12 16:54:58 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/data/db/jstests_fastsync-s2/journal"
m31002| Wed Sep 12 16:54:58 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
m31002| Wed Sep 12 16:54:58 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open local.system.namespaces
m31002| Wed Sep 12 16:54:58 [initandlisten] opening db: local
m31002| Wed Sep 12 16:54:58 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.namespaces
m31002| Wed Sep 12 16:54:58 [initandlisten] opening db: admin
m31002| Wed Sep 12 16:54:58 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open foo.system.namespaces
m31002| Wed Sep 12 16:54:58 [initandlisten] opening db: foo
m31002| Wed Sep 12 16:54:58 [initandlisten] waiting for connections on port 31002
m31002| Wed Sep 12 16:54:58 [initandlisten] connection accepted from 174.121.165.194:46575 #1 (1 connection now open)
m31002| Wed Sep 12 16:54:58 [conn1] _DEBUG ReadContext db wasn't open, will try to open local.system.replset
m31002| Wed Sep 12 16:54:58 [conn1] opening db: local
m31002| Wed Sep 12 16:54:58 [rsStart] replSet load config ok from self
m31002| Wed Sep 12 16:54:58 [rsStart] replSet REMOVED
m31002| Wed Sep 12 16:54:58 [rsStart] replSet info self not present in the repl set configuration:
m31002| Wed Sep 12 16:54:58 [rsStart] { _id: "jstests_fastsync", version: 2, members: [ { _id: 0, host: "tp1.10gen.cc:31000", priority: 2.0 }, { _id: 1, host: "tp1.10gen.cc:31001" } ] }
m31002| Wed Sep 12 16:54:58 [initandlisten] connection accepted from 127.0.0.1:49570 #2 (2 connections now open)
m31000| Wed Sep 12 16:54:59 [conn2] replSet replSetReconfig config object parses ok, 3 members specified
m31002| found a constructor in the call tree: mongo::ReplSetConfig::ReplSetConfig(mongo::HostAndPort const&)
m31002| _ZN5mongo13ReplSetConfigC1ERKNS_11HostAndPortE
m31002| Wed Sep 12 16:54:59 [rsStart] local.system.replset Fatal Assertion 16236
m31002| 0xd24f97 0xd02718 0xcf0d44 0x974ec3 0xbb6042 0x8deae8 0x8df1f9 0x9ecd87 0xb0b12f 0xb0b3a0 0xb0c694 0xb0d6a3 0xb0fb8b 0xa91751 0xa927e7 0xa95b69 0x926cd3 0x8fe2d9 0xa95d54 0x8fca6f 
m31002| Wed Sep 12 16:54:59 [initandlisten] connection accepted from 174.121.165.194:46577 #3 (3 connections now open)
m31000| Wed Sep 12 16:54:59 [conn2] replSet replSetReconfig [2]
m31000| Wed Sep 12 16:54:59 [conn2] replSet info saving a newer config version to local.system.replset
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0xd24f97]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo10logContextEPKc+0x5e) [0xd02718]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo13fassertFailedEi+0x65) [0xcf0d44]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo7fassertEib+0x25) [0x974ec3]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo6Record10_accessingEv+0x11c) [0xbb6042]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo6Record4dataEv+0x18) [0x8deae8]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo7BSONObj4makeEPKNS_6RecordE+0x21) [0x8df1f9]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo11BasicCursor7currentEv+0x3d) [0x9ecd87]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo21ResponseBuildStrategy7currentEb+0x229) [0xb0b12f]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo20OrderedBuildStrategy11handleMatchERbRNS_12MatchDetailsE+0x114) [0xb0b3a0]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0xd6) [0xb0c694]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_17ShardChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE+0x49d) [0xb0d6a3]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x113b) [0xb0fb8b]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod [0xa91751]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x33b) [0xa927e7]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x91) [0xa95b69]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo14DBClientCursor4initEv+0xdd) [0x926cd3]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xe9) [0x8fe2d9]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo14DBDirectClient5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x6e) [0xa95d54]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo17DBClientInterface5findNERSt6vectorINS_7BSONObjESaIS2_EERKSsNS_5QueryEiiPKS2_i+0xcb) [0x8fca6f]
m31002| Wed Sep 12 16:54:59 [conn1] end connection 174.121.165.194:46575 (2 connections now open)
m31002| Wed Sep 12 16:54:59 [rsStart] 
m31002| 
m31002| ***aborting after fassert() failure
m31002| 
m31002| 
m31002| Wed Sep 12 16:54:59 Got signal: 6 (Aborted).
m31002| 
m31002| Wed Sep 12 16:54:59 [conn1] thread conn1 stack usage was 7456 bytes, which is the most so far
m31002| Wed Sep 12 16:54:59 Backtrace:
m31002| 0xd24f97 0x8b9d82 0x7fb7884e2530 0x7fb7884e24b5 0x7fb7884e5f50 0xcf0d85 0x974ec3 0xbb6042 0x8deae8 0x8df1f9 0x9ecd87 0xb0b12f 0xb0b3a0 0xb0c694 0xb0d6a3 0xb0fb8b 0xa91751 0xa927e7 0xa95b69 0x926cd3 
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0xd24f97]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo10abruptQuitEi+0x130) [0x8b9d82]
m31002| /lib/libc.so.6 [0x7fb7884e2530]
m31002| /lib/libc.so.6(gsignal+0x35) [0x7fb7884e24b5]
m31002| /lib/libc.so.6(abort+0x180) [0x7fb7884e5f50]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo9uassertedEiRKSs+0) [0xcf0d85]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo7fassertEib+0x25) [0x974ec3]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo6Record10_accessingEv+0x11c) [0xbb6042]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo6Record4dataEv+0x18) [0x8deae8]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo7BSONObj4makeEPKNS_6RecordE+0x21) [0x8df1f9]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo11BasicCursor7currentEv+0x3d) [0x9ecd87]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZNK5mongo21ResponseBuildStrategy7currentEb+0x229) [0xb0b12f]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo20OrderedBuildStrategy11handleMatchERbRNS_12MatchDetailsE+0x114) [0xb0b3a0]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0xd6) [0xb0c694]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_17ShardChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE+0x49d) [0xb0d6a3]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x113b) [0xb0fb8b]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod [0xa91751]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x33b) [0xa927e7]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x91) [0xa95b69]
m31002| /home/yellow/buildslave/Linux_64bit_debug_dur_off/mongo/mongod(_ZN5mongo14DBClientCursor4initEv+0xdd) [0x926cd3]
m31002| 



 Comments   
Comment by auto [ 20/Sep/12 ]

Author:

{u'date': u'2012-09-20T14:55:51-07:00', u'email': u'aaron@10gen.com', u'name': u'Aaron'}

Message: SERVER-7028 compile linux, in owned_pointer_vector_test define DestructionLogger::_destroyedName.
Branch: master
https://github.com/mongodb/mongo/commit/7e0408a17daf0b5c09a0b2d315752a958c53eda3

Comment by auto [ 20/Sep/12 ]

Author:

{u'date': u'2012-09-20T11:29:42-07:00', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7028 Move complex initialization (that may trigger page fault exceptions) out of ReplSetConfig and ReplSet constructors.
Branch: master
https://github.com/mongodb/mongo/commit/fda584530032dd7792789b1db6c1d88ab29c2f75

Comment by auto [ 20/Sep/12 ]

Author:

{u'date': u'2012-09-20T11:28:50-07:00', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7028 Add OwnedPointerVector.
Branch: master
https://github.com/mongodb/mongo/commit/b659b3031e5ced047b490641dc9793620394b880

Generated at Thu Feb 08 03:13:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.