[SERVER-16567] extend data directory metadata to hold storage engine creation settings for startup validation Created: 16/Dec/14  Updated: 03/Mar/15  Resolved: 09/Jan/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.8.0-rc2
Fix Version/s: 2.8.0-rc5

Type: Improvement Priority: Major - P3
Reporter: Asya Kamsky Assignee: Benety Goh
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-15986 Starting with different storage engin... Closed
Related
related to SERVER-16785 Cannot toggle directoryperdb on mmapv1 Closed
is related to SERVER-965 Store the indexes of a collection on ... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Extend data directory metadata (introduced in SERVER-15986) to hold creation settings (derived from startup options). These creation settings will be validated by the storage engine on restart.

On startup, the storage engine will check the current configuration/startup options against the creation settings stored in the metadata. If an inconsistency is detected, the server will fail to start up.

---------------

In attempt to emulate something a user might inadvertently do, I started mongod with directoryForIndexes and added an index to an existing collection.

1) mongod now had some indexes in top level data directory and some in "index" subdirectory of data directory

I now emulated a partial backup by bringing down mongod, and removing "index" directory - this is what would happen if index was mounted on another physical device and a snapshot of just data directory was taken (leaving all or some indexes behind). I now brought up mongod and ran db.collection.getIndexes and db.collection.find( <query-using-new-index>).explain() and same find with explain(true).

The first two commands succeeded with correct output, but actually forcing the running of the indexed query crashed the server with:

2014-12-16T13:40:33.565-0500 E STORAGE  [conn1] WiredTiger (2) [1418755233:565872][29294:0x111c67000], file:index/1--3930778720722776466.wt, session.open_cursor: /data/wt/index/1--3930778720722776466.wt: No such file or directory
2014-12-16T13:40:33.566-0500 E STORAGE  [conn1] no cursor for uri: table:index/1--3930778720722776466
2014-12-16T13:40:33.567-0500 F -        [conn1] Invalid access at address: 0x30
2014-12-16T13:40:33.599-0500 F -        [conn1] Got signal: 11 (Segmentation fault: 11).
 
 0x10ec99839 0x10ec993fe 0x7fff8e7fe5aa 0xc0 0x10eac8810 0x10e80daf4 0x10e7839b8 0x10e783d1c 0x10e77575e 0x10e94fcc7 0x10e9503b8 0x10e91939d 0x10e6cc9b9 0x10e6c5268 0x10e7444fa 0x10e74561e 0x10e74626c 0x10e91eaf1 0x10e8202b9 0x10e574d31 0x10ec5d376 0x10eccd551 0x7fff8ef59899 0x7fff8ef5972a 0x7fff8ef5dfc9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"10E56A000","o":"72F839"},{"b":"10E56A000","o":"72F3FE"},{"b":"7FFF8E7FB000","o":"35AA"},{"b":"0","o":"C0"},{"b":"10E56A000","o":"55E810"},{"b":"10E56A000","o":"2A3AF4"},{"b":"10E56A000","o":"2199B8"},{"b":"10E56A000","o":"219D1C"},{"b":"10E56A000","o":"20B75E"},{"b":"10E56A000","o":"3E5CC7"},{"b":"10E56A000","o":"3E63B8"},{"b":"10E56A000","o":"3AF39D"},{"b":"10E56A000","o":"1629B9"},{"b":"10E56A000","o":"15B268"},{"b":"10E56A000","o":"1DA4FA"},{"b":"10E56A000","o":"1DB61E"},{"b":"10E56A000","o":"1DC26C"},{"b":"10E56A000","o":"3B4AF1"},{"b":"10E56A000","o":"2B62B9"},{"b":"10E56A000","o":"AD31"},{"b":"10E56A000","o":"6F3376"},{"b":"10E56A000","o":"763551"},{"b":"7FFF8EF58000","o":"1899"},{"b":"7FFF8EF58000","o":"172A"},{"b":"7FFF8EF58000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc2", "gitVersion" : "f77869e01a6ed235695a46fad441c0dff15202f6", "uname" : { "sysname" : "Darwin", "release" : "13.4.0", "version" : "Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/asya/bins/mongodb-osx-x86_64-2.8.0-rc2/bin/mongod", "machType" : 2, "b" : "10E56A000", "buildId" : "5E5E1684499B308383920314F3A9D5EA" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF89164000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8FD8B000", "buildId" : "2CB96D101E3E39B1B90DC56C3818321E" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF933AC000", "buildId" : "8C1EFC4F3F513DE9A973360B461F3D65" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF890C2000", "buildId" : "79B8E80FE59633028243EC479B9546CA" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF88E61000", "buildId" : "2A8900040A783E339AC5A7D63BD4F879" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF8B211000", "buildId" : "CF29DFF605893590834C82E2316612E8" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF890F5000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF95CD7000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF890CD000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8AF23000", "buildId" : "D7045F25BFA43840847DAF95DF8B6CCA" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8F941000", "buildId" : "A40A0C7B321639B48AE0B5D3BAF1DA8A" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8D7A5000", "buildId" : "B2BE3C25CF1F309BAB991F0B54621445" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF88E5E000", "buildId" : "973BE51D6465392F9099D4AB21BF0D25" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8D29E000", "buildId" : "26266E3FFDDC3CFCB27F78B49BDC9BDC" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF90FA1000", "buildId" : "B983CA60F418317BB14248A6376564FC" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8E302000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8B094000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF890BF000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF949CE000", "buildId" : "17B03FFD92C532829981EBB28B456207" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8F010000", "buildId" : "395D8CD6616A3BD3A195C6D68EB9AB22" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8A192000", "buildId" : "9EDE872E2A9E3A788E1DAB790794A098" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8CF24000", "buildId" : "9D96AACD58703A5CB8497F9C0F6EBB09" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF8B13A000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF93D81000", "buildId" : "A499D688916537768C8EC018897B5B13" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF90247000", "buildId" : "9B34B4FEF5AD3F09A5F046AFF3571323" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF8E7FB000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8EF58000", "buildId" : "9DA50FD0D9AC3051AD4BBA0D745BC49C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF942DC000", "buildId" : "0D0B13EA6B7A3AC8BE60B548543BEB77" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF88B12000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF90856000", "buildId" : "C3737C9AC06F310CB78C7D8D882A35DE" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF8C25C000", "buildId" : "95D4D11833683474989D271DE18C8365" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8E26F000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8F0EE000", "buildId" : "F21C57427B9C31F1BBAE1717BC6C2F1B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8EFC0000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8FDEF000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8CAD0000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10ec99839]
 mongod(_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv+0x11E) [0x10ec993fe]
 libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff8e7fe5aa]
 ??? [0xc0]
 mongod(_ZN5mongo15WiredTigerIndex11IndexCursor9advanceToERKNS_7BSONObjEibRKNSt3__16vectorIPKNS_11BSONElementENS5_9allocatorIS9_EEEERKNS6_IbNSA_IbEEEE+0x60) [0x10eac8810]
 mongod(_ZN5mongo16BtreeIndexCursor4seekERKNSt3__16vectorIPKNS_11BSONElementENS1_9allocatorIS5_EEEERKNS2_IbNS6_IbEEEE+0x44) [0x10e80daf4]
 mongod(_ZN5mongo9IndexScan13initIndexScanEv+0x338) [0x10e7839b8]
 mongod(_ZN5mongo9IndexScan4workEPm+0x4C) [0x10e783d1c]
 mongod(_ZN5mongo10FetchStage4workEPm+0x10E) [0x10e77575e]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x97) [0x10e94fcc7]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x58) [0x10e9503b8]
 mongod(_ZN5mongo7Explain13explainStagesEPNS_12PlanExecutorENS_13ExplainCommon9VerbosityEPNS_14BSONObjBuilderE+0x6D) [0x10e91939d]
 mongod(_ZNK5mongo7FindCmd7explainEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEERKNS_7BSONObjENS_13ExplainCommon9VerbosityEPNS_14BSONObjBuilderE+0x2A9) [0x10e6cc9b9]
 mongod(_ZN5mongo10CmdExplain3runEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderEb+0x248) [0x10e6c5268]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_7BSONObjEiRSA_RNS_14BSONObjBuilderEb+0x3A) [0x10e7444fa]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xBDE) [0x10e74561e]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x47C) [0x10e74626c]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_b+0xF61) [0x10e91eaf1]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0x9D9) [0x10e8202b9]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE1) [0x10e574d31]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x526) [0x10ec5d376]
 mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x10eccd551]
 libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8ef59899]
 libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8ef5972a]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff8ef5dfc9]
-----  END BACKTRACE  -----

So it would be nice to make sure that on startup _mdb metadata matches expected files? Also not sure that allowing a mix with in subdirectory and not in subdirectory indexes is a good idea (same problem would probably happen if I mistakenly start mongod without directoryForIndexes flag - it would be like index files disappeared while being in metadata).



 Comments   
Comment by Githook User [ 09/Jan/15 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-16567 removed StorageEngineMetadata::updateIfMissing
Branch: master
https://github.com/mongodb/mongo/commit/94fafc43f893854a17d0a7c711e3c90cefce4660

Comment by Githook User [ 09/Jan/15 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-16567 check startup options against creation settings in data directory metadata
Branch: master
https://github.com/mongodb/mongo/commit/95042383af1e5fa46bb79b7ed70e71eed4c8ae67

Comment by Githook User [ 09/Jan/15 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-16567 added StorageEngineMetadata support for optional storage engine BSON document
Branch: master
https://github.com/mongodb/mongo/commit/354fcc2846cdef3666ed8aab7247193e2f665317

Comment by Matt Kangas [ 19/Dec/14 ]

I concur. When --wiredTigerDirectoryForIndexes is used, we should verify that the subdirectories still exist and the tables _mdb_metadata.wt expects are still in them. The feature exists so users can put indexes on a separate partition, and failure to mount the correct partition will no doubt happen in the field.

Also, when a user initializes a dbpath with --wiredTigerDirectoryForIndexes, we should require the same option for future uses of that dbpath. We can extend SERVER-15986 to support this.

Generated at Thu Feb 08 03:41:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.