[SERVER-14198] Std::set<pointer> and Windows Heap Allocation Reuse produces non-deterministic results Created: 07/Jun/14  Updated: 02/Aug/18  Resolved: 09/Jun/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.7.1
Fix Version/s: 2.6.4, 2.7.2

Type: Task Priority: Major - P3
Reporter: Mark Benvenuto Assignee: Mark Benvenuto
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-13729 Reads & Writes are blocked during dat... Closed
Backwards Compatibility: Fully Compatible
Backport Completed:
Sprint: Server 2.7.2
Participants:

 Description   

std::set<pointer> based types are unreliable if the memory addresses are not unique, but could be reused during the lifetime of the process. This also applies to other types such as like std::map.

In the dump below from Visual Studio 2013 of the mmfiles std::set, we see element 6 contains a paradox.

-		thingsToFlushWrapper	{_vector={ size=7 } }	mongo::OwnedPointerVector<mongo::MongoFile::Flushable>
-		_vector	{ size=7 }	std::vector<mongo::MongoFile::Flushable *,std::allocator<mongo::MongoFile::Flushable *> >
		[size]	7	__int64
		[capacity]	9	__int64
+		[0]	0x000000d48e6802c0 {_theFile=0x000000d48a5a5500 {_view_write=0x00000057d6000000 _view_private=0x00000057da000000 ...} ...}	mongo::MongoFile::Flushable *
+		[1]	0x000000d48e680470 {_theFile=0x000000d48a5a65e0 {fd=0xdddddddddddddddd maphandle=0xdddddddddddddddd ...} ...}	mongo::MongoFile::Flushable *
+		[2]	0x000000d48e680a10 {_theFile=0x000000d48a5a6ca0 {_view_write=0x0000004014000000 _view_private=0x0000004018000000 ...} ...}	mongo::MongoFile::Flushable *
+		[3]	0x000000d48e681700 {_theFile=0x000000d48a642288 {_view_write=0x0000000000000000 _view_private=0x0000000000000000 ...} ...}	mongo::MongoFile::Flushable *
+		[4]	0x000000d48e680740 {_theFile=0x000000d48a643d88 {_view_write=0x000000401c000000 _view_private=0x000000401d000000 ...} ...}	mongo::MongoFile::Flushable *
+		[5]	0x000000d48e686650 {_theFile=0x000000d48aa07ce8 {_view_write=0x00000057d4000000 _view_private=0x00000057d5000000 ...} ...}	mongo::MongoFile::Flushable *
-		[6]	0x000000d48e684a30 {_theFile=0x000000d48e2f2a38 {_view_write=0x00000057de000000 _view_private=0x00000057df000000 ...} ...}	mongo::MongoFile::Flushable *
-		[mongo::WindowsFlushable]	{_theFile=0x000000d48e2f2a38 {_view_write=0x00000057de000000 _view_private=0x00000057df000000 _willNeedRemap=...} ...}	mongo::WindowsFlushable
+		mongo::MongoFile::Flushable	{...}	mongo::MongoFile::Flushable
-		_theFile	0x000000d48e2f2a38 {_view_write=0x00000057de000000 _view_private=0x00000057df000000 _willNeedRemap=true ...}	mongo::MemoryMappedFile *
+		[mongo::DurableMappedFile]	{_view_write=0x00000057de000000 _view_private=0x00000057df000000 _willNeedRemap=true ...}	mongo::DurableMappedFile
+		mongo::MongoFile	{_filename="D:/data/db/foob.ns" }	mongo::MongoFile
		fd	0x0000000000000370	void *
		maphandle	0x00000000000002c0	void *
-		views	{ size=2 }	std::vector<void *,std::allocator<void *> >
		[size]	2	__int64
		[capacity]	2	__int64
		[0]	0x00000057de000000	void *
		[1]	0x00000057df000000	void *
+		[Raw View]	0x000000d48e2f2a78 {...}	std::vector<void *,std::allocator<void *> > *
		len	16777216	unsigned __int64
		_uniqueId	0	const unsigned __int64
+		_flushMutex	{...}	boost::mutex
		_view	0x00000057ca000000	void *
		_fd	0x0000000000000270	void *
+		_filename	"D:/data/db/foob.ns"	std::basic_string<char,std::char_traits<char>,std::allocator<char> >
+		_flushMutex	{...}	boost::mutex &
+		__vfptr	0x00007ff68bced080 {mongod.exe!const mongo::WindowsFlushable::`vftable'} {0x00007ff689dc421d {mongod.exe!mongo::WindowsFlushable::`vector deleting destructor'(unsigned int)}, ...}	void * *

The inner class of type mongo::MemoryMappedFile contains a fd of 0x370 while the outer class of mongo::WindowsFlushable contains a fd of 0x270.

We assume the following invariant will hold:
mongo::WindowsFlushable::_fd == mongo::MemoryMappedFile::fd

This is logical since mongo::WindowsFlushable::_fd is simply a copy of mongo::MemoryMappedFile::fd.

But this invariant will fail to hold under the following scenario.
1. mongo::WindowsFlushable::_fd <= mongo::MemoryMappedFile::fd
2. mongo::MemoryMappedFile is deleted
3. a new mongo::MemoryMappedFile is allocated at the same exact location
4. mongo::MemoryMappedFile::fd now is set to a new value
5. Crash since
mmap_win.cpp contains the following check

                if ( MongoFile::getAllFiles().count(_theFile) == 0 ) {
                    // this was deleted while we were unlocked
                    return;
                }

which will incorrectly pass because it believes the file is still allocated.

In this example, I can validate that this is not just dirty memory because
1. the item in question exists in mmfiles set
2. the handle in this case is valid via WinDBG

0:002> !handle 370 f
Handle 370
  Type         	File
  Attributes   	0
  GrantedAccess	0x12019f:
         ReadControl,Synch
         Read/List,Write/Add,Append/SubDir/CreatePipe,ReadEA,WriteEA,ReadAttr,WriteAttr
  HandleCount  	2
  PointerCount 	65536
  No Object Specific Information available



 Comments   
Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-14198: Fix memory allocation reuse

In std::set<Pointer>, Pointer to a memory address is not a unique key.
Mmap_posix already used a unique key in the case memory was reused.
Added the same change to mmap_win.

(cherry picked from commit af4f086490d6ae51aa0c252df26751385683eaaa)
Branch: v2.6
https://github.com/mongodb/mongo/commit/fa5b6b290aedc35492c20fa4cb549f9cf660eb62

Comment by Githook User [ 09/Jun/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-14198: Fix memory allocation reuse

In std::set<Pointer>, Pointer to a memory address is not a unique key.
Mmap_posix already used a unique key in the case memory was reused.
Added the same change to mmap_win.
Branch: master
https://github.com/mongodb/mongo/commit/af4f086490d6ae51aa0c252df26751385683eaaa

Generated at Thu Feb 08 03:34:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.