Details
Description
While upgrading our replica set from MongoDB 2.6.8 to 3.0.6, I received the below error. I also tried 3.0.7 today and still received the error. I originally received the error about 10 minutes after upgrading a secondary. I then restarted it and the error occurred about 1 hour later. I then played the restart game for a few more hours and it seemed to keep happening. I ended up performing a db.repairDatabase() on the high volume databases on that server and I thought it fixed the issue - it ran for about 24 hours without a crash. However, today, after updating the binaries from 3.0.6 to 3.0.7 I did receive the error again after running for less than one minute.
I had also tried 3.0.6 on the other secondary, and deleted its data to let it just replicate over, thinking it was a data format issue (since the repair appeared to have helped). However either during the final index build or while catching up with replication, it had the error again.
We are running on Win 2012 R2 on AWS. Our secondaries have 32GB of RAM. We have been using this production DB for a few years now. I currently have the system split between 2.6.8 and 3.0.7 as completing the upgrade puts us at risk of crashing.
Error from a MongoDB 3.0.7 secondary, about 20 seconds after launching it:
2015-10-14T15:36:04.990+0000 I CONTROL [durability] MapViewOfFileEx for d:/mongo/troidb2.79 failed with error errno:487 Attempt to access invalid address. (file size is 2146435072) in MemoryMappedFile::remapPrivateView
|
2015-10-14T15:36:04.990+0000 I ACCESS [conn2239] Successfully authenticated as principal mongodev on troidb2
|
2015-10-14T15:36:04.990+0000 I - [durability] Fatal Assertion 16148
|
|
|
|
|
2015-10-14T15:36:05.573+0000 I NETWORK [initandlisten] connection accepted from 54.204.126.8:58766 #2395 (561 connections now open)
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\stacktrace_win.cpp(175) mongo::printStackTrace+0x43
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\log.cpp(134) mongo::logContext+0x97
|
2015-10-14T15:36:05.579+0000 I NETWORK [conn2395] end connection 54.204.126.8:58766 (560 connections now open)
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\assert_util.cpp(166) mongo::fassertFailed+0x80
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\mmap_win.cpp(425) mongo::MemoryMappedFile::remapPrivateView+0x327
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\durable_mapped_file.cpp(76) mongo::DurableMappedFile::remapThePrivateView+0x2a
|
2015-10-14T15:36:05.579+0000 I NETWORK [conn2394] end connection 54.204.247.20:64774 (559 connections now open)
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(388) mongo::dur::`anonymous namespace'::remapPrivateViewImpl+0x1e8
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(639) mongo::dur::remapPrivateView+0x69
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(826) mongo::dur::durThread+0x797
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] MSVCR120.dll beginthreadex+0x107
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] MSVCR120.dll endthreadex+0x192
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability] KERNEL32.DLL BaseThreadInitThunk+0x1a
|
2015-10-14T15:36:05.579+0000 I CONTROL [durability]
|
2015-10-14T15:36:05.580+0000 I - [durability]
|
|
|
***aborting after fassert() failure
|
Error from a MongoDB 3.0.6 secondary performing an inital sync (about 9 hours in):
2015-10-14T09:08:47.910+0000 I CONTROL [durability] MapViewOfFileEx for d:/mongo/DBNamehidden.27 failed with error errno:487 Attempt to access invalid address. (file size is 2146435072) in MemoryMappedFile::remapPrivateView
|
2015-10-14T09:08:47.910+0000 I - [durability] Fatal Assertion 16148
|
|
|
|
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\stacktrace_win.cpp(175) mongo::printStackTrace+0x43
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\log.cpp(134) mongo::logContext+0x97
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\assert_util.cpp(166) mongo::fassertFailed+0x80
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\util\mmap_win.cpp(425) mongo::MemoryMappedFile::remapPrivateView+0x327
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\durable_mapped_file.cpp(76) mongo::DurableMappedFile::remapThePrivateView+0x2a
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(388) mongo::dur::`anonymous namespace'::remapPrivateViewImpl+0x1e8
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(639) mongo::dur::remapPrivateView+0x69
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\mongo\db\storage\mmap_v1\dur.cpp(826) mongo::dur::durThread+0x797
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] MSVCR120.dll beginthreadex+0x107
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] MSVCR120.dll endthreadex+0x192
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability] KERNEL32.DLL BaseThreadInitThunk+0x1a
|
2015-10-14T09:08:51.876+0000 I CONTROL [durability]
|
2015-10-14T09:08:51.876+0000 I - [durability]
|
|
|
***aborting after fassert() failure
|
Attachments
Issue Links
- related to
-
SERVER-23977 Failure of MapViewOfFileEx in MemoryMappedFile::remapPrivateView with "errno:487 Attempt to access invalid address"
-
- Closed
-