[SERVER-2695] Server crash during restore - cannot allocate memory Created: 07/Mar/11  Updated: 30/Mar/12  Resolved: 09/Mar/11

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 1.8.0-rc1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Bickell Assignee: Dwight Merriman
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 5 64-bit VM, 2GB RAM, dataset 2.1 GB


Operating System: Linux
Participants:

 Description   

When restoring a snapshot from production (1.6.2) to 1.8.0-rc1 the server crashes with the following error.

The server was running with the --dur option, and the restore was performed with --drop and --indexesLast.

Mon Mar 7 09:19:43 [conn1] building new index on

{ segmentTypeId: 1 }

for metaframe.segment
Mon Mar 7 09:19:43 [dur] lsn set 531582
854400/6203222 13%
1287700/6203222 20%
1950900/6203222 31%
2296100/6203222 37%
3000000/6203222 48%
3589500/6203222 57%
4066900/6203222 65%
4770000/6203222 76%
5282300/6203222 85%
5907300/6203222 95%
Mon Mar 7 09:21:34 [conn1] external sort used : 7 files in 111 secs
2414000/6203222 38%
5365700/6203222 86%
Mon Mar 7 09:21:57 [conn1] done building bottom layer, going to commit
Mon Mar 7 09:22:06 [conn1] ERROR: 13601 Couldn't remap private view: errno:12 Cannot allocate memory
Mon Mar 7 09:22:06 [conn1] aborting
Mon Mar 7 09:22:06 Got signal: 6 (Aborted).

Mon Mar 7 09:22:06 Backtrace:
0x8a4c29 0x3cdf8302d0 0x3cdf830265 0x3cdf831d10 0x5bc4df 0x629d7d 0x62de32 0x62e225 0x62e944 0x6852ea 0x6858bd 0x77702b 0x777a78 0x7794d0 0x77a91f 0x75585e 0x75b33c 0x8a80ae 0x8b84f0 0x3ce040673d
mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a4c29]
/lib64/libc.so.6 [0x3cdf8302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3cdf830265]
/lib64/libc.so.6(abort+0x110) [0x3cdf831d10]
mongod(_ZN5mongo16MemoryMappedFile16remapPrivateViewEPv+0x10f) [0x5bc4df]
mongod(_ZN5mongo8MongoMMF19remapThePrivateViewEv+0x4d) [0x629d7d]
mongod(_ZN5mongo3dur17_REMAPPRIVATEVIEWEv+0x212) [0x62de32]
mongod(_ZN5mongo3dur16REMAPPRIVATEVIEWEv+0x25) [0x62e225]
mongod [0x62e944]
mongod(_ZN5mongo12BtreeBuilder14buildNextLevelENS_7DiskLocE+0x12a) [0x6852ea]
mongod(_ZN5mongo12BtreeBuilder6commitEv+0x1d) [0x6858bd]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x12cb) [0x77702b]
mongod [0x777a78]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf60) [0x7794d0]
mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x5f) [0x77a91f]
mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x37e) [0x75585e]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x18ec) [0x75b33c]
mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a80ae]
mongod(thread_proxy+0x80) [0x8b84f0]
/lib64/libpthread.so.0 [0x3ce040673d]

      • glibc detected *** mongod: free(): invalid pointer: 0x0000000014199df0 ***
        ======= Backtrace: =========
        /lib64/libc.so.6[0x3cdf87230f]
        /lib64/libc.so.6(cfree+0x4b)[0x3cdf87276b]
        mongod(_ZNK5mongo3Top8cloneMapERSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS2_EEE+0x64)[0x5ff214]
        mongod(_ZN5mongo9Snapshots12takeSnapshotEv+0x51)[0x5fa101]
        mongod(_ZN5mongo14SnapshotThread3runEv+0x5b)[0x5fac2b]
        mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b)[0x54578b]
        mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x547044]
        mongod(thread_proxy+0x80)[0x8b84f0]
        /lib64/libpthread.so.0[0x3ce040673d]
        /lib64/libc.so.6(clone+0x6d)[0x3cdf8d3f6d]
        ======= Memory map: ========
        00400000-00b43000 r-xp 00000000 fd:00 2591568 /home/vs/mongo/bin/mongod
        00d42000-00d5f000 rw-p 00742000 fd:00 2591568 /home/vs/mongo/bin/mongod
        00d5f000-00d6e000 rw-p 00d5f000 00:00 0
        14154000-1f8b7000 rw-p 14154000 00:00 0 [heap]
        403fc000-403fd000 ---p 403fc000 00:00 0
        403fd000-40dfd000 rw-p 403fd000 00:00 0
        41212000-41213000 ---p 41212000 00:00 0
        41213000-41c13000 rw-p 41213000 00:00 0
        41fcd000-41fce000 ---p 41fcd000 00:00 0
        41fce000-429ce000 rw-p 41fce000 00:00 0
        429ce000-429cf000 ---p 429ce000 00:00 0
        429cf000-433cf000 rw-p 429cf000 00:00 0
        433cf000-433d0000 ---p 433cf000 00:00 0
        433d0000-43dd0000 rw-p 433d0000 00:00 0
        43dd0000-43dd1000 ---p 43dd0000 00:00 0
        43dd1000-447d1000 rw-p 43dd1000 00:00 0
        447d1000-447d2000 ---p 447d1000 00:00 0
        447d2000-451d2000 rw-p 447d2000 00:00 0
        451d2000-451d3000 ---p 451d2000 00:00 0
        451d3000-45bd3000 rw-p 451d3000 00:00 0
        45bd3000-45bd4000 ---p 45bd3000 00:00 0
        45bd4000-465d4000 rw-p 45bd4000 00:00 0
        3cdf400000-3cdf41c000 r-xp 00000000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf61b000-3cdf61c000 r--p 0001b000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf61c000-3cdf61d000 rw-p 0001c000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf800000-3cdf94e000 r-xp 00000000 fd:00 3474889 /lib64/libc-2.5.so
        3cdf94e000-3cdfb4d000 ---p 0014e000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb4d000-3cdfb51000 r--p 0014d000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb51000-3cdfb52000 rw-p 00151000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb52000-3cdfb57000 rw-p 3cdfb52000 00:00 0
        3ce0000000-3ce0082000 r-xp 00000000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0082000-3ce0281000 ---p 00082000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0281000-3ce0282000 r--p 00081000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0282000-3ce0283000 rw-p 00082000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0400000-3ce0416000 r-xp 00000000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0416000-3ce0615000 ---p 00016000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0615000-3ce0616000 r--p 00015000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0616000-3ce0617000 rw-p 00016000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0617000-3ce061b000 rw-p 3ce0617000 00:00 0
        3ce4400000-3ce440d000 r-xp 00000000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce440d000-3ce460d000 ---p 0000d000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce460d000-3ce460e000 rw-p 0000d000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce7000000-3ce70e6000 r-xp 00000000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce70e6000-3ce72e5000 ---p 000e6000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72e5000-3ce72eb000 r--p 000e5000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72eb000-3ce72ee000 rw-p 000eb000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72ee000-3ce7300000 rw-p 3ce72ee000 00:00 0
        2aaaaaaab000-2aaaaaaac000 rw-p 2aaaaaaab000 00:00 0
        2aaaaaaac000-2aaaabaac000 rw-s 00000000 fd:00 3637252 /data/metaframe.ns
        2aaaabaac000-2aaaacaac000 rw-p 00000000 fd:00 3637252 /data/metaframe.ns
        2aaaacaac000-2aaab0aac000 rw-s 00000000 fd:00 3637253 /data/metaframe.0
        2aaab0aac000-2aaab4aac000 rw-p 00000000 fd:00 3637253 /data/metaframe.0
        2aaab4aac000-2aaabcaac000 rw-s 00000000 fd:00 3637255 /data/metaframe.1
        2aaabcaac000-2aaac4aac000 rw-p 00000000 fd:00 3637255 /data/metaframe.1
        2aaac4aac000-2aaad4aac000 rw-s 00000000 fd:00 3637257 /data/metaframe.2
        2aaad4aac000-2aaae4aac000 rw-p 00000000 fd:00 3637257 /data/metaframe.2
        2aaae4aac000-2aab04aac000 rw-s 00000000 fd:00 3637258 /data/metaframe.3
        2aab04aac000-2aab24aac000 rw-p 00000000 fd:00 3637258 /data/metaframe.3
        2aab24aac000-2aab64aac000 rw-s 00000000 fd:00 3637260 /data/metaframe.4
        2aab64aac000-2aaba4aac000 rw-p 00000000 fd:00 3637260 /data/metaframe.4
        2aaba4aac000-2aac249ac000 rw-s 00000000 fd:00 3637261 /data/metaframe.5
        2aaca496e000-2aaca4b70000 rw-p 2aaca496e000 00:00 0
        2aaca8000000-2aacafa20000 rw-p 2aaca8000000 00:00 0
        2aacafa20000-2aacb0000000 ---p 2aacafa20000 00:00 0
        2aacb0000000-2aacb19c0000 rw-s 00000000 fd:00 3637267 /data/_tmp/esort.1299507583.490791297/file.0
        2aacb19c0000-2aacb3380000 rw-s 00000000 fd:00 3637268 /data/_tmp/esort.1299507583.490791297/file.1
        2aacb3380000-2aacb4d40000 rw-s 00000000 fd:00 3637269 /data/_tmp/esort.1299507583.490791297/file.2
        2aacb4d40000-2aacb6700000 rw-s 00000000 fd:00 3637270 /data/_tmp/esort.1299507583.490791297/file.3
        2aacb6700000-2aacb80c0000 rw-s 00000000 fd:00 3637271 /data/_tmp/esort.1299507583.490791297/file.4
        2aacb80c0000-2aacb9a80000 rw-s 00000000 fd:00 3637272 /data/_tmp/esort.1299507583.490791297/file.5
        2aacb9a80000-2aacb9fbc000 rw-s 00000000 fd:00 3637273 /data/_tmp/esort.1299507583.490791297/file.6
        2aacc55d7000-2aaccd5da000 rw-p 2aacc55d7000 00:00 0
        2b10e993f000-2b10e9940000 rw-p 2b10e993f000 00:00 0
        2b10e994d000-2b10e9951000 rw-p 2b10e994d000 00:00 0
        7fff36b21000-7fff36b36000 rw-p 7ffffffe9000 00:00 0 [stack]
        ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
        Mon Mar 7 09:22:11 Got signal: 6 (Aborted).

Mon Mar 7 09:22:11 Backtrace:
0x8a4c29 0x3cdf8302d0 0x3cdf830265 0x3cdf831d10 0x3cdf86a84b 0x3cdf87230f 0x3cdf87276b 0x5ff214 0x5fa101 0x5fac2b 0x54578b 0x547044 0x8b84f0 0x3ce040673d 0x3cdf8d3f6d
mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a4c29]
/lib64/libc.so.6 [0x3cdf8302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3cdf830265]
/lib64/libc.so.6(abort+0x110) [0x3cdf831d10]
/lib64/libc.so.6 [0x3cdf86a84b]
/lib64/libc.so.6 [0x3cdf87230f]
/lib64/libc.so.6(cfree+0x4b) [0x3cdf87276b]
mongod(_ZNK5mongo3Top8cloneMapERSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS2_EEE+0x64) [0x5ff214]
mongod(_ZN5mongo9Snapshots12takeSnapshotEv+0x51) [0x5fa101]
mongod(_ZN5mongo14SnapshotThread3runEv+0x5b) [0x5fac2b]
mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b) [0x54578b]
mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x547044]
mongod(thread_proxy+0x80) [0x8b84f0]
/lib64/libpthread.so.0 [0x3ce040673d]
/lib64/libc.so.6(clone+0x6d) [0x3cdf8d3f6d]

      • glibc detected *** mongod: double free or corruption (fasttop): 0x0000000014175640 ***
        ======= Backtrace: =========
        /lib64/libc.so.6[0x3cdf87230f]
        /lib64/libc.so.6(cfree+0x4b)[0x3cdf87276b]
        /usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x3a)[0x3ce709db6a]
        mongod[0x5fa6ee]
        /lib64/libc.so.6(exit+0xe5)[0x3cdf8333a5]
        mongod(_ZN5mongo10abruptQuitEi+0x5ee)[0x8a4e7e]
        /lib64/libc.so.6[0x3cdf8302d0]
        /lib64/libc.so.6(gsignal+0x35)[0x3cdf830265]
        /lib64/libc.so.6(abort+0x110)[0x3cdf831d10]
        mongod(_ZN5mongo16MemoryMappedFile16remapPrivateViewEPv+0x10f)[0x5bc4df]
        mongod(_ZN5mongo8MongoMMF19remapThePrivateViewEv+0x4d)[0x629d7d]
        mongod(_ZN5mongo3dur17_REMAPPRIVATEVIEWEv+0x212)[0x62de32]
        mongod(_ZN5mongo3dur16REMAPPRIVATEVIEWEv+0x25)[0x62e225]
        mongod[0x62e944]
        mongod(_ZN5mongo12BtreeBuilder14buildNextLevelENS_7DiskLocE+0x12a)[0x6852ea]
        mongod(_ZN5mongo12BtreeBuilder6commitEv+0x1d)[0x6858bd]
        mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x12cb)[0x77702b]
        mongod[0x777a78]
        mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf60)[0x7794d0]
        mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x5f)[0x77a91f]
        mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x37e)[0x75585e]
        mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x18ec)[0x75b33c]
        mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e)[0x8a80ae]
        mongod(thread_proxy+0x80)[0x8b84f0]
        /lib64/libpthread.so.0[0x3ce040673d]
        /lib64/libc.so.6(clone+0x6d)[0x3cdf8d3f6d]
        ======= Memory map: ========
        00400000-00b43000 r-xp 00000000 fd:00 2591568 /home/vs/mongo/bin/mongod
        00d42000-00d5f000 rw-p 00742000 fd:00 2591568 /home/vs/mongo/bin/mongod
        00d5f000-00d6e000 rw-p 00d5f000 00:00 0
        14154000-16968000 rw-p 14154000 00:00 0 [heap]
        403fc000-403fd000 ---p 403fc000 00:00 0
        403fd000-40dfd000 rw-p 403fd000 00:00 0
        41212000-41213000 ---p 41212000 00:00 0
        41213000-41c13000 rw-p 41213000 00:00 0
        41fcd000-41fce000 ---p 41fcd000 00:00 0
        41fce000-429ce000 rw-p 41fce000 00:00 0
        429ce000-429cf000 ---p 429ce000 00:00 0
        429cf000-433cf000 rw-p 429cf000 00:00 0
        433cf000-433d0000 ---p 433cf000 00:00 0
        433d0000-43dd0000 rw-p 433d0000 00:00 0
        43dd0000-43dd1000 ---p 43dd0000 00:00 0
        43dd1000-447d1000 rw-p 43dd1000 00:00 0
        447d1000-447d2000 ---p 447d1000 00:00 0
        447d2000-451d2000 rw-p 447d2000 00:00 0
        451d2000-451d3000 ---p 451d2000 00:00 0
        451d3000-45bd3000 rw-p 451d3000 00:00 0
        45bd3000-45bd4000 ---p 45bd3000 00:00 0
        45bd4000-465d4000 rw-p 45bd4000 00:00 0
        3cdf400000-3cdf41c000 r-xp 00000000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf61b000-3cdf61c000 r--p 0001b000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf61c000-3cdf61d000 rw-p 0001c000 fd:00 3474888 /lib64/ld-2.5.so
        3cdf800000-3cdf94e000 r-xp 00000000 fd:00 3474889 /lib64/libc-2.5.so
        3cdf94e000-3cdfb4d000 ---p 0014e000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb4d000-3cdfb51000 r--p 0014d000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb51000-3cdfb52000 rw-p 00151000 fd:00 3474889 /lib64/libc-2.5.so
        3cdfb52000-3cdfb57000 rw-p 3cdfb52000 00:00 0
        3ce0000000-3ce0082000 r-xp 00000000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0082000-3ce0281000 ---p 00082000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0281000-3ce0282000 r--p 00081000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0282000-3ce0283000 rw-p 00082000 fd:00 3473463 /lib64/libm-2.5.so
        3ce0400000-3ce0416000 r-xp 00000000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0416000-3ce0615000 ---p 00016000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0615000-3ce0616000 r--p 00015000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0616000-3ce0617000 rw-p 00016000 fd:00 3474894 /lib64/libpthread-2.5.so
        3ce0617000-3ce061b000 rw-p 3ce0617000 00:00 0
        3ce4400000-3ce440d000 r-xp 00000000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce440d000-3ce460d000 ---p 0000d000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce460d000-3ce460e000 rw-p 0000d000 fd:00 3474905 /lib64/libgcc_s-4.1.2-20080825.so.1
        3ce7000000-3ce70e6000 r-xp 00000000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce70e6000-3ce72e5000 ---p 000e6000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72e5000-3ce72eb000 r--p 000e5000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72eb000-3ce72ee000 rw-p 000eb000 fd:00 2832757 /usr/lib64/libstdc++.so.6.0.8
        3ce72ee000-3ce7300000 rw-p 3ce72ee000 00:00 0
        2aaaaaaab000-2aaaaaaac000 rw-p 2aaaaaaab000 00:00 0
        2aaaaaaac000-2aaaabaac000 rw-s 00000000 fd:00 3637252 /data/metaframe.ns
        2aaaabaac000-2aaaacaac000 rw-p 00000000 fd:00 3637252 /data/metaframe.ns
        2aaaacaac000-2aaab0aac000 rw-s 00000000 fd:00 3637253 /data/metaframe.0
        2aaab0aac000-2aaab4aac000 rw-p 00000000 fd:00 3637253 /data/metaframe.0
        2aaab4aac000-2aaabcaac000 rw-s 00000000 fd:00 3637255 /data/metaframe.1
        2aaabcaac000-2aaac4aac000 rw-p 00000000 fd:00 3637255 /data/metaframe.1
        2aaac4aac000-2aaad4aac000 rw-s 00000000 fd:00 3637257 /data/metaframe.2
        2aaad4aac000-2aaae4aac000 rw-p 00000000 fd:00 3637257 /data/metaframe.2
        2aaae4aac000-2aab04aac000 rw-s 00000000 fd:00 3637258 /data/metaframe.3
        2aab04aac000-2aab24aac000 rw-p 00000000 fd:00 3637258 /data/metaframe.3
        2aab24aac000-2aab64aac000 rw-s 00000000 fd:00 3637260 /data/metaframe.4
        2aab64aac000-2aaba4aac000 rw-p 00000000 fd:00 3637260 /data/metaframe.4
        2aaba4aac000-2aac249ac000 rw-s 00000000 fd:00 3637261 /data/metaframe.5
        2aaca8000000-2aacafa20000 rw-p 2aaca8000000 00:00 0
        2aacafa20000-2aacb0000000 ---p 2aacafa20000 00:00 0
        2aacb0000000-2aacb19c0000 rw-s 00000000 fd:00 3637267 /data/_tmp/esort.1299507583.490791297/file.0
        2aacb19c0000-2aacb3380000 rw-s 00000000 fd:00 3637268 /data/_tmp/esort.1299507583.490791297/file.1
        2aacb3380000-2aacb4d40000 rw-s 00000000 fd:00 3637269 /data/_tmp/esort.1299507583.490791297/file.2
        2aacb4d40000-2aacb6700000 rw-s 00000000 fd:00 3637270 /data/_tmp/esort.1299507583.490791297/file.3
        2aacb6700000-2aacb80c0000 rw-s 00000000 fd:00 3637271 /data/_tmp/esort.1299507583.490791297/file.4
        2aacb80c0000-2aacb9a80000 rw-s 00000000 fd:00 3637272 /data/_tmp/esort.1299507583.490791297/file.5
        2aacb9a80000-2aacb9fbc000 rw-s 00000000 fd:00 3637273 /data/_tmp/esort.1299507583.490791297/file.6
        2b10e993f000-2b10e9940000 rw-p 2b10e993f000 00:00 0
        2b10e994d000-2b10e9951000 rw-p 2b10e994d000 00:00 0
        7fff36b21000-7fff36b36000 rw-p 7ffffffe9000 00:00 0 [stack]
        ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
        Mon Mar 7 09:22:12 Got signal: 6 (Aborted).

Mon Mar 7 09:22:12 Backtrace:
0x8a4c29 0x3cdf8302d0 0x3cdf830265 0x3cdf831d10 0x3cdf86a84b 0x3cdf87230f 0x3cdf87276b 0x3ce709db6a 0x5fa6ee 0x3cdf8333a5 0x8a4e7e 0x3cdf8302d0 0x3cdf830265 0x3cdf831d10 0x5bc4df 0x629d7d 0x62de32 0x62e225 0x62e944 0x6852ea
mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a4c29]
/lib64/libc.so.6 [0x3cdf8302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3cdf830265]
/lib64/libc.so.6(abort+0x110) [0x3cdf831d10]
/lib64/libc.so.6 [0x3cdf86a84b]
/lib64/libc.so.6 [0x3cdf87230f]
/lib64/libc.so.6(cfree+0x4b) [0x3cdf87276b]
/usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x3a) [0x3ce709db6a]
mongod [0x5fa6ee]
/lib64/libc.so.6(exit+0xe5) [0x3cdf8333a5]
mongod(_ZN5mongo10abruptQuitEi+0x5ee) [0x8a4e7e]
/lib64/libc.so.6 [0x3cdf8302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3cdf830265]
/lib64/libc.so.6(abort+0x110) [0x3cdf831d10]
mongod(_ZN5mongo16MemoryMappedFile16remapPrivateViewEPv+0x10f) [0x5bc4df]
mongod(_ZN5mongo8MongoMMF19remapThePrivateViewEv+0x4d) [0x629d7d]
mongod(_ZN5mongo3dur17_REMAPPRIVATEVIEWEv+0x212) [0x62de32]
mongod(_ZN5mongo3dur16REMAPPRIVATEVIEWEv+0x25) [0x62e225]
mongod [0x62e944]
mongod(_ZN5mongo12BtreeBuilder14buildNextLevelENS_7DiskLocE+0x12a) [0x6852ea]

Mon Mar 7 09:22:12 [snapshotthread] ERROR: Client::shutdown not called: snapshotthread



 Comments   
Comment by Eliot Horowitz (Inactive) [ 09/Mar/11 ]

That last error is a warning and is normal.
The output has already been improved.

Comment by Aaron Bickell [ 09/Mar/11 ]

This issue may have been the result of an error on the disk array used for VM storage. Several other file system related issues arose yesterday, and I'm told they are now resolved. I was able to import the dataset with the --dur option successfully, and the server did not crash. I did however receive this error when the restore was complete:

Wed Mar 9 05:38:52 [conn1] metaframe.system.indexes Assertion failure btBuilder.getn() == nkeys || dropDups db/pdfile.cpp 1194
0x55127e 0x5628cf 0x7775ae 0x777a78 0x7794d0 0x77a91f 0x75585e 0x75b33c 0x8a80ae 0x8b84f0 0x374d20673d 0x374cad3f6d
mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x55127e]
mongod(_ZN5mongo9wassertedEPKcS1_j+0x10f) [0x5628cf]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x184e) [0x7775ae]
mongod [0x777a78]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf60) [0x7794d0]
mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x5f) [0x77a91f]
mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x37e) [0x75585e]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x18ec) [0x75b33c]
mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a80ae]
mongod(thread_proxy+0x80) [0x8b84f0]
/lib64/libpthread.so.0 [0x374d20673d]
/lib64/libc.so.6(clone+0x6d) [0x374cad3f6d]

All of the data and indexes appear to be intact, so there seem to be no lasting ramifications of this error. I apologize if the original bug was a red herring, and please let me know if the above issue should be tracked as a separate bug. Thanks for the quick responses.

Comment by Mathias Stearn [ 08/Mar/11 ]

Could you send the output of "cat /proc/sys/vm/overcommit_memory"

if it is 0 or 2 run " echo 1 | sudo tee /proc/sys/vm/overcommit_memory" and try again

(Edited to use correct numbers)

Comment by Aaron Bickell [ 07/Mar/11 ]

(vs)[vs@abickell ~]$ top | grep Swap
Swap: 1048568k total, 274376k used, 774192k free, 1627676k cached

Comment by Dwight Merriman [ 07/Mar/11 ]

how much swap? e.g., "top | grep Swap"

Comment by Aaron Bickell [ 07/Mar/11 ]

Below is the startup info for my mongo instance.

Mon Mar 7 09:10:27 [initandlisten] MongoDB starting : pid=7950 port=27017 dbpath=/data/ 64-bit
Mon Mar 7 09:10:27 [initandlisten] db version v1.8.0-rc1, pdfile version 4.5
Mon Mar 7 09:10:27 [initandlisten] git version: 87dd5720e48e875487d5fbd78cbe120d5a610973
Mon Mar 7 09:10:27 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Mon Mar 7 09:10:27 [initandlisten] journal dir=/data/journal
Mon Mar 7 09:10:27 [initandlisten] recover : no journal files present, no recovery needed

Yes, I have swap enabled my OS.

I did precisely the workaround you suggest, restore without --dur, restart with --dur and everything is functioning

Comment by Dwight Merriman [ 07/Mar/11 ]

not necessarily the entirety of the issue, but a fix that is related:
commit fd3f3569c02f7a097768f7b75463b93f7698a098

Comment by Dwight Merriman [ 07/Mar/11 ]

32 bit or 64 bit build?
do you have swap enabled on your OS instance?
if you need a quick workaround, you can restore the snapshot without --dur, then restart with --dur. we will fix of course just if you want a fix in the next few minutes...

Generated at Thu Feb 08 03:00:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.