-
Type: Bug
-
Resolution: Incomplete
-
Priority: Critical - P2
-
None
-
Affects Version/s: 2.0.2
-
Component/s: None
-
Labels:None
-
Environment:CentOS 5.7, single-node deployment, 32GB RAM, 16 cores, 100GB Vertex SSD drive
-
Linux
The following code is used to save 1GB file to GridFS:
public <T> void saveMeta(String key, T data) { GridFS fileSpace = getFileSpace(META_SPACE_PREFIX); // create gridFS: ... fs = new GridFS(mongoDB, fsName); ... String journalFilename = key + "_JOURNAL"; GridFSInputFile journalFile = fileSpace.createFile(journalFilename); // some kind of pre-commit, write new version of data to a 'journal' file try { journalFile.getOutputStream().write(SerializationUtil.serialize(data)); journalFile.getOutputStream().close(); } catch (IOException e) { throw new RuntimeException(e); } deleteFromStorage(key, META_SPACE_PREFIX); // delete old version using GridFS#remove(key); GridFSInputFile file = fileSpace.createFile(key); try { file.getOutputStream().write(SerializationUtil.serialize(data)); // write new data file.getOutputStream().close(); } catch (IOException e) { throw new RuntimeException(e); } deleteFromStorage(journalFilename, META_SPACE_PREFIX); // delete the journal file }
This constantly crashes server with the following problem:
Tue May 10 22:50:54 [clientcursormon] mem (MB) res:1189 virt:53753 mapped:26564 Tue May 10 22:53:18 [journal] old journal file will be removed: /var/lib/mongo/journal/j._0 Tue May 10 22:54:30 [journal] warning assertion failure a <= 256*1024*1024 util/alignedbuilder.cpp 90 0x57a926 0x58434a 0x75b2d3 0x768b0a 0x76920f 0x769445 0x760f0f 0x76160d 0x76192d 0x7620bb 0xaa80b0 0x3705e0673d 0x37056d44bd /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926] /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x58434a] /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x75b2d3] /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a] /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f] /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445] /usr/bin/mongod(_ZN5mongo3dur28_groupCommitWithLimitedLocksEv+0x15f) [0x760f0f] /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x1d) [0x76160d] /usr/bin/mongod [0x76192d] /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x10b) [0x7620bb] /usr/bin/mongod(thread_proxy+0x80) [0xaa80b0] /lib64/libpthread.so.0 [0x3705e0673d] /lib64/libc.so.6(clone+0x6d) [0x37056d44bd] Tue May 10 22:54:38 [conn4] insert red-aril.dimensionsMETA.chunks 6385ms Tue May 10 22:54:40 [FileAllocator] allocating new datafile /var/lib/mongo/test-base.18, filling with zeroes... Tue May 10 22:54:40 [conn4] insert red-aril.dimensionsMETA.chunks 1153ms Tue May 10 22:54:42 [conn4] old journal file will be removed: /var/lib/mongo/journal/j._1 Tue May 10 22:54:56 [FileAllocator] done allocating datafile /var/lib/mongo/test-basel.18, size: 2047M Tue May 10 22:54:56 [conn4] DR101 latency warning on journal file open 5473ms Tue May 10 22:54:56 [conn4] insert red-aril.dimensionsMETA.chunks 14983ms Tue May 10 22:54:59 [conn4] insert red-aril.dimensionsMETA.chunks 2455ms Tue May 10 22:55:01 [conn4] remove red-aril.dimensionsMETA.chunks query: { files_id: ObjectId('4dca0b39e4b0a48e8532560b') } 721ms Tue May 10 22:55:13 [conn4] warning assertion failure a <= 256*1024*1024 util/alignedbuilder.cpp 90 0x57a926 0x58434a 0x75b2d3 0x768b0a 0x76920f 0x769445 0x75f179 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926] /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x58434a] /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x75b2d3] /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a] /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f] /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445] /usr/bin/mongod [0x75f179] /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da] /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7] /lib64/libpthread.so.0 [0x3705e0673d] /lib64/libc.so.6(clone+0x6d) [0x37056d44bd] Tue May 10 22:55:13 [conn4] rate limiting wassert Tue May 10 22:55:14 [conn4] Assertion failure a <= 512*1024*1024 util/alignedbuilder.cpp 91 0x57a926 0x5857db 0x75b2ed 0x768b0a 0x76920f 0x769445 0x75f179 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926] /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x5857db] /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x7d) [0x75b2ed] /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a] /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f] /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445] /usr/bin/mongod [0x75f179] /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da] /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7] /lib64/libpthread.so.0 [0x3705e0673d] /lib64/libc.so.6(clone+0x6d) [0x37056d44bd] Tue May 10 22:55:14 [conn4] dbexception in groupCommit causing immediate shutdown: 0 assertion util/alignedbuilder.cpp:91 Tue May 10 22:55:14 gc1 Tue May 10 22:55:14 Got signal: 6 (Aborted). Tue May 10 22:55:14 Backtrace: 0xa8d669 0x37056302d0 0x3705630265 0x3705631d10 0x881427 0x75f4a1 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa8d669] /lib64/libc.so.6 [0x37056302d0] /lib64/libc.so.6(gsignal+0x35) [0x3705630265] /lib64/libc.so.6(abort+0x110) [0x3705631d10] /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x881427] /usr/bin/mongod [0x75f4a1] /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da] /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7] /lib64/libpthread.so.0 [0x3705e0673d] /lib64/libc.so.6(clone+0x6d) [0x37056d44bd] Logstream::get called in uninitialized state Tue May 10 22:55:14 ERROR: Client::shutdown not called: conn