Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-4893

Mongo 2.0.2 crashes when a large file is written to GridFS : "Assertion failure a <= 512*1024*1024 util/alignedbuilder.cpp"

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.0.2
    • Component/s: None
    • Labels:
      None
    • Environment:
      CentOS 5.7, single-node deployment, 32GB RAM, 16 cores, 100GB Vertex SSD drive
    • Linux

      The following code is used to save 1GB file to GridFS:

           public <T> void saveMeta(String key, T data) {
              GridFS fileSpace = getFileSpace(META_SPACE_PREFIX); // create gridFS: ... fs = new GridFS(mongoDB, fsName); ...
      
              String journalFilename = key + "_JOURNAL";
              GridFSInputFile journalFile = fileSpace.createFile(journalFilename);    // some kind of pre-commit, write new version of data to a 'journal' file 
      
              try {
                  journalFile.getOutputStream().write(SerializationUtil.serialize(data));
                  journalFile.getOutputStream().close();
              } catch (IOException e) {
                  throw new RuntimeException(e);
              }
      
              deleteFromStorage(key, META_SPACE_PREFIX);    // delete old version using GridFS#remove(key);
      
              GridFSInputFile file = fileSpace.createFile(key);  
      
              try {
                  file.getOutputStream().write(SerializationUtil.serialize(data));  // write new data
                  file.getOutputStream().close();
              } catch (IOException e) {
                  throw new RuntimeException(e);
              }
      
              deleteFromStorage(journalFilename, META_SPACE_PREFIX);  // delete the journal file
          }
      

      This constantly crashes server with the following problem:

      Tue May 10 22:50:54 [clientcursormon] mem (MB) res:1189 virt:53753 mapped:26564
      Tue May 10 22:53:18 [journal] old journal file will be removed: /var/lib/mongo/journal/j._0
      Tue May 10 22:54:30 [journal]   warning assertion failure a <= 256*1024*1024 util/alignedbuilder.cpp 90
      0x57a926 0x58434a 0x75b2d3 0x768b0a 0x76920f 0x769445 0x760f0f 0x76160d 0x76192d 0x7620bb 0xaa80b0 0x3705e0673d 0x37056d44bd 
       /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926]
       /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x58434a]
       /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x75b2d3]
       /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a]
       /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f]
       /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445]
       /usr/bin/mongod(_ZN5mongo3dur28_groupCommitWithLimitedLocksEv+0x15f) [0x760f0f]
       /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x1d) [0x76160d]
       /usr/bin/mongod [0x76192d]
       /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x10b) [0x7620bb]
       /usr/bin/mongod(thread_proxy+0x80) [0xaa80b0]
       /lib64/libpthread.so.0 [0x3705e0673d]
       /lib64/libc.so.6(clone+0x6d) [0x37056d44bd]
      Tue May 10 22:54:38 [conn4] insert red-aril.dimensionsMETA.chunks 6385ms
      Tue May 10 22:54:40 [FileAllocator] allocating new datafile /var/lib/mongo/test-base.18, filling with zeroes...
      Tue May 10 22:54:40 [conn4] insert red-aril.dimensionsMETA.chunks 1153ms
      Tue May 10 22:54:42 [conn4] old journal file will be removed: /var/lib/mongo/journal/j._1
      Tue May 10 22:54:56 [FileAllocator] done allocating datafile /var/lib/mongo/test-basel.18, size: 2047M
      
      Tue May 10 22:54:56 [conn4] DR101 latency warning on journal file open 5473ms
      Tue May 10 22:54:56 [conn4] insert red-aril.dimensionsMETA.chunks 14983ms
      Tue May 10 22:54:59 [conn4] insert red-aril.dimensionsMETA.chunks 2455ms
      Tue May 10 22:55:01 [conn4] remove red-aril.dimensionsMETA.chunks query: { files_id: ObjectId('4dca0b39e4b0a48e8532560b') } 721ms
      Tue May 10 22:55:13 [conn4]   warning assertion failure a <= 256*1024*1024 util/alignedbuilder.cpp 90
      0x57a926 0x58434a 0x75b2d3 0x768b0a 0x76920f 0x769445 0x75f179 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd 
       /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926]
       /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x58434a]
       /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x75b2d3]
       /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a]
       /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f]
       /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445]
       /usr/bin/mongod [0x75f179]
       /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da]
       /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808]
       /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95]
       /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6]
       /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7]
       /lib64/libpthread.so.0 [0x3705e0673d]
       /lib64/libc.so.6(clone+0x6d) [0x37056d44bd]
      Tue May 10 22:55:13 [conn4] rate limiting wassert
      Tue May 10 22:55:14 [conn4]   Assertion failure a <= 512*1024*1024 util/alignedbuilder.cpp 91
      0x57a926 0x5857db 0x75b2ed 0x768b0a 0x76920f 0x769445 0x75f179 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd 
       /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a926]
       /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x5857db]
       /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x7d) [0x75b2ed]
       /usr/bin/mongod(_ZN5mongo3dur21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE+0x2fa) [0x768b0a]
       /usr/bin/mongod(_ZN5mongo3dur15prepBasicWritesERNS_14AlignedBuilderE+0x6f) [0x76920f]
       /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderE+0x75) [0x769445]
       /usr/bin/mongod [0x75f179]
       /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da]
       /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808]
       /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95]
       /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6]
       /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7]
       /lib64/libpthread.so.0 [0x3705e0673d]
       /lib64/libc.so.6(clone+0x6d) [0x37056d44bd]
      Tue May 10 22:55:14 [conn4] dbexception in groupCommit causing immediate shutdown: 0 assertion util/alignedbuilder.cpp:91
      Tue May 10 22:55:14 gc1
      Tue May 10 22:55:14 Got signal: 6 (Aborted).
      
      Tue May 10 22:55:14 Backtrace:
      0xa8d669 0x37056302d0 0x3705630265 0x3705631d10 0x881427 0x75f4a1 0x7640da 0x888808 0x88ac95 0xaa00c6 0x635bb7 0x3705e0673d 0x37056d44bd 
       /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa8d669]
       /lib64/libc.so.6 [0x37056302d0]
       /lib64/libc.so.6(gsignal+0x35) [0x3705630265]
       /lib64/libc.so.6(abort+0x110) [0x3705631d10]
       /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x881427]
       /usr/bin/mongod [0x75f4a1]
       /usr/bin/mongod(_ZN5mongo9writelockD1Ev+0xba) [0x7640da]
       /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x468) [0x888808]
       /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1155) [0x88ac95]
       /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa00c6]
       /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635bb7]
       /lib64/libpthread.so.0 [0x3705e0673d]
       /lib64/libc.so.6(clone+0x6d) [0x37056d44bd]
      
      Logstream::get called in uninitialized state
      Tue May 10 22:55:14 ERROR: Client::shutdown not called: conn
      

        1. log.7z
          99 kB

            Votes:
            3 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: