Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19879

WT_SESSION.checkpoint: WiredTiger.wt read error: failed to read 4096 bytes at offset 32768: WT_ERROR: non-specific WiredTiger error

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: 3.1.6
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Run YCSB load phase (I suspect a workload that does a large number of inserts with similar sized documents: ~1K would do the same) on a standalone mongod.

      Show
      Run YCSB load phase (I suspect a workload that does a large number of inserts with similar sized documents: ~1K would do the same) on a standalone mongod.
    • Linked BF Score:
      0

      Description

      While running the load phase of a YCSB test, wiredTiger encountered an error and panic. This has happened 3 out of 4 times so it seems fairly reproducible. The workload is doing a large number of inserts of documents that are 1K each. The mongod log reports some slow inserts (>100 msec) and then showed the following line about wiredTiger. This was seen on an EC2 instance running Enterprise version of mongod (3.1.7-pre, git version: 1b5b55cb48ff9daca74177b86d336613d207c5d1).

      015-08-11T08:53:47.184+0000 E STORAGE  [thread1] WiredTiger (-31802) [1439283227:184443][18776:0x7fdf70a65700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.wt read error: failed to read 4096 bytes at offset 32768: WT_ERROR: non-specific WiredTiger error
       
      2015-08-11T08:53:47.184+0000 E STORAGE  [thread1] WiredTiger (-31802) [1439283227:184629][18776:0x7fdf70a65700], checkpoint-server: checkpoint server error: WT_ERROR: non-specific WiredTiger error
       
      2015-08-11T08:53:47.184+0000 E STORAGE  [thread1] WiredTiger (-31804) [1439283227:184661][18776:0x7fdf70a65700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
       
      2015-08-11T08:53:47.184+0000 I -        [conn9] Fatal Assertion 28559
       
      2015-08-11T08:53:47.184+0000 I -        [conn16] Fatal Assertion 28559
      

      One the console where the YCSB test were run, these error repeated many times around the problem and are mostly related. Part of this error message that is interesting is where it says "Caused by: java.io.EOFException), which could be related to the error in mongod log.

      com.mongodb.MongoException$Network: Operation on server localhost:27017 failed
              at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:215)
              at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:461)
              at com.mongodb.DBCollectionImpl.insertWithCommandProtocol(DBCollectionImpl.java:426)
              at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:188)
              at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:167)
              at com.mongodb.DBCollection.insert(DBCollection.java:93)
              at com.mongodb.DBCollection.insert(DBCollection.java:78)
              at com.mongodb.DBCollection.insert(DBCollection.java:120)
              at com.yahoo.ycsb.db.MongoDbClient.insert(MongoDbClient.java:246)
              at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
              at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
              at com.yahoo.ycsb.ClientThread.run(Client.java:269)
      Caused by: java.io.EOFException
              at org.bson.io.Bits.readFully(Bits.java:75)
              at org.bson.io.Bits.readFully(Bits.java:50)
              at org.bson.io.Bits.readFully(Bits.java:37)
              at com.mongodb.Response.<init>(Response.java:42)
              at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:532)
              at com.mongodb.DBCollectionImpl.access$300(DBCollectionImpl.java:48)
              at com.mongodb.DBCollectionImpl$2.execute(DBCollectionImpl.java:474)
              at com.mongodb.DBCollectionImpl$2.execute(DBCollectionImpl.java:461)
              at com.mongodb.DBPort.doOperation(DBPort.java:187)
              at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:208)
              ... 11 more
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: