Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8358

"Move chunk commit failed" shutdown leaves mongod with no journal files and mongod.lock file

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.3.2
    • Component/s: Sharding
    • None
    • ALL

      This makes it difficult to recover from.

      Mon Jan 28 09:25:29.328 [conn27] moveChunk commit outcome ongoing: { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "test.not.hashed-user_id_"user327"", lastmod: Timestamp 30000|0, lastmodEpoch: ObjectId('510681b502471f5418db3e35'), ns: "test.not.hashed", min: { user_id: "user327" }, max: { user_id: "user3332" }, shard: "test-rs2" }, o2: { _id: "test.not.hashed-user_id_"user327"" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "test.not.hashed-user_id_"user3332"", lastmod: Timestamp 30000|1, lastmodEpoch: ObjectId('510681b502471f5418db3e35'), ns: "test.not.hashed", min: { user_id: "user3332" }, max: { user_id: "user3392" }, shard: "test-rs4" }, o2: { _id: "test.not.hashed-user_id_"user3332"" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "test.not.hashed" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 29000|1 } } ] } for command :{ $err: "SyncClusterConnection::findOne prepare failed: 10276 DBClientBase::findN: transport error: ubuntu:28001 ns: admin.$cmd query: { fsync: 1 } ubuntu:28001:{}", code: 13104 }
      Mon Jan 28 09:25:30.312 [conn88230] end connection 127.0.0.1:36609 (18 connections now open)
      Mon Jan 28 09:25:30.312 [initandlisten] connection accepted from 127.0.0.1:36631 #88232 (19 connections now open)
      Mon Jan 28 09:25:39.328 [conn27] SyncClusterConnection connecting to [ubuntu:28000]
      Mon Jan 28 09:25:39.328 [conn27] SyncClusterConnection connecting to [ubuntu:28001]
      Mon Jan 28 09:25:39.328 [conn27] SyncClusterConnection connecting to [ubuntu:28002]
      Mon Jan 28 09:25:39.329 [conn27] moveChunk commit failed: version is at29|1||000000000000000000000000 instead of 30|1||510681b502471f5418db3e35
      Mon Jan 28 09:25:39.329 [conn27] TERMINATING
      Mon Jan 28 09:25:39.329 dbexit: 
      Mon Jan 28 09:25:39.329 [conn27] shutdown: going to close listening sockets...
      Mon Jan 28 09:25:39.329 [conn27] closing listening socket: 9
      Mon Jan 28 09:25:39.329 [conn27] closing listening socket: 11
      Mon Jan 28 09:25:39.329 [conn27] closing listening socket: 12
      Mon Jan 28 09:25:39.329 [conn27] removing socket file: /tmp/mongodb-31500.sock
      Mon Jan 28 09:25:39.329 [conn27] shutdown: going to flush diaglog...
      Mon Jan 28 09:25:39.329 [conn27] shutdown: going to close sockets...
      Mon Jan 28 09:25:39.329 [conn27] shutdown: waiting for fs preallocator...
      Mon Jan 28 09:25:39.329 [conn27] shutdown: closing all files...
      Mon Jan 28 09:25:39.330 [conn35376] end connection 127.0.0.1:47546 (18 connections now open)
      Mon Jan 28 09:25:39.330 [initandlisten] now exiting
      Mon Jan 28 09:25:39.330 dbexit: ; exiting immediately
      
      
      ***** SERVER RESTARTED *****
      
      
      Mon Jan 28 10:01:49.172 [initandlisten] MongoDB starting : pid=19657 port=31500 dbpath=/data/db/test-rs4-0 64-bit host=ubuntu
      Mon Jan 28 10:01:49.172 [initandlisten] 
      Mon Jan 28 10:01:49.172 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB.
      Mon Jan 28 10:01:49.172 [initandlisten] **       Not recommended for production.
      Mon Jan 28 10:01:49.172 [initandlisten] 
      Mon Jan 28 10:01:49.172 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5
      Mon Jan 28 10:01:49.172 [initandlisten] git version: e27538de7afa9f63ca0a0a5be97dc69f26143466
      Mon Jan 28 10:01:49.172 [initandlisten] build info: Linux ubuntu 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 BOOST_LIB_VERSION=1_49
      Mon Jan 28 10:01:49.172 [initandlisten] allocator: tcmalloc
      Mon Jan 28 10:01:49.172 [initandlisten] options: { dbpath: "/data/db/test-rs4-0", fork: true, logappend: true, logpath: "/home/k/dbs/test-rs4-0.log", port: 31500, replSet: "test-rs4" }
      Mon Jan 28 10:01:49.172 [initandlisten] Unable to check for journal files due to: boost::filesystem::directory_iterator::construct: No such file or directory: "/data/db/test-rs4-0/journal"
      ************** 
      Unclean shutdown detected.
      Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
      *************
      Mon Jan 28 10:01:49.172 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
      Mon Jan 28 10:01:49.172 dbexit: 
      Mon Jan 28 10:01:49.172 [initandlisten] shutdown: going to close listening sockets...
      Mon Jan 28 10:01:49.172 [initandlisten] shutdown: going to flush diaglog...
      Mon Jan 28 10:01:49.172 [initandlisten] shutdown: going to close sockets...
      Mon Jan 28 10:01:49.172 [initandlisten] shutdown: waiting for fs preallocator...
      Mon Jan 28 10:01:49.172 [initandlisten] shutdown: closing all files...
      Mon Jan 28 10:01:49.172 [initandlisten] closeAllFiles() finished
      Mon Jan 28 10:01:49.172 dbexit: really exiting now
      

      Rest of log attached. Not sure if this is actually sharding, might be a shutdown problem.

        1. log.tar.bz2
          1.20 MB
          Kristina Chodorow

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            kristina Kristina Chodorow (Inactive)
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: