Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19796

Cannot stop and backup hidden secondary without risk of inconsistent data.

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.4.14
    • Component/s: Admin, Replication
    • Labels:
      None
    • Environment:
      Ubuntu 12.04
    • Linux
    • Hide

      Using a replicated MongoD environment with one Primary, on Secondary, and one Hidden Secondary:
      (backup process)

      • Issue "db.adminCommand( {shutdown : 1}

        )" to the Hidden Secondary

      • Clone the data under the database folder to an alternate location
      • Start the Hidden Secondary and allow it to catch up.

      (restore process)
      1. Stop all MongoD hosts under the replicaset
      2. Remove the database data on all three MongoD hosts
      3. On the Hidden Secondary, restore the backup data to its original location
      4. Start the Hidden Secondary MongoD process
      5. Force a new config via rs.reconfig() which only includes the hidden secondary
      6. Add the original Primary and Secondary to the replicaset to replicate back
      7. Reconfigure the replicaset back to Primary, Secondary, Hidden Secondary.

      This will work 95% of the time. The other 5%, you cannot get past step 4 of the restore because the data that was backed up was still behind. The shutdown command did not wait for the hidden secondary to be caught up to the primary before it shut down.

      Show
      Using a replicated MongoD environment with one Primary, on Secondary, and one Hidden Secondary: (backup process) Issue "db.adminCommand( {shutdown : 1} )" to the Hidden Secondary Clone the data under the database folder to an alternate location Start the Hidden Secondary and allow it to catch up. (restore process) 1. Stop all MongoD hosts under the replicaset 2. Remove the database data on all three MongoD hosts 3. On the Hidden Secondary, restore the backup data to its original location 4. Start the Hidden Secondary MongoD process 5. Force a new config via rs.reconfig() which only includes the hidden secondary 6. Add the original Primary and Secondary to the replicaset to replicate back 7. Reconfigure the replicaset back to Primary, Secondary, Hidden Secondary. This will work 95% of the time. The other 5%, you cannot get past step 4 of the restore because the data that was backed up was still behind. The shutdown command did not wait for the hidden secondary to be caught up to the primary before it shut down.

      Our offsite backup process operates by stopping the MongoD processes on our Hidden Secondary hosts, then capturing their data by cloning their database folder and copying it offsite.

      The restore process works by stopping the MongoD process, removing the existing data within the database folder, then cloning the offsite data back into the database folder.

      On several occasions now we have had the restored MongoD process never get out of 'startup2' as it believes there is data still to be replicated.

      We have followed the documented recommendations for stopping a replica host by issuing "db.adminCommand(

      {shutdown : 1}

      )" to the MongoD process instead of stopping the process through upstart.

      The documentation for the shutdown command states that it will not run unless "a" secondary has caught up with the primary. If the wording matches the logic, I suspect that the shutdown command proceeds with stopping the Hidden Secondary MongoD host, even when it is behind, because "A" Secondary is up to date (i.e. the non-hidden secondary witihin the replicaset).

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            dave.muysson@360pi.com Dave Muysson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: