Hi Team,

We are facing an problem and would request your help to resolve the issue:

Requirement :

We have an replica set with three members in Prodcution and Net file back up snap shots are taken at regular intervals.

We wanted this net files backed up for the entire db folders in the db path and wanted to copy it across to a stand alone machine where mongo is set up, to be used for
development pusposes on top of the production data copy or as a mechanism for disaster recovery when a fail over happens on all the nodes in the replica set.

Please note that We had taken these net file back ups from Secondary nodes while its up and running. The replica set is configured to be journal enabled.

Problem :

Tried the following steps :

1. Copied the backed up Net file Back Up as is to the – db path folder to the Standalone machine when the mongod instance is down.
2. The stand alone machine has the following config set up : Please refer Mongo.config
3. Received the error on the mongo shell :

MONGO SHELL LOG:

about to fork child process, waiting until server is ready for connections.
forked process: 20608
ERROR: child process failed, exited with error number 100

4. Remove the mongod.lock and started once again and it says the error

MONGO SHELL LOG :

[ppoker@in1cod2vir235 db]$ mongod --config "/home/release/release_independent/dbdata/coredfs/conf/coredfs.conf"
about to fork child process, waiting until server is ready for connections.
forked process: 20668
ERROR: child process failed, exited with error number 51

5. Could see the following lines from the log files :

MONGO SHELL LOG:

2018-03-02T06:19:50.513-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1519989590:513360][20668:0x7ffaa98b4e40], file:coredb/collection-11--3459593218143293763.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 1546698752: block header checksum of 2826665503 doesn't match expected checksum of 4213193626
2018-03-02T06:19:50.513-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1519989590:513402][20668:0x7ffaa98b4e40], file:coredb/collection-11--3459593218143293763.wt, WT_SESSION.open_cursor: coredb/collection-11--3459593218143293763.wt: encountered an illegal file format or internal value
2018-03-02T06:19:50.513-0500 E STORAGE  [initandlisten] WiredTiger error (-31804) [1519989590:513415][20668:0x7ffaa98b4e40], file:coredb/collection-11--3459593218143293763.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-03-02T06:19:50.513-0500 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-03-02T06:19:50.513-0500 I -        [initandlisten]

6. The config details used for start up in the stand alone Mongo instance are as follows :

systemLog:
   destination: file
   path: "/home/release/release_independent/dbdata/coredfs/log/coredfs.log"
   quiet: false
   logAppend: true
   traceAllExceptions: true
   logRotate: "rename"
   verbosity: 0
storage:
   journal:
      enabled: true
   dbPath: "/home/release/release_independent/dbdata/coredfs/data/db"
   engine: "wiredTiger"
   directoryPerDB: true
processManagement:
   fork: true

7. Repair of the mongod was not possible as it was journal enabled .. Tried with --no journal during start up but resulted in the same Check Point Error as mentioned above.

I have the following questions :

1. How do we resolve this . Is this approach of net file back up from a secondary in a replica to a stand alone is fine. Am i missing any thing here.
2. How do we copy consistent copies from Production system to development environment using Net file Backup ONLY without using copy Database or Mongo Dump and Restore options.

Humbly request your kind inputs here.

Thanks in advance.