[SERVER-28006] Unexpected Initial Sync on Replicaset Created: 14/Feb/17  Updated: 31/May/17  Resolved: 10/Mar/17

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Ralph Decatrel Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 1c.log     Text File 1e-26.log     File mongod3-0.conf     File mongodb2-6.conf     Text File original_mail_comment.txt    
Issue Links:
Related
related to SERVER-17710 Do not automatically wipe existing da... Backlog
Participants:

 Description   

Trying to add a MongoDB 3.0 instance to an existing MongoDB 2.6 replica set. Following the instructions on how to do this I tried the following steps.

From the Primary, called 1e-26, I added the new instance "1c:27107" to the replica set.
Took an EBS snapshot of data volume of 1e-26
Stopped the mongod service on new instance, 1c.
Created an EBS volume of from the 1e-26 snapshot.
Attached the volume to 1c as the new data volume
Restarted the mongod service

I expected 1c to use the data from the snapshot to speed up the sync with 1e-26, but instead it removes all the data and starts and initial sync. We cannot upgrade to MongoDB 3+ until we can solve the problem of how to restore a MongoDB instance from an EBS Snapshot.

With MongoDB 2.6 and prior. It was a simple matter to start a mongd service with the latest EBS snapshot and add the new instance from the primary. This approach no longer works.

Attached are the log files from 1e-26, 1c and the configs



 Comments   
Comment by Kelsey Schubert [ 10/Mar/17 ]

Thank you for the update.

Comment by Ralph Decatrel [ 27/Feb/17 ]

Here's what I found works, so this issue can be closed:

1. Be prepared either with a DNS solution or by using /etc/hosts to create
a hostname for a new member. It won't be possible to use the AWS internal
dns name because this would involve predicting the IP address.

2. From the PRIMARY of a multi-member replica set, add the new member
before it exists using "rs.add('<new member hostname>:27017')". This will
fail if there is only one reachable member of the replica set, otherwise it
will add the member with the status of 'unreachable'. If the replica set
contains only the PRIMARY it will complain that the replica set must have
most of it's members present.

3. Take an EBS snapshot of the data volume any replica member that has the
new rs.conf() containing the newly added non-existent member.

3. Use that snapshot to create the data volume of a new instance with the
hostname that was already added to the rs.conf().

4. start mongod service on the new member

5. Add the new member's name to DNS or /etc/hosts for all replica members
and all other instances that will want to connect if the new server should
become PRIMARY.

After recovering the data the new member will join the replica set as a
secondary.

On Thu, Feb 23, 2017 at 7:04 PM Thomas Schubert (JIRA) <jira@mongodb.org>


Ralph <http://about.me/justronin>
ralph@y12k.com

Comment by Kelsey Schubert [ 24/Feb/17 ]

Hi justronin,

Thank you for the additional information.

Would you please execute the following steps, and provide the request output?

  1. Shutdown mongod on 1e-26
  2. Provide output of md5sum * executed on the dbpath of 1e-26
  3. WIth mongod still shutdown, take EBS snapshot
  4. mongod on 1e-26 can now be restarted
  5. Restore EBS snapshot
  6. Provide output of md5sum * executed on the dbpath of 1c
  7. Start mongod on 1c

Thanks again,
Thomas

Comment by Ralph Decatrel [ 22/Feb/17 ]

After setting up 1e-26 as primary:

production-rs:PRIMARY> rs.status()
{
"set" : "production-rs",
"date" : ISODate("2017-02-22T17:40:46Z"),
"myState" : 1,
"members" : [

{ "_id" : 1, "name" : "1e-26:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1172, "optime" : Timestamp(1487761150, 1), "optimeDate" : ISODate("2017-02-22T10:59:10Z"), "electionTime" : Timestamp(1487785236, 1), "electionDate" : ISODate("2017-02-22T17:40:36Z"), "self" : true }

],
"ok" : 1
}

rdecatrel@1e-26:~$ ls -lsa /var/lib/mongodb
total 337609984
20 drwxrwxrwx 3 mongodb mongodb 16384 Feb 22 17:21 .
4 drwxr-xr-x 45 root root 4096 Feb 22 17:20 ..
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 21:20 cust1_production.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 20 19:27 cust1_production.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 20 19:27
cust1_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Feb 16 14:31 cust1_sandbox.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 20 18:56 cust1_sandbox.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 20 18:56 cust1_sandbox.ns
65536 rw------ 1 mongodb nogroup 67108864 Feb 16 19:19 cust2_production.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 22 00:01 cust2_production.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 22 00:01
cust2_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Feb 15 13:22 cust2_sandbox.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 17 20:53 cust2_sandbox.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 17 20:53 cust2_sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 20 22:19 cust3_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 20 22:19 cust3_production.1
16384 rw------ 1 mongodb mongodb 16777216 Feb 20 22:19
cust3_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 20 22:19 cust4_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 20 22:19 cust4_production.1
16384 rw------ 1 mongodb mongodb 16777216 Feb 20 22:19
cust4_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Nov 10 17:11 cust5_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 20 22:19 cust5_production.1
262144 rw------ 1 mongodb mongodb 268435456 Feb 8 22:09 cust5_production.2
524288 rw------ 1 mongodb mongodb 536870912 Nov 10 17:11 cust5_production.3
1048576 rw------ 1 mongodb mongodb 1073741824 Oct 20 20:31 cust5_production.4
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 20 22:19 cust5_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 24 16:14 cust5_production.6
16384 rw------ 1 mongodb mongodb 16777216 Feb 20 22:19
cust5_production.ns
65540 rw------ 1 mongodb mongodb 67108864 Jan 27 21:21 cust6_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 21:21 cust6_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 21:21
cust6_production.ns
65536 rw------ 1 mongodb mongodb 67108864 May 26 2016 cust7_production.0
131076 rw------ 1 mongodb mongodb 134217728 Dec 29 18:16 cust7_production.1
16384 rw------ 1 mongodb mongodb 16777216 Dec 29 18:16
cust7_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Nov 10 17:11 cust8_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 10 00:02 cust8_production.1
262144 rw------ 1 mongodb mongodb 268435456 Nov 15 21:40 cust8_production.2
16384 rw------ 1 mongodb mongodb 16777216 Feb 10 00:02
cust8_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Oct 26 16:40 cust8_sandbox.0
131076 rw------ 1 mongodb mongodb 134217728 Nov 15 19:20 cust8_sandbox.1
262148 rw------ 1 mongodb mongodb 268435456 Nov 15 19:20 cust8_sandbox.2
16384 rw------ 1 mongodb mongodb 16777216 Nov 15 19:20 cust8_sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Dec 3 2015 cust9_production.0
131072 rw------ 1 mongodb mongodb 134217728 Jan 27 21:21 cust9_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 21:21
cust9_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 8 08:03
cust10_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 22 09:58
cust10_production.1
262148 rw------ 1 mongodb mongodb 268435456 Feb 21 13:17
cust10_production.2
524292 rw------ 1 mongodb mongodb 536870912 Feb 21 13:17
cust10_production.3
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 22 09:58
cust10_production.4
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 21 13:17
cust10_production.5
2096128 rw------ 1 mongodb nogroup 2146435072 Jan 13 22:25
cust10_production.6
2096132 rw------ 1 mongodb nogroup 2146435072 Feb 22 09:58
cust10_production.7
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 09:58
cust10_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 21 23:51 demo_production.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015 demo_production.1
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 21 23:51 demo_production.10
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015 demo_production.2
524288 rw------ 1 mongodb mongodb 536870912 Apr 28 2015 demo_production.3
1048576 rw------ 1 mongodb mongodb 1073741824 Apr 28 2015 demo_production.4
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015 demo_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 21 21:42 demo_production.6
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 21 23:51 demo_production.7
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 10:56 demo_production.8
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 10:56 demo_production.9
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 10:56 demo_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 13 22:53
cust11_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 22 17:28
cust11_production.1
2096128 rw------ 1 mongodb mongodb 2146435072 Mar 7 2016
cust11_production.10
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 15 2016
cust11_production.11
2096128 rw------ 1 mongodb mongodb 2146435072 May 17 2016
cust11_production.12
2096128 rw------ 1 mongodb mongodb 2146435072 Jun 9 2016
cust11_production.13
2096128 rw------ 1 mongodb mongodb 2146435072 Jul 1 2016
cust11_production.14
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 17:28
cust11_production.15
2096128 rw------ 1 mongodb mongodb 2146435072 Jul 22 2016
cust11_production.16
2096128 rw------ 1 mongodb mongodb 2146435072 Aug 5 2016
cust11_production.17
2096128 rw------ 1 mongodb mongodb 2146435072 Aug 25 12:47
cust11_production.18
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 2 18:58
cust11_production.19
262144 rw------ 1 mongodb mongodb 268435456 Feb 22 17:28
cust11_production.2
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 14 22:38
cust11_production.20
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 22 03:16
cust11_production.21
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 27 23:58
cust11_production.22
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 4 21:22
cust11_production.23
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 8 22:31
cust11_production.24
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 14 14:18
cust11_production.25
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 19 19:47
cust11_production.26
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 25 01:41
cust11_production.27
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 28 08:36
cust11_production.28
2096128 rw------ 1 mongodb mongodb 2146435072 Nov 3 05:36
cust11_production.29
524288 rw------ 1 mongodb mongodb 536870912 Feb 21 21:55
cust11_production.3
2096128 rw------ 1 mongodb mongodb 2146435072 Nov 8 13:16
cust11_production.30
2096128 rw------ 1 mongodb mongodb 2146435072 Nov 12 01:59
cust11_production.31
2096128 rw------ 1 mongodb nogroup 2146435072 Nov 17 17:31
cust11_production.32
2096128 rw------ 1 mongodb nogroup 2146435072 Nov 22 22:03
cust11_production.33
2096128 rw------ 1 mongodb nogroup 2146435072 Nov 25 11:29
cust11_production.34
2096128 rw------ 1 mongodb nogroup 2146435072 Nov 30 07:14
cust11_production.35
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 3 03:35
cust11_production.36
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 6 00:58
cust11_production.37
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 8 20:21
cust11_production.38
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 14 08:46
cust11_production.39
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 21 16:29
cust11_production.4
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 18 07:11
cust11_production.40
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 22 10:53
cust11_production.41
2096128 rw------ 1 mongodb nogroup 2146435072 Dec 28 23:55
cust11_production.42
2096128 rw------ 1 mongodb nogroup 2146435072 Jan 1 12:53
cust11_production.43
2096128 rw------ 1 mongodb nogroup 2146435072 Jan 6 14:18
cust11_production.44
2096128 rw------ 1 mongodb nogroup 2146435072 Jan 25 13:59
cust11_production.45
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 1 22:22
cust11_production.46
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 7 21:05
cust11_production.47
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 14 11:40
cust11_production.48
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 17 22:30
cust11_production.49
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 08:24
cust11_production.5
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 21 23:38
cust11_production.50
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 08:24
cust11_production.6
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 08:24
cust11_production.7
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 08:24
cust11_production.8
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 09:04
cust11_production.9
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 17:28
cust11_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 18:41
cust12-preview_production.0
131072 rw------ 1 mongodb nogroup 134217728 Jan 27 18:41
cust12-preview_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 18:41
cust12-preview_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 22:48
cust12_production.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015
cust12_production.1
2096128 rw------ 1 mongodb mongodb 2146435072 Aug 13 2015
cust12_production.10
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015
cust12_production.2
524288 rw------ 1 mongodb mongodb 536870912 Apr 28 2015
cust12_production.3
1048576 rw------ 1 mongodb mongodb 1073741824 Apr 28 2015
cust12_production.4
2096128 rw------ 1 mongodb mongodb 2146435072 May 3 2016
cust12_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust12_production.6
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust12_production.7
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 30 2015
cust12_production.8
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 27 22:48
cust12_production.9
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 22:48
cust12_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 10 2016 f2development.0
131072 rw------ 1 mongodb mongodb 134217728 Oct 20 17:08 f2development.1
16384 rw------ 1 mongodb mongodb 16777216 Oct 20 17:08 f2development.ns
65536 rw------ 1 mongodb mongodb 67108864 Apr 28 2015
f2-printer_production.0
16384 rw------ 1 mongodb mongodb 16777216 Apr 28 2015
f2-printer_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Nov 18 2015 f2production.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015 f2production.1
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015 f2production.2
524288 rw------ 1 mongodb mongodb 536870912 Feb 4 20:01 f2production.3
1048576 rw------ 1 mongodb mongodb 1073741824 Feb 22 02:27 f2production.4
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 02:27 f2production.5
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 02:27 f2production.ns
65536 rw------ 1 mongodb mongodb 67108864 Apr 28 2015 f2sandbox.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015 f2sandbox.1
2096128 rw------ 1 mongodb mongodb 2146435072 Oct 26 15:48 f2sandbox.10
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 16 15:23 f2sandbox.11
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 21 16:24 f2sandbox.12
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015 f2sandbox.2
524288 rw------ 1 mongodb mongodb 536870912 Apr 28 2015 f2sandbox.3
1048576 rw------ 1 mongodb mongodb 1073741824 Feb 21 16:24 f2sandbox.4
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 07:32 f2sandbox.5
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 07:32 f2sandbox.6
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 18 16:24 f2sandbox.7
2096128 rw------ 1 mongodb mongodb 2146435072 Jun 21 2016 f2sandbox.8
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 07:32 f2sandbox.9
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 07:32 f2sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 18:42
cust13_production.0
131072 rw------ 1 mongodb mongodb 134217728 Jan 27 22:48
cust13_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 18:42
cust13_production.ns
65540 rw------ 1 mongodb mongodb 67108864 Jan 27 18:43
cust14_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 22:48
cust14_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 18:43
cust14_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 18:45
cust15_production.0
131072 rw------ 1 mongodb mongodb 134217728 Jan 27 22:48
cust15_production.1
262144 rw------ 1 mongodb mongodb 268435456 Jan 27 18:45
cust15_production.2
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 18:45
cust15_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jun 13 2016
cust16_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 22 02:38
cust16_production.1
262148 rw------ 1 mongodb mongodb 268435456 Feb 22 02:11
cust16_production.2
524292 rw------ 1 mongodb mongodb 536870912 Feb 22 02:18
cust16_production.3
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 22 02:54
cust16_production.4
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 02:18
cust16_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 02:18
cust16_production.6
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 02:18
cust16_production.7
2096132 rw------ 1 mongodb nogroup 2146435072 Feb 22 02:54
cust16_production.8
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 22 02:54
cust16_production.9
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 02:54
cust16_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 19:05 hotel_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 19:05 hotel_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 19:05
hotel_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Aug 25 18:51
cust17_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 22 10:48
cust17_production.1
262148 rw------ 1 mongodb mongodb 268435456 Feb 22 09:11
cust17_production.2
524292 rw------ 1 mongodb mongodb 536870912 Feb 21 23:15
cust17_production.3
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 22 10:48
cust17_production.4
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 22 00:05
cust17_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 10:48
cust17_production.6
2096132 rw------ 1 mongodb mongodb 2146435072 Feb 21 23:40
cust17_production.7
2096132 rw------ 1 mongodb nogroup 2146435072 Feb 22 00:05
cust17_production.8
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 10:48
cust17_production.ns
0 drwxr-xr-x 2 mongodb mongodb 27 Feb 22 17:35 journal
65536 rw------ 1 mongodb nogroup 67108864 Feb 10 21:04
cust18_production.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 10 21:04
cust18_production.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 10 21:04
cust18_production.ns
65540 rw------ 1 mongodb mongodb 67108864 Jan 27 22:48
cust19_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 21 20:25
cust19_production.1
262148 rw------ 1 mongodb mongodb 268435456 Feb 2 16:02
cust19_production.2
524292 rw------ 1 mongodb mongodb 536870912 Feb 21 20:25
cust19_production.3
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 2 16:02
cust19_production.4
16384 rw------ 1 mongodb mongodb 16777216 Feb 21 20:25
cust19_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Oct 27 20:29 cust19_sandbox.0
131076 rw------ 1 mongodb mongodb 134217728 Oct 27 20:25 cust19_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 Oct 5 19:31 cust19_sandbox.ns
131008 rw------ 1 mongodb mongodb 67108864 Feb 22 17:35 local.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015 local.1
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 13 17:40 local.10
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 17 13:47 local.11
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 21 15:38 local.12
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 28 06:10 local.13
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 1 01:50 local.14
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 6 03:41 local.15
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 11 23:01 local.16
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 20 16:08 local.17
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 27 23:07 local.18
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 31 20:51 local.19
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015 local.2
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 1 22:30 local.20
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 2 20:51 local.21
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 3 14:38 local.22
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 4 00:27 local.23
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 7 23:31 local.24
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 9 22:24 local.25
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 13 22:08 local.26
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 17:28 local.27
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 17:40 local.3
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 17 21:41 local.4
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 21 17:07 local.5
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 17:28 local.6
2096128 rw------ 1 mongodb mongodb 2146435072 Feb 22 17:28 local.7
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 4 19:35 local.8
2096128 rw------ 1 mongodb mongodb 2146435072 Dec 8 00:47 local.9
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 17:40 local.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 19:46
master_demo_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 12 21:54
master_demo_production.1
16384 rw------ 1 mongodb mongodb 16777216 Feb 12 21:54
master_demo_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 21 23:33
cust20_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 21 23:33
cust20_production.1
262148 rw------ 1 mongodb nogroup 268435456 Jan 27 19:47
cust20_production.2
16384 rw------ 1 mongodb mongodb 16777216 Feb 21 23:33
cust20_production.ns
65536 rw------ 1 mongodb mongodb 67108864 May 5 2016 cust20_sandbox.0
131076 rw------ 1 mongodb mongodb 134217728 Oct 27 20:29 cust20_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 Sep 13 19:19 cust20_sandbox.ns
4 -rwxr-xr-x 1 mongodb mongodb 6 Feb 22 17:21 mongod.lock
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 19:49
cust21_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 22:48
cust21_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 19:50
cust21_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 19:50
cust22_production.0
131076 rw------ 1 mongodb mongodb 134217728 Nov 10 22:04
cust22_production.1
262148 rw------ 1 mongodb mongodb 268435456 Jan 27 19:50
cust22_production.2
524292 rw------ 1 mongodb mongodb 536870912 Jan 27 19:50
cust22_production.3
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 19:50
cust22_production.ns
65540 rw------ 1 mongodb mongodb 67108864 Jan 27 19:51
cust23_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 22:48
cust23_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 19:51
cust23_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 20:28
cust24_production.0
131072 rw------ 1 mongodb mongodb 134217728 Aug 26 19:57
cust24_production.1
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.10
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.11
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.12
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.13
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.14
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.15
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.16
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.17
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.18
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.19
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015
cust24_production.2
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.20
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.21
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.22
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.23
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.24
2096128 rw------ 1 mongodb mongodb 2146435072 Jun 30 2015
cust24_production.25
2096128 rw------ 1 mongodb mongodb 2146435072 Sep 18 2015
cust24_production.26
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 29 2016
cust24_production.27
2096128 rw------ 1 mongodb mongodb 2146435072 Mar 4 2016
cust24_production.28
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 27 2016
cust24_production.29
524288 rw------ 1 mongodb mongodb 536870912 Jan 27 23:09
cust24_production.3
2096128 rw------ 1 mongodb mongodb 2146435072 Jun 28 2016
cust24_production.30
1048576 rw------ 1 mongodb mongodb 1073741824 Apr 28 2015
cust24_production.4
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.5
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.6
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 27 20:28
cust24_production.7
2096128 rw------ 1 mongodb mongodb 2146435072 Jan 27 20:28
cust24_production.8
2096128 rw------ 1 mongodb mongodb 2146435072 Apr 28 2015
cust24_production.9
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 20:28
cust24_production.ns
65540 rw------ 1 mongodb mongodb 67108864 Feb 22 06:44
cust25_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 22 10:54
cust25_production.1
262148 rw------ 1 mongodb nogroup 268435456 Feb 22 10:54
cust25_production.2
16384 rw------ 1 mongodb mongodb 16777216 Feb 22 10:53
cust25_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Aug 9 2016 cust25_sandbox.0
131076 rw------ 1 mongodb mongodb 134217728 Nov 4 20:04 cust25_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 Nov 4 20:04 cust25_sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 20 17:22
shared_demo_production.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015
shared_demo_production.1
262144 rw------ 1 mongodb mongodb 268435456 Apr 28 2015
shared_demo_production.2
524288 rw------ 1 mongodb mongodb 536870912 Feb 21 21:39
shared_demo_production.3
1048580 rw------ 1 mongodb mongodb 1073741824 Feb 21 21:39
shared_demo_production.4
2096132 rw------ 1 mongodb nogroup 2146435072 Feb 15 17:02
shared_demo_production.5
16384 rw------ 1 mongodb mongodb 16777216 Feb 21 21:39
shared_demo_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 20:28
cust26_production.0
131076 rw------ 1 mongodb nogroup 134217728 Jan 27 20:28
cust26_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 20:28
cust26_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Feb 13 22:53
sso-azure-adfs_sandbox.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 15 19:09
sso-azure-adfs_sandbox.1
16384 rw------ 1 mongodb nogroup 16777216 Feb 15 19:09
sso-azure-adfs_sandbox.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 18:42
sso-okta_production.0
131072 rw------ 1 mongodb nogroup 134217728 Jan 27 18:42
sso-okta_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 18:42
sso-okta_production.ns
65536 rw------ 1 mongodb mongodb 67108864 May 10 2016 sso-okta_sandbox.0
131072 rw------ 1 mongodb mongodb 134217728 Apr 28 2015 sso-okta_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 May 10 2016
sso-okta_sandbox.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 18:42
sso-onelogin_production.0
131072 rw------ 1 mongodb nogroup 134217728 Jan 27 18:42
sso-onelogin_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 18:42
sso-onelogin_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 14 16:48
sso-onelogin_sandbox.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 14 16:48
sso-onelogin_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 Feb 14 16:48
sso-onelogin_sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Feb 17 21:22
cust27_production.0
131072 rw------ 1 mongodb mongodb 134217728 Feb 21 20:34
cust27_production.1
262148 rw------ 1 mongodb mongodb 268435456 Feb 21 20:34
cust27_production.2
16384 rw------ 1 mongodb mongodb 16777216 Feb 21 20:34
cust27_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 20:28
cust28_production.0
131076 rw------ 1 mongodb nogroup 134217728 Jan 27 20:28
cust28_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 20:28
cust28_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 27 18:42
tryonelogin_production.0
131072 rw------ 1 mongodb nogroup 134217728 Jan 27 18:42
tryonelogin_production.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 27 18:42
tryonelogin_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 17 19:55
tryonelogin_sandbox.0
131072 rw------ 1 mongodb mongodb 134217728 Jan 17 19:55
tryonelogin_sandbox.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 17 19:55
tryonelogin_sandbox.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 20:28
usability_production.0
131072 rw------ 1 mongodb mongodb 134217728 Jan 27 20:28
usability_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 20:28
usability_production.ns
65536 rw------ 1 mongodb mongodb 67108864 Jan 27 20:28
cust29_production.0
131076 rw------ 1 mongodb mongodb 134217728 Jan 27 20:28
cust29_production.1
16384 rw------ 1 mongodb mongodb 16777216 Jan 27 20:28
cust29_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Feb 21 23:38
cust30_production.0
131076 rw------ 1 mongodb nogroup 134217728 Feb 22 10:37
cust30_production.1
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 8 23:22
cust30_production.10
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 14 14:37
cust30_production.11
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 17 20:50
cust30_production.12
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 22 00:41
cust30_production.13
262148 rw------ 1 mongodb nogroup 268435456 Feb 22 00:45
cust30_production.2
524292 rw------ 1 mongodb nogroup 536870912 Feb 22 10:37
cust30_production.3
1048576 rw------ 1 mongodb nogroup 1073741824 Feb 17 16:58
cust30_production.4
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 22 00:41
cust30_production.5
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 21 19:14
cust30_production.6
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 22 00:41
cust30_production.7
2096132 rw------ 1 mongodb nogroup 2146435072 Feb 22 00:41
cust30_production.8
2096128 rw------ 1 mongodb nogroup 2146435072 Feb 6 16:29
cust30_production.9
16384 rw------ 1 mongodb nogroup 16777216 Feb 22 10:37
cust30_production.ns
65536 rw------ 1 mongodb nogroup 67108864 Jan 6 20:02 cust30_sandbox.0
131076 rw------ 1 mongodb nogroup 134217728 Jan 6 21:41 cust30_sandbox.1
16384 rw------ 1 mongodb nogroup 16777216 Jan 6 21:41 cust30_sandbox.ns

production-rs:PRIMARY> db.getSiblingDB("local").oplog.rs.stats()
{
"ns" : "local.oplog.rs",
"count" : 5545163,
"size" : 51393540916,
"avgObjSize" : 9268,
"storageSize" : 51495505552,
"numExtents" : 24,
"nindexes" : 0,
"lastExtentSize" : 2127687680,
"paddingFactor" : 1,
"systemFlags" : 0,
"userFlags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"capped" : true,
"max" : NumberLong("9223372036854775807"),
"ok" : 1
}

1e-26 after adding 1c to replica set:

production-rs:PRIMARY> rs.status()
{
"set" : "production-rs",
"date" : ISODate("2017-02-22T19:44:23Z"),
"myState" : 1,
"members" : [

{ "_id" : 1, "name" : "1e-26:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 8589, "optime" : Timestamp(1487792659, 1), "optimeDate" : ISODate("2017-02-22T19:44:19Z"), "electionTime" : Timestamp(1487785236, 1), "electionDate" : ISODate("2017-02-22T17:40:36Z"), "self" : true }

,

{ "_id" : 2, "name" : "1c:27017", "health" : 1, "state" : 5, "stateStr" : "STARTUP2", "uptime" : 4, "optime" : Timestamp(0, 0), "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2017-02-22T19:44:21Z"), "lastHeartbeatRecv" : ISODate("2017-02-22T19:44:22Z"), "pingMs" : 1 }

],
"ok" : 1
}

1c after adding 1c to replica set:
production-rs:STARTUP2> rs.status()
{
"set" : "production-rs",
"date" : ISODate("2017-02-22T19:45:27.894Z"),
"myState" : 5,
"syncingTo" : "1e-26:27017",
"members" : [

{ "_id" : 1, "name" : "1e-26:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 67, "optime" : Timestamp(1487792659, 1), "optimeDate" : ISODate("2017-02-22T19:44:19Z"), "lastHeartbeat" : ISODate("2017-02-22T19:45:26.499Z"), "lastHeartbeatRecv" : ISODate("2017-02-22T19:45:27.790Z"), "pingMs" : 0, "electionTime" : Timestamp(1487785236, 1), "electionDate" : ISODate("2017-02-22T17:40:36Z"), "configVersion" : 91106 }

,

{ "_id" : 2, "name" : "1c:27017", "health" : 1, "state" : 5, "stateStr" : "STARTUP2", "uptime" : 447, "optime" : Timestamp(0, 0), "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "syncingTo" : "1e-26:27017", "configVersion" : 91106, "self" : true }

],
"ok" : 1
}

1c after shutdown:

rdecatrel@1c:~$ ls -lsa /var/lib/mongodb
total 53910668
4 drwxrwxrwx 4 mongodb mongodb 4096 Feb 22 19:46 .
4 drwxr-xr-x 45 root root 4096 Feb 22 19:37 ..
65536 rw------ 1 mongodb mongodb 67108864 Feb 22 19:44 cust3_production.0
131076 rw------ 1 mongodb mongodb 134217728 Feb 22 19:44 cust3_production.1
16384 rw------ 1 mongodb mongodb 16777216 Feb 2...

Comment by Ralph Decatrel [ 21/Feb/17 ]

I will gather the information your requested. The EBS snapshot is an exact
copy of the dbpath of 1e-26 when the 1c is brought back online. Would you
expect local.oplog.rs to have something different than whatever was in
1e-26?

On Tue, Feb 21, 2017, 4:43 PM Thomas Schubert (JIRA) <jira@mongodb.org>


Ralph <http://about.me/justronin>
ralph@y12k.com

Comment by Kelsey Schubert [ 21/Feb/17 ]

Hi justronin,

It appears that there were no entries in the local.oplog.rs collection. As a result, MongoDB performed an initial sync. Please review SERVER-17710 for additional details around this behavior and suggested improvements.

If you would like us to continue to investigate, please provide the following:

  1. a complete listing of the dbpath (e.g. ls -lsa) of 1e-26
  2. a complete listing of the dbpath of 1c before restart
  3. a complete listing of dbpath of 1c after the restart
  4. output of the following command executed against 1e-26:

     db.getSiblingDB("local").oplog.rs.stats() 

Thank you,
Thomas

Generated at Thu Feb 08 04:16:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.