[SERVER-2290] Broken master-slave replication Created: 27/Dec/10  Updated: 12/Jul/16  Resolved: 29/Dec/10

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 1.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Pawe? T?cza Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu Lucid 10.04 LTS as Xen DomU under control of Linux 2.6.32-5-xen-amd64


Attachments: File mongodb.log.gz    
Operating System: Linux
Participants:

 Description   

I already reported my problem with master-slave replication on mongodb-users mailing list:

http://groups.google.com/group/mongodb-user/browse_thread/thread/55e4c9afbd97dedc?hl=en

Now I file a new issue for uploading the log file of my slave, because Eliot Horowitz asked me for it



 Comments   
Comment by Pawe? T?cza [ 29/Dec/10 ]

I just used your instruction

http://www.mongodb.org/display/DOCS/Starting+and+Stopping+Mongo

Sending a Unix INT or TERM signal
You can cleanly stop mongod using a SIGINT or SIGTERM signal on Unix-like systems. Either ^C, "kill -2 PID," or kill -15 PID will work.

Comment by Eliot Horowitz (Inactive) [ 29/Dec/10 ]

You should just be using kill, not kill -2. or kill -3 is ok

Comment by Pawe? T?cza [ 29/Dec/10 ]

I was thinking that SIGINT signal cleanly stops mongod and removes the lock file, but I was wrong:

  1. kill -2 `pidof mongod`
  2. pidof mongod
    #
  3. ls
    local local.1 local.ns mdstore.0 mdstore.ns mydb mydb.1
    local.0 local.2 mdstore mdstore.1 mongod.lock mydb.0 mydb.ns

It seems that --repair also creates the lock file:

  1. rm -f mongod.lock local.*
  2. mongod --repair --dbpath /srv/mongodb/
    [...]
    Tue Dec 28 22:18:58 [initandlisten] shutdown: going to close listening sockets...
    Tue Dec 28 22:18:58 [initandlisten] shutdown: going to flush oplog...
    Tue Dec 28 22:18:58 [initandlisten] shutdown: going to close sockets...
    Tue Dec 28 22:18:58 [initandlisten] shutdown: waiting for fs preallocator...
    Tue Dec 28 22:18:58 [initandlisten] shutdown: closing all files...
    Tue Dec 28 22:18:58 closeAllFiles() finished

Tue Dec 28 22:18:58 [initandlisten] shutdown: removing fs lock...
Tue Dec 28 22:18:58 dbexit: really exiting now

  1. ls
    local mdstore mdstore.0 mdstore.1 mdstore.ns mongod.lock mydb mydb.0 mydb.1 mydb.ns _tmp

but probably it's empty. Unfortunately I didn't check it. I guess that existence of the lock file is not a problem for Ubuntu init script if it doesn't contain PID of mongod.

Probably I found the source of my issues with starting/stopping mongod using the init script. I don't know a reason, but here mongoDB files belonged to root:root instead of mongodb:nogroup:

  1. cd /srv/mongodb/
  2. ls -l
    total 1687576
    drwxr-xr-x 2 root root 6 2010-12-28 15:56 local
    rw------ 1 root root 67108864 2010-12-28 15:57 local.0
    rw------ 1 root root 134217728 2010-12-28 15:57 local.1
    rw------ 1 root root 1073741824 2010-12-28 22:15 local.2
    rw------ 1 root root 16777216 2010-12-28 22:15 local.ns
    drwxr-xr-x 2 root root 6 2010-12-28 11:46 mdstore
    rw------ 1 root root 67108864 2010-12-28 15:57 mdstore.0
    rw------ 1 root root 134217728 2010-12-28 15:57 mdstore.1
    rw------ 1 root root 16777216 2010-12-28 15:57 mdstore.ns
    -rwxr-xr-x 1 mongodb nogroup 6 2010-12-28 15:59 mongod.lock
    drwxr-xr-x 2 root root 6 2010-12-28 11:46 mydb
    rw------ 1 root root 67108864 2010-12-28 15:57 mydb.0
    rw------ 1 root root 134217728 2010-12-28 15:57 mydb.1
    rw------ 1 root root 16777216 2010-12-28 15:57 mydb.ns

I fixed it and now I don't need to run mongod via `/usr/bin/mongod --config /etc/mongodb.conf`

Comment by Eliot Horowitz (Inactive) [ 28/Dec/10 ]

Yes, before or after running repair you can remove the lock file.

Comment by Pawe? T?cza [ 28/Dec/10 ]

Should I delete the lock file manually before --repair command? I left mongod.lock and it seems that --repair didn't removed it.

  1. cd /srv/mongodb/
  2. ls
    local.0 local.2 mdstore mdstore.1 mongod.lock mydb.0 mydb.ns
    local.1 local.ns mdstore.0 mdstore.ns mydb mydb.1 _tmp
  1. mongod --repair --dbpath /srv/mongodb/
    Tue Dec 28 15:57:53 MongoDB starting : pid=24773 port=27017 dbpath=/srv/mongodb/ 64-bit
    Tue Dec 28 15:57:53 db version v1.6.4, pdfile version 4.5
    Tue Dec 28 15:57:53 git version: 4f5c02f8d92ff213b71b88f5eb643b7f62b50abc
    Tue Dec 28 15:57:53 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] need to upgrade database mydb with pdfile version 4.5, new version: 4.5
    Tue Dec 28 15:57:53 [initandlisten] starting upgrade
    Tue Dec 28 15:57:53 [initandlisten] mydb repairDatabase mydb
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.ns, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.ns, size: 16MB, took 0.046 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.0, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.0, size: 64MB, took 0.001 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.1, filling with zeroes...
    Tue Dec 28 15:57:53 [initandlisten] building new index on { _id: 1 }

    for mydb.mydb
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mydb.foo
    Tue Dec 28 15:57:53 [initandlisten] done for 1 records 0.001secs
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mydb.1, size: 128MB, took 0.022 secs
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] need to upgrade database mdstore with pdfile version 4.5, new version: 4.5
    Tue Dec 28 15:57:53 [initandlisten] starting upgrade
    Tue Dec 28 15:57:53 [initandlisten] mdstore repairDatabase mdstore
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.ns, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.ns, size: 16MB, took 0 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.0, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.0, size: 64MB, took 0 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.1, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/mdstore.1, size: 128MB, took 0 secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mdstore.metadata
    Tue Dec 28 15:57:53 [initandlisten] done for 6 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mdstore.a4486da6-5cdb-44b4-823d-de0ad015b998_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mdstore.4040cd6e-2da3-46fa-80eb-58ea3fd32d02_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mdstore.69cee54d-6f18-4b0e-86fd-c75e24bfe476_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for mdstore.e36423ad-2c05-4f42-bd47-187e5d6f0893_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { id: 1 }

    for mdstore.a4486da6-5cdb-44b4-823d-de0ad015b998_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { timestamp: 1 }

    for mdstore.a4486da6-5cdb-44b4-823d-de0ad015b998_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { originalId: 1 }

    for mdstore.a4486da6-5cdb-44b4-823d-de0ad015b998_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { id: 1 }

    for mdstore.4040cd6e-2da3-46fa-80eb-58ea3fd32d02_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { timestamp: 1 }

    for mdstore.4040cd6e-2da3-46fa-80eb-58ea3fd32d02_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { originalId: 1 }

    for mdstore.4040cd6e-2da3-46fa-80eb-58ea3fd32d02_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 2 records 0secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { id: 1 }

    for mdstore.e36423ad-2c05-4f42-bd47-187e5d6f0893_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { timestamp: 1 }

    for mdstore.e36423ad-2c05-4f42-bd47-187e5d6f0893_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { originalId: 1 }

    for mdstore.e36423ad-2c05-4f42-bd47-187e5d6f0893_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { id: 1 }

    for mdstore.69cee54d-6f18-4b0e-86fd-c75e24bfe476_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { timestamp: 1 }

    for mdstore.69cee54d-6f18-4b0e-86fd-c75e24bfe476_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.001secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { originalId: 1 }

    for mdstore.69cee54d-6f18-4b0e-86fd-c75e24bfe476_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==
    Tue Dec 28 15:57:53 [initandlisten] done for 237 records 0.002secs
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] ****
    Tue Dec 28 15:57:53 [initandlisten] need to upgrade database local with pdfile version 4.5, new version: 4.5
    Tue Dec 28 15:57:53 [initandlisten] starting upgrade
    Tue Dec 28 15:57:53 [initandlisten] local repairDatabase local
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/local.ns, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/local.ns, size: 16MB, took 0.017 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/local.0, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/local.0, size: 64MB, took 0 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/local.1, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/local.1, size: 128MB, took 0 secs
    Tue Dec 28 15:57:53 allocating new datafile /srv/mongodb/$tmp_repairDatabase_0/local.2, filling with zeroes...
    Tue Dec 28 15:57:53 done allocating datafile /srv/mongodb/$tmp_repairDatabase_0/local.2, size: 1024MB, took 0.001 secs
    Tue Dec 28 15:57:53 [initandlisten] building new index on

    { _id: 1 }

    for local.slaves
    Tue Dec 28 15:57:53 [initandlisten] done for 1 records 0secs
    Tue Dec 28 15:57:53 [initandlisten] finished checking dbs
    Tue Dec 28 15:57:53 dbexit:

Tue Dec 28 15:57:53 [initandlisten] shutdown: going to close listening sockets...
Tue Dec 28 15:57:53 [initandlisten] shutdown: going to flush oplog...
Tue Dec 28 15:57:53 [initandlisten] shutdown: going to close sockets...
Tue Dec 28 15:57:53 [initandlisten] shutdown: waiting for fs preallocator...
Tue Dec 28 15:57:53 [initandlisten] shutdown: closing all files...
Tue Dec 28 15:57:53 closeAllFiles() finished

Tue Dec 28 15:57:53 [initandlisten] shutdown: removing fs lock...
Tue Dec 28 15:57:53 dbexit: really exiting now

  1. ls
    local local.1 local.ns mdstore.0 mdstore.ns mydb mydb.1 _tmp
    local.0 local.2 mdstore mdstore.1 mongod.lock mydb.0 mydb.ns

I'm not sure if I still need to remove local.* files...

I had a problem with starting mongoDB daemon using the init script:

  1. /etc/init.d/mongodb start
    Rather than invoking init scripts through /etc/init.d, use the service(8)
    utility, e.g. service mongodb start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start mongodb
mongodb start/running, process 24876

  1. pidof mongod
    #
  2. ps uax |grep mongod
    root 24940 0.0 0.0 7620 912 pts/1 S+ 15:59 0:00 grep --color=auto mongod

Finally I ran the daemon manually:

  1. /usr/bin/mongod --config /etc/mongodb.conf -vvvv
    all output going to: /var/log/mongodb/mongodb.log

^Z
[1]+ Stopped /usr/bin/mongod --config /etc/mongodb.conf -vvvv

  1. bg
    [1]+ /usr/bin/mongod --config /etc/mongodb.conf -vvvv &
  2. pidof mongod
    24963
Comment by Eliot Horowitz (Inactive) [ 28/Dec/10 ]

No - you should run --repair now!
If the server crashed and you don't, you could be making the situation worse.

Comment by Pawe? T?cza [ 28/Dec/10 ]

Good to know it. Thanks! I promise I will run --repair next time

Comment by Eliot Horowitz (Inactive) [ 28/Dec/10 ]

You should never remove the lock file without repairing.
I would stop mongod and run a repair asap

Comment by Pawe? T?cza [ 28/Dec/10 ]

I restarted my slave. Please note that I had to delete the lock file manually, because the init script didn't do it:

  1. ps aux |grep mongo
    root 3389 0.0 0.0 7620 904 pts/0 S+ 08:40 0:00 grep --color=auto mongo
    mongodb 10641 0.0 1.3 400472 57112 ? Ssl Dec27 0:19 /usr/bin/mongod --config /etc/mongodb.conf
  1. /etc/init.d/mongodb stop
    Rather than invoking init scripts through /etc/init.d, use the service(8)
    utility, e.g. service mongodb stop

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the stop(8) utility, e.g. stop mongodb
mongodb stop/waiting

  1. ps aux |grep mongo
    root 3411 0.0 0.0 7620 904 pts/0 S+ 08:41 0:00 grep --color=auto mongo
  1. ls /srv/mongodb/
    local.0 local.1 local.ns mdstore.0 mdstore.1 mdstore.ns mongod.lock mydb.0 mydb.1 mydb.ns _tmp
  1. less /var/log/mongodb/mongodb.log
    [...]
    Tue Dec 28 08:40:52 [conn8] end connection 127.0.0.1:51479
    Tue Dec 28 08:40:55 [replslave] repl: from host:masterhost
    Tue Dec 28 08:40:59 [replslave] repl: from host:masterhost
    Tue Dec 28 08:41:03 [replslave] repl: from host:masterhost
    Tue Dec 28 08:41:07 [replslave] repl: from host:masterhost
    Tue Dec 28 08:41:12 [replslave] repl: from host:masterhost
    Tue Dec 28 08:41:13 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends
    Tue Dec 28 08:41:13 [interruptThread] now exiting
    Tue Dec 28 08:41:13 dbexit:

Tue Dec 28 08:41:13 [interruptThread] shutdown: going to close listening sockets...
Tue Dec 28 08:41:13 [interruptThread] closing listening socket: 5
Tue Dec 28 08:41:13 [interruptThread] closing listening socket: 6
Tue Dec 28 08:41:13 [interruptThread] closing listening socket: 7
Tue Dec 28 08:41:13 [interruptThread] closing listening socket: 8
Tue Dec 28 08:41:13 [interruptThread] shutdown: going to flush oplog...
Tue Dec 28 08:41:13 [interruptThread] shutdown: going to close sockets...
Tue Dec 28 08:41:13 [interruptThread] shutdown: waiting for fs preallocator...
Tue Dec 28 08:41:13 [interruptThread] shutdown: closing all files...
Tue Dec 28 08:41:13 closeAllFiles() finished

Tue Dec 28 08:41:13 [interruptThread] shutdown: removing fs lock...
Tue Dec 28 08:41:13 dbexit: really exiting now

  1. rm -f /srv/mongodb/mongod.lock
  1. /usr/bin/mongod -vvvv --config /etc/mongodb.conf
    all output going to: /var/log/mongodb/mongodb.log
    ^Z
    [1]+ Stopped /usr/bin/mongod -vvvv --config /etc/mongodb.conf
  1. bg
    [1]+ /usr/bin/mongod -vvvv --config /etc/mongodb.conf &
  1. less /var/log/mongodb/mongodb.log
    [...]
    Tue Dec 28 08:44:31 MongoDB starting : pid=3609 port=27017 dbpath=/srv/mongodb slave=1 64-bit
    Tue Dec 28 08:44:31 db version v1.6.4, pdfile version 4.5
    Tue Dec 28 08:44:31 git version: 4f5c02f8d92ff213b71b88f5eb643b7f62b50abc
    Tue Dec 28 08:44:31 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009
    x86_64 BOOST_LIB_VERSION=1_41
    Tue Dec 28 08:44:31 [initandlisten] query: local.system.namespaces { name: /^local.temp./ }

    Tue Dec 28 08:44:31 [initandlisten] Accessing: local for the first time
    Tue Dec 28 08:44:31 [initandlisten] query local.system.namespaces reslen:36 nreturned:0 0ms
    Tue Dec 28 08:44:31 [initandlisten] enter repairDatabases (to check pdfile version #)
    Tue Dec 28 08:44:31 [initandlisten] local
    Tue Dec 28 08:44:31 [initandlisten] mdstore
    Tue Dec 28 08:44:31 [initandlisten] Accessing: mdstore for the first time
    Tue Dec 28 08:44:31 [initandlisten] mydb
    Tue Dec 28 08:44:31 [initandlisten] Accessing: mydb for the first time
    Tue Dec 28 08:44:31 [initandlisten] done repairDatabases
    Tue Dec 28 08:44:31 [initandlisten] waiting for connections on port 27017
    Tue Dec 28 08:44:31 [initandlisten] Accessing: local for the first time
    Tue Dec 28 08:44:31 [initandlisten] slave=true
    Tue Dec 28 08:44:31 [initandlisten] fd limit hard:1024 soft:1024 max conn: 819
    Tue Dec 28 08:44:31 [websvr] web admin interface listening on port 28017
    Tue Dec 28 08:44:31 [websvr] fd limit hard:1024 soft:1024 max conn: 819
    Tue Dec 28 08:44:32 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:32 [replslave] replHandshake res not: 1 res:

    { ok: 1.0 }

    Tue Dec 28 08:44:32 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: local.oplog.$main.find(

    Unknown macro: { ts}

    )
    Tue Dec 28 08:44:36 [replslave] repl: first op time received: 4cefaa71:1
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:36 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 129077515300 0|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:36 [replslave] repl: applied 0 operations
    Tue Dec 28 08:44:36 [replslave] repl: end sync_pullOpLog syncedTo: Nov 26 12:39:13 4cefaa71:1
    Tue Dec 28 08:44:36 [replslave] repl: sleep 2sec before next pass
    Tue Dec 28 08:44:38 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:42 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: tailing=true
    repl: tailing & no new activity
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:42 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 129077515300 0|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:42 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:46 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: tailing=true
    repl: tailing & no new activity
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:46 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 1290775153000|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:46 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:50 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: tailing=true
    repl: tailing & no new activity
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:50 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 1290775153000|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:50 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:54 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: tailing=true
    repl: tailing & no new activity
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:54 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 1290775153000|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:54 [replslave] repl: from host:masterhost
    Tue Dec 28 08:44:59 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1
    repl: tailing=true
    repl: tailing & no new activity
    updated _lastSavedLocalTs to: 0:0
    Tue Dec 28 08:44:59 [replslave] Saving repl source:

    { host: "masterhost", source: "main", syncedTo: Timestamp 1290775153000|1, localLogTs: Timestamp 0|0 }

    Tue Dec 28 08:44:59 [replslave] repl: from host:masterhost
    Tue Dec 28 08:45:03 [replslave] repl: sync_pullOpLog local.oplog.$main syncedTo:Nov 26 12:39:13 4cefaa71:1

Generated at Thu Feb 08 02:59:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.