-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.4.7
-
Component/s: WiredTiger
-
Labels:None
-
ALL
When I checked my logfiles on the primary server I noticed errors that my 3rd server in the 3 server cluster could not be reached. When checking on the server I found mongod was not running, so I tried to restart the service and got a 34433 fatal assertion.
This server is running as an arbiter only, so I am a bit surprised to see this WiredTiger error on this machine.
The output of /var/log/mongodb/mongod.log:
/var/log/mongodb/mongod.log
2017-12-07T12:26:32.253+0100 I CONTROL [main] ***** SERVER RESTARTED *****
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] MongoDB starting : pid=18337 port=27017 dbpath=/var/lib/mongodb 64-bit host=xxxxxxxxxxxxx
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] db version v3.4.7
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] allocator: tcmalloc
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] modules: none
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] build environment:
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] distmod: ubuntu1604
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] distarch: x86_64
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] target_arch: x86_64
2017-12-07T12:26:32.270+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, replication: { replSetName: "rsym" }, security: { keyFile: "/opt/mongo/mongo-keyfile" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-12-07T12:26:32.270+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-12-07T12:26:32.296+0100 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-12-07T12:26:32.296+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2017-12-07T12:26:32.296+0100 I STORAGE [initandlisten]
2017-12-07T12:26:32.296+0100 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-07T12:26:32.296+0100 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-12-07T12:26:32.296+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2017-12-07T12:26:32.311+0100 I - [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 796
2017-12-07T12:26:32.311+0100 I - [initandlisten]
***aborting after fassert() failure
Running 'mongod --dbpath /var/lib/mongodb --repair' resulted in an aborted repair, but after restarting the mongo service the fatal assertion changed to 28578.
/var/log/mongodb/mongod.log after --repair
2017-12-07T12:44:58.954+0100 I CONTROL [main] ***** SERVER RESTARTED *****
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] MongoDB starting : pid=19822 port=27017 dbpath=/var/lib/mongodb 64-bit host=xxxxxxxxxxxxx
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] db version v3.4.7
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] allocator: tcmalloc
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] modules: none
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] build environment:
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] distmod: ubuntu1604
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] distarch: x86_64
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] target_arch: x86_64
2017-12-07T12:44:58.971+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, replication: { replSetName: "rsym" }, security: { keyFile: "/opt/mongo/mongo-keyfile" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-12-07T12:44:58.971+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-12-07T12:44:58.996+0100 E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock Operation not permitted
2017-12-07T12:44:58.996+0100 I - [initandlisten] Fatal Assertion 28578 at src/mongo/util/net/listen.cpp 195
2017-12-07T12:44:58.996+0100 I - [initandlisten]
***aborting after fassert() failure
After removing the '/tmp/mongodb-27017.sock' file and chowning all files in '/var/lib/mongodb' to 'mongodb:mongodb' I am back to 34433 error when restarting the service. I did edit the logfile output and removed only my external IP address.