Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33178

Server Crashes on Startup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: 3.4.10, 3.4.11
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL

      Description

      I recently upgraded an Ubuntu 14 server, running Mongo 3.4.10, to Ubuntu 16. I re-installed the identical version of mongodb-org from your PPA, but now when I start mongo, it's doesn't respond to any connections, and I see this error in `/var/log/mongodb/mongodb.log`:

          2018-02-06T18:40:15.680+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] MongoDB starting : pid=15925 port=27017 dbpath=/var/lib/mongodb 64-bit host=proddb1
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] db version v3.4.11
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] git version: 34f5bec2c9d827d71828fe858167f89a28b29a2a
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] allocator: tcmalloc
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] modules: none
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] build environment:
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten]     distarch: x86_64
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten]     target_arch: x86_64
          2018-02-06T18:40:15.684+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
          2018-02-06T18:40:15.684+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
          2018-02-06T18:40:15.709+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
          2018-02-06T18:40:15.709+0000 I STORAGE  [initandlisten] 
          2018-02-06T18:40:15.709+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
          2018-02-06T18:40:15.709+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
          2018-02-06T18:40:15.709+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15574M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
          2018-02-06T18:40:16.468+0000 I STORAGE  [initandlisten] WiredTiger message [1517942416:468088][15925:0x7f4d31632d00], txn-recover: Main recovery loop: starting at 12168/128
          2018-02-06T18:40:16.468+0000 I STORAGE  [initandlisten] WiredTiger message [1517942416:468689][15925:0x7f4d31632d00], txn-recover: Recovering log 12168 through 12169
          2018-02-06T18:40:16.528+0000 I STORAGE  [initandlisten] WiredTiger message [1517942416:528271][15925:0x7f4d31632d00], txn-recover: Recovering log 12169 through 12169
          2018-02-06T18:40:17.875+0000 E STORAGE  [initandlisten] WiredTiger error (24) [1517942417:875171][15925:0x7f4d31632d00], file:collection-43442-4253276309270751377.wt, WT_SESSION.open_cursor: /var/lib/mongodb/collection-43442-4253276309270751377.wt: handle-open: open: Too many open files
          2018-02-06T18:40:17.875+0000 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError: 24: Too many open files at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 95
          2018-02-06T18:40:17.875+0000 I -        [initandlisten] 
       
          ***aborting after invariant() failure
      

      Googling, I found some people suggesting I need to increase my limits, so I tried modifying sudo nano /etc/security/limits.conf by adding:

      • soft fsize unlimited
      • hard fsize unlimited
      • soft cpu unlimited
      • hard cpu unlimited
      • soft as unlimited
      • hard as unlimited
      • soft nofile 64000
      • hard nofile 64000
      • soft rss unlimited
      • hard rss unlimited
      • soft nproc 64000
      • hard nproc 64000

      I also added `session required pam_limits.so` to `/etc/pam.d/common-session` and `/etc/pam.d/common-session-noninteractive`, and then rebooted. However, I get the same error.

      Think it might be data corruption, I tried running `sudo -u mongodb mongod --repair --dbpath /var/lib/mongodb` but after several minutes of rebuilding indexes, that too failed with a similar crash report:

          2018-02-07T21:55:46.255+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-43442-4253276309270751377. Not salvaging.
          2018-02-07T21:55:46.757+0000 E STORAGE  [initandlisten] WiredTiger error (24) [1518040546:757829][1803:0x7f90823bcd00], WT_SESSION.create: /var/lib/mongodb/: directory-sync: open: Too many open files
          2018-02-07T21:55:47.025+0000 I -        [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195
          2018-02-07T21:55:47.025+0000 I -        [initandlisten] 
       
          ***aborting after invariant() failure
       
       
          2018-02-07T21:55:47.046+0000 F -        [initandlisten] Got signal: 6 (Aborted).
      

        Attachments

          Activity

            People

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            chrisspen Chris Spencer
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: