Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33119

Could not start Mongo DB on Centos 7 running on ESXi 5.5

    • Type: Icon: Question Question
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • None

      I deployed Mongo 3.2.13 on Centos-release-7-2.1511.el7.centos.2.10.x86_64 (3.10.0-327.el7.x86_64) running on ESXi 5.5 (this ESXi is using Local ATA Disk, see "
      Datastore ATA.jpg" image) but i could not start it.

      The log displayed :

      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] MongoDB starting : pid=13932 port=27017 dbpath=/opt/Test/data/mongodb 64-bit host=test
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] db version v3.2.13
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] modules: none
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] build environment:
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distmod: amazon
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distarch: x86_64
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] options: { config: "/opt/Test/services/mongodb/mongodb.conf", net: { bindIp: "127.0.0.1", http: { RESTInterfaceEnabled: true, enabled: true }, ipv6: true, port: 27017 }, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/opt/Test/services/mongodb/ovmongodb.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/opt/Test/data/mongodb", journal: { commitIntervalMs: 300, enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/opt/Test/services/mongodb/logs/mongodb.log", quiet: true } }
      2018-01-19T02:52:58.926-0800 I -        [initandlisten] Detected data files in /opt/Test/data/mongodb created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten]
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/Test/data/mongodb is set to 4096KB
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
      2018-01-19T02:52:58.937-0800 I JOURNAL  [initandlisten] journal dir=/opt/Test/data/mongodb/journal
      2018-01-19T02:52:59.121-0800 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
      2018-01-19T02:53:03.355-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 70.14
      2018-01-19T02:53:06.407-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 55.64
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 27.48
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster check took 10.64 secs
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.0
      2018-01-19T02:53:52.409-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.1
      2018-01-19T02:54:30.367-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.2
      2018-01-19T02:55:14.470-0800 I JOURNAL  [durability] Durability thread started
      2018-01-19T02:55:14.480-0800 I JOURNAL  [journal writer] Journal writer thread started
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] ** WARNING: The server is started with the web server interface and access control.
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          The web interfaces (rest, httpinterface and/or jsonp) are insecure
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          and should be disabled unless required for backward compatibility.
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.760-0800 I -        [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 410
      2018-01-19T02:55:14.760-0800 I -        [initandlisten]
      
      ***aborting after invariant() failure
      
      
      2018-01-19T02:55:14.854-0800 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x1328da2 0x1327cc9 0x13284d2 0x7f5031f7c100 0x7f5031be15f7 0x7f5031be2ce8 0x12a502b 0xa9eb62 0xa9ecd8 0xaa0864 0xaa4b4c 0x946a53 0x9498ca 0x94c83d 0x7f5031bcdb15 0x944d89
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F28DA2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27CC9"},{"b":"400000","o":"F284D2"},{"b":"7F5031F6D000","o":"F100"},{"b":"7F5031BAC000","o":"355F7","s":"gsignal"},{"b":"7F5031BAC000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA502B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"69EB62","s":"_ZNK5mongo8Database13getCollectionENS_10StringDataE"},{"b":"400000","o":"69ECD8","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"6A0864","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"6A4B4C","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"546A53"},{"b":"400000","o":"5498CA"},{"b":"400000","o":"54C83D","s":"main"},{"b":"7F5031BAC000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"544D89"}],"processInfo":{ "mongodbVersion" : "3.2.13", "gitVersion" : "23899209cad60aaafe114f6aea6cb83025ff51bc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.el7.x86_64", "version" : "#1 SMP Thu Nov 19 22:10:57 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D425B16F216CF164333E71CEE765F2A21A45B069" }, { "b" : "7FFE39876000", "elfType" : 3, "buildId" : "0F488C85309E4AB293D3B758748A3037B01F9885" }, { "b" : "7F5032E95000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7F5032AAD000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "6A997DC6D2CFD6702B987BF1B4926ABBA91691B8" }, { "b" : "7F50328A5000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4D22B5631EAF7D1EB6E72A42315105D14C70DC50" }, { "b" : "7F50326A1000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7F503239F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8DEE792EF5B31B20A3CD95985E8BE374D5C6A8C2" }, { "b" : "7F5032189000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7F5031F6D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4C0F15C537CF112511B6BE97DE1627C350652AA2" }, { "b" : "7F5031BAC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "B04A54C443D36058702AB4060C63F4AB3273EAE9" }, { "b" : "7F5033102000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D2E4A501823D041413EEB652B89044D1F680EE" }, { "b" : "7F5031960000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7F503167B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7F5031477000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7F5031245000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7F503102F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7F5030E20000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7F5030C1C000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F5030A02000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7F50307DD000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F503057C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7F5030357000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1328da2]
       mongod(+0xF27CC9) [0x1327cc9]
       mongod(+0xF284D2) [0x13284d2]
       libpthread.so.0(+0xF100) [0x7f5031f7c100]
       libc.so.6(gsignal+0x37) [0x7f5031be15f7]
       libc.so.6(abort+0x148) [0x7f5031be2ce8]
       mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12a502b]
       mongod(_ZNK5mongo8Database13getCollectionENS_10StringDataE+0x42) [0xa9eb62]
       mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0x28) [0xa9ecd8]
       mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xaa0864]
       mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x40C) [0xaa4b4c]
       mongod(+0x546A53) [0x946a53]
       mongod(+0x5498CA) [0x9498ca]
       mongod(main+0x15D) [0x94c83d]
       libc.so.6(__libc_start_main+0xF5) [0x7f5031bcdb15]
       mongod(+0x544D89) [0x944d89]
      -----  END BACKTRACE  -----
      

      How can i resolve this issue ?
      This is a fresh installation, and i faced this issue when starting mongo first time. i also tried to deploy the mongo on other Centos VM located on the same ESXi, but still faced the same issue.
      One strange thing, when i deployed the same mongo on Centos VM located on an ESXi 6.0 (this host is using SCSI disk), it worked fine. So i suspect the issue relates to Local DATA disk or the partition.

      Thanks,

        1. Datastore ATA.jpg
          Datastore ATA.jpg
          85 kB
        2. mongo data files.jpg
          mongo data files.jpg
          54 kB

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            qa John
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: