Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33119

Could not start Mongo DB on Centos 7 running on ESXi 5.5

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Storage
    • None

    Description

      I deployed Mongo 3.2.13 on Centos-release-7-2.1511.el7.centos.2.10.x86_64 (3.10.0-327.el7.x86_64) running on ESXi 5.5 (this ESXi is using Local ATA Disk, see "
      Datastore ATA.jpg" image) but i could not start it.

      The log displayed :

      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] MongoDB starting : pid=13932 port=27017 dbpath=/opt/Test/data/mongodb 64-bit host=test
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] db version v3.2.13
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] modules: none
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] build environment:
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distmod: amazon
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distarch: x86_64
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] options: { config: "/opt/Test/services/mongodb/mongodb.conf", net: { bindIp: "127.0.0.1", http: { RESTInterfaceEnabled: true, enabled: true }, ipv6: true, port: 27017 }, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/opt/Test/services/mongodb/ovmongodb.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/opt/Test/data/mongodb", journal: { commitIntervalMs: 300, enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/opt/Test/services/mongodb/logs/mongodb.log", quiet: true } }
      2018-01-19T02:52:58.926-0800 I -        [initandlisten] Detected data files in /opt/Test/data/mongodb created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten]
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/Test/data/mongodb is set to 4096KB
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
      2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
      2018-01-19T02:52:58.937-0800 I JOURNAL  [initandlisten] journal dir=/opt/Test/data/mongodb/journal
      2018-01-19T02:52:59.121-0800 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
      2018-01-19T02:53:03.355-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 70.14
      2018-01-19T02:53:06.407-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 55.64
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 27.48
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster check took 10.64 secs
      2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.0
      2018-01-19T02:53:52.409-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.1
      2018-01-19T02:54:30.367-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.2
      2018-01-19T02:55:14.470-0800 I JOURNAL  [durability] Durability thread started
      2018-01-19T02:55:14.480-0800 I JOURNAL  [journal writer] Journal writer thread started
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] ** WARNING: The server is started with the web server interface and access control.
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          The web interfaces (rest, httpinterface and/or jsonp) are insecure
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          and should be disabled unless required for backward compatibility.
      2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
      2018-01-19T02:55:14.760-0800 I -        [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 410
      2018-01-19T02:55:14.760-0800 I -        [initandlisten]
       
      ***aborting after invariant() failure
       
       
      2018-01-19T02:55:14.854-0800 F -        [initandlisten] Got signal: 6 (Aborted).
       
       0x1328da2 0x1327cc9 0x13284d2 0x7f5031f7c100 0x7f5031be15f7 0x7f5031be2ce8 0x12a502b 0xa9eb62 0xa9ecd8 0xaa0864 0xaa4b4c 0x946a53 0x9498ca 0x94c83d 0x7f5031bcdb15 0x944d89
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F28DA2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27CC9"},{"b":"400000","o":"F284D2"},{"b":"7F5031F6D000","o":"F100"},{"b":"7F5031BAC000","o":"355F7","s":"gsignal"},{"b":"7F5031BAC000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA502B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"69EB62","s":"_ZNK5mongo8Database13getCollectionENS_10StringDataE"},{"b":"400000","o":"69ECD8","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"6A0864","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"6A4B4C","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"546A53"},{"b":"400000","o":"5498CA"},{"b":"400000","o":"54C83D","s":"main"},{"b":"7F5031BAC000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"544D89"}],"processInfo":{ "mongodbVersion" : "3.2.13", "gitVersion" : "23899209cad60aaafe114f6aea6cb83025ff51bc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.el7.x86_64", "version" : "#1 SMP Thu Nov 19 22:10:57 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D425B16F216CF164333E71CEE765F2A21A45B069" }, { "b" : "7FFE39876000", "elfType" : 3, "buildId" : "0F488C85309E4AB293D3B758748A3037B01F9885" }, { "b" : "7F5032E95000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7F5032AAD000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "6A997DC6D2CFD6702B987BF1B4926ABBA91691B8" }, { "b" : "7F50328A5000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4D22B5631EAF7D1EB6E72A42315105D14C70DC50" }, { "b" : "7F50326A1000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7F503239F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8DEE792EF5B31B20A3CD95985E8BE374D5C6A8C2" }, { "b" : "7F5032189000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7F5031F6D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4C0F15C537CF112511B6BE97DE1627C350652AA2" }, { "b" : "7F5031BAC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "B04A54C443D36058702AB4060C63F4AB3273EAE9" }, { "b" : "7F5033102000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D2E4A501823D041413EEB652B89044D1F680EE" }, { "b" : "7F5031960000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7F503167B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7F5031477000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7F5031245000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7F503102F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7F5030E20000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7F5030C1C000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F5030A02000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7F50307DD000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F503057C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7F5030357000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1328da2]
       mongod(+0xF27CC9) [0x1327cc9]
       mongod(+0xF284D2) [0x13284d2]
       libpthread.so.0(+0xF100) [0x7f5031f7c100]
       libc.so.6(gsignal+0x37) [0x7f5031be15f7]
       libc.so.6(abort+0x148) [0x7f5031be2ce8]
       mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12a502b]
       mongod(_ZNK5mongo8Database13getCollectionENS_10StringDataE+0x42) [0xa9eb62]
       mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0x28) [0xa9ecd8]
       mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xaa0864]
       mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x40C) [0xaa4b4c]
       mongod(+0x546A53) [0x946a53]
       mongod(+0x5498CA) [0x9498ca]
       mongod(main+0x15D) [0x94c83d]
       libc.so.6(__libc_start_main+0xF5) [0x7f5031bcdb15]
       mongod(+0x544D89) [0x944d89]
      -----  END BACKTRACE  -----
      

      How can i resolve this issue ?
      This is a fresh installation, and i faced this issue when starting mongo first time. i also tried to deploy the mongo on other Centos VM located on the same ESXi, but still faced the same issue.
      One strange thing, when i deployed the same mongo on Centos VM located on an ESXi 6.0 (this host is using SCSI disk), it worked fine. So i suspect the issue relates to Local DATA disk or the partition.

      Thanks,

      Attachments

        1. Datastore ATA.jpg
          Datastore ATA.jpg
          85 kB
        2. mongo data files.jpg
          mongo data files.jpg
          54 kB

        Activity

          People

            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            qa John
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: