[SERVER-33119] Could not start Mongo DB on Centos 7 running on ESXi 5.5 Created: 05/Feb/18  Updated: 21/Mar/18  Resolved: 27/Feb/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: John Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: JPEG File Datastore ATA.jpg     JPEG File mongo data files.jpg    
Participants:

 Description   

I deployed Mongo 3.2.13 on Centos-release-7-2.1511.el7.centos.2.10.x86_64 (3.10.0-327.el7.x86_64) running on ESXi 5.5 (this ESXi is using Local ATA Disk, see "
Datastore ATA.jpg" image) but i could not start it.

The log displayed :

2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] MongoDB starting : pid=13932 port=27017 dbpath=/opt/Test/data/mongodb 64-bit host=test
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] db version v3.2.13
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] modules: none
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] build environment:
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distmod: amazon
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-19T02:52:58.901-0800 I CONTROL  [initandlisten] options: { config: "/opt/Test/services/mongodb/mongodb.conf", net: { bindIp: "127.0.0.1", http: { RESTInterfaceEnabled: true, enabled: true }, ipv6: true, port: 27017 }, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/opt/Test/services/mongodb/ovmongodb.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/opt/Test/data/mongodb", journal: { commitIntervalMs: 300, enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/opt/Test/services/mongodb/logs/mongodb.log", quiet: true } }
2018-01-19T02:52:58.926-0800 I -        [initandlisten] Detected data files in /opt/Test/data/mongodb created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten]
2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/Test/data/mongodb is set to 4096KB
2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2018-01-19T02:52:58.937-0800 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
2018-01-19T02:52:58.937-0800 I JOURNAL  [initandlisten] journal dir=/opt/Test/data/mongodb/journal
2018-01-19T02:52:59.121-0800 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2018-01-19T02:53:03.355-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 70.14
2018-01-19T02:53:06.407-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 55.64
2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster=true 27.48
2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocateIsFaster check took 10.64 secs
2018-01-19T02:53:09.762-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.0
2018-01-19T02:53:52.409-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.1
2018-01-19T02:54:30.367-0800 I JOURNAL  [initandlisten] preallocating a journal file /opt/Test/data/mongodb/journal/prealloc.2
2018-01-19T02:55:14.470-0800 I JOURNAL  [durability] Durability thread started
2018-01-19T02:55:14.480-0800 I JOURNAL  [journal writer] Journal writer thread started
2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] ** WARNING: The server is started with the web server interface and access control.
2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          The web interfaces (rest, httpinterface and/or jsonp) are insecure
2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten] **          and should be disabled unless required for backward compatibility.
2018-01-19T02:55:14.509-0800 I CONTROL  [initandlisten]
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-01-19T02:55:14.510-0800 I CONTROL  [initandlisten]
2018-01-19T02:55:14.760-0800 I -        [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 410
2018-01-19T02:55:14.760-0800 I -        [initandlisten]
 
***aborting after invariant() failure
 
 
2018-01-19T02:55:14.854-0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1328da2 0x1327cc9 0x13284d2 0x7f5031f7c100 0x7f5031be15f7 0x7f5031be2ce8 0x12a502b 0xa9eb62 0xa9ecd8 0xaa0864 0xaa4b4c 0x946a53 0x9498ca 0x94c83d 0x7f5031bcdb15 0x944d89
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F28DA2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27CC9"},{"b":"400000","o":"F284D2"},{"b":"7F5031F6D000","o":"F100"},{"b":"7F5031BAC000","o":"355F7","s":"gsignal"},{"b":"7F5031BAC000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA502B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"69EB62","s":"_ZNK5mongo8Database13getCollectionENS_10StringDataE"},{"b":"400000","o":"69ECD8","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"6A0864","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"6A4B4C","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"546A53"},{"b":"400000","o":"5498CA"},{"b":"400000","o":"54C83D","s":"main"},{"b":"7F5031BAC000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"544D89"}],"processInfo":{ "mongodbVersion" : "3.2.13", "gitVersion" : "23899209cad60aaafe114f6aea6cb83025ff51bc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.el7.x86_64", "version" : "#1 SMP Thu Nov 19 22:10:57 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D425B16F216CF164333E71CEE765F2A21A45B069" }, { "b" : "7FFE39876000", "elfType" : 3, "buildId" : "0F488C85309E4AB293D3B758748A3037B01F9885" }, { "b" : "7F5032E95000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7F5032AAD000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "6A997DC6D2CFD6702B987BF1B4926ABBA91691B8" }, { "b" : "7F50328A5000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4D22B5631EAF7D1EB6E72A42315105D14C70DC50" }, { "b" : "7F50326A1000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7F503239F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8DEE792EF5B31B20A3CD95985E8BE374D5C6A8C2" }, { "b" : "7F5032189000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7F5031F6D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4C0F15C537CF112511B6BE97DE1627C350652AA2" }, { "b" : "7F5031BAC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "B04A54C443D36058702AB4060C63F4AB3273EAE9" }, { "b" : "7F5033102000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D2E4A501823D041413EEB652B89044D1F680EE" }, { "b" : "7F5031960000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7F503167B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7F5031477000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7F5031245000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7F503102F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7F5030E20000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7F5030C1C000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F5030A02000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7F50307DD000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F503057C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7F5030357000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1328da2]
 mongod(+0xF27CC9) [0x1327cc9]
 mongod(+0xF284D2) [0x13284d2]
 libpthread.so.0(+0xF100) [0x7f5031f7c100]
 libc.so.6(gsignal+0x37) [0x7f5031be15f7]
 libc.so.6(abort+0x148) [0x7f5031be2ce8]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12a502b]
 mongod(_ZNK5mongo8Database13getCollectionENS_10StringDataE+0x42) [0xa9eb62]
 mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0x28) [0xa9ecd8]
 mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xaa0864]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x40C) [0xaa4b4c]
 mongod(+0x546A53) [0x946a53]
 mongod(+0x5498CA) [0x9498ca]
 mongod(main+0x15D) [0x94c83d]
 libc.so.6(__libc_start_main+0xF5) [0x7f5031bcdb15]
 mongod(+0x544D89) [0x944d89]
-----  END BACKTRACE  -----

How can i resolve this issue ?
This is a fresh installation, and i faced this issue when starting mongo first time. i also tried to deploy the mongo on other Centos VM located on the same ESXi, but still faced the same issue.
One strange thing, when i deployed the same mongo on Centos VM located on an ESXi 6.0 (this host is using SCSI disk), it worked fine. So i suspect the issue relates to Local DATA disk or the partition.

Thanks,



 Comments   
Comment by Ramon Fernandez Marina [ 27/Feb/18 ]

qa, with the information you provided here's my assessment:

  • If you used the exact same OVF file to deploy this VM on both ATA and SCSI disks, then that would indicate that the MongoDB data in that OVF file is healthy
  • The error message you're seeing is common when there's on-disk corruption, which you originally pointed out as a possibility

Given the information thus far I don't believe there's enough evidence of a bug in the server, so I'm going to close this ticket.

I'd recommend checking the health of the local ATA disk(s) you were trying to use. It could also be the issue is on the VMWare driver, but I think that's a lot more unlikely.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Comment by John [ 12/Feb/18 ]

Hi Bruce,

Sorry for late reply.

We built an OVF file, it contains Centos 7, our app and MongoDB. There are some data files built-in at /opt/Test/data/mongodb

>>>Can you clarify what you mean by "fresh installation"?
I means we deployed OVF file on an EXSi host, and when the deploy was successfully, we started this new virtual machine, and Mongo service will be started, but it was failed.

>>>Can you confirm that in the cases where you started it successfully you were using precisely the same db content?
Yes, the data files are the same because we used the same OVF file to deploy on another EXSi (I noticed this EXSi is using SCSI Disk instead of Local ATA Disk), and everything worked fine without any issue.
Thanks and Regards,
John.

Comment by Bruce Lucas (Inactive) [ 05/Feb/18 ]

Hi John,

Can you clarify what you mean by "fresh installation"? The log line "Detected data files in /opt/Test/data/mongodb created by the 'mmapv1' storage engine" indicates that the data directory was not empty when mongod started - is that what you intended? The message indicates likely some kind of corruption in the db; how was the copy of the db used here created? Can you confirm that in the cases where you started it successfully you were using precisely the same db content?

Bruce

Generated at Thu Feb 08 04:32:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.