[SERVER-29246] IDL doesn't support pathnames with dots Created: 17/May/17  Updated: 30/Oct/23  Resolved: 01/Jun/17

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: 3.5.9

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Gabriel Russell (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Platforms 2017-06-19
Participants:

 Description   

The compilation line is:

/usr/bin/python2.7 buildscripts/scons.py all --disable-warnings-as-errors --ssl -j4 --opt=off --dbg=off CC=/opt/mongodbtoolchain/v2/bin/gcc CXX=/opt/mongodbtoolchain/v2/bin/g++ --link-
model=dynamic

In file included from src/mongo/db/logical_session_record.h:38:0,
                 from src/mongo/db/logical_session_record_test.cpp:37:
build/7a67446c/mongo/db/logical_session_record_gen.h:23:23: fatal error: mongo_gen.h: No such file or directory
compilation terminated.
scons: *** [build/7a67446c/mongo/db/logical_session_record_test.o] Error 1
scons: building terminated because of errors.
build/7a67446c/mongo/db/logical_session_record_test.o failed: Error 1



 Comments   
Comment by Githook User [ 01/Jun/17 ]

Author:

{u'username': u'gabrielrussell', u'name': u'Gabriel Russell', u'email': u'gabriel.russell@mongodb.com'}

Message: SERVER-29246 idl compiler supports dots in pathnames
Branch: master
https://github.com/mongodb/mongo/commit/a0ddbc70bc5f91ff9e27cd83d16302f99ffbbad7

Comment by Eric Milkie [ 30/May/17 ]

I figured it out. There is a bug in this line in compiler.py:

	include_h_file_name = resolved_file_name.split('.')[0] + args.output_suffix + ".h"

In my case, the resolved file name happens to have a dot in one of the directory names that makes up the path:

/data/src/mongo.master/src/mongo/db/logical_session_id.idl

In this case, the split is making an incorrect assumption about where dots appear in filenames.

Comment by Eric Milkie [ 26/May/17 ]

Looks like the bug is something to do with this function:

def _update_import_includes(args, spec, header_file_name):
    # type: (CompilerArgs, syntax.IDLSpec, unicode) -> None
    """Update the list of imports with a list of include files for each import with structs."""
    # This function is fragile:
    # In order to try to generate headers with an "include what you use" set of headers, the IDL
    # compiler needs to generate include statements to headers for imported files with structs. The
    # problem is that the IDL compiler needs to make the following assumptions:
    # 1. The layout of build vs source directory.
    # 2. The file naming suffix rules for all IDL invocations are consistent.
    if not spec.imports:
        return
 
    if args.output_base_dir:
        base_include_h_file_name = os.path.relpath(
            os.path.normpath(header_file_name), os.path.normpath(args.output_base_dir))
    else:
        base_include_h_file_name = os.path.abspath(header_file_name)
 
    # Normalize to POSIX style for consistency across Windows and POSIX.
    base_include_h_file_name = base_include_h_file_name.replace("\\", "/")
 
    # Modify the includes list of the root_doc to include all of its direct imports
    if not spec.globals:
        spec.globals = syntax.Global(args.input_file, -1, -1)
 
    first_dir = base_include_h_file_name.split('/')[0]
 
    for resolved_file_name in spec.imports.resolved_imports:
        # Guess: the file naming rules are consistent across IDL invocations
        include_h_file_name = resolved_file_name.split('.')[0] + args.output_suffix + ".h"

The final line is producing, for some idl, "mongo_gen.h" with no path ("_gen" is args.output_suffix). This is always the last include in a list of all the generated includes; for example:

/**
 * WARNING: This is a generated file. Do not modify.
 *
 * Source: buildscripts/idl/idlc.py --include src --base_dir build/7a67446c --header build/7a67446c/mongo/db/logical_session_record_gen.h --output build/7a67446c/mongo/db/logical_session_record_gen.cpp src/mongo/db/logical_session_record.idl
 */
 
#pragma once
 
#include <algorithm>
#include <boost/optional.hpp>
#include <cstdint>
#include <string>
#include <vector>
 
#include "mongo/base/data_range.h"
#include "mongo/base/string_data.h"
#include "mongo/bson/bsonobj.h"
#include "mongo/bson/bsonobjbuilder.h"
#include "mongo/db/logical_session_id.h"
#include "mongo/db/namespace_string.h"
#include "mongo/idl/idl_parser.h"
#include "mongo/util/uuid.h"
#include "mongo_gen.h"

Comment by Gabriel Russell (Inactive) [ 24/May/17 ]

milkie All of these failures are from the same file being missing, mongo_gen.h . mongo_gen.h is not generated in a successful build of master. It's not mentioned in unittest_gen.h, or any of the other files that I can find, and mongo.idl is not in the repo.

I'm assuming that you're not working on a patch where mongo.idl is being added, so I guess that mongo_gen.h dependency is actually a mistake resulting from a bug in the idl compiler.

How consistently does this fail? And where? I'm not aware of a steam of build failures that look similar to this. Do all of these failures happen in your working environment on some branch of yours? If so, I can have access to the branch?

Gabriel

Comment by Eric Milkie [ 23/May/17 ]

This continues to fail, and shows symptoms of being a missing dependency, in that it's not always the same translation unit that fails:

/opt/mongodbtoolchain/v2/bin/g++ -o build/7a67446c/mongo/idl/unittest_gen.o -c -Woverloaded-virtual -Wno-maybe-uninitialized -std=c++14 -fno-omit-frame-pointer -
fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -O0 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-de
clarations -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp -fPIE -DPCRE_STATIC -DNDEBUG -DBOOST_THREAD_VERSION=4 -D
BOOST_THREAD_DONT_PROVIDE_VARIADIC_THREAD -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DBOOST_THREAD_DONT_PROVIDE_INTERRUPTIONS -DBOO
ST_THREAD_HAS_NO_EINTR_BUG -Isrc/third_party/pcre-8.39 -Isrc/third_party/boost-1.60.0 -Ibuild/7a67446c -Isrc build/7a67446c/mongo/idl/unittest_gen.cpp
In file included from src/mongo/idl/idl_test.cpp:33:0:
build/7a67446c/mongo/idl/unittest_gen.h:23:23: fatal error: mongo_gen.h: No such file or directory
compilation terminated.
scons: *** [build/7a67446c/mongo/idl/idl_test.o] Error 1
/usr/bin/python2.7 buildscripts/idl/idlc.py --include src --base_dir build/7a67446c --header build/7a67446c/mongo/idl/unittest_import_gen.h --output build/7a6744
6c/mongo/idl/unittest_import_gen.cpp src/mongo/idl/unittest_import.idl
In file included from build/7a67446c/mongo/idl/unittest_gen.cpp:7:0:
build/7a67446c/mongo/idl/unittest_gen.h:23:23: fatal error: mongo_gen.h: No such file or directory
compilation terminated.
scons: *** [build/7a67446c/mongo/idl/unittest_gen.o] Error 1
scons: building terminated because of errors.
build/7a67446c/mongo/idl/idl_test.o failed: Error 1
build/7a67446c/mongo/idl/unittest_gen.o failed: Error 1

Generated at Thu Feb 08 04:20:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.