[SERVER-70672] Merge enterprise repo into 10gen/mongo Created: 03/Oct/22  Updated: 23/Jan/24

Status: In Code Review
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.24, 4.4.28, 6.0.14

Type: Improvement Priority: Major - P3
Reporter: Steve Tarzia Assignee: Steve Gross
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File copy.bara.sky     File test.py    
Issue Links:
Backports
Depends
Problem/Incident
Related
Assigned Teams:
Correctness
Backport Requested:
v7.3
Sprint: Build and Correctness OnDeck
Participants:
Linked BF Score: 154

 Description   

This ticket should follow the plan laid out here on bullet 2f. This should use the git commands laid out here. This will be timing critical so this work should be done during off hours to minimize disruption.



 Comments   
Comment by Githook User [ 29/Nov/23 ]

Author:

{'name': 'Juan Gu', 'email': 'juan.gu@mongodb.com', 'username': 'juangugit'}

Message: SERVER-70672 Merge enterprise repo into 10gen/mongo

GitOrigin-RevId: f495673c3b44d8892e478a228a7449fa10b233a6
Branch: v4.4
https://github.com/mongodb/mongo/commit/571369c1fd3962d79565cd10f0406ddfb62c16b7

Comment by Alex Neben [ 29/Nov/23 ]

For those following along the changes in 4.4 have been reverted with a force push. We are attempting again tomorrow with a branch created with the following instructions.

I / juan.gu@mongodb.com got it working with the following script / commands

# copy.bara.sky core.workflow(
    name = "default",
    origin = git.origin(
        url = "git@github.com:10gen/mongo-enterprise-modules.git",
        ref = "v5.0",
    ),
    destination = git.destination(
        url = "/Users/alexander.neben/workspace/mongo4",  # your local repo
        push = "merge",
        # Your branch - this branch cannot exist before starting
        # if you rename this the command below must also be modified
        fetch = "v5.0",
    ),
    # ask_for_confirmation = True,
    origin_files = glob(["**"]),
    destination_files = glob(["src/mongo/db/modules/enterprise/**"]),
    authoring = authoring.pass_thru("MongoDB <mongodb@mongodb.com>"),
    mode = "ITERATIVE",
    # Change the path here to the folder you want to publish publicly
    transformations = [
        core.move("", "src/mongo/db/modules/enterprise"),
    ],
)

then run

cd /Users/alexander.neben/workspace/mongo4 && git branch -D merge && cd /Users/alexander.neben/workspace/copybara && bazel run //java/com/google/copybara -- migrate /Users/alexander.neben/workspace/mongo4/copy.bara.sky --init-history 

So if you just put up the same PR after having run that then i think we are going to be good to merge.

Then we ran the follow script to confirm all the files are still there

import os
import difflibdef read_file(filepath):
    with open(filepath, 'r', errors='replace') as file:
        return file.readlines()def get_file_list(base_dir):
    file_list = []
    for root, _, files in os.walk(base_dir):
        for file in files:
            file_list.append(os.path.join(root, file))
    return file_listdef compare_files(file1, file2):
    content1 = read_file(file1)
    content2 = read_file(file2)    diff = difflib.unified_diff(content1, content2, lineterm='')
    return list(diff)def compare_branches(base_dir_a, base_dir_b):
    files_a = get_file_list(base_dir_a)
    files_b = get_file_list(base_dir_b)    # Check if file counts are the same
    if len(files_a) != len(files_b):
        print(f"File count differs: Branch A has {len(files_a)} files, Branch B has {len(files_b)} files")    for file_b in files_b:
        rel_path = os.path.relpath(file_b, base_dir_b)
        file_a = os.path.join(base_dir_a, rel_path)
        if os.path.exists(file_a):
            differences = compare_files(file_a, file_b)
            if differences:
                print(f"juanfixme file_a is {file_a}\n rel_path is {rel_path} \n file_b is {file_b}")
                print(f"Differences in file {rel_path}:\n")
                print('\n'.join(differences))
        else:
            print(f"juanfixme file_a is {file_a}\n rel_path is {rel_path} \n file_b is {file_b}")
            print(f"File {rel_path} in Branch A does not exist in Branch B at {file_b}")base_dir_a = '/Users/alexander.neben/workspace/mongo4/src/mongo/db/modules/enterprise' 
base_dir_b = '/Users/alexander.neben/workspace/mongo-enterprise-modules4'compare_branches(base_dir_a, base_dir_b) 

Juan will post a larger announcement but it seems to have been a success. These steps will need to be repeated for v5.0, v6.0, v7.0, and master. Since this is weird about the formatting the original files have been attached to this ticket.

Comment by Alex Neben [ 28/Jan/23 ]

One idea here is to use https://github.com/google/copybara to have our 10gen github organization of mongo contain the enterprise module and the our public organization (mongodb/) not contain the enterprise module.

Comment by Alex Neben [ 18/Oct/22 ]

That build parameter exists, it is `–modules`.

Comment by Steve Tarzia [ 17/Oct/22 ]

Note that there are cases when we want to compile without enterprise.  Currently I think that's done by simply not cloning the enterprise module.  If they are merged into one repo then a build parameter may be needed to disable the enterprise repo.

Comment by Zakhar Kleyman [ 04/Oct/22 ]

Sending this to SDP because it sounds server development related.

Generated at Thu Feb 08 06:16:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.