[SERVER-32295] Support python 3 Created: 13/Dec/17  Updated: 30/Oct/23  Resolved: 08/Apr/19

Status: Closed
Project: Core Server
Component/s: Build, Packaging
Affects Version/s: 3.6.0
Fix Version/s: 4.1.10

Type: Improvement Priority: Major - P3
Reporter: Marek Skalický Assignee: Mathew Robinson (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File python3-buildscripts-tests.patch    
Issue Links:
Depends
Problem/Incident
Related
related to SERVER-42623 sched module in Python 3 causes close... Closed
is related to SERVER-11367 The buildsystem doesn't support pytho... Closed
is related to SERVER-52687 MongoDB 4.0 - building with Python 3 Closed
Backwards Compatibility: Fully Compatible
Sprint: Dev Tools 2019-03-11, Dev Tools 2019-03-25, Dev Tools 2019-04-08, Dev Tools 2019-04-22
Participants:
Linked BF Score: 0

 Description   

Hi,
python2 is closing to end of its life - https://pythonclock.org/ . MongoDB relases has about 2 years life cycle, so it would be great to switch to python3.

Scons 3.0 now support python3. So it this change is possible now.

I've attached a patch to add python3 support for MongoDB building and testing. It is not ideal patch - most of it is done using 2to3 utility and probably some parts will be unnecessary (explicit creating of lists from iterable objects, binary files and encoding,...). But I didn't want to create big changes in code now.

Not all can be done with python3 with this patch. Some generators and similar scripts still use python2. I can help with converting them.

It would be great to see python3 support in MongoDB build scripts.



 Comments   
Comment by Githook User [ 05/Jun/19 ]

Author:

{'name': 'Mathew Robinson', 'email': 'chasinglogic@gmail.com', 'username': 'chasinglogic'}

Message: SERVER-32295 Support Python 3 and 2 for upload.py
Branch: PROG-1649-WT-backports-default-assignee
https://github.com/10gen/kernel-tools/commit/311ed14e215b99ccc76be08c2ae95b47a66f9b00

Comment by Githook User [ 09/Apr/19 ]

Author:

{'email': 'chasinglogic@gmail.com', 'name': 'Mathew Robinson', 'username': 'chasinglogic'}

Message: SERVER-32295 Support Python 3 and 2 for upload.py
Branch: master
https://github.com/10gen/kernel-tools/commit/311ed14e215b99ccc76be08c2ae95b47a66f9b00

Comment by Githook User [ 08/Apr/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-32295 Remove references to Python 2
Branch: master
https://github.com/mongodb/mongo/commit/8eae618a76e8a454e491f1a7ebaf6e66a229798e

Comment by Githook User [ 08/Apr/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-32295 Support Python 3 builds in Perf configs
Branch: master
https://github.com/mongodb/mongo/commit/415dfa9ec016d8a1e37d950adda64a17efc950c1

Comment by Githook User [ 08/Apr/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-32295 Support Python 3
Branch: master
https://github.com/mongodb/mongo/commit/8dd6d4755734ed37c1b98dfdefce3ca6bc65f1f6

Comment by Githook User [ 08/Apr/19 ]

Author:

{'email': 'chasinglogic@gmail.com', 'name': 'Mathew Robinson', 'username': 'chasinglogic'}

Message: SERVER-32295 Support Python 3 on Windows
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/0e223d6571e76d3e19560d29124bcffed14c9c1a

Comment by Githook User [ 29/Mar/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'}

Message: SERVER-32295 Support python 3
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/6a8e6bf7f1d536d0a4370d9061c4cbdedd382f25

Comment by Bernie Hackett [ 15/Nov/18 ]

The release notes for the first RHEL Beta are out. Python 3.6 is the system Python.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html-single/8.0_beta_release_notes/

Comment by Marek Skalický [ 24/Apr/18 ]

acm Thanks. I only wanted to be sure

Comment by Andrew Morrow (Inactive) [ 17/Apr/18 ]

Hi mskalick - I hear you. I had really hoped we would get it done for 4.0, it would have been a nice win. Unfortunately, we had some late breaking must-land 4.0 work that absorbed all the slack time I'd hoped would go towards getting 2+3 support in for this release (and for quite a few other things too). Such is the reality of software development, I suppose...

Comment by Marek Skalický [ 17/Apr/18 ]

acm Thanks for the update. I'm looking forward python3 only MongoDB...

I understand that migrating internal inftrastructure to python3 takes time. But is there any chance to support for both python3 and python2 in testsuite and scons scripts in Mongodb 4.0?

Comment by Andrew Morrow (Inactive) [ 16/Apr/18 ]

mskalick - I wanted to catch you up on this. First, the bad news: we are too close to the MongoDB 4.0 ship date to safely undertake this work. The good news: my team is proposing a project for MongoDB 4.2 that will officially migrate us to python 3 only. Assuming the project is approved, MongoDB 4.0 will be the last release that requires or is compatible with python 2.

Comment by Andrew Morrow (Inactive) [ 26/Jan/18 ]

Send us a PR for the version against master, and we will give it a look. The biggest problem for our side would be locking in python 3 support, since all of our CI system only uses python 2 right now. If a python3 breakage were to slip in, we might not notice, and we generally try to avoid situations where features can silently degrade. That said, maybe we have an environment available where it would be easy to spin up with python3?

Comment by Marek Skalický [ 26/Jan/18 ]

So I finally managed to create a patch which works for me with both python3 and python2. It can be found in https://github.com/omron93/mongo/tree/python3-pull . It is based on r3.6.2 release. I'm testing it in Fedora 28.

I've also rebased the patch to rawhide - https://github.com/omron93/mongo/tree/python3-pull-master . But I don't build mongodb master right now, so it is not tested.

Few points:

  • six python package is required after this patch for python2/3 support for metaclasses
  • in the patch I used explicit calling of list(). It creates list from iterable object. This approach is probably slower than leaving iterable objects (generators). But I don't know if that iterables are modified or not so I chose this approach. Please take a look in it and remove list() in cases it's not needed.

I've compared this with bmanojlovic PR on github. It is different mainly because of missing python2 compatibility and python3 conversion of some packaging scripts (in ./buildscripts/) is missing. But even in my patch not all these scripts are converted.

What should be the next step with this patch?

Comment by Andrew Morrow (Inactive) [ 26/Jan/18 ]

mskalick - I'm not actually sure what the best way would be for the two of you to coordinate your efforts here. Unfortunately python 3 compatibility is not something I'm expert in so I can't really give much helpful guidance. I will say retaining python 2 support is pretty much a hard requirement for us right now. We have the idea of a future project to make a full migration to python 3, at which point python 2 compatibility would no longer be a requirement. If adding python 3 support without breaking python 2 compatibility isn't achievable, I suppose I would recommend suspending efforts here and waiting for us to make the wholesale move to python 3.

Comment by Marek Skalický [ 18/Jan/18 ]

Sorry for my long inactivity.

I've started to work on this again, but tomorrow I'm not working. So I'll finish it after weekend.

What is a best way how do it? Create a pull request into bmanojlovic:python3 to update https://github.com/mongodb/mongo/pull/1195?

Comment by Andrew Morrow (Inactive) [ 16/Jan/18 ]

mskalick - I was curious if you and the submitter of https://github.com/mongodb/mongo/pull/1195 ever got around to forming a unified patch? I'm still interested in getting this done - it would be nice if the build system were 2/3 agnostic, so we were better positioned to make the leap to pure python 3 in the next release or so.

Comment by Andrew Morrow (Inactive) [ 28/Dec/17 ]

mskalick - FYI we have recently received a GitHub pull request that shares a lot with your proposed changes. Would you be willing to work through the differences in your proposal and theirs on the PR?

https://github.com/mongodb/mongo/pull/1195

Comment by Andrew Morrow (Inactive) [ 13/Dec/17 ]

Thanks mskalick - This has been on my personal TODO list for a while now. We will review the patch and see if we can make this happen.

Generated at Thu Feb 08 04:29:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.