ActiveState continues to provide Python 2 support to help organizations safely run their Python 2 applications, services, and systems, at a fraction of the cost they spend in developer hours maintaining them.
Every programming language must continue to evolve over time to stay relevant, and Python is no exception. Unfortunately, that means many organizations have found themselves with bulky legacy applications written in older versions of Python—most notably Python 2.7. Newer releases of Python 3 have gained a lot of ground over Python 2, and now feature faster runtimes and larger support communities than their pre-v3 counterparts. Combine this with the impending end of support for Python 2 (currently scheduled for Jan. 1, 2020), and it’s easy to see why organizations feel the need to migrate their existing codebases.
Porting code from an older to a newer version can be a difficult and intimidating process. In this article, I’ll discuss the various migration libraries that exist to help convert Python 2.x applications to Python 3.x. Using code samples that leverage three methods for Python migration (2to3, python-future, six), I’ll show how you can get started quickly on application conversion.
If you want to try converting code from Python 2 to 3 yourself, make sure you have a recent version of Python installed along with the 2to3, six and python-future packages. To get started quickly, you can either:
- Download and install the pre-built “Migrating 2 to 3” runtime environment for Windows 10, macOS, CentOS 7, ?utm_source=activestate.com&utm_medium=referral&utm_content=blog-migrating-2-to-3-python&utm_campaign=user-acquisitionor…
- If you’re on a different OS, you can automatically create a custom Python runtime (click on the Get Started button once you log in) with just the packages required (such as six), and then automatically install it into a virtual environment using ActiveState’s CLI, the State Tool.
Automated Python Migration with 2to3
2to3 is a Python program that utilizes the library lib2to3 to transform Python 2 application source code into that of a Python 3 application. It does so by utilizing functionality called fixers. These are essentially functions that detect specific syntax differences between Python 2 and Python 3, and refactor code to make it compatible with versions of Python 3.
Let’s take a look at a sample Python program written using Python 2.7 to see how we can leverage 2to3 to transform previously incompatible or deprecated source code into code that is compatible with the latest versions of Python:
import unittest
def add_values(val1, val2):
print('val1: ' + str(val1)),
print 'val2: ' + str(val2)
return val1+val2
class SampleTests(unittest.TestCase):
def test_add_values(self):
self.assertEquals(add_values(1,2), 3)
def test_add_values_not_equal(self):
self.assertNotEquals(add_values(1,2),4)
if __name__ == '__main__':
unittest.main()
The above program, sample_tests.py, is fairly simplistic in terms of functionality. First, we define a Python function called add_values(val1, val2). When called, this function prints out the values passed to the function (on the same line due to the trailing comma at the conclusion of the first print statement) and returns the sum of the two values. When running sample_tests.py, this program executes two unit tests to test the add_values function:
- The first function tests add_values using the deprecated function
assertEquals(nowassertEqual)
- The second function tests add_values using the deprecated function
assertNotEquals(nowassertNotEqual)
Running this program with Python 2.7 yields the following output:
.val1: 1 val2: 2 . ----------------------------------------------------------------- Ran 2 tests in 0.000s OK
But when attempting to run this program with Python 3.6, we get the following output:
File "sample_tests.py", line 10 print 'val2: ' + str(val2) ^ SyntaxError: invalid syntax
As it stands, this program is compatible with Python 2.7 but not compatible with Python 3.6. In order to convert this application, you must first install 2to3. Once you’ve installed it, the command to convert is as follows:
2to3 -w sample_tests.py
The optional -w flag indicates that the necessary modifications detected by the 2to3 fixers will be written directly to the sample_tests.py file. In other words, after running this command, the file sample_tests.py will have the compatibility changes reflected in the code itself.
Here’s the output from the command:
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored sample_tests.py
--- sample_tests.py (original)
+++ sample_tests.py (refactored)
@@ -6,16 +6,16 @@
import unittest
def add_values(val1, val2):
- print('val1: ' + str(val1)),
- print 'val2: ' + str(val2)
+ print(('val1: ' + str(val1)), end=' ')
+ print('val2: ' + str(val2))
return val1+val2
class SampleTests(unittest.TestCase):
def test_add_values(self):
- self.assertEquals(add_values(1,2), 3)
+ self.assertEqual(add_values(1,2), 3)
def test_add_values_not_equal(self):
- self.assertNotEquals(add_values(1,2),4)
+ self.assertNotEqual(add_values(1,2),4) if __name__ == '__main__':
RefactoringTool: Files that were modified:
RefactoringTool: sample_tests.py
This output indicates that various fixers, notably the print fixer and the asserts fixer, have detected and modified several lines on the basis of deprecation and incompatibility with Python 3 (as can be seen in the documentation).
The print lines in the add_values function were changed to reflect the lack of a print statement in Python 3, as well as the move to differing functionality for preventing a new line, which is no longer acceptable to do with a trailing comma. In addition, the deprecated assertEquals and assertNotEquals have been modified to reference the new function names, assertEqual and assertNotEqual. After these modifications are made via 2to3, sample_tests.py will now execute successfully in Python 3.6.
The biggest advantage of using 2to3 is its ease of use due to the automated nature of the conversion: 2to3 does the work for you. Outdated code can be translated, and these translations can be applied through the use of a single command. Finally, it’s important to mention that 2to3 is mostly valid for cases where backwards compatibility is not important. For environments running a version of Python 2, converting using 2to3 will often break your program. If compatibility between environments is important for your organization, you will want to consider other libraries with an existing compatibility layer between Python 2 and 3.
Migration Using Python-future
When considering porting an application from Python 2.6 or Python 2.7 to Python 3.3+, you should consider utilizing python-future. This particular library carries several similarities to 2to3 in terms of its usage, but also contains some stark differences that can make it more valuable in certain cases. Let’s take the original sample program from the above section (sample_tests.py) and demonstrate the functionality behind python-future for converting this application from one written in 2.7 to a version for use with Python 3.3+.
The python-future library is leveraged against a Python 2-compatible codebase using the futurize command, allowing for the detection and translation of Python 2.6 or 2.7-specific code, which in turn allows for execution in a Python 3.3+ environment. Similar to 2to3, the futurize command can be leveraged to automate the conversion of your application’s source code, enabling it to be run with Python 3.3+ through the use of the -w flag. It even leverages the lib2to3 fixers for some of its conversion processes. That being said, there is one additional benefit of python-future which does not exist in 2to3 that is important to mention: python-future provides a compatibility layer between Python 2.6 and 2.7 environments and those running 3.3+. In other words, when your application has been “futurized,” it can still be run in environments running Python 2.6 or 2.7 where python-future is installed.
The following command will enable our application to be executed in supported environments:
futurize -w sample_tests.py
After running the above command, here is the output that we receive in our Python 3.6 environment:
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored sample_tests.py
--- sample_tests.py (original)
+++ sample_tests.py (refactored)
@@ -3,11 +3,13 @@
@author: Scott
'''
+from __future__ import print_function
+from builtins import str
import unittest
def add_values(val1, val2):
- print('val1: ' + str(val1)),
- print 'val2: ' + str(val2)
+ print(('val1: ' + str(val1)), end=' ')
+ print('val2: ' + str(val2))
return val1+val2
class SampleTests(unittest.TestCase):
RefactoringTool: Files that were modified:
RefactoringTool: sample_tests.py
Here is our new sample_tests.py file following the call to futurize:
from __future__ import print_function
from builtins import str
import unittest
def add_values(val1, val2):
print(('val1: ' + str(val1)), end=' ')
print('val2: ' + str(val2))
return val1+val2
class SampleTests(unittest.TestCase):
def test_add_values(self):
self.assertEquals(add_values(1,2), 3)
def test_add_values_not_equal(self):
self.assertNotEquals(add_values(1,2),4)
if __name__ == '__main__':
unittest.main()
There are a few things to dig into here, the first of which is the import lines at the top of the new sample_tests.py file. In particular, the import of the print_function from the python-future library enables these lines to behave identically between all the supported versions of Python mentioned above (2.6, 2.7, 3.3+). In addition, running futurize left the deprecated aliases for assertEqual and assertNotEqual functions in Python 3. Since the deprecated alias can still be interpreted in Python 3, compatibility between Python 2 and 3 has been maintained in this case, as well. This library is meant for organizations who wish to maintain backwards compatibility with past versions of Python 2.
With that said, sample_tests.py can now be executed successfully in our 2.7 environment, as well as in our 3.6 environment when python-future is installed in the environment in which you are running the program (critical for accessing the necessary python-future modules to allow compatibility).
Migrating with Python Six Provides Dual Compatibility
The last migration method we’ll look at in this article uses Python’s six library (named for 2*3 = 6). Much like python-future, six allows for the translation of source code in a manner that allows an application to be run successfully in both Python 2 and Python 3 environments. In the case of six, however, all versions of Python 2.6 and higher are supported, rather than being limited to just Python 2.6, 2.7, 3.3+ as is the case with python-future. In addition, six is utilized in a slightly different manner that involves manually applying wrapper functions from the six API, leveraging their library for compatibility between versions.
To demonstrate six, we’ll use a different code example than the one we’ve been using. The new program (sample_reduce.py) represents a version of the program solely compatible with versions of Python prior to 3:
import unittest def multiply_values(val1, val2): return val1*val2 def add_values(val1, val2): return val1+val2 class SampleTests(unittest.TestCase): def test_sets_equal(self): setValue1 = reduce(multiply_values, [1,2]) setValue2 = reduce(add_values, [1,2]) self.assertItemsEqual([2,3], [setValue1,setValue2]) if __name__ == '__main__': unittest.main()
While this program runs successfully in Python 2.7, it does not fare so well in Python 3.6. See the output below:
================================================================== ERROR: test_sets_equal (__main__.SampleTests) ------------------------------------------------------------------ Traceback (most recent call last): File "sample_reduce.py", line 16, in test_sets_equal setValue1 = reduce(multiply_values, [1,2]) NameError: name 'reduce' is not defined ------------------------------------------------------------------ Ran 1 test in 0.001s FAILED (errors=1)
While the first reported error traced the failure back to the lack of recognition of the reduce function, this is only one of the two issues preventing compatibility. The other is the lack of an assertItemsEqual method, which was re-implemented in Python 3.2 as assertCountEqual.
As is the case with any of these tools, we need to install the six library, and then we can fix these errors with the six API. The bolded lines below indicate the changes we need to make to the sample_reduce.py to add compatibility for Python 3.
import six from six.moves import reduce import unittest def multiply_values(val1, val2): return val1*val2 def add_values(val1, val2): return val1+val2 class SampleTests(unittest.TestCase): def test_sets_equal(self): setValue1 = reduce(multiply_values, [1,2]) setValue2 = reduce(add_values, [1,2]) six.assertCountEqual(self, [2,3], [setValue1,setValue2]) if __name__ == '__main__': unittest.main()
The first step is to import the six library. Next, we need to import reduce from six.moves. Importing specific functionality in this manner is often necessary with six, since the release of Python 3 resulted in a refactor that included the relocation of certain Python functionality to different modules. Finally, we reference the API function from the six library assertCountEqual. This allows for translation between assertCountEqual in Python 3.2+ and assertItemsEqual in environments running an earlier version of Python. With these changes, we can now successfully run this program in Python 2 or 3 environments. Keep in mind that six is recommended for organizations wishing to add compatibility for Python 3 to existing versions of code written in Python 2. In other words, it’s more for adding Python 3 compatibility to an existing Python 2 application than it is for actually porting code forward.
Conclusions
None of the migration libraries exercised in this post are going to be perfect for every situation. And no matter which one(s) you choose to work with, keep in mind that significant testing will always be required when porting source code from Python 2 to Python 3. However, utilizing these libraries will make your job much easier when it comes to identifying and translating incompatible source code within an outdated application. The more difficult part will be deciding which library (or combination of libraries) should be utilized in any particular case.
As discussed above, choosing a library can be dependent upon several factors: these include the version of Python in which an organization expects to write their application code in the future, and the versions of Python that the application in question needs to be able to run against. If backwards compatibility with versions of Python 2 is important, then either python-future or six might be the best choice; if not, then 2to3 might do the trick. This will need to be analyzed on a case-by-case basis.
- If you’re thinking of migrating because Python 2 support will soon expire, learn how ActiveState can help you migrate to Python 3 by requesting your free assessment.
- For more information on how to migrate from Python 2 to Python 3, read our Migration Datasheet.
Related Blogs:
