Python for .NET

Python for .NET is a completely new implementation of Python. Although many facilities could be implemented simply by extending Python (that is, by using the existing CPython implementation and writing an extension module), there were two primary reasons for creating a completely new implementation:

• Full integration with .NET. The only effective way to have another .NET language derive from a Python implemented class is for Python to use .NET facilities when creating classes.

• Contractual obligations. The initial port was done under contract to Microsoft, largely to prove to Microsoft that the .NET CLR was indeed capable of supporting a large number of languages.

An alternative strategy that could be employed is discussed later in this appendix.

Current Status

The current state of the compiler and the runtime systems shows that it is possible to have compiled Python programs fully supported within the .NET Framework. Most of the basic infrastructure is in place, with full bidirectional cross-language support (i.e., Python can inherit or call objects created in other languages, and other languages can inherit from or call objects created in Python).

There are three categories of problems that preclude Python for .NET from being truly useful to a large number of people:

• There is no support for some of the features of .NET that other frameworks will require, such as custom attributes, PInvoke, or ASP.NET. This means that Python for .NET users are not currently able to write classes that interact with tools requiring those features. Related topics are the mismatch between the class/instance semantics, module/package semantics, and exception systems.

• The speed of the current system is so low as to render the current implementation useless for anything beyond demonstration purposes. This speed problem applies to both the compiler itself and the code generated by the compiler. Given that part of the appeal of Python programming is a quick edit-compile-run cycle, the speed issues severely limit the utility of Python on this platform. Some of the blame for this slow performance lies in the domain of .NET internals and Reflection::Emit , but some of it is due to the simple implementation of the Python for .NET compiler.

• There is no support for some Python features that some programs will require. Most of these are fairly obscure, but are a limitation. Examples include string formatting; core language features, such as long integers, complex numbers , and built-in object methods ; and the standard Python library.

If these issues are addressed, we feel that Python for .NET would become a viable , interesting technology to use within .NET.

Architecture

The Python for .NET implementation consists of three semi-discrete areas: the compiler, the runtime, and the library. Although there is obviously significant interaction between these parts , each has a distinct architecture.

The Compiler

The Python for .NET compiler is written using CPython. It compiles Python source code and uses the .NET Reflection::Emit library to generate a .NET assembly. The COM Interoperability features of .NET are used to access the Reflection::Emit library.

This particular strategy was chosen to minimize the implementation time. Python's parser is built into the CPython runtime, and the existing Python2C project ^[3] had infrastructure we could borrow . Greg Stein was involved in the Python2C project and so provided the expertise needed to get up and running quickly.

^[3] http://lima.mudlib.org/~rassilon/p2c/.

The key benefits to this approach are the rapid implementation of a simple compiler and the rapid development obtained by coding the compiler in CPython.

The primary drawback is the speed of the compiler. Much of the abstract syntax tree manipulation code is also written in Python code, and as this is one of the most CPU- intensive areas of the compiler, we suffer a significant speed penalty. However, the core Python2C code has been integrated into a standard Python module, and it is hoped that there will be general improvements in the speed of this library as more users turn to this standard module.

Further, the use of Reflection::Emit via COM is also causing some performance problems. Some of these problems are due to the speed of the Python COM bindings, but Reflection::Emit itself and/or the COM Interoperability layers are costing significant time as well.

Although currently implemented using CPython, it is envisioned that in the future this compiler will be capable of running under Python for .NET. This will allow the dynamic Python language features, such as exec and eval() , to be exploited by having the compiler invoked at runtime. However, the functionality provided by the Python for .NET compiler must be enhanced and a number of existing C-implemented Python modules must be ported before this is possible.

Python for .NET Runtime

Due to the dynamic nature of Python, the compiler will often generate code that references the Python for .NET runtime. Even for a simple Python expression such as " a + b ", if the types of the variables are not known by the compiler, it will generate code to ensure that Python for .NET determines the correct semantics at runtime.

One of the most important jobs of the runtime environment is to ensure that the Python language semantics are faithfully implemented. For this reason, the design and implementation of the Python for .NET runtime borrows heavily from the CPython implementation. The Python for .NET runtime is written in C#.

The Python for .NET runtime defines a .NET interface ( IPyType ) that captures Python's semantics. The definition of this interface is almost identical to the existing CPython type object , which is the object primarily responsible for object semantics in CPython.

The IPyType interface defines the semantics for a Python type independent of an object instance. For example, the type definition for a string or integer defines the behavior of strings and integers without reference to a specific string or specific integer. Thus, to operate on any .NET object, the Python runtime needs two components : an instance of an IPyType interface that describes the Python semantics (such as defining string or integer semantics) and the .NET object itself (that is, a reference to the specific string or integer being operated on). For this reason, the Python for .NET runtime defines a PyObject structure (that is, a C# value type), which consists of references to an IPyType interface and a .NET object. Almost all runtime functions work with PyObject s.

The Python for .NET runtime also exposes an API for use by the compiler, which works primarily with PyObject structures. It provides a function for creating a new PyObject at runtime, given nothing except an anonymous .NET object reference. The compiler will frequently generate calls to create these PyObject structures (often storing the result in a variable) and pass these PyObject structures back into the runtime environment as needed.

The Python Library

The role of the Python for .NET library is to implement the standard Python library ”the modules Python guarantees will be available to a program at runtime. Python programs reference library modules with the import statement.

One of the biggest challenges with a new implementation is this library. In general, there are two distinct categories of library modules in the existing CPython implementation:

• Extension modules are written in C/C++. They take advantage of the CPython runtime system to provide new facilities for Python.

• Python modules are written in Python. In general, these modules make use of the facilities provided by the extension modules written in C. For example, Python has an os module written purely in Python. It still makes use of the C-implemented functions in a different extension module native to the platform.

As the CPython API is defined in terms of the CPython implementation (that is, the extension module interface is specific to an implementation rather than part of the language specification), it is not practical to port existing C/C++ extension modules to the .NET implementation while having the CLR consider these modules verifiable . For this reason, there is a two-stage approach to implementing the Python library for .NET:

Reimplement all standard Python extension modules. These modules can be reimplemented either by creating an extension module in C# or by implementing these modules in Python and thereby taking advantage of the .NET Framework facilities.

Compile existing Python extension modules under the .NET compiler. Assuming that all extension modules available for CPython have been re-implemented for .NET, and that the compiler is capable, all existing Python modules should compile under the new implementation.

At the time of this writing, the library for the Python for .NET implementation is quite sparse, and only enough modules required to run basic tests have been implemented. It is hoped that this library will grow as use of the compiler expands.