Python and C interaction: Part III - Advanced use of Ctypes

We will discuss in this topic a completely Pythonic way to communicate C and Python and make the end result completely Object Oriented, not usually explained.

So far, we have discussed about how to connect Python and C through ctypes to speed up Python code¹. The main goal was not only to be able to run C code, but also to do so transparently. This way, we could call C code and wrap it properly so the library user would never know she or he is actually using C code. Both with values and references we were able to do so, but we actually accomplished the transparency of Python functions. We know that, though Python is multi-paradigm, its use is heavily focused in Object Oriented Programming. Wouldn’t it be nice if we could somehow mimic the object oriented behaviour through ctypes?

Structures

Among the many types that ctypes exposes to Python, it exposes the structure. So, let’s say we have a Rectangle structure in C and a functions that takes a rectangle as an argument as follows:

/* file: rectangle.c */

struct _rect {
  float height, width;
};

typedef struct _rect Rectangle;

float area(Rectangle rect) {
  return rect.height * rect.width;
}

and build it as a dynamic library

$ gcc -c -fPIC rectangle.c
$ gcc -shared rectangle.o -o libgeometry.so 

A C structure in Python is an object that inherits from the Structure in ctypes, and the variables of the structure (in this case, height and width) are called _fields_ in the ctypes structure. So a minimal Python library can be:

# file: geometry_minimal.py

import ctypes as C

CLIB = C.CDLL('./libgeometry.so')
CLIB.area.argtypes = [C.Structure]
CLIB.area.restype = C.c_float

class Rectangle(C.Structure):
  _fields_ = [("height", C.c_float),
              ("width", C.c_float)]

def area(rect):
  return CLIB.area(rect)

were we encapsulated the area function from the C library.

We can use now this library completely seamlessly from python:

>>> import geometry_minimal
>>> r = geometry_minimal.Rectangle()
>>> r.width = 10
>>> r.height = 30
>>> geometry_minimal.area(r)
300.0

Object Oriented Python and C through Ctypes

Memory Layout

When we set the fields of the ctypes structure, we have to be very cautious: the order has to be the same as the order in the original C structure. Studying the reason behind this, we will answer an actual larger question: Why on Earth does this work? First of all: a structure in C is actually a fancy way to name relative memory positions². In this case, height is a float (that takes 4 bytes) that is located 0 bytes relative to the memory location of Rectangle. Similarly, width is a float that is located 4 bytes relative to the memory location of Rectangle (since the first 4 bytes are taken by height)³. So actually the C function area just takes the first 4 bytes starting from rect and the second 4 bytes starting from rect, interpret the data as floats and then multiply each other. How do we take advantage of this behaviour from Python? When we define a Python class inheriting from a C structure, we are saying that the first bytes will be occupied by the _fields_ and the class itself. Every other method or attribute we add, will be added below. So the C function area, when going to the locations we named earlier, will actually find height and width. This is, obviously, as long as we put both the C structure and the ctypes fields in the same order.

Implementation of the Library

If we modify the Python class that inherits from the ctypes structure adding methods or new attributes, the first bytes will remain unchanged. This means that, for the C functions, adding methods and attributes will not change the structure (at least, in the memory positions that were originally available in the C structure). This is what we will use in order to completely encapsulate the C structure as a Python object. As a first example, we can add a simple __init__ function⁴. But the most interesting example is adding methods that were originally C functions. From the explanation above, it is quite evident that we can encapsulate the C area function as a method of the Python object, and the argument that we will use to call the area function has to be the structure itself. So now we can create a more advanced Python library:

# file: geometry.py

import ctypes as C

CLIB = C.CDLL('./libgeometry.so')
CLIB.area.argtypes = [C.Structure]
CLIB.area.restype = C.c_float

class Rectangle(C.Structure):
  _fields_ = [("height", C.c_float),
              ("width", C.c_float)]

  def __init__(self, height, width):
    self.height = height
    self.width = width

  def area(self):
    return CLIB.area(self)

How do we use this library from Python?

>>> import geometry
>>> r = geometry.Rectangle(10, 20)
>>> r.area()
200.0
>>> r.width = 400
>>> r.area()
4000.0

And, finally, the C structure and functions are completely encapsulated. For the end user, there is no difference between this library and a pure Python library, but we are actually running C code.

Conclusions

We were able to encapsulate completely the behaviour of C functions that act on structures as methods of a Python object. Therefire, we have a completely object oriented code, that looks completely pythonic, yet does, in the background, all its calculations in C.