My visualization library [1] is written in C and exposes a visualization API in C. It is packaged as a Python wheel using auto-generated ctypes bindings, which includes the shared library (so, dylib, or dll) and a few dependencies. This setup works very well, with no need to compile against each Python version. I only need to build it for the supported platforms, which is handled automatically by GitHub Actions. The library is designed to minimize the number of C calls, making the ctypes overhead negligible in practice.

[1] https://datoviz.org/

I realize I'm talking about C++ not C, but coincidentally just today I ported our 7 year old library's Swig/Python interface to nanobind. What a fragile c9k Swig has been all these years (don't touch it!) and the nanobind transformation is so refreshing and clean, and lots of type information suddenly available to Python programs. One day of effort and our tests all pass, and now nanobind seems able to allow us to improve the ergonomics (from the Python pov) of our lib.

Lots of articles focus on Cython and optimizing Python using C code.

This article is about embedding Python scripts inside a C codebase

I did a lot of this for my raytracer, c-ray [1]. Originally it was just a self-contained C program, but I got tired of writing buggy and limited asset import/export code, so eventually I put together a minimal public C API [2] that I then wrapped with CPython bindings [3] and some additional python code [4] to expose a more 'pythonic' API. It's all still a WIP, but it has already allowed me to write a Blender plugin [5], so now I can play around with my renderer directly in Blender, and test with more complex scenes others have made.

Fun project, and it's really cool to see my little renderer in the interactive viewport in Blender, but I have also learned that I don't particularly enjoy working with non-trivial amounts of Python code.

[1] https://github.com/vkoskiv/c-ray [2] https://github.com/vkoskiv/c-ray/blob/51a742b2ee4d0b570975cd... [3] https://github.com/vkoskiv/c-ray/tree/51a742b2ee4d0b570975cd... [4] https://github.com/vkoskiv/c-ray/tree/51a742b2ee4d0b570975cd... [5] https://github.com/vkoskiv/c-ray/tree/51a742b2ee4d0b570975cd...

Once I needed to implement a simple python plugin engine in a C/C++ software, I've been successfully using the official guide [1].

[1] https://docs.python.org/3/extending/embedding.html

This is one of the "killer apps" for Nim. Nim makes makes it easy to wrap C and easy to talk to Python (via Nimpy).

The title of the article is misleading. Making C and python talk to each other implies, calling python from C and calling C from python. The article only covers the former.

Just a little nitpick but using Py_BuildValue is much better to generate PyObjects from C if you are doing more than one of them instead of populating a list and converting it to a tuple to pass to the python function.

My general rule (when doing the opposite of passing C values to python as part of an extension) is to use the dedicated function, like PyLong_FromLong, when there is a single return and Py_BuildValue (with its format string for automagic conversion) when the function returns a tuple.

Oh, and if you are checking your return values from python (you are, right?) using Py_XDECREF isn't all that good of an idea since it will mask some flawed logic. I pretty much only use it when a PyObject pointer could validly be NULL (like with an optional function argument) and I'm cleaning up right before throwing an exception. Tracking python reference counts is a whole blog post in itself since some functions steal references and others don't and if you get it wrong you can easily crash the interpreter and/or create memory leaks.

Thanks a lot for the article. Here's a QQ: did you measure the time of some basic operations python vs C? (e.g. if I do a loop of 10 billion iterations, just dividing numbers in C and do the same in python, and then import these operations into one another as libraries, does anything change?)

I'm a beginner engineer so please don't judge me if my question is not making perfect sense.

Lots of people argue that AI R&D is currently done in Python because of the benefits of the rich library ecosystem. This makes me realize that's actually a poor reason for everything to be in Python since the actually useful libraries for things like visualization could easily be called from lower level languages if they're off the hot path.

I’ve been curious, what are the motivations for most projects to use Lua for enabling scripting in C over this? Is the concern around including an entire Python interpreter in a project and Lua is lighter?

Or this?

https://docs.python.org/3/extending/embedding.html

Useful, I’m going to be doing something similar w/C++ soon.

How does this compare to pybind11?

Isn't this the whole point to cffi and cython?