TL;DR

PyPI provides a neat way of distributing binaries from other languages, and Python venvs make it easy to run different versions side by side. This post takes a look at how to do that with Dart, and the next steps necessary to do a proper job of it.

Background

A few days ago I wrote about Using a Python venv to run different versions of CMake, which works because the CMake binaries are poured into a Python package that’s hosted on the Python Package Index (PyPI)[1].

CMake isn’t the only thing that’s distributed that way. Zizmor is a Rust tool for analysing GitHub Actions that offers PyPI as an installation option (perhaps because its author William Woodruff had a big hand in creating the trusted publisher mechanism for PyPI). Pull the thread on Zizmor, and it leads to Maturin, a tool specifically for putting Rust stuff into Python packages, with a whole bunch of projects using it.

CMake can also be used to build Python packages with py-build-cmake.

There’s more… Simon Willison wrote “Bundling binary tools in Python wheels” when he discovered he could get Zig from PyPI.

Why?

• The Dart (and Flutter) package manager pub.dev is source code only, and doesn’t deal with binaries[3].
• Python venvs provide a nice way to run different versions of the same binaries side by side.
• I’m hopeful that Python packaging provides a stepping stone to apt packages on Kali (and maybe Ubuntu, Debian etc.) as there’s a mature process and tooling for Python stuff, but I’ve not yet fully unpicked whether binaries are (re)created from source or downloaded from PyPI.

Example repo

After some experimentation I threw together a test repo: cpswan/dart_echo_py. The key pieces are:

build.sh

A simple script that compiles the Dart source in the src directory and creates binaries in bin. Then runs python -m build to create packages in the dist directory.

pyproject.toml

This provides all the project metadata that’s used by python -m build to construct a package.

Most importantly the [project.scripts] section provides the entry points to the binaries so that they’re presented on the path for whatever (virtual) environment the package is installed into.

__init.py__

This provides a wrapper around the binaries so that command line arguments are passed into the correct binary in the place where it’s been installed.

Todo

So far I’ve got the bare bones working, but there are a few more things I need to get straightened out before I start putting packages onto proper PyPI rather than TestPyPI:

Multi-arch

The simple build script creates binaries on whatever platform it runs on (Linux x64 in my case) and packages them up into a wheel named dart_echo_py-0.1.5-py3-none-any.whl. That py3-none-any suffix is incorrect, as the binary is very definitely not platform independent. The correct suffix is (something like) manylinux_2_12_x86_64.manylinux2010_x86_64, but I should probably also be building binaries for the other Dart AOT platforms. Arm64 is now easy given that Dart 3.8 does cross compilation, but armv7 and riscv64 are also in reach (if I do builds in Docker).

It should also be possible to target musl based environments (like Alpine) using dart-musl.

Source without binary pollution

When I look at the source tarball created by python -m build it also includes the binaries :(

Automation

A simple build script works for a single architecture build, but to support multi architectures it’s going to make sense to use some automation, which naturally takes me to GitHub actions.

And once the build process is automated that opens the door to build attestations for SLSA etc. so that the resulting packages have some provenance.

It should also be possible to ditch Twine in favour of trusted publishing to PyPI.

Notes

[1] Pronounced ‘pie pee eye’ by those in the know. PyPy ‘pie pie’ is another Python thing altogether.
[2] I could have used that with Dart, but I didn’t want to add to the complexity and confusion by bringing in another tool.
[3] This makes perfect sense in the context of Dart mostly being used for Flutter, and Flutter apps mostly ending up in the Apple App Store and Google Play.