Multi architecture automated builds for Dart binaries

17May23

One of my favourite features of Dart is its ability to create executables (aka ahead of time [AOT] binaries)[1].

Creating binaries for the platform you’re running on is very straightforward, just dart compile exe but Dart doesn’t presently support cross compilation for command line binaries, unlike Rust and Go, which have also surged in popularity. This is a great shame, as Dart clearly can cross compile when it creates Flutter apps for iOS and Android[2].

For stuff we release at Atsign we often need to create binaries for a wide variety of platforms and architectures. In part that’s because we make things for the Internet of Things (IoT) market, and there’s a wide variety of dev boards etc. out there to suit different needs and (power) budgets. Our SSH No Ports tool is packaged for:

macOS
- arm64
- x64
Linux
- arm
- arm64
- riscv64
- x64

There are things where we might also add Windows (x64 and maybe even arm64) to that list.

x64 is easy

Using GitHub Actions it’s relatively easy to build the x64 stuff. All I need is a matrix:

jobs:
  x64_build:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, macOS-latest]
        include:
          - os: ubuntu-latest
            output-name: sshnp-linux-x64
          - os: macOS-latest
            output-name: sshnp-macos-x64

and then I can run through the steps to build something e.g.:

    steps:
      - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
      - uses: dart-lang/setup-dart@d6a63dab3335f427404425de0fbfed4686d93c4f # v1.5.0
      - run: mkdir sshnp
      - run: mkdir tarball
      - run: dart pub get
      - run: dart compile exe bin/activate_cli.dart -v -o sshnp/at_activate
      - run: dart compile exe bin/sshnp.dart -v -o sshnp/sshnp
      - run: dart compile exe bin/sshnpd.dart -v -o sshnp/sshnpd
      - run: cp scripts/* sshnp
      - run: tar -cvzf tarball/${{ matrix.output-name }}.tgz sshnp
      - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
        with:
          name: x64_binaries
          path: tarball

Everything else needs a bit more effort

At the moment GitHub Actions only supports x64. Arm64 for MacOS is on the roadmap for public beta in Q4-23, but that leaves a bunch of other platform:architecture combinations that can’t be done natively in a hosted Actions runner.

Of course we could use self hosted runners, but that throws up a bunch of other issues:

They’re not recommended for open source projects, due to security issues.
We’d be running a bunch of VMs (or even hardware) full time, and hardly using it.
RISC-V isn’t supported yet.

I also took a look at Actuated, which makes use of ephemeral Firecracker VMs to get around security concerns. It certainly helped with arm64, but wasn’t able to move the needle on armv7 and RISC-V.

For Linux at least Docker Buildx can help

Multi-platform images provide a convenient way to harness QEMU to build Linux images for a variety of architectures, and is obviously the way to go for Docker stuff. But in this case we want a binary rather than a Docker image. It turns out that Buildx can help with this too, as it has a variety of output formats. So I can construct a multi stage build that results in a tarball:

FROM atsigncompany/buildimage:automated@sha256:9abbc3997700117914848e6c3080c4c6ed3b07adbd9a44514ce42129a203a3c5 AS build
# Using atsigncompany/buildimage until official dart image has RISC-V support
WORKDIR /sshnoports
COPY . .
RUN set -eux; \
    case "$(dpkg --print-architecture)" in \
        amd64) \
            ARCH="x64";; \
        armhf) \
            ARCH="arm";; \
        arm64) \
            ARCH="arm64";; \
        riscv64) \
            ARCH="riscv64";; \
    esac; \
    mkdir sshnp; \
    mkdir tarball; \
    dart pub get; \
    dart compile exe bin/activate_cli.dart -v -o sshnp/at_activate; \
    dart compile exe bin/sshnp.dart -v -o sshnp/sshnp; \
    dart compile exe bin/sshnpd.dart -v -o sshnp/sshnpd; \
    cp scripts/* sshnp; \
    tar -cvzf tarball/sshnp-linux-${ARCH}.tgz sshnp
    
FROM scratch
COPY --from=build /sshnoports/tarball/* /

What I actually get there is a bunch of architecture specific tarballs, inside a tarball, but it’s easy enough to winkle them out with an Actions workflow job:

  other_build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
      - uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0
      - uses: docker/setup-buildx-action@4b4e9c3e2d4531116a6f8ba8e71fc6e2cb6e6c8c # v2.5.0
      - run: |
          docker buildx build -t atsigncompany/sshnptarball -f Dockerfile.package \
          --platform linux/arm/v7,linux/arm64,linux/riscv64 -o type=tar,dest=bins.tar .
      - run: mkdir tarballs
      - run: tar -xvf bins.tar -C tarballs
      - run: mkdir upload
      - run: cp tarballs/*/*.tgz upload/
      - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
        with:
          name: other_binaries
          path: upload

The main problem with this is that emulation makes things super slow. Each AOT binary takes about 10m to compile, leading to a 30m build process for a package with 3 binaries. As this guide illustrates, things would be much faster with a tool chain that supports cross-compilation (like Golang). We work around this by sending a chat notification when the building is done:

  notify_on_completion:
    needs: [x64_build, other_build]
    runs-on: ubuntu-latest
    steps:
      - name: Google Chat Notification
        uses: Co-qn/google-chat-notification@3691ccf4763537d6e544bc6cdcccc1965799d056 # v1
        with:
          name: SSH no ports binaries were built by GitHub Action ${{ github.run_number }}
          url: ${{ secrets.GOOGLE_CHAT_WEBHOOK }}
          status: ${{ job.status }}

If you’ve got this far you might also want to check out Bret Fisher’s guide to multi-platform-docker-build, which opened my eyes to a few things.

Notes

[1] I’ve talked about this (and some of the trade offs) at QCon Plus in my Full Stack Dart presentation, and QCon SF in my (yet to be published) Backends in Dart.
[2] J-P Nurmi has documented how the SDK can be hacked for Cross-compiling Dart apps, but it’s not something I personally want to spend time maintaining.

Filed under: Dart | Leave a Comment
Tags: Actions, architecture, ARM, Arm64, Armv7, binary, Buildx, Dart, Docker, github, Linux, MacOS, matrix, multi, platform, RISC-V, riscv64, tar, tarball, x64

No Responses Yet to “Multi architecture automated builds for Dart binaries”

Feed for this Entry Trackback Address

Leave a Comment

	Tim Coote on Ross Anderson RIP
	iain on Skiing in Espace Killy (Val…
	Chris Swan on Getting more from a British Ga…
	Murray Cowell on Getting more from a British Ga…
	JL on AI MacGuffin