Publishing binaries is annoying. Do it anyway

When I was in grad school, I studied particulate matter and how its concentration could be estimated at ground level. I did so with various machine learning algorithms to combine meteorological and satellite data. Kinda fun. Kinda stressful as grad school is.

All of the algorithms I used came from open-source software. Almost all of them were implementations in scikit-learn. XGBoost, though, is the package I remember most. Not just because it's a cool algorithm (it really is), but because it's implemented primarily in C++ and CUDA with bindings for Python, R, Java, and others. Every time I'd look at the docs to check an API or feature, I'd also spend a little time staring at the installation options, because I'd never delivered software in multiple languages. I truly didn't understand how this happened. Did the authors rewrite the package in each language? How do you keep the package versions and features the same? Can you even maintain feature parity? For that matter, how do you actually publish a package to any package manager? I'd never done that. Since then, I've always wanted to write a library that can be used in multiple languages.

Now, it's true that many other Python packages I was using at the time were the same way, but somehow the bigger packages like pandas and numpy feel like general-purpose software, and I expected them to be usable on any system I had access to. Because XGBoost seemed more specific and likely had a smaller team with less financial support, it didn't feel as likely to be available in many places.

At the time, the only software delivery experience I really had came from my first software consulting job, where we developed primarily .NET APIs deployed only on Windows. Single platform, single architecture, really nothing all that complicated. Also, sometimes our deployment process involved kicking off a release build and manually copying files over. Super exciting and very official. Only a couple of projects had autodeployment.

While my education at A&M had been reasonably expansive in terms of algorithms, parallel computing, and whatever else computer science students do, I'd never learned how to deliver a real, installable binary. In my first job, I was hoping to see how you go from a greenfield project idea to an official, distributable binary. The two deployment paths we had showed me that, and honestly, I found the delivery quite lackluster.

The lack of tooling, though, was not an indication of a lack of DevOps skills by that company. They simply employ just enough technology to get the job done well. For many clients, manual deployment was sufficient, especially since every step was documented and easy to follow. For more complex projects that span multiple servers under higher load, auto-deployment from a CI/CD system is often employed. However, a business-to-business (B2B) software product is not the same as an open-source library deployed across many package repositories across languages, which is the world I live in now, and I had to struggle to learn.

Part of the reason I never went through the process of learning to distribute software is that my professional job didn't need it, nor did any of my personal projects. No need and lack of motivation meant no experience.

When I joined NCAR in 2021, that changed. I began working on open-source atmospheric chemistry modeling software. Specifically, the project I work on (surprise surprise, it's an acronym) is called MUSICA, or the Multiscale Interface for Chemistry and Aerosols. The project's purpose and user base are very different from a B2B application.

Our software is designed to be a general-purpose library for studying any atmospheric chemistry process for use in what we call 0-dimensional box models and global 3-dimensional weather prediction models. We started off with three tenants:

1. All components must be built as a standalone library
2. All components must be runtime configurable
3. All components must have at least 80% test coverage

Somehow, these points have led to quite an extensible, composable, and reasonably stable library and have allowed me to push for an unofficial fourth target:

4. Deliver interfaces of MUSICA to every language where a use case exists

The fourth tenet arose from the success of how we built the project and feels like a happy consequence, also addressing pain points in the scientific modeling community that some projects can't address.

I'll spare you the gory details, but MUSICA's purpose is to allow atmospheric modelers to study the exact same chemical processes at any time and space scale, across models. Naturally, this sounds like a software library with an interface that is reused across models. For mostly historical reasons, this is often not how atmospheric modeling works.

MUSICA is implemented primarily in C++, but we need to interface with 3-D models, which are often written in Fortran. Because of this, we have a C interface, which we wrap with ISO bindings to do language interoperability with Fortran. This was our first cross-language interface and what inspired all of our other language bindings. Some call the C interface, some the C++ interface, and some a mix of both.

While I would love to be installable via a Linux package manager, it is often more useful for models to compile code directly, which is how MUSICA is typically included. Still, it's better to push towards more reusable, composable software.

Since part of MUSICA's purpose is to provide a library for atmospheric chemistry, we do need to make it easy to install and ingest. Below are some of our deliverables, language bindings, and comments on my experience getting our software into these packages.

Spack

Because I want MUSICA to be usable for scientific modeling, I created a spack installer for MUSICA. If you're unfamiliar, spack is a package manager targeting the supercomputing community. I know of several supercomputing administration teams that now prefer to build all of the packages for their supers through spack. This is because spack makes it very easy to build and install specific versions of software for specific compilers, and to manage them on one machine. Each spack package lists all its dependencies, which can make for a smooth installation.

The first PR (the spack community has since moved their packages to their own repository) was easy enough to write. However, I don't use spack on a day-to-day basis, and I found that testing locally was kinda difficult. I had to point the spack tool to the spack packages directory on my local machine, figure out how to configure all the compilers for spack to use, and then I could test. What I actually found and still find most annoying about spack is that you seem to have to tag maintainers directly to get your package merged. I don't like having to bug people to do something that could be automated.

Another pain point, the checks in the spack CI don't actually check that your software can be built and installed. You have to do that yourself, which is very fair. This means the stability of the install and compiler support is still, rightfully, on the library maintainers. However, MUSICA is very well tested, and I have found that the compilers we test with in our CI were failing to install MUSICA simply because some options were missing in spack.

Still, this was the first path we set up that could deliver our software library! Now, I don't think anyone has actually used it outside of our team yet, but it's there as an option! In the future, I would like to integrate MUSICA with a Linux package manager, but this is the closest we are to that for now.

Python

The C interface we developed for Fortran is quite well tested. Right around the time we were finalizing the C interface, I began working with a group of students at Texas A&M. They ended up wrapping our C and C++ interface to create a Python package, musica, which makes our chemistry software available in arguably the most popular language for students in the sciences today.

Delivering to PyPI for a mixed-language project is quite interesting. Because we have compiled code, we do have to compile for each operating system (OS) and computer architecture that we want to support. I learned A LOT about supporting a binary for multiple targets when creating our installable Python wheels. We use a tool called cibuildwheel (see the actual musica workflow here), which helps you build and test your binary Python distributions across all of your OS/architecture targets.

If you only have C/C++ code, this isn't all that hard. The complicating factors for MUSICA are that we have a CUDA-enabled ODE solver and Fortran components. I'll leave the particulars on supporting each language for later, but both of these require special care for specific OS/architecture pairs. Spoiler alert, I have found that compiling Fortran for Windows is as fun as eating glass, but maybe I'm doing it wrong, and someone can point out a better way.

Either way, cibuildwheel's job is to create the wheels for all targets so that you can upload them to PyPI for distribution. This was WAY easier than getting our package into spack. Once you set up trusted publishing with PyPI, your deployments become automatic; it's as easy as triggering a release workflow, which runs whenever we create a GitHub release.

One caveat: someone else used to have a PyPI package called musica. They had not published any release in over 10 years when we started our Python journey. I reached out to the owner after finding their contact information and politely asked if they would transfer the name to us. They did! I was surprised to get the name, but if you don't ask, the answer is always no.

Delivering the Python wheels with only our C/C++ code took me less than a month. Including the CUDA and Fortran components took me about 6 months of work, partly because I had things to learn and partly because I wasn't able to devote my full attention to getting the build right. However, when we finally included all components, I felt a great sense of accomplishment.

Javascript

I'll also leave the details of this for a future post, since this is possible with our WebAssembly build of @ncar/musica (our wrapper code is here), and delivering to npmjs was also relatively simple with trusted publishing.

Admittedly, a JavaScript interface to atmospheric chemistry modeling software is probably mostly useful by my team specifically. We plan to use it for software targeting education and outreach, but we also have some practical box-modeling purposes we can do in the browser that directly support atmospheric chemistry research.

Julia

By far the most annoying of our interfaces was the Julia interface. We set this up in early 2026 to support an intern this summer who will be integrating our chemistry software into CliMA, a Julia research model.

It seems that the Julia community decided to do nearly everything on GitHub. On the surface, that sounds good, but in practice, it leads to PRs that need to be approved by random humans whom I have to bother to get packages merged. Some things can be automated, but the initial setup requires human approval. This was very much like the spack build, but one positive of the Julia delivery over spack is that your package is built for each OS/architecture pair when you submit to their system, much like cibuildwheel. This does provide some comfort that, once your package is accepted, a downstream user can install it without issue.

I also plan to talk about this more in-depth because there were MANY steps from start to finish.

I feel quite content that I'm working on software that will hopefully provide utility to researchers and students studying atmospheric chemistry. I feel even more empowered by how many platforms and languages we deliver this in, so that hopefully there's some new student in their academic journey who sees what my team has done and also feels inspired.

Science and the tools we use to study it are only useful if they are easily accessible. There is a problem in scientific modeling, tooling, and publishing where something works likely only for the original developer (who is often the only user). This can lead to duplicated efforts across research teams and unreplicable science.

As a software engineer supporting scientists, I can help address this problem by creating language wrappers and delivery targets as I listed above. Almost all of the problems I faced, which I'll detail more in the future, were DevOps problems. The community of scientists I work with is wicked smart. However, figuring out the nuances of multiple package managers and cross-language feature support may not always be the most appealing problem for them to solve, because it isn't science.

Also, it's neat to see how the work we do goes from code to an easily installable package in multiple languages. That brings me far too much joy. I hope that if you are in a position to support open-source science, you undertake part of this journey as well!

AI Disclaimer:

I did not use AI to generate or plan any part of this post. The only writing tool I used was Grammarly. Grammarly corrected several typos, verb tenses, punctuation mistakes, and occasionally suggested a word that I liked better. Also, some sentences were restructured by Grammarly. However, all of the ideas and the flow of the piece came from my own meat-space brain.