A new release for soundfile

[bastibe]

This is a continuation of the discussion in #310, where I'm working on packaging a new version of soundfile.

It appears that the packaging problem is more complicated than anticipated. Our existing build system will probably need to be replaced, which will take some time.

If anyone would like to help me package soundfile, I'd be grateful!

The problem is, we need to package four wheels for

1. Windows 32 Bit
2. Windows 64 Bit
3. macOS Intel
4. macOS Arm

each containing their own libsndfile{.dll,.dylib}, and making sure that the right version gets installed on the right platform.

We have a build script in place for building packages for 1-3, but I don't yet know how to build wheels specifically for macOS Arm, and it seems that we should convert our build system to use PEP 517 (pyproject.toml/build et al).

I only had an hour or so today to work on this, so there wasn't much progress. I'll keep you updated.

[chuma9615]

Thanks for taking time to solve this, If u need any helping hands let me know

[HexDecimal]

pyproject.toml is super nice but I don't think it will resolve your current issues with wheels. What it can do is make it so that cffi is installed before the setup.py script is run which would crash otherwise.

I'm not sure about the specific CLI flags. I think it's adding -arch arm64 to the C/link flags will work it if the tools support it. Or -arch arm64 -arch x86_64 for a Universal 2 library.

The platform tags would be something like macosx_11_0_arm64. If the library supports both arm64 and x86_64 then the platform tag would be macosx_10_9_universal2 instead (pulling examples from my own projects, I don't know how to figure out the best version numbers.)

For my own projects I have to rely on GitHub Actions to build MacOS packages/libraries for me as downloadable artifacts. I can write a workflow for you if you're unable to compile the arm64 library yourself.

[bastibe]

Thank you so much for your help!

If you could point me to an example of how to have Github compile the macOS libraries, that would be fantastic! That might actually allow us to compile the wheels separately on each OS, instead of the current franken-system that packages precompiled libraries with the generic python code.

[HexDecimal]

This would be an example of a workflow. This script would be stored in a path like .github/workflows/build-libs.yml in the repository, and will take affect as soon as it's pushed to the repo.

name: Compile libraries

on:
  push:
  pull_request:

jobs:
  build-libs:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ["macos-11"]
      fail-fast: true

    steps:
      - uses: actions/checkout@v2
      - name: Compile library
        run: ./mac_build.sh
      - uses: actions/upload-artifact@v2
        with:
          name: libs
          path: "*.dylib"
          retention-days: 7
          if-no-files-found: error 

I imagine this running the mac_build.sh script from bastibe/libsndfile-binaries, so this workflow would be added there for this example. You can also see the docs for actions/upload-artifact.

GitHub has good documentation on making workflows. Build artifacts uploaded this way can be downloaded from another job, such as a job that builds the wheel.

[bastibe]

This worked amazingly well, thank you a whole lot!

At the moment, Github runners do not yet provide M1 machines, but that's not super important at the moment, since @chuma9615 and @jurihock have already gracefully provided pre-build libraries in libsndfile-binaries#7 and libsndfile-binaries#8, respectively.

I'll investigate cross-compilation once I've managed to build wheels.

[bastibe]

I have drafted a new release at releases/0.11.0b1.

There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.

If anyone would like to contribute up-to-date Python libraries compiled with Visual Studio 14+, I'd be grateful for a pull request in bastibe/libsndfile-libraries. Or better yet, integrate it into the CI scripts to build the binaries automatically.

Also, you would help me tremendously if anyone could test the provided wheels on macOS and M1!

[SpotlightKid]

Will you provide source distribution archives (tarballs) for the latest and upcoming releases or should Linux distro packagers use the source archives automatically created by Github?

[bastibe]

I will upload a source distribution.

The reason I didn't attach it to the draft release linked above is that the current build script still bundles all the libraries with the source dist, making it unnecessarily big. This will need fixing before the final release is cut.

[HexDecimal]

There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.

Looks like if you compile to v140 then that binary will support all versions of Python after 3.5. The instructions I've found for telling MinGW to use specific runtimes seems convoluted so maybe it would be easier to use Vcpkg to get these builds.

[HaHeho]

Or better yet, integrate it into the CI scripts to build the binaries automatically.

I'm inexperienced with all the required compilation steps and CI. But I remembered that Mathias uses a script which builds a small version of libsndfile on Mac during some Github actions, build-small-libsndfile.sh. Forgive me if this doesn't help at all. :)

[bastibe]

@SpotlightKid I just uploaded a fixed sdist to the latest tag.

Right now, all that's missing for a release is

• someone should test the M1 wheels.
• I just noticed that the macos x64 wheel does not install. Any help as to why would be greatly appreciated.
• I've opened a pull request for the new release in Release 0.11.0 #326 and noticed that the automated tests no longer run. Does anyone know why?

@HaHeho thank you for your response. Thanks to @HexDecimal, we now have automated builds for Windows and macOS! That's wild!

[HexDecimal]

... and noticed that the automated tests no longer run. Does anyone know why?

If you check the commits on GitHub then TravisCI's last status check was Nov 26, 2019. It stopped working sometime after that, probably when they dropped support for the travis-ci.org site. You'd have to migrate to travis-ci.com, but I don't think that site actually gives free time for FOSS projects. You might get a one time allotment to work with while you move to something else. I remember dropping TravisCI for GitHub Actions over this.

More info: https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567

[SpotlightKid]

@SpotlightKid I just uploaded a fixed sdist to the latest tag.

Thanks for the heads up. The URLs for the attached files still have 0.11.0b1 in the path, which is slightly inconvenient. But I'm assuming this will be correct when a proper release is made.

[chuma9615]

@bastibe Tested the wheel on a M1 machine and it's working

Note that I tested soundfile-0.11.0b2-py2.py3-none-any.whl