Dump buildbot for GitHub Actions.

[icculus]

Task list

This is what the buildbot covered, plus some other wishlist items. Some we may want to drop (QNX? Windows Phone?), others might be covered in other ways (for example, the multiple macOS architectures might all be caught by the Xcode work, but we might still want to make sure we have configure and cmake coverage on those platforms generally, so these don't all have to be separate GitHub Actions).

On every commit/pull request

• Linux/amd64, configure (no longer in SDL3)
• Windows/amd64, CMake
• macOS/amd64, CMake

Occasionally, on a schedule

• Android/arm64
• Emscripten/wasm
• FreeBSD/amd64
• Haiku/amd64
• iOS
• Linux/x86
• Linux static analysis
• macOS/amd64
• macOS/arm64
• macOS with xcode project files
• macOS static analysis
• mingw64/amd64
• mingw64/x86
• OpenBSD/amd64
• OS/2 with OpenWatcom
• QNX/amd64 (still available in SDL3 but a builder is impractical, see comments.)
• QNX/x86 (still available in SDL3 but a builder is impractical, see comments.)
• QNX/arm64 (still available in SDL3 but a builder is impractical, see comments.)
• QNX/arm32 (still available in SDL3 but a builder is impractical, see comments.)
• Raspberry Pi
• WinRT/UWP
• Windows/amd64 with Visual Studio project files
• Windows/x86 with Visual Studio project files
• Windows Phone/arm32 (no longer offered in SDL3)
• PSP
• PS Vita
• Nintendo 3DS
• RiscOS
• CMake where possible
• configure script where possible (no longer offered in SDL3)
• static analysis where it dramatically improves code coverage (Windows, Linux, macOS...others less so.)
• upload builds where possible

Original comments

(This is just a brain dump for now.)

This is not for the 2.0.18 milestone, but I've been thinking about the time I spend making sure the buildbots continue to stay up and running, and watching things like the PS Vita port get added to GitHub Action's CI system with a few lines of configuration, and wondering why we're messing with this thing and not just making it another service we moved to GitHub so it Just Works and is Nicely Integrated.

Also, even though the buildbot master is hosted on libsdl.org, the actual builders are on a workstation in my office; if I get hit by a bus tomorrow, the builders last until the next power outage, or Windows update automatic reboot, or whatever. Since this is some beefy hardware, running about a bazillion vmware instances, it wasn't really technically or financially feasible to run it on Digital Ocean with the rest of libsdl.org.

Some of the Buildbot builders are obviously redundant now--Linux, macOS, Windows, even Emscripten--and can be shut off right now in favor of the GitHub Actions counterparts...but there are some useful tools and esoteric platforms on Buildbot that are less trivial to duplicate on GitHub.

Notably:

• Static analysis isn't hooked up to a GitHub Action, and it's worth noting that on pretty beefy hardware, this takes quite some time per-commit...it's not uncommon for static analysis to take 8-10 minutes, but it's also not clear how much of that is competing for CPU time verses other builders and virtual machines, and would be less of an issue on GitHub's server farm. Once you cook through all of GitHub's free minutes, you have to pay for CPU time, and this might be the sort of thing that pushes it over the limit, though. In practice, we could limit static analysis to just pushes to main and not pull requests (although it would be useful especially on pull requests!). We might have to cross this bridge once we come to it. Right now we have static analysis on macOS and Linux, but I would love to get it on as many platforms as humanly possible. The more code coverage you can get, the more free proactive debugging labor this thing generates. In practice, this should work anywhere that has a reasonable Clang install.
• Less mainstream platforms: right now we have FreeBSD, OpenBSD, and Haiku in vmware on the buildbot, but these platforms are not available on GitHub's servers. GitHub has a tool where you run your own builders but otherwise hook into GitHub Actions, but we'd have to turn off Pull Requests for those platforms (otherwise these can run arbitrary code, which we don't care about on GitHub because the OS instance has no meaningful data to steal or live on a private LAN and is blown away after each build), and crucially, we'd have to continue to host the builders. Also, GitHub's tool doesn't work on any of these platforms, although apparently this third-party app is an option: https://github.com/ChristopherHX/github-act-runner ...but it might be better to move these to cross-compilers on a standard Linux install or something and hope it's "close enough" with a Haiku sysroot and not a full Haiku install.
• Some of these are building weird targets on a standard Linux install; the OS/2 port is built with the Linux version of OpenWatcom, for example. This part is absolutely doable, but we'd have to figure out how to get those tools in-place on each iteration. That's a matter of research but probably isn't hard.
• QNX's builds can be done on (Intel) macOS or Linux, but it's with non-public tools and requires a license file, so that's absolutely hitting the end of the line here, as we probably can't get that onto GitHub's servers in a reasonable way. But as I've said recently, it's probably time to ditch the QNX builds, as the tools I have are years out of date now.
• Several (but not all) targets on the buildbot upload binaries for almost every commit, and I think it's useful to be able to tell people "these are presented as-is, but if you just need a DLL fast, you can grab it here," and I'd like to keep that if we dump buildbot (but doing so is not a dealbreaker for me). Likewise for uploading the static analysis HTML output; if that can't be hosted at GitHub, we'll need to upload it to libsdl.org somehow for viewing.

Anyhow, these are things to think about. In a perfect world, we have all the existing functionality migrated to GitHub Actions and the buildbot turned off in the 2.0.20 timeframe, but we'll have to see what our options are.

[sezero]

Some of the Buildbot builders are obviously redundant now--Linux, macOS, Windows, even Emscripten--and can be shut off right now in favor of the GitHub Actions counterparts...

Not exactly true: Buildbot uses autotools, Github Actions only use CFake:
Github actions must be updated to also use autotools (which would most
importantly cover MinGW), then you can consider buildbot redundant.

[icculus]

Using the configure scripts wouldn't be a problem, I think; I assume we used CMake on GitHub Actions because it was what the original implementor chose, but also because one small config could handle a bunch of platforms. But I assume changing this is easy enough.

(If we want to get wild, we could have it build the appropriate platforms with both to catch build project bugs.)

[sezero]

(If we want to get wild, we could have it build the appropriate platforms with both to catch build project bugs.)

That would be preferable

[slime73]

Several (but not all) targets on the buildbot upload binaries for almost every commit, and I think it's useful to be able to tell people "these are presented as-is, but if you just need a DLL fast, you can grab it here," and I'd like to keep that if we dump buildbot

Github Actions can upload artifacts to Github as part of the build process. Here's an example of that in action, at the bottom of the page here: https://github.com/love2d/love/actions/runs/1203406489 (I imagine it wouldn't be hard to upload HTML files when applicable, too).

However I'm not sure if there's an easy way to grab the latest artifact URLs from a certain branch, or to take a commit hash as input and get the github actions and artifact URLs from that. Maybe there are other tools which can do that...

[ericoporto]

for HTML files you could push to a new repository and make GH Pages activated there, so you can read directly.

[InfoTeddy]

However I'm not sure if there's an easy way to grab the latest artifact URLs from a certain branch, or to take a commit hash as input and get the github actions and artifact URLs from that. Maybe there are other tools which can do that...

There is also the issue that GitHub Actions artifacts are only available to people who have logged in to GitHub.

The solution to both of these issues is to publish a new release from Actions for each run, which is also possible to do.

[icculus]

Some of these are building weird targets on a standard Linux install; the OS/2 port is built with the Linux version of OpenWatcom, for example. This part is absolutely doable, but we'd have to figure out how to get those tools in-place on each iteration. That's a matter of research but probably isn't hard.

I took care of this one. The magic is the "actions/cache@v2" thing which basically makes the OpenWatcom install instantaneous if we've successfully run before. If the cache is removed--which happens periodically--we fall back to installing it again, which is also not slow since it's just unzipping a file downloaded from GitHub's web servers.

We can tweak build-scripts/os2-buildbot.sh to use binl64 instead of binl (the buildbot was running on the 32-bit Linux install, just to spread builds across virtual machines more evenly, but this isn't an issue with GitHub Actions), but otherwise, OS/2 is hooked up and good to go now.

Anyhow, I just wanted to try this out real fast; I'll hit GitHub Actions hard after 2.0.20 is finalized in the next few days.

[icculus]

We can tweak build-scripts/os2-buildbot.sh to use binl64 instead of binl

Done in ae9e214.

[icculus]

I think we're going to abandon QNX. It needs a commercial license key and I don't think that's something we can get away with in a public repo, since we'd have to put it and the SDK someplace visible for the build machine to download it.

Also my SDK is years out of date.

Obviously we'll still accept patches and aren't removing anything, we just won't have a builder for it.

[icculus]

@pionere has a patch for build artifacts; apply this or something like this before we're done: pionere@bf08b8f

[icculus]

So something that Sam and I discussed: we really want the per-push and per-pull-request actions to just be a fast check...maybe just Linux/Mac/Windows in a reasonable default, so we can see right away if a commit is obviously incorrect in the common cases.

GitHub Actions lets you run things on a schedule (literally using cron * * * * * syntax!), so it would be useful to use that once a day and just go nuts with tons of time-consuming builds...all the platforms we can, static analysis, nightly build artifacts, etc. So once a day, or once a week, whatever, if there was a commit to check, it'll cook through everything and we can clean up the trash in the morning.

[pionere]

@icculus : you might want to look at this example.
It checks every day if there was a commit in 24 hours, and runs only if necessary (should_run).