Unable to use LTO in Bazel.

[PiotrSikora]

For context, I'm working on updating Proxy-Wasm C++ SDK from our custom toolchain to @emsdk (see: proxy-wasm/proxy-wasm-cpp-sdk#132) and I'm seeing some regressions in terms of features.

$ bazel build -c opt //example/...

[...]
Target //example:http_wasm_example.wasm up-to-date:
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm.map
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js.mem
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.fetch.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.worker.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.data
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js.symbols
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm.debug.wasm
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.html

bazel build -s -c opt --linkopt=-flto //example/...

[...]
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/lto/crt1_reactor.o" 

Same without --no-entry for commands:

Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/lto/crt1.o"

[sbc100]

I think the problem is that we don't build all libraries in all configurations when building the emsdk tar archives.

In order to fix this we would need to have bazel create additional libraries, perhaps in a supplemental archive, which are also archived somewhere. I'm not sure what the best way to do that might be? @dschuff @kripken .. should be look into adding pre-built lto versions of all the standard libraries to the emsdk download? How much do we care about marginal increases in download size?

[PiotrSikora]

But that used to work, since previously those missing libraries were built on demand.

However, @emsdk sets FROZEN_CACHE=True, and Bazel stores Emscripten's cache on a read-only mountpoint, which prevents that from happening.

[sbc100]

Yes, the bazel toolchain does not support the "build libraries on demand" feature of emscripten. I think this is generally a good thing.. one does not generally want the contents of the sysroot to change during a build.

[PiotrSikora]

Yeah, I agree it's a good thing, once things are working :)

[kripken]

If there is a strong reason to add more builds by default that might make sense. But the matrix of combinations is constantly increasing...

In general I think the right workflow would be to invoke embuilder.py to generate the necessary libraries before anything else, using something like this for LTO:

embuilder.py build MINIMAL --lto

That is, to fully construct the necessary sysroot first. Could the bazel support do that automatically perhaps? (Does it have the necessary information? Sorry, I'm not familiar with bazel.)

[sbc100]

I think the problem is that bazel pulls the .tar.gz files that we generate.

@PiotrSikora @walkingeyerobot do you know if we can run a kind of "post-download" step on the downloaded archive? That would be a good solution I think.. as long as it was a just a one-off thing that run only on first download.

[PiotrSikora]

You can override the patch_cmds to do whatever post-processing you want, but it's a bit of a hack.

This is what we're currently doing:

http_archive(
    name = "emscripten_toolchain",
    build_file = "@proxy_wasm_cpp_sdk//:emscripten-toolchain.BUILD",
    patch_cmds = [
        "./emsdk install 2.0.7",
        "./emsdk activate --embedded 2.0.7",
    ],
    strip_prefix = "emsdk-2.0.7",
    url = "https://github.com/emscripten-core/emsdk/archive/2.0.7.tar.gz",
    sha256 = "ce7a5c76e8b425aca874cea329fd9ac44b203b777053453b6a37b4496c5ce34f"
)

[sbc100]

In that case I think you should be able to run ./upstream/emscripten/embuilder build SYSTEM --lto to get the libraries you need pre-built.

We should also do this for the un-patched toolchain I think.

[PiotrSikora]

In that case I think you should be able to run ./upstream/emscripten/embuilder build SYSTEM --lto to get the libraries you need pre-built.

This actually doesn't work. While patch_cmds could be used when building @emscripten_toolchain from sources, this is not something that can be easily used with @emsdk, which downloads prebuilt binaries.

I tried to make this work, patching @emsdk to use patched @emscripten_toolchain instead of the prebuilt binaries, but re-building everything and installing npm dependencies twice, and maintaining hacks on top of @emsdk doesn't seem to be worth the trouble.

Is there a chance that we could include LTO system in the official binaries? Looks like I'm not the only one that asked about this (see: #807).

[sbc100]

emsdk still relies on shipping just a subset of libraries pre-built, while leaving some libraries to be built on demand. This allows is to strike a balance between build time and download size and user convenience. Even if we start shipping LTO libraries there are also maybe other configuration for the system libraries: LTO + debug, -fPIC, -fPIC + debug, -fPIC + LTO + debug, ThinLTO, -fPIC + ThinLTO...etc.. I think we want avoid shipping all combinations of all libraries all the time.

I wonder if we can come up with a way to allow bazel to allow built-on-demand libraries? Or maybe to allow users to inject their own overlay of extra pre-built libraries.

Possible solutions:

1. Allow bazel users to inject extra/overlay archives of system libraries
2. Allow bazel builds to build libraries on demand in a temporary location
3. Describe the system libraries themselves as BUILD files and have them build by Bazel just like user code (I'm guess bazel has some kind of clever cache here that makes this efficient, unlike (2))
4. Build a separate tar archive of a hyper-sysroot that includes all the prebuilt libraries (not sure how/when to build and upload this).

Having said all that as a short term solution we could consider adding just one more configuration (i.e. doubling the amount of libraries we include in the tar archive).

[PiotrSikora]

• Describe the system libraries themselves as BUILD files and have them build by Bazel just like user code (I'm guess bazel has some kind of clever cache here that makes this efficient, unlike (2))

I think this would be the perfect solution (even ignoring LTO issue, prebuilt libraries should be avoided in Bazel ecosystem).

Could we get this done? cc @trybka @walkingeyerobot

[trybka]

This is definitely on our TODO list, but basically I was the only person who thought this was useful. Having another use-case is motivating.