Fix most ports not being overrideable with EMCC_LOCAL_PORTS

[SkyLeite]

Fixes #8466

[SkyLeite]

Huh, this actually doesn't seem to work for some reason. Attempting to use multiple ports gives me an error about the cache being locked:

srb2kart> Traceback (most recent call last):
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 4436, in <module>
srb2kart>     sys.exit(main(sys.argv))
srb2kart>   File "/nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/lib/python3.10/contextlib.py", line 79, in inner
srb2kart>     return func(*args, **kwds)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 4429, in main
srb2kart>     ret = run(args)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 1295, in run
srb2kart>     linker_inputs = phase_compile_inputs(options, state, newargs, input_files)
srb2kart>   File "/nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/lib/python3.10/contextlib.py", line 79, in inner
srb2kart>     return func(*args, **kwds)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 3111, in phase_compile_inputs
srb2kart>     compile_source_file(i, input_file)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 3089, in compile_source_file
srb2kart>     cmd = get_clang_command(input_file)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 3023, in get_clang_command
srb2kart>     return get_compiler(src_file) + get_cflags(state.orig_args, use_cxx(src_file)) + compile_args + [src_file]
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/emcc.py", line 1032, in get_cflags
srb2kart>     ports.add_cflags(cflags, settings)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/tools/ports/__init__.py", line 416, in add_cflags
srb2kart>     port.get(Ports, settings, shared)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/tools/ports/ogg.py", line 18, in get
srb2kart>     ports.fetch_project('ogg', f'https://github.com/emscripten-ports/ogg/archive/{TAG}.zip', sha512hash=HASH)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/tools/ports/__init__.py", line 236, in fetch_project
srb2kart>     with cache.lock('unpack local port'):
srb2kart>   File "/nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/lib/python3.10/contextlib.py", line 135, in __enter__
srb2kart>     return next(self.gen)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/tools/cache.py", line 61, in lock
srb2kart>     acquire_cache_lock(reason)
srb2kart>   File "/nix/store/gj1xsg304vkh26bj23ygzk02rg2i3ipd-emscripten-3.1.42/share/emscripten/tools/cache.py", line 35, in acquire_cache_lock
srb2kart>     assert 'EM_CACHE_IS_LOCKED' not in os.environ, f'attempt to lock the cache while a parent process is holding the lock ({reason})'
srb2kart> AssertionError: attempt to lock the cache while a parent process is holding the lock (unpack local port)

[sbc100]

Yes this is not a well documented or tested, or used feature. I wonder if we need to re-think it or remove it?

[sbc100]

Can you show the command that generated that error?

[SkyLeite]

Sure. The reason I need this patch is because I'm attempting to package an application with Nix, which does not allow internet access in builds. This seems to be related to dealing with dependencies in local ports, since my other ports seem to build fine and Vorbis immediately fails (which is the only one I'm using that also pulls dependencies as far as I can tell).

The entire build log is here: https://gist.github.com/SkyLeite/4da587b439ef13f379fe3f553282e1ed

If there's a different way to point Emscripten to vendored ports I'd be happy to pivot to that as well

[SkyLeite]

Fwiw simply removing the with cache.lock() line when building local ports seems to have resolved this particular issue. I'm not knowledgeable enough about the project to know if that can cause issues though.

[sbc100]

Sure. The reason I need this patch is because I'm attempting to package an application with Nix, which does not allow internet access in builds. This seems to be related to dealing with dependencies in local ports, since my other ports seem to build fine and Vorbis immediately fails (which is the only one I'm using that also pulls dependencies as far as I can tell).

It should be possible to avoid downloading anything at compile time. Are you using emsd, or some other distribution of emscripten that comes with prebuilt ports? Can you share the command line that you are running than triggers the internet access?

The entire build log is here: https://gist.github.com/SkyLeite/4da587b439ef13f379fe3f553282e1ed

If there's a different way to point Emscripten to vendored ports I'd be happy to pivot to that as well

Sadly no there is no way other way today.

[sbc100]

Ah yes, it seems that emsdk ships with pre-built system libraries but not with any pre-built ports.

One option would be to popular that ports tree ahead of time using ./embuild build libvorbis .. that will cause the download and build to happen before you run emcc. Does that work for you?

[SkyLeite]

Ah yes, it seems that emsdk ships with pre-built system libraries but not with any pre-built ports.

One option would be to popular that ports tree ahead of time using ./embuild build libvorbis .. that will cause the download and build to happen before you run emcc. Does that work for you?

Not really, unfortunately. Ideally I would leave downloading and managing these dependencies to Nix, as it gives me guarantees that are outside of Emscripten's scope. The current implementation of EMCC_LOCAL_PORTS does seem to work besides the cache locking issue, fortunately, which is more than good enough imo :)

[sbc100]

If you are pre-downloading these archive anyway I think you can just place in them in the cache/ports directory and emscripten will not download them (assuming the hashes match). For example: cache/ports/sdl2.24.2.zip.

It get access the specific archives with the right hashes you can use embuilder. e.g. ./embuilder build sdl2 will cause cache/ports/sdl2.24.2.zip to be downloaded.

[SkyLeite]

It worked! I took a look at the code and it seems to check for a file .emscripten_url containing the file's URL, so this is what I ended up doing:

export EM_CACHE=$(pwd)/.emscriptencache
mkdir -p $EM_CACHE/ports

OGG_URL="https://github.com/emscripten-ports/ogg/archive/version_1.zip"

mkdir ogg
pushd ogg
echo $OGG_URL > .emscripten_url
wget $OGG_URL
unar $(basename $OGG_URL .zip)
popd