Out-links for remote stores on shared filesystems

[mupdt]

Is your feature request related to a problem? Please describe.

We are using nix's remote store feature in the following setup:

• Machine A and B share filesystems (via NFS).
• Machine A is where users log in and do their work.
• Machine B is where the nix-daemon runs (this is the only host that can write
  to the nix store on the shared filesystem).

Users on machine A will be using nix like this:

machine-a % nix build --store ssh-ng://machine-b --out-link result-foo ...

Right now the out link is not created at all (neither on the remote store or
locally).

Describe the solution you'd like

The out link should be created on the remote store if we're sure that the
filesystems are shared.

We are aware that we cannot assume remote stores have a local filesystem
available. Moreover, we cannot assume the remote store shares its filesystem
with the client. We therefore propose to add an option to remote stores that
gives a hint to nix that the remote store should be treated as "shared" and that
it should create out-links. Something along these lines:

machine-a % nix build --store ssh-ng://machine-b?shared-fs=true --out-link result-foo ...

Describe alternatives you've considered

Some alternatives we considered:

• lift the concept of out-links all the way to the Store class. This means
  that all stores, not only LocalFSStore would have a way to create out links.

• a wrapper around our nix tooling that creates out-links out of band (this has
  a race condition with the GC unfotunately).

cc: @thufschmitt @rickynils

[mupdt]

If you agree with the proposed solution, we'd love to try and implement it.

However, if you believe an alternative solution would be better, please cast your vote. 🙏

[rickynils]

@mupdt The main issue is GC, right? Like, you want to be sure that the store path you just built remotely is not immediately garbage collected? If you could be given some guarantee about the liveness of the store path, you could create whatever symlinks you'd like (locally or remotely).

Maybe the store abstraction should somehow include GC roots?

That could also fix a related issue: When you copy store paths to a remote store there is no way to avoid them being garbage collected remotely before you can register a GC root.

[mupdt]

Yes, the main issue is GC.

However, there's also an UX issue: it would be nice if nix build behaved as similarly as possible on local FS stores and remote stores (since we use them in both contexts). Hmm, to be honest, I think the UX issue is perhaps even more important to us.

Forgot to ask, @rickynils, but would you have any recommendations on the GC roots abstraction?

[rickynils]

No, I don't have any direct suggestions for GC roots abstractions. I suspect that is something that could be discussed at some length... So maybe your suggestion about adding "out links" to the store abstraction is actually a better way to get started. They will represent GC roots, but only for stores where local fs symlinks makes sense.

Would you have to provide the shared-fs parameter? Couldn't you just opportunistically try to create the out link? If the store doesn't support out links, the user could just be informed that the link couldn't be created?

[mupdt]

We proposed that conservative solution due to caution as it doesn't change existing behaviour (unless you opt-in with the shared-fs=true flag) and is backwards compatible.

I must admit, I do like the idea of lifting out-links higher into a more general abstraction.

Perhaps we can do it in stages? First do the conservative solution and then extend it to be a "general abstraction"? What do you think?

[rickynils]

@mupdt Starting with the conservative solution sounds good to me. Now I'm just slightly puzzled over the shared-fs name. For the out-link, does it matter that the remote store shares its filesystem? We could still want the remote to register an outlink on its side, regardless of sharing?

[mupdt]

@rickynils: Ah, I see what you mean. Yes, shared-fs is not great. How about out-links=true?

[rickynils]

@mupdt Yeah, out-links sounds better to me. I think if that is implemented as a feature of the ssh-ng store implementation, we can see how useful it turns out and then think about a more general solution.

[mupdt]

@rickynils: After reading the code more carefully and while implementing this, it turned out that there is already a concept that fits this feature perfectly. I believe it makes the out-links flag unnecessary. In the end it's a very simple and clean patch. I'm using it now and running it through its paces. It looks great so far. I'll push up a PR very soon.