pkgsLinux: set crossSystem instead of localSystem #317651
Conversation
|
The last tpm2-tss commit would likely cause a lot of rebuilds since it’s systemd dependency. I’ll rebase to the staging branch later. |
tie
force-pushed
the
nixos-darwin-cross
branch
2 times, most recently
from
d36910e
to
d64c253
Compare
ofborg
bot
added
10.rebuild-darwin: 101-500
10.rebuild-linux: 501+
10.rebuild-linux: 5001+
and removed
10.rebuild-darwin: 1-10
10.rebuild-linux: 0
labels
|
OK, apparently some Linux-specific packages are still getting pulled from |
As the doc implies, we should be using crossSystem to build *for* the Linux platform natively instead of hoping that the system has a remote builder or substituer with the right set of packages. While this could be considered a breaking change, pkgsLinux attribute is used exclusively for nixosTests and is a relatively new addition so there shouldn’t be many external users that would be affected by this change.
Fixes pkgsLinux.tpm2-tss build on macOS since shadow package is Linux-specific.
|
Ah, I see #293573. That still seems like a really weird and niche use case to me — I mean, it already requires linux builder, so why not run tests there? Instead, I think we should be able to cross-compile NixOS from macOS with some effort. |
| @@ -249,7 +249,7 @@ let | |||
| if stdenv.hostPlatform.isLinux | |||
| then self | |||
| else nixpkgsFun { | |||
| localSystem = lib.systems.elaborate "${stdenv.hostPlatform.parsed.cpu.name}-linux"; | |||
| crossSystem = lib.systems.elaborate "${stdenv.hostPlatform.parsed.cpu.name}-linux"; | |||
There was a problem hiding this comment.
This is meant to be a natively built Linux package set, ie a non-cross build that's potentially done on a different machine. Do you think we non-cross would be a clearer description than natively built in the doc above?
Note that the "machine" is easy to set up with for example linux-builder.
By changing to cross compilation, we make the user experience worse and increase cost.
- Cross compilation, despite how well it works in Nixpkgs, does not permit tests to be run in package builds
- We don't have a cache for darwin->linux cross builds, afaik. These builds would be unnecessary extra builds, costing compute and cache storage (ofborg, hydra, cache.nixos.org, etc)
Also note that if you're in a team using a mix of Linux and Darwin and you're doing deployment with cross builds, your performing unnecessary redeployments depending on who's initiating it, causing unnecessary disruptions as system services are "updated" to an equivalent package on a different store path. You could solve this by deploying from dedicated infrastructure, but then you might as well use that for remote builds as well.
More importantly though, non-cross builds are more likely to work.
All in all, non-cross deployments are simpler and more robust, so I would not default to cross compilation anywhere, including here.
Perhaps your goal could be achieved by adding pkgsLinuxCross, but then I still wouldn't make the test framework use that for the reasons above.
There was a problem hiding this comment.
That sounds reasonable given the current state of cross-compilation of Nixpkgs (though I wouldn’t say it works well, rather it works for well-tested {local,cross}System pairs, but that doesn’t cover glibc → glibc static cross-build, NixOS built from macOS, and a lot of other cases).
I think we’ve had a similar discussion in NixOS/nix#10291 about this use case. W.r.t. tests, these do not change the resulting package output. So it doesn’t really matter where the tests are run as long the closures are byte-for-byte identical and reproducible. That is, as long as we can test that a certain set of packages is identical independent of the localSystem, it doesn’t really matter which derivation runs the tests (assuming that at least one does). Sure, that requires content-addressed derivations to get this right, but then there are a lot of other places that should be fixed before that.
I understand your point, but I do want to improve the current state of affairs. I’ll open separate PR for package-specific commits I’ve pushed to this branch, without this particular change.
|
I'd previously asked the infra team to provide a Linux builder, but they were concerned about a deadlock issue. I've now opened an issue with alternatives that avoid the problem (and it's also actually properly visible, unlike chat in Matrix). |
Description of changes
As the doc implies, we should be using crossSystem to build for the Linux platform. Using localSystem just happens to occasionally hit cached builds. See also #316659 (comment) and https://github.com/NixOS/nixpkgs/runs/25855763929
See also #282401, #294725
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.