Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better uv integration #668

Open
1 of 4 tasks
mitsuhiko opened this issue Feb 17, 2024 · 22 comments
Open
1 of 4 tasks

Better uv integration #668

mitsuhiko opened this issue Feb 17, 2024 · 22 comments

Comments

@mitsuhiko
Copy link
Collaborator

mitsuhiko commented Feb 17, 2024

This is a meta task to look into actually leveraging uv properly.

@brunorpinho
Copy link

Is there a config option to start using rye with uv instead of pip today?

@golgor
Copy link

golgor commented Feb 17, 2024

@brunorpinho
See the documentation about the configuration:

# When set to `true` enables experimental support of uv as a replacement
# for pip-tools. Learn more about uv here: https://github.com/astral-sh/uv
use-uv = false

This was also asked during the installation and I have been using it and testing it out today. No real heavy testing, but it works great for me so far.

@mitsuhiko
Copy link
Collaborator Author

Note that for this to work on windows you need the not yet released version as it's disabled for safety reasons on windows still since I did not have time yet to verify basic functionality.

@mjclarke94
Copy link

One that might be worth including here for feature completeness is the dependency overrides functionality. A common complaint with poetry is when you have a chain of git repositories dependant on one another (say, pointing at main), an update to one repo breaks everything downstream of it as poetry locks to a specific commit hash.

The overrides functionality in uv could be a fairly lightweight way to bypass this whilst all the repos are in active development.

@mitsuhiko mitsuhiko pinned this issue Feb 18, 2024
@sbarrios93
Copy link
Contributor

Does using uv not take into account sources listed in pyproject.toml?

I have the following on my pyproject.toml file:

[[tool.rye.sources]]
name = "pytorch"
type = "index"
url = "https://download.pytorch.org/whl/nightly/cu121"

The lastest stable release of torch is 2.2.0 but I'm trying to download 2.3.0 which is prerelease.

Does not work:

❯ rye add torch
Added torch>=2.2.0 as regular dependency

Does not work:

❯ rye add --pre torch
Added torch>=2.2.0 as regular dependency

Seems like rye is constructing a similar command argument as the following. Based on the add.rs file

❯ VIRTUAL_ENV=.venv uv pip compile --no-deps --no-header --python-version 3.11.7 --prerelease=allow -
torch
Resolved 1 package in 1.54s
torch==2.2.0

But if try to install directly with uv, it does work:

❯ uv pip install torch --prerelease allow --index-url https://download.pytorch.org/whl/nightly/cu121
Resolved 22 packages in 570ms
Downloaded 11 packages in 28.06s
Installed 22 packages in 117ms
 + filelock==3.9.0
 + fsspec==2023.4.0
 + jinja2==3.1.2
 + markupsafe==2.1.3
 + mpmath==1.2.1
 + networkx==3.0rc1
 + nvidia-cublas-cu12==12.1.3.1
 + nvidia-cuda-cupti-cu12==12.1.105
 + nvidia-cuda-nvrtc-cu12==12.1.105
 + nvidia-cuda-runtime-cu12==12.1.105
 + nvidia-cudnn-cu12==8.9.2.26
 + nvidia-cufft-cu12==11.0.2.54
 + nvidia-curand-cu12==10.3.2.106
 + nvidia-cusolver-cu12==11.4.5.107
 + nvidia-cusparse-cu12==12.1.0.106
 + nvidia-nccl-cu12==2.19.3
 + nvidia-nvjitlink-cu12==12.1.105
 + nvidia-nvtx-cu12==12.1.105
 + pytorch-triton==3.0.0+901819d2b6
 + sympy==1.11.1
 + torch==2.3.0.dev20240219+cu121
 + typing-extensions==4.8.0

@mitsuhiko
Copy link
Collaborator Author

@sbarrios93 yes, looks like a bug. I will fix that.

@sbarrios93
Copy link
Contributor

sbarrios93 commented Feb 21, 2024

@mitsuhiko, I'm assuming that a fix would involve setting the default source in config.toml for rye (usually PyPi) as --index-url, with any additional index sources as --extra-index-url. However, based on astral-sh/uv#1451 (comment), I'm guessing that such a fix on rye's side wouldn't fix the resolution issue.

I've added some insights specific to PyTorch in astral-sh/uv#1723, which I closed in favor of astral-sh/uv#1451.

I don't want to hijack this thread – let me know if it's better to open a separate issue for this discussion.

@osuchw
Copy link

osuchw commented Feb 21, 2024

rye add with --features currently does not work when uv is enabled.

$ rye --version
rye 0.25.0
commit: 0.25.0 (d8e00cea1 2024-02-19)
platform: linux (x86_64)
self-python: [email protected]
symlink support: true
uv enabled: true

$ rye add flask --features dotenv
error: failed to resolve packages:
thread 'main' panicked at crates/uv-resolver/src/resolution.rs:136:29:
Every package should have metadata: NameVersion(PackageName("flask"), "3.0.2")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

$ rye config --set-bool behavior.use-uv=false

$ rye add flask --features dotenv
Added flask[dotenv]>=3.0.2 as regular dependency

Installing with uv by itself works.

$ uv --version
uv 0.1.6
$ uv venv
Using Python 3.12.1 interpreter at /home/osuchw/.rye/py/[email protected]/install/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

$ . .venv/bin/activate

$ uv pip install flask[dotenv]
Resolved 8 packages in 316ms
Downloaded 8 packages in 83ms
Installed 8 packages in 17ms
 + blinker==1.7.0
 + click==8.1.7
 + flask==3.0.2
 + itsdangerous==2.1.2
 + jinja2==3.1.3
 + markupsafe==2.1.5
 + python-dotenv==1.0.1
 + werkzeug==3.0.1

@mitsuhiko
Copy link
Collaborator Author

mitsuhiko commented Feb 21, 2024

This at the very least is triggering a weird code path in uv. I filed it there for now: astral-sh/uv#1801

Could be trivially worked around on the rye side though if needed.

@osuchw
Copy link

osuchw commented Feb 23, 2024

Thank you for looking into it. There is no more panic with rye 0.26.0 but --features is being ignored.
There is #745 now with more details.

@utkarshgupta137
Copy link

Not sure if this is the right place, but I can't figure out an easy way to get rye (with uv enabled) to update my rust deps.
Currently (rye 0.29.0, no uv), when I run rye sync --no-lock, it re-installs all workspace members, including re-building all the rust dependencies:

$ rye sync --no-lock -v
Python version already downloaded. Skipping.
Reusing already existing virtualenv
Installing dependencies
Using pip-tools configuration defaults found in 'pyproject.toml'.
Found existing installation: proj 0.1.0
Uninstalling proj-0.1.0:
  Successfully uninstalled proj-0.1.0
Found existing installation: proj-entrypoint 0.1.0
Uninstalling proj-entrypoint-0.1.0:
  Successfully uninstalled proj-entrypoint-0.1.0
Found existing installation: proj-toolbox 0.1.0
Uninstalling proj-toolbox-0.1.0:
  Successfully uninstalled proj-toolbox-0.1.0
Looking in indexes: https://pypi.org/simple/
Obtaining file:///lib/proj-entrypoint (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 1))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///lib/proj-toolbox (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 2))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///src (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 3))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: proj-entrypoint, proj-toolbox, proj
  Building editable for proj-entrypoint (pyproject.toml) ... done
  Created wheel for proj-entrypoint: filename=proj_entrypoint-0.1.0-cp312-cp312-macosx_11_0_arm64.whl size=1004 sha256=fdf8bc908d40ca6bb2bf4b302790cd34f4fef726aa9e5321e325b03807ac5e2b
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/10/c1/9a/4b146280476e5340ceac7f2cc7fe6722d555ff0209eabc2f15
  Building editable for proj-toolbox (pyproject.toml) ... done
  Created wheel for proj-toolbox: filename=proj_toolbox-0.1.0-cp312-cp312-macosx_11_0_arm64.whl size=999 sha256=c896f9950182d5fec3d9171a3bf2b9bf20364b40345b046ed2dd15506d168bc2
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/d2/12/22/425ebf9a3f9b82245195a3d48f64aaa3c63d365f24108c023d
  Building editable for proj (pyproject.toml) ... done
  Created wheel for proj: filename=proj-0.1.0-0.editable-py3-none-any.whl size=2649 sha256=549df1a8990cbb11dd8d2d09e1c8a9e14f4e9c02405745d08c2ca8209e0972b1
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/be/16/c1/367c544384017c548a18cd5b8ec9d158b60362a7ae1a22dff3
Successfully built proj-entrypoint proj-toolbox proj
Installing collected packages: proj-toolbox, proj-entrypoint, proj
Successfully installed proj-0.1.0 proj-entrypoint-0.1.0 proj-toolbox-0.1.0
Done!

With uv enabled, I get the following:

$ rye sync --no-lock -v
Python version already downloaded. Skipping.
Reusing already existing virtualenv
Installing dependencies
    0.002595s DEBUG uv_client::registry_client Using registry request timeout of 300s
 uv::requirements::from_source source=/Users/utkarsh/work/rye-proj/requirements-dev.lock
    0.137214s DEBUG uv_interpreter::python_environment Found a virtualenv through VIRTUAL_ENV at: /Users/utkarsh/work/rye-proj/.venv
    0.138290s DEBUG uv_interpreter::interpreter Cached interpreter info for Python 3.12.1, skipping probing: /Users/utkarsh/work/rye-proj/.venv/bin/python
    0.138299s DEBUG uv::commands::pip_sync Using Python 3.12.1 environment at /Users/utkarsh/work/rye-proj/.venv/bin/python
 uv_client::flat_index::from_entries
    0.144290s DEBUG uv_installer::plan Treating editable requirement as immutable: proj-entrypoint==0.1.0 (from file:///Users/utkarsh/work/rye-proj/lib/proj-entrypoint)
    0.144330s DEBUG uv_installer::plan Treating editable requirement as immutable: proj-toolbox==0.1.0 (from file:///Users/utkarsh/work/rye-proj/lib/proj-toolbox)
    0.144332s DEBUG uv_installer::plan Treating editable requirement as immutable: proj==0.1.0 (from file:///Users/utkarsh/work/rye-proj/src)
    0.144366s DEBUG uv_installer::plan Requirement already satisfied: alabaster==0.7.16
    0.144369s DEBUG uv_installer::plan Requirement already satisfied: asciitree==0.3.3
    0.144370s DEBUG uv_installer::plan Requirement already satisfied: asttokens==2.4.1

Where proj-entrypoint & proj-toolbox are rust crates & proj depends on both of them.
It would be nice if I could pass a flag to force rye to re-build my rust dependencies or better yet, have it automatically figure it out.

@geoqiao
Copy link

geoqiao commented Apr 1, 2024

Will rye add -r requirements.txt be supported, just like uv pip install -r requirements.txt or something like that?

It's needed when clone some projects which not init by rye

@bluss
Copy link
Contributor

bluss commented Apr 3, 2024

@geoqiao Rye init already supports importing requirements files

Edit: Indeed, as you say, it seems like it's not documented.

@bluss
Copy link
Contributor

bluss commented Apr 3, 2024

I'll suggest here that uv's --offline flag could be made available on rye lock and rye sync.

@geoqiao
Copy link

geoqiao commented Apr 3, 2024

@geoqiao Rye init already supports importing requirements files

Sorry, I didn't find in doc. Could you teach me how to do that? Thanks!

@geoqiao
Copy link

geoqiao commented Apr 3, 2024

@geoqiao Rye init already supports importing requirements files

Thanks for your reply, I found the -r option in the command line section of the document

idlsoft added a commit to idlsoft/rye that referenced this issue Apr 13, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 19, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 20, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 20, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
charliermarsh pushed a commit that referenced this issue Apr 24, 2024
The `build` package supports uv since version 1.2.0 - it can use either
uv or pip as installer.

The main way to use it would be to install `build[uv]` but rye already
has uv installed separately, build will find it on the path as a
fallback, so this can work for rye.

build is implicitly tested by test_publish.

Relates to #668
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 24, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Apr 26, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue May 5, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue May 6, 2024
@MilesCranmer
Copy link
Contributor

Friendly ping. Is there any way to pass --prerelease=allow to rye? I got hit with this today and literally can't install a package with rye because of it:

Reusing already existing virtualenv
Generating production lockfile: /Users/mcranmer/PermaDocuments/spock/requirements.lock
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of exoplanet-core==0.3.0rc2 and celmech==1.5.0 depends on exoplanet-core==0.3.0rc2, we can conclude that celmech==1.5.0 cannot be used.
      And because you require celmech==1.5.0, we can conclude that the requirements are unsatisfiable.

      hint: exoplanet-core was requested with a pre-release marker (e.g., exoplanet-core==0.3.0rc2), but pre-releases weren't enabled (try: `--prerelease=allow`)
error: could not write production lockfile for project

Caused by:
    Failed to run uv compile /var/folders/1h/xyppkvx52cl6w3_h8bw_gdqh0000gr/T/.tmpgCC8Dh/requirements.txt. uv exited with status: exit status: 1

but alas, I cannot seem to pass this option through to uv.

@charliermarsh
Copy link
Member

@MilesCranmer - I can look into adding it. Note that you can work around this by telling uv to allow pre-releases for exoplanet-core specifically, by adding a requirement like:

exoplanet-core >= 0.3.0rc2

As long as you include some pre-release segment in the specifier, we'll enable pre-releases for that dependency.

@MilesCranmer
Copy link
Contributor

Awesome. Thanks!

@bluss
Copy link
Contributor

bluss commented May 19, 2024

@MilesCranmer have you had any success using rye sync --pre ?

@MilesCranmer
Copy link
Contributor

Yep, that did it, thanks! (Unfortunately it was a dependency of a dependency, so putting it as an explicit dependency wasn't optimal)

Maybe the error message could point out the --pre flag? Right now the error is from uv and only mentions the --prerelease one.

idlsoft added a commit to idlsoft/rye that referenced this issue May 21, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue May 22, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue May 22, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Jun 17, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Jun 17, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Jun 17, 2024
idlsoft added a commit to idlsoft/rye that referenced this issue Jun 17, 2024
@bzm3r
Copy link

bzm3r commented Jun 30, 2024

@MilesCranmer have you had any success using rye sync --pre ?

Thanks for the hint.

I found that the --pre flag can also be passed to rye add (not just rye sync). So that's nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests