Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading the embedding model with NPU does not work #2364

Open
Nicogs43 opened this issue Sep 5, 2024 · 3 comments
Open

Loading the embedding model with NPU does not work #2364

Nicogs43 opened this issue Sep 5, 2024 · 3 comments
Assignees
Labels

Comments

@Nicogs43
Copy link

Nicogs43 commented Sep 5, 2024

Describe the bug
In the llm-rag-langchain.ipynb notebooks when I try to compile the embeddings model using the NPU I get an error back. Expected behavior
It should be able to compile the model with NPU without returning errors
Screenshots
image

Installation instructions (Please mark the checkbox)
[ ] I followed the installation guide at https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide to install the notebooks.

** Environment information **
System information:
Python executable: C:\Users\nico\openvino_env\Scripts\python.exe
Pip version: 24.2
OpenVINO source: C:\Users\nico\openvino_env\lib\site-packages\openvino
OpenVINO IE version: 2024.5.0-16610-41394bb0e6a
OpenVINO environment activated: OK
Jupyter kernel installed for openvino_env: NOT OK
Python version: 3.10 OK
OpenVINO pip package installed: OK
OpenVINO import succeeds: OK
OpenVINO development tools installed: NOT OK
OpenVINO not installed globally: OK
No broken requirements: NOT OK

The Python version in openvino_env does not match the openvino_env Jupyter kernel.
This may not be an issue. If you experience issues, please follow the instructions
in the README to reinstall the kernel.

OpenVINO development tools are not installed in this Python environment.
Please follow the instructions in the README to install openvino-dev

pip check shows broken requirements:
CompletedProcess(args=['pip', 'check'], returncode=1, stdout='ninja 1.11.1.1 is not supported on this platform\n')

The README.md file is located in the openvino_notebooks directory
and at https://github.com/openvinotoolkit/openvino_notebooks
Broken requirements are often harmless, but could cause issues.
Additional context
Add any other context about the problem here.

@Nicogs43 Nicogs43 changed the title Loading the embedding template with NPU does not work Loading the embedding model with NPU does not work Sep 5, 2024
@brmarkus
Copy link

brmarkus commented Sep 5, 2024

Have a look under "https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html#limitations". At least currently, for the NPU only static shapes are supported.

See also "https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html#dynamic-shapes" for the GPU plugin: "Currently, dynamic shape support for GPU is a preview feature and has the following limitations"

@Nicogs43
Copy link
Author

Nicogs43 commented Sep 5, 2024

Have a look under "https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html#limitations". At least currently, for the NPU only static shapes are supported.

See also "https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html#dynamic-shapes" for the GPU plugin: "Currently, dynamic shape support for GPU is a preview feature and has the following limitations"

Thank you, I don't understand then the point of making a notebook with an example like this if it doesn't work. I don't understand then this part either:
if USING_NPU: embedding.ov_model.reshape(1, 512) embedding.ov_model.compile()

@brmarkus
Copy link

brmarkus commented Sep 5, 2024

Not all notebooks, not all models are for all sorts of accelerators...

This notebook "llm-rag-langchain" specifically seems to be made for CPU and (d)GPUs according to the comments inside.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants