Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conventional-changelog-conventionalcommits v8.0.0 breaks semantic release #633

Closed
ryancausey opened this issue May 4, 2024 · 28 comments · Fixed by #645
Closed

conventional-changelog-conventionalcommits v8.0.0 breaks semantic release #633

ryancausey opened this issue May 4, 2024 · 28 comments · Fixed by #645
Labels

Comments

@ryancausey
Copy link

It looks like v8.0.0 of conventional-changelog-conventionalcommits was released about an hour ago. This causes the semantic-release "generateNotes" step of plugin "@semantic-release/release-notes-generator" to fail:

$ npm --version
10.5.0
$ semantic-release --version
23.0.8
$ semantic-release
[11:48:10 PM] [semantic-release] › ℹ  Running semantic-release version 23.0.8
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "verifyConditions" from "@semantic-release/changelog"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "verifyConditions" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "verifyConditions" from "@semantic-release/git"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "verifyConditions" from "@semantic-release/gitlab"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "analyzeCommits" from "@semantic-release/commit-analyzer"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "analyzeCommits" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "verifyRelease" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "generateNotes" from "@semantic-release/release-notes-generator"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "generateNotes" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "prepare" from "@semantic-release/changelog"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "prepare" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "prepare" from "@semantic-release/git"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "publish" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "publish" from "@semantic-release/gitlab"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "addChannel" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "success" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "success" from "@semantic-release/gitlab"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "fail" from "@semantic-release/exec"
[11:48:11 PM] [semantic-release] › ✔  Loaded plugin "fail" from "@semantic-release/gitlab"
[11:48:15 PM] [semantic-release] › ✔  Run automated release from branch master on repository [https://gitlab-ci-token:[secure]@gitlab.com/foo/bar.git](https://gitlab-ci-token:%5Bsecure%[email protected]/foo/bar.git)
[11:48:15 PM] [semantic-release] › ✔  Allowed to push to the Git repository
[11:48:15 PM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/changelog"
[11:48:15 PM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/changelog"
[11:48:15 PM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/git"
[11:48:15 PM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/git"
[11:48:15 PM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/gitlab"
[11:48:15 PM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://gitlab.com/api/v4)
[11:48:15 PM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/gitlab"
[11:48:15 PM] [semantic-release] › ℹ  Found git tag v1.4.0 associated with version 1.4.0 on branch master
[11:48:15 PM] [semantic-release] › ℹ  Found 1 commits since last release
[11:48:15 PM] [semantic-release] › ℹ  Start step "analyzeCommits" of plugin "@semantic-release/commit-analyzer"
[11:48:15 PM] [semantic-release] [@semantic-release/commit-analyzer] › ℹ  Analyzing commit: fix: make the esg rule name not clash with other domains
This was not applying properly because the rule has the same name as a
rule created by a different AWS account on the same bus.
[11:48:15 PM] [semantic-release] [@semantic-release/commit-analyzer] › ℹ  The release type for the commit is patch
[11:48:15 PM] [semantic-release] [@semantic-release/commit-analyzer] › ℹ  Analysis of 1 commits complete: patch release
[11:48:15 PM] [semantic-release] › ✔  Completed step "analyzeCommits" of plugin "@semantic-release/commit-analyzer"
[11:48:15 PM] [semantic-release] › ℹ  Start step "analyzeCommits" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ✔  Completed step "analyzeCommits" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ℹ  The next release version is 1.4.1
[11:48:15 PM] [semantic-release] › ℹ  Start step "verifyRelease" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ✔  Completed step "verifyRelease" of plugin "@semantic-release/exec"
[11:48:15 PM] [semantic-release] › ℹ  Start step "generateNotes" of plugin "@semantic-release/release-notes-generator"
[11:48:15 PM] [semantic-release] › ✘  Failed step "generateNotes" of plugin "@semantic-release/release-notes-generator"
[11:48:15 PM] [semantic-release] › ✘  An error occurred while running semantic-release: RangeError: Invalid time value
    at committerDate (/usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/index.js:80:30)
    at /usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/lib/util.js:202:17
    at Array.forEach (<anonymous>)
    at processCommit (/usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/lib/util.js:198:31)
    at /usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/index.js:123:32 {
  pluginName: '@semantic-release/release-notes-generator'
}
RangeError: Invalid time value
    at committerDate (/usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/index.js:80:30)
    at /usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/lib/util.js:202:17
    at Array.forEach (<anonymous>)
    at processCommit (/usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/lib/util.js:198:31)
    at /usr/local/lib/node_modules/semantic-release/node_modules/conventional-changelog-writer/index.js:123:32 {
  pluginName: '@semantic-release/release-notes-generator'
}

Reverting conventional-changelog-conventionalcommits back to the v7.x.x series makes this release run fine again.

@jedwards1211
Copy link

I had the error even with conventional-changelog-conventionalcommits v7.x.x; it seems like it was being caused because the preset I was using with @semantic-release/commit-analyzer didn't match the preset I was using with @semantic-release/release-notes-generator. When I made them match, the error went away with v8.0.0.

@ryancausey
Copy link
Author

I had the error even with conventional-changelog-conventionalcommits v7.x.x; it seems like it was being caused because the preset I was using with @semantic-release/commit-analyzer didn't match the preset I was using with @semantic-release/release-notes-generator. When I made them match, the error went away with v8.0.0.

I'm not sure what that means. I simply have a preset: conventionalcommits at the top of my .releaserc.yml file, so I would assume they all share the same preset?

@jedwards1211
Copy link

@ryancausey Oh, I'm not sure, I'm configuring all the plugins explicitly, and the plugins accept the preset option.

levibostian added a commit to levibostian/Wendy-iOS that referenced this issue May 4, 2024
Seems like there is a bug with semantic-release plugins conflicting with latest major conventional commits. Reverting as issue suggests: semantic-release/release-notes-generator#633

commit-id:b1f4a00c
levibostian added a commit to levibostian/Wendy-iOS that referenced this issue May 4, 2024
Seems like there is a bug with semantic-release plugins conflicting with latest major conventional commits. Reverting as issue suggests: semantic-release/release-notes-generator#633

commit-id:b1f4a00c
@travi
Copy link
Member

travi commented May 5, 2024

the conventional-changelog packages released new major versions. that means they include breaking changes. since they were just released, no work has been done to make adjustments to account for the changes in semantic-release.

if someone would be willing to investigate what changes would be needed, that could help us adjust for these changes sooner

@jedwards1211
Copy link

jedwards1211 commented May 5, 2024

@travi yeah that makes sense. If we need to customize the preset though, we have to install it, and then it's easy to install the wrong version... so should we make this package check the version of the preset at runtime, and at least print a warning? I would almost suggest we put the compatible version in optionalDependencies, but package managers, confoundingly, install optional dependencies by default. So the best I can think to do is a runtime check. I'd be happy to make a PR.

@travi
Copy link
Member

travi commented May 7, 2024

@travi yeah that makes sense. If we need to customize the preset though, we have to install it, and then it's easy to install the wrong version... so should we make this package check the version of the preset at runtime, and at least print a warning? I would almost suggest we put the compatible version in optionalDependencies, but package managers, confoundingly, install optional dependencies by default. So the best I can think to do is a runtime check. I'd be happy to make a PR.

I understand the desire, but npm doesn't provide a great way to define these sorts of relationships. I'm open to proposals as a separate issue if there are options that I'm not aware of, but this is a difficult one to solve. I think the closest might be peerDependencies with meta defining the optional part, but I think there are complexities that make that not ideal too

@travi
Copy link
Member

travi commented May 7, 2024

For folks landing here for various breakages related to updating conventional-commits packages before semantic-release is updated to handle the breaking changes, I'm trying to consolidate the conversation in this thread.

Until I get more organized about that, please see my suggestions in semantic-release/semantic-release#3292 (comment)

@jedwards1211
Copy link

jedwards1211 commented May 8, 2024

@travi not sure you saw my suggestion to check and warn at runtime if the preset package being used is not a known compatible version, would you be okay with a PR to do that?

@travi
Copy link
Member

travi commented May 8, 2024

not sure you saw my suggestion to check and warn at runtime if the preset package being used is not a known compatible version, would you be okay with a PR to do that?

sorry. i did, but forgot to comment about that approach. it doesnt seem to me like there is a clean approach there either that is maintainable and still accomplishes the goal for new breaking changes. our approach to true peer dependencies and node versions is to keep them open ended, so we would have to release a new version with updated ranges once we are aware that a new major version is incompatible, which also requires responding fairly quickly to be most helpful. each of the conventional-commits packages has its own individual major version (an approach that i agree with), so we would have to handle each one individually. this also means more investigation when trying to release a version for such incompatibility communication.

in the end, i think this almost entirely comes down to folks installing latest versions in pipelines without pinning a major version. would be open to suggestions to make this suggestion more clear and discoverable in docs or elsewhere in a general way that doesnt require maintainer action each time that breaking changes are released in conventional-commits packages.

@jedwards1211
Copy link

jedwards1211 commented May 8, 2024

think this almost entirely comes down to folks installing latest versions in pipelines without pinning a major version.

In my case it wasn't that - I was installing conventional-changelog-conventionalcommits for the first time so that I could try to customize the behavior for a monorepo.

I can understand where you're coming from though. I can also understand the way all these different packages are structured. But - figuring out how to even customize everything and follow the control flow of release notes generation has left me thinking the work of fetching and parsing git commits and generating a changelog from them is spread out across too many packages for its own good. Maybe I would feel differently if these were all TS projects or the configuration was done via a strongly typed TS API rather than an unvalidated JSON DSL. It's pretty hairy to dig into how the configuration is getting passed around and used. And at least half of the code is for implementing that DSL that isn't really capable of doing what I want anyway, I wish I could just provide a function for filtering and categorizing commits by type and scope, etc, which would be way more flexible and require way less library code.

@DenisBalan
Copy link

We have the same issue..
Decided to downgrade for only package which is causing troubles to [email protected]

...
npm install semantic-release \
@semantic-release/changelog \
@semantic-release/git \
[email protected]
...

@anthony-nhs
Copy link

in the end, i think this almost entirely comes down to folks installing latest versions in pipelines without pinning a major version. would be open to suggestions to make this suggestion more clear and discoverable in docs or elsewhere in a general way that doesnt require maintainer action each time that breaking changes are released in conventional-commits packages.

I agree that is what people should do. However, the problem I found is that is currently hard to test if a major version bump will cause something to fail as we have semantic-release only running on our main branch. Implementing semantic-release/semantic-release#3278 would allow us to at least run with dry-run in a branch where we upgrade versions and see if anything breaks

ashvardanian added a commit to unum-cloud/usearch that referenced this issue May 26, 2024
@travi
Copy link
Member

travi commented May 26, 2024

@ashvardanian the new beta of semantic-release is intended to work with the latest majors of conventional-changelog presets. As mentioned in the release notes, that means older version will no longer work. For the eslint preset, it looks like v6.0.0 is the latest version, but you're project is still configured to use v3.

@alexey-sh
Copy link

So what is the plan?

@strausmann
Copy link

I just asked myself that. currently my Semantic release is no longer running. my gitlab ci pipelines are standing still.

@travi
Copy link
Member

travi commented May 29, 2024

There are two options available

  • Pin the major versions of the packages you are installing alongside semantic-release until we release a new version that supports the latest conventional-changelog packages
  • Try out the beta that should support those new versions and let us know if it works in your context. That will help give us the confidence to promote the fix for everyone to use (even after we promote the fix, you should still pin the major version of those extra packages to avoid this situation next time)

@ashvardanian
Copy link

Hi @travi! Thanks for suggestions!

I've bumped all the versions in my package.json, which now starts like this:

{
    "name": "simsimd-ci",
    "version": "1.0.0",
    "devDependencies": {
        "@semantic-release/exec": "^6.0.3",
        "@semantic-release/git": "^10.0.1",
        "@semantic-release/commit-analyzer": "^12.0.0",
        "@semantic-release/release-notes-generator": "^13.0.0",
        "@semantic-release/github": "^10.0.5",
        "conventional-changelog-eslint": "^6.0.0",
        "semantic-release": "^24.0.0-beta.2"
    },
    "release": {
        "debug": true,
        "ci": false,
        "dryRun": false,
        "private": true,
        "branches": [
            "main"
        ],
        "plugins": [

Then, when I run the following command locally:

npx --prefix .github/workflows semantic-release  --no-ci --no-npm --debug

I'm still getting errors like:

[7:51:24 PM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/npm"
[7:51:24 PM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/github"
[7:51:24 PM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[7:51:24 PM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/github"
[7:51:24 PM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/github"
[7:51:24 PM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[7:51:24 PM] [semantic-release] › ✘  Failed step "fail" of plugin "@semantic-release/github"
[7:51:24 PM] [semantic-release] › ✘  ENOGHTOKEN No GitHub token specified.
A GitHub personal token must be created and set in the GH_TOKEN or GITHUB_TOKEN environment variable on your CI environment.

Please make sure to create a GitHub personal token and to set it in the GH_TOKEN or GITHUB_TOKEN environment variable on your CI environment. The token must allow to push to the repository ashvardanian/SimSIMD.

[7:51:25 PM] [semantic-release] › ✘  ENOGHTOKEN No GitHub token specified.
A GitHub personal token must be created and set in the GH_TOKEN or GITHUB_TOKEN environment variable on your CI environment.

Same happens if I remove any GitHub-related dependencies or steps. How can I "dry-run" the semantic release pipeline locally to avoid polluting the release history and the main branch of my projects?

@travi
Copy link
Member

travi commented May 29, 2024

You do not need to/should not install the commit-analyzer or release-notes-generator plugins as direct dependencies, since they are already dependencies of semantic-release itself. If you test with npm ls <package-name>, you'll see that has resulted in you having multiple versions installed. Removing those direct dependencies should resolve your problem

@ashvardanian
Copy link

@travi, I've just tried that locally and still face same issues.

$ npm cache clean --force
$ rm -rf .github/workflows/node_modules .github/workflows/package-lock.json
$ nm install --ignore-scripts --legacy-peer-deps --prefix .github/workflows
$ npx --prefix .github/workflows semantic-release  --no-ci --no-npm --debug
...
[1:03:58 AM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/npm"
[1:03:58 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[1:03:58 AM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[1:03:58 AM] [semantic-release] › ✘  Failed step "fail" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] › ✘  ENOGHTOKEN No GitHub token specified.
A GitHub personal token must be created and set in the GH_TOKEN or GITHUB_TOKEN environment variable on your CI environment.

What else would you recommend trying?

@ryancausey
Copy link
Author

https://github.com/semantic-release/semantic-release/releases/tag/v24.0.0-beta.2 should resolve this problem and enable use of the latest conventional-changelog packages.

it would be appreciated if folks following this issue that have experienced problems with the new major versions of the conventional-changelog plugins and let us know if this version resolves the problem for you

I've tested with this and it resolved my issue.

@HenrikPoulsen
Copy link

Would be nice if this was something that wasn't always discovered aftter an update was being made in a repo.
Some sort of sanity check somewhere in the pipeline would be nice.
I had a similar issue, but in my case it didn't get as far as the release-notes-generator and ended up not working in the commit-analyzer, which just didn't detect any changes and I spent a day trying to figure out what I had done wrong when tryring to setup conventionalcommits on the repo.
And this is a setup I was planning to setup in a 100+ repos across many teams. It would be extremely costly if Dependabot or Renovate or whatever suggests they update to a new version of one of the plugins, and then several weeks after they've merged they notice that it's actually not working anymore because there's no errors etc.
Pinning won't really solve the type of issue I am proposing there. Because you will have to update eventually. How are you supposed to do that safely? Preferably in an automated way. Like having a semantic-releasee verify that checks all the plugins and configs and complains if things look weird.

achingbrain added a commit to ipfs/aegir that referenced this issue May 30, 2024
Fixes the breakage from the latest commit analyzer release.

Refs: semantic-release/release-notes-generator#633
@LeleDallas
Copy link

Would be nice if this was something that wasn't always discovered aftter an update was being made in a repo. Some sort of sanity check somewhere in the pipeline would be nice. I had a similar issue, but in my case it didn't get as far as the release-notes-generator and ended up not working in the commit-analyzer, which just didn't detect any changes and I spent a day trying to figure out what I had done wrong when tryring to setup conventionalcommits on the repo. And this is a setup I was planning to setup in a 100+ repos across many teams. It would be extremely costly if Dependabot or Renovate or whatever suggests they update to a new version of one of the plugins, and then several weeks after they've merged they notice that it's actually not working anymore because there's no errors etc. Pinning won't really solve the type of issue I am proposing there. Because you will have to update eventually. How are you supposed to do that safely? Preferably in an automated way. Like having a semantic-releasee verify that checks all the plugins and configs and complains if things look weird.

A command like semantic-releasee verify will be a great feature!

@BinToss
Copy link

BinToss commented May 31, 2024

@travi, I've just tried that locally and still face same issues.

$ npm cache clean --force
$ rm -rf .github/workflows/node_modules .github/workflows/package-lock.json
$ nm install --ignore-scripts --legacy-peer-deps --prefix .github/workflows
$ npx --prefix .github/workflows semantic-release  --no-ci --no-npm --debug
...
[1:03:58 AM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/npm"
[1:03:58 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[1:03:58 AM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] › ℹ  Start step "fail" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] [@semantic-release/github] › ℹ  Verify GitHub authentication
[1:03:58 AM] [semantic-release] › ✘  Failed step "fail" of plugin "@semantic-release/github"
[1:03:58 AM] [semantic-release] › ✘  ENOGHTOKEN No GitHub token specified.
A GitHub personal token must be created and set in the GH_TOKEN or GITHUB_TOKEN environment variable on your CI environment.

What else would you recommend trying?

A. Create and securely store a GitHub Personal Access Token that has repo scope (classic token) or "Contents" repository permissions (write) (fine-grained). Then, try npx --prefix .github/workflows cross-env GITHUB_TOKEN=**** semantic-release --no-ci --no-npm --debug
B. Remove the github plugin from your main config.
C. Create a separate config file which imports the main one and then options.plugins = options.plugins.filter(pluginSpec =>typeof pluginSpec === 'string' ? pluginSpec !== '@semantic-release/github' : pluginSpec[0] !== '@semantic-release/github'); before export default options. You may need to suppress a TypeScript 'readonly' error. Then, add --extends './SRNoGithub.js' to your CLI command.

@imliuruiqi
Copy link

FYI, tried on self-hosted gitlab with python project and it works as expected. 🎉

script:
    - npm install @semantic-release/gitlab @semantic-release/exec @semantic-release/changelog conventional-changelog-conventionalcommits @semantic-release/git -D
    - npx [email protected]

configuration of semantic-release:

plugins:
  - "@semantic-release/commit-analyzer"
  - "@semantic-release/release-notes-generator"
  - - "@semantic-release/changelog"
    - changelogFile: CHANGELOG.md
  - - "@semantic-release/git"
    - assets:
        - CHANGELOG.md
  - - "@semantic-release/exec"
    - prepareCmd: "poetry version ${nextRelease.version} && poetry build"
  - - "@semantic-release/gitlab"
    - assets:
      - path: "dist/*.whl"
        type: "package"
      - path: "dist/*.gz"
        type: "package"
      - path: "CHANGELOG.md"
        type: "runbook"


preset: "conventionalcommits"

Copy link

🎉 This issue has been resolved in version 14.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@travi
Copy link
Member

travi commented May 31, 2024

Thank you @sherbakovdev for contributing to getting this out the door and to @ryancausey and @imliuruiqi for testing out the fix and confirming that it solved the problem in your projects 🎉

ashvardanian added a commit to unum-cloud/usearch that referenced this issue Aug 6, 2024
* Improve: Swift test for issue #399 (#400)

* Fix: Integer overflow in aligned-alloc

Fixes ClickHouse/ClickHouse#61780

Co-authored-by: Antonio Andelic <[email protected]>

* Make: Disable Windows NPM builds

Relates to the #377 and the comment:
#377 (comment)

This temporarily disables the failing CI pipeline to generate
and update docs.

* Fix: Going beyond level 0 in clustering

* Improve: Error handling in `index_dense_gt`

This commit drops `std::vector` dependency,
making compilation time shorter and error
handling universal across abstraction layers.

* Improve: Remove `std::function` calls

* Improve: Remove `std::thread` from `index_dense_gt`

* Improve: `std::vector` -> `buffer_gt` in plugins

* Add: `usearch_change_threads_search`

* Fix: `index_dense_t::make(path)`

* Fix: Exhastive Search

In the past, if we got "too lucky" traversing the graph,
we could exit early before accumulating K top matches,
even if the index had more than K entries. This patch
changes that behavior, making output more predicatable.

* Fix: Replacement leaves isolated nodes

This patch addresses the issue #399, originally
observed in the Swift layer. Reimplementing it in
C++ helped locate the issue and lead to refactoring
the `update` procedure in the lowest-layer `index_gt`.

Now, `add` and `update` share less code. The `add`
is one branch shorter (not that it would be noticeable),
and `update` brings additional logic to avoid spilling
`updated_slot` into top-results and consequently
introducing self-loops.

Closes #399

* Fix: Misc warnings & compilation issues

* Fix: Misc warnings & compilation issues

* Fix: Detect `ring_gt` being full

Relates to #355

* Fix: Re-init `available_threads_` after load

Both `view` and `load` would `reset` the thread contexts.
After that, the very first `search` and `add` would fail, as
no thread-local contexts are initialized. It would require
a `reserve` call with a non-zero second arcgument to
define the number of concurrent threads, for which the
queues & buffers need to be allocated.

That design is counter-intuitive, so this patch re-inits the
same number of threads as before the `load` & `view` or
one, if none existed.

* Fix: `uint32_t` to `uint40_t` cast (#404)

Co-authored-by: Ash Vardanian <[email protected]>

* Docs: Mention `b1` in `README.md`

Co-authored-by: Adolfo Garcia <[email protected]>

* Docs: Cover new users

* Improve: Updates stability & catch bug

* Fix: Dereferencing `member_iterator_t`

* Add: Java `get` API (#407)

* Fix: Compilation with `uint40_t` keys

* Add: `AutoClosable` using `c_destroy` for Java (#408)

* Fix: Rare deadlock on tiny collections

* Improve: `enable_key_lookups=false` memory usage

As noted by Robert Schulze, we can avoid populating `slot_lookup_`
during insertions, if `enable_key_lookups` is not set. This would lead
to lower memory consumption for large indexes of tiny vectors,
particularly common in GIS.

Co-authored-by: Robert Schulze <[email protected]>

* Fix: Preset `enable_key_lookups=true` in C

* Fix: `std::is_pod` deprecated in C++20

* Fix: Unused type aliases

* Improve: Avoid `#pragma region` pre GCC 13 (#386)

* Do not use #pragma region if not supported by the compiler
* `pragma region` supported by GCC 13+

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85487

---------

Co-authored-by: Mikhail Bautin <[email protected]>
Co-authored-by: Ash Vardanian <[email protected]>

* Fix: Capacity checks in C# tests

* Docs: Add doc-strings in C#

* Make: Verbose C# logging in debug builds

* Docs: Describe serialization in GoLang

* Make: Drop Java & JS API references

Compiling and introspecting docstrings in Sphinx
is extremely flaky. It's safer to simply describe the
usage patterns in the `README.md`.

* Add: Load from path in Java (#410)

* Improve: Avoid sorting on small "refines"

* Docs: Cover JIT-compiled Python examples

* Fix: `index.copy()` trying to `memcpy(*, NULL, *)`

* Docs: UB in C++ Example (#415)

Co-authored-by: Ash Vardanian <[email protected]>

* Improve: Catch UB with tests

* Docs: Rearrange

* Fix: Reserving contexts post-reload

* Improve: Detect more failures in tests

* Improve: Log failing lines

* Fix: Clamp before down-casting to `i8_t` (#422)

Previously we were down-casting floats to the target type (e.g. int8_t), and then
clamping to [-100, 100] range. This means that e.g. 129 would be cast to -127 and
then converted to -100, in stead of becoming 100

The fix does clamping first, and then casts the resulting number
(which is guaranteed to be in range [-100, 100], due to clamping)
from source type to target int8_t. Given the clamping, this will never overflow.

---------

Co-authored-by: Ash Vardanian <[email protected]>

* Fix: Concurrency bug in high-K search

When calling `index_dense_gt`, the thread lock was not propagating
with the `search_result_t`. That is a an error-prone API. When too many
threads are running in parallel (ideally, more than physical CPU cores)
another thread may start reusing the `context_t` before the original
caller finishes exporting entries with `dump_to`.

This solution is backwards compatible and passes the tests.

* Make: Manually bump version to 2.13.0

We can't yet rely on the SemVer tool
semantic-release/release-notes-generator#633 (comment)

* Improve: Attempt to implement batch-metrics

* Add: Batch-capable metrics

* Add: `misaligned_ptr_gt` comparators

* Improve: Separate `error_t` constructors

This makes debugging easier, as it's simpler to trace
where the error message is being set.

* Improve: Support `enum` slots

Tracing implicit conversions of `std::uint32_t` and other
primitive types isn't always easy in concurrent apps. This
commit adds support for `enum` types to be used for
safer implementation of `index_gt` specializations.

* Improve: Ranges-V3 compatibility

* Add: Preliminary support for batch metrics

* Add: Batch-parallel refinements

* Add: `MANIFEST.in` for `py.typed` (#425)

Adding type annotation for Python native modules
solves the `Skipping analyzing "usearch.index" module` 
warning due to `missing library stubs or py.typed marker`.

Closes #424

* Fix: Clear cast buffer before bitwise ORs for `b1x8_t` (#428)

When converting floating point arrays to binary, we use bitwise OR
operations to set the relevant bits in the output buffer to 1. We do
nothing if the bit is zero, so we assume that the bit is zero to start
with. The `memset` statement makes sure this assumption holds.

* Fix: `esm` duplicate import bug in `jest` (#420)

Closes #418
Closes #426

* Fix: build.gradle deprecations

* Fix: ESM build support (#433)

Closes #426
Relates to #420

* Fix: `capacity()` assertion in Rust (#436)

Closes #432

* Fix: Computing `stats(i).max_edges`

* Add: Returning `computed_distances_in_refines`

In high-connectivity graphs, the number of distance
computations can be dominated by the number of "refine"
heuristic computations performed by the core structure.
The extended `add_result_t` now includes both:

- `computed_distances_in_refines`
- `computed_distances_in_reverse_refines`

This commit also extends the documentation.

* Add: Profiling attributes for `index_gt`

* Fix: Preserve thread limits after `fork()`

* Improve: Benchmarks self-recall support and `.bbin`

This patch adds support for partial datasets, without
ground truth neighborhood data. It also adds support
for `.bbin` binary, `.hbin` half-precision, and `.dbin`
double-precision input vector files/

* Fix: Printing progress between exit

* Docs: Fix spelling

* Fix: Wolfram bindings (#437)

Co-authored-by: Ash Vardanian <[email protected]>

* Fix: Pre-reserve enough threads for C users

This indirectly fixes the crash in C# layer

* Revert: Parallel metrics

* Fix: Updating the `entry_slot_` node

* Improve: Enable single-threaded update tests

* Make: Bump version

* Fix: `flat_hash_multi_set_gt::reset` double-free

* Fix: Not enough memory in `fork()`

* Make: Bump to SimSIMD v5

* Fix: Missing SimSIMD v5 capability names

* Improve: Detecting copy & move issues

* Fix: Compilation w. explicit `template class`

* Improve: Bypass UBSan NULL dereferencing warning

* Improve: Minimize alignment issues

* Make: Disable `bf16` on MacOS

* Make: Link to GitHub repo

* Fix: Conditional `call_key` compilation in MSVC

* Fix: Unary minus on unsigned distances

* Make: Disable `bf16` in JS

* Fix: Compatibility with pre-v2.10

Closes #423

* Improve: Test wrong number of dimensions in Rust (#413)

Closes #412

---------

Co-authored-by: Julius Brummack <[email protected]>

* Fix: Handle wrong dimensionality in Rust

* Fix: Overwriting `SIMSIMD_DYNAMIC_DISPATCH`

* Make: Upgrade Java version

* Fix: Spelling mistakes

* Fix: `sprintf` deprecated

* Fix: `-Wpass-failed=transform-warning`

* Fix: Memory pinning on `Add` in C#

* Make: Specify Java distribution

* Fix: Pin memory in gets (C#)

* Make: Skip `PersistAndRestore` in CI on MacOS

* Make: Upgrade Docker action

This fixes a GitHub CI warning about the
deprecated NodeJS version.

* Fix: `view_from_buffer` is unsafe in Rust

Closes #453

Co-authored-by: Andrew Dirksen <[email protected]>"

* Fix: `view_from_buffer` is unsafe in Rust

Closes #453

Co-authored-by: Andrew Dirksen <[email protected]>

* Make: Upgrade SimSIMD

* Docs: Index header has no capacity

The lack of capacity data is intended.
Reserving memory is a non-persistent operation
by nature, and we shouldn't save that metadata
on the disk.

Closes #452

Co-authored-by: Christopher Yim <[email protected]>

* Fix: Aggressive neighborhood checks on updates

* Fix: `update()` bug detect with non-POD keys

This bug was tough to spot. Apple Clang was the only one
that caught it. The `-O0` flags were explicitly added to expose
more symbols for debugging. More `uint40_t` tests were added.

* Fix: Align vector type w index in C# (#456)

Co-authored-by: Ash Vardanian <[email protected]>

* Make: Versioning in pre-release CI

* Make: Switch from SemanticRelease to TinySemVer

---------

Co-authored-by: Jaysen Marais <[email protected]>
Co-authored-by: Antonio Andelic <[email protected]>
Co-authored-by: Narek Galstyan <[email protected]>
Co-authored-by: Adolfo Garcia <[email protected]>
Co-authored-by: Trevor McCulloch <[email protected]>
Co-authored-by: Robert Schulze <[email protected]>
Co-authored-by: Mikhail Bautin <[email protected]>
Co-authored-by: Mikhail Bautin <[email protected]>
Co-authored-by: SheldonFung <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: cinchen <[email protected]>
Co-authored-by: Mark Reed <[email protected]>
Co-authored-by: John <[email protected]>
Co-authored-by: batracos <[email protected]>
Co-authored-by: Ziyang Hu <[email protected]>
Co-authored-by: Julius Brummack <[email protected]>
Co-authored-by: Julius Brummack <[email protected]>
Co-authored-by: Andrew Dirksen <[email protected]>
Co-authored-by: Christopher Yim <[email protected]>
Co-authored-by: Britt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.