Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document rake build:checksum #325

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

pboling
Copy link
Contributor

@pboling pboling commented Oct 27, 2022

Related to rubygems/rubygems#5942 and rubygems/rubygems#6022

Script uses a good regex, /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/, for pulling out a version match:
https://rubular.com/r/9QuDiGVjvlHOrt

@pboling pboling changed the title Document rake dbuild:checksum Document rake build:checksum Oct 27, 2022
security.md Outdated
rake build:checksum

The checksums will be placed in the `checksums/` directory. It is recommended to commit checksums so others can verify
the authenticity of a release.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would migrate the previous instructions to use the builtin task and that's it. Officially recommending it should be a different discussion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I'll update soon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the "recommendation" implication. Also improved the script to what I am currently using as I'm not sure when I'll get back around to the rake task, since in its current form it isn't useful to me at all, and I have a deep todo list. I'll remove the script portion if preferred.

@pboling pboling force-pushed the document-checksum branch 2 times, most recently from acc0500 to f9be437 Compare October 10, 2023 03:55
security.md Outdated Show resolved Hide resolved
@pboling
Copy link
Contributor Author

pboling commented Oct 10, 2023

Very strangely Gem::Version.new(version) compared to another of the same isn't working for pre-release versions. Outside pre-release versions, the script works. 😖

rubygems/rubygems#3086 (comment)

security.md Outdated

require "digest/sha2"

VERSION_REGEX = /\d+\.\d+\.\d+([-.].+)*/.freeze
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect when applied to a filename since it will include .gem in the version number, making everything a pre-release.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oooh, that's why it fails... interesting. I will see if I can find what rubygems uses for parsing gem names.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I knew it included the .gem, but it hadn't ever mattered until I had to deal with prereleases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martinemde Fixed!

@pboling
Copy link
Contributor Author

pboling commented Oct 11, 2023

# Final clause of Regex `(?=\.gem)` is a positive lookahead assertion
# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
# Used to pattern match against a gem package name, which always ends with .gem.
# The positive lookahead ensures it is present, and prevents it from being captured.
VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze

A good regex is hard to find. 🤣

@@ -72,61 +72,137 @@ Building Gems
### Sign with: `gem cert`

1) Create self-signed gem cert

```shell
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The enumerated list was not proper markdown, and after a few syntax changes, which should also allow language-specific code highlighting, it is now proper markdown.

@@ -141,7 +217,7 @@ Reporting Security vulnerabilities

### Reporting a security vulnerability with someone else's gem

If you spot a security vulnerability in someone else's gem, then you
If you spot a security vulnerability in someone else's gem, then your
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a typo you => your

# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
# Used to pattern match against a gem package name, which always ends with .gem.
# The positive lookahead ensures it is present, and prevents it from being captured.
VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regex works perfectly.

https://rubular.com/r/9QuDiGVjvlHOrt

pboling added a commit to pboling/require_bench that referenced this pull request Oct 12, 2023
@pboling
Copy link
Contributor Author

pboling commented Oct 20, 2023

Unfortunate discovery. The process of running a checksum script or task outside of the rake release results in checksums that do not match the actual released gem.

❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909
❯ bin/checksums
Looking for: "pkg/*.gem"
Found: 4 gems; latest is kettle-soup-cover-1.0.2.gem
[ GEM: kettle-soup-cover-1.0.2.gem ]
[ VERSION: 1.0.2 ]
[ GEM PKG LOCATION: pkg/kettle-soup-cover-1.0.2.gem ]
[ CHECKSUM SHA-256: 677d0f4778e065be83d9e77b0514b64731cb4313fa2b085056e9ec45f92991a5 ]
[ CHECKSUM SHA-512: de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909 ]
[ CHECKSUM SHA-256 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha256 ]
[ CHECKSUM SHA-512 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha512 ]

... Running ...

git add checksums/* && git commit -m "🔒️ Checksums for v1.0.2"

[main e62712c] 🔒️ Checksums for v1.0.2
 2 files changed, 2 insertions(+)
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha256
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha512
❯ bundle exec rake release
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
Tagged v1.0.2.
Pushed git commits and release tag.
Pushing gem to https://rubygems.org...
You have enabled multi-factor authentication. Please enter OTP code.
Code:   077236
Successfully registered gem: kettle-soup-cover (1.0.2)
Pushed kettle-soup-cover 1.0.2 to rubygems.org
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

80971e7cd27a4d22b8a8d93e869e7a2543a21455e5ae0700b0321268f555cc81381bec88b024017e9e74fa4a0cbfa4bbfe2ad1db2f5fc0b77259906fce15198d
❯ bundle exec rake build
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

12354158c4a5c116aacb24cb16efd9b9a6240673c551042814b9f7e82d95de8d7cc87b5875a52e33bac660a5e182a581683af33cf7caef86930536d3006b2fe8

It seems that the only solution would be to run the checksums after release, but not running a new build of the gem package. The checksum commits would thus be after the release, and not in the git tag for the release. 😿

@martinemde
Copy link
Member

martinemde commented Oct 20, 2023

Are you including the checksums files in the gem? Just a guess? (You would want to not do that.)

Is having the checksum in the tag even correct? How can a commit contain its own digest? Inception?

@pboling
Copy link
Contributor Author

pboling commented Oct 20, 2023

Are you including the checksums files in the gem?

Nope, I wondered the same thing, because if there were file changes between the gem builds, then they should have different checksums. Alas, no such luck.

Is having the checksum in the tag even correct?

I thought so, because the checksums are checksums of the built gem package. They have nothing to do with git.

How can a commit contain its own digest? Inception?

LOL, yeah it is a mind F8cker. It's really just that the gem package is wholly unrelated to the tracking of the project in git.

So the thing I don't understand is this - if the files are not changing, why does the checksum change everytime the gem is built? Either that's what is happening, or the packaged gem that results from rake build is fundamentally different than the packaged gem that results from rake release, and that lands somewhere between moderately and wildly disconcerting. 🤣 For a while I thought I was losing my mind, and then a pang of shame, realizing that every gem I have pushed, every checksum I have pushed does not match any of my releases, for the last year+. 🦺

I think I've found a bug. Because I assume that rake build and rake release should create identical packages, which would necessarily result in identical checksums.

@pboling
Copy link
Contributor Author

pboling commented Oct 20, 2023

To state as simply as possible:

Here's the repo I've been doing this evaluation in.

Somehow rake build and rake release create .gem packages that have different checksums. 😱

IMO, both tools are useful, but they must build identical packages...

@segiddins
Copy link
Member

They do build identical packages, but with different timestamps unless you're setting the same SOURCE_DATE_EPOCH

@pboling
Copy link
Contributor Author

pboling commented Oct 20, 2023

Hero shit. ❤️ @segiddins. Thanks! I'll update to reflect that. 🔥

@pboling
Copy link
Contributor Author

pboling commented Oct 20, 2023

I think this should be prominently documented for both tasks, as for me, this is quite a violation of POLS. I'll work on PRs for that too. 🍔

Update

I haven't looked recently, so it if already is prominent, then I'm a victim of POLS assumptions. I may have never looked at the documentation for the tasks beyond the rake desc, because they seem extremely clearly named and obvious in function.

It seemed self-evident that both tasks should build an identical package, and I never considered timestamps, because that has never been on my radar... But this did cause me quite a problem, as I now have hundreds? of gems published with bad checksums, and bad release instructions. Since updated docs likely wouldn't have even helped me, I need to think about what a better solution might be.

pboling added a commit to pboling/rspec-block_is_expected that referenced this pull request Jun 23, 2024
pboling added a commit to oauth-xx/ruby-openid2 that referenced this pull request Sep 4, 2024
pboling added a commit to pboling/gem_bench that referenced this pull request Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants