Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Several upstream bottles cause segfault. related to old patchelf.rb bugs. #163826

Open
4 tasks done
redspot opened this issue Feb 22, 2024 · 61 comments
Open
4 tasks done

Several upstream bottles cause segfault. related to old patchelf.rb bugs. #163826

redspot opened this issue Feb 22, 2024 · 61 comments
Labels
bug Reproducible Homebrew/homebrew-core bug help wanted Task(s) needing PRs from the community or maintainers in progress Stale bot should stay away

Comments

@redspot
Copy link

redspot commented Feb 22, 2024

brew gist-logs <formula> link OR brew config AND brew doctor output

$ brew config
HOMEBREW_VERSION: 4.2.9-88-g94e987e
ORIGIN: https://github.com/Homebrew/brew
HEAD: 94e987ee65fa6ae564f86d37fe21b5ad81a258d4
Last commit: 26 minutes ago
Core tap HEAD: 3e47b3bac60e610ea8fcdfcf9693c25812ef1727
Core tap last commit: 3 months ago
Core tap JSON: 22 Feb 21:25 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
SUDO_ASKPASS: /home/wmartin45/bin/askpass-kdewallet
Homebrew Ruby: 3.1.4 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: octa-core 64-bit skylake
Clang: 17.0.6
Git: 2.39.3 => /bin/git
Curl: 7.61.1 => /bin/curl
Kernel: Linux 4.18.0-513.5.1.el8_9.x86_64 x86_64 GNU/Linux
OS: Red Hat Enterprise Linux release 8.9 (Ootpa)
Host glibc: 2.28
/usr/bin/gcc: 8.5.0
/usr/bin/ruby: N/A
glibc: 2.35_1
gcc@11: N/A
gcc: 13.2.0
xorg: N/A

$ brew doctor
Your system is ready to brew.
$ brew update --auto-update  # no output

Verification

  • My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
  • I ran brew update and am still able to reproduce my issue.
  • I have resolved all warnings from brew doctor and that did not fix my problem.
  • I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

I was attempting to use several formulae installed via brew.

  • ssdeep
  • go
  • pkg-config

There are several related issues:
#137991 related to old patchelf.rb bug
#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL

What happened (include all command output)?

$ brew reinstall --force-bottle ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/manifests/2.14.1
Already downloaded: /home/wmartin45/.cache/Homebrew/downloads/5d5c962b71849c67a5e3e6c0df39b82fd0b823ac868d942f93e6a1bec0a6a03b--ssdeep-2.14.1.bottle_manifest.json
==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:debece05c49ee73f650afa080d3f0953ac64449c79169525001e871d0888edef
Already downloaded: /home/wmartin45/.cache/Homebrew/downloads/89333f5f972034d0bbcfbdab464d33f04ced2b2f6e86f532cf6e6c95484a2b7e--ssdeep--2.14.1.x86_64_linux.bottle.tar.gz
==> Reinstalling ssdeep
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 194.9KB
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
Segmentation fault
$ dmesg | grep ssdeep
[4717362.289573] 3149001 (ssdeep): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

Note the segment that causes the problem, 0x0000000000401000

Now, take a look at the on-disk segments:

$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep
Elf file type is EXEC (Executable file)
Entry point 0x4028e0
There are 11 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x00000000003ff040 0x00000000003ff040
                 0x0000000000000268 0x0000000000000268  R E    0x8
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  LOAD           0x0000000000000000 0x00000000003ff000 0x00000000003ff000
                 0x00000000000022ae 0x00000000000022ae  RW     0x1000
  INTERP         0x0000000000001308 0x0000000000400308 0x0000000000400308
                 0x0000000000000025 0x0000000000000025  R      0x1
      [Requesting program interpreter: /home/linuxbrew/.linuxbrew/lib/ld.so]
  NOTE           0x0000000000001330 0x0000000000400330 0x0000000000400330
                 0x0000000000000020 0x0000000000000020  R      0x4
  NOTE           0x0000000000001350 0x0000000000400350 0x0000000000400350
                 0x0000000000000024 0x0000000000000024  R      0x4
  LOAD           0x00000000000022ae 0x00000000004012ae 0x00000000004012ae
                 0x0000000000009917 0x0000000000009917  R E    0x1000
  GNU_EH_FRAME   0x000000000000a580 0x0000000000409580 0x0000000000409580
                 0x00000000000002cc 0x00000000000002cc  R      0x4
  LOAD           0x000000000000bda0 0x000000000060ada0 0x000000000060ada0
                 0x00000000000004f4 0x00000000000005a8  RW     0x1000
  GNU_RELRO      0x000000000000bda0 0x000000000060ada0 0x000000000060ada0
                 0x0000000000000260 0x0000000000000260  R      0x1
  DYNAMIC        0x000000000000bdf8 0x000000000060adf8 0x000000000060adf8
                 0x0000000000000200 0x0000000000000200  RW     0x8

$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep | grep '0x0000000000401... '
  LOAD           0x00000000000022ae 0x00000000004012ae 0x00000000004012ae

Note the segment 0x00000000004012ae

Also, of interest, when run using brew's interpreter, programs seem to work:

$ /home/linuxbrew/.linuxbrew/lib/ld.so /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1

Here is the results when built from source:

$ brew install --build-from-source ssdeep
==> Fetching ssdeep
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/11dc5d3906c536d30245ec1bace1b73338857c9b/Formula/s/ssdeep.rb
######################################################################################################################################### 100.0%
==> Downloading https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz
==> Downloading from https://objects.githubusercontent.com/github-production-release-asset-2e65be/93355216/ae5313b0-c3b0-11e7-88a8-251ebd913c9d?
######################################################################################################################################### 100.0%
==> ./configure
==> make install
🍺  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 201.6KB, built in 9 seconds
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep | grep '0x0000000000401... '  # no matches
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1

What did you expect to happen?

I expect the upstream pre-built bottles to not have lingering issues with patchelf.rb.

Now, the issues with patchelf.rb have been fixed, and maybe those Centos/RHEL related issues as well.

It's that the bottles still have the problem.

Step-by-step reproduction instructions (by running brew commands)

As mentioned in this comment: #132852 (comment)

Some formulas work:

$ /home/linuxbrew/.linuxbrew/bin/xz --version
xz (XZ Utils) 5.4.5
liblzma 5.4.5

Some formulas fail:

$ /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
Segmentation fault
$ dmesg | grep pkg-config
[4719592.428429] 3158801 (pkg-config): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already
$ readelf -l /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config | grep '0x0000000000401... '
  LOAD           0x0000000000006dd2 0x0000000000401dd2 0x0000000000401dd2
$ /home/linuxbrew/.linuxbrew/lib/ld.so /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
0.29.2
$ readlink -f /home/linuxbrew/.linuxbrew/lib/ld.so
/home/linuxbrew/.linuxbrew/Cellar/glibc/2.35_1/lib/ld-linux-x86-64.so.2

Note the kernel reporting the same segment, 0x0000000000401000, and readelf showing that segment. And, it strangely works when run with brew's interpreter.

when building from source:

$ brew reinstall --build-from-source pkg-config
Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
==> Fetching pkg-config
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/11dc5d3906c536d30245ec1bace1b73338857c9b/Formula/p/pkg-config.rb
######################################################################################################################################### 100.0%
==> Downloading https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
######################################################################################################################################### 100.0%
==> Reinstalling pkg-config
==> ./configure --disable-host-tool --with-internal-glib --with-pc-path=/home/linuxbrew/.linuxbrew/lib/pkgconfig:/home/linuxbrew/.linuxbrew/shar
==> make
==> make install
🍺  /home/linuxbrew/.linuxbrew/Cellar/pkg-config/0.29.2_3: 11 files, 750.6KB, built in 40 seconds
==> Running `brew cleanup pkg-config`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ readelf -l /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config | grep '0x0000000000401... '  # no matches
$ /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
0.29.2

It also seems reliable to detect the bad packages by looking for a segment that matches the regex 0x0000000000401....

@redspot redspot added the bug Reproducible Homebrew/homebrew-core bug label Feb 22, 2024
@redspot
Copy link
Author

redspot commented Feb 22, 2024

I don't know why all of the Markdown blocks got flipped around.

Here are the related issues and comment:
#137991 related to old patchelf.rb bug
#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL

#132852 (comment)

@redspot
Copy link
Author

redspot commented Feb 25, 2024

This issue seems related: #136432

@redspot
Copy link
Author

redspot commented Feb 25, 2024

I think the root cause related to many homebrew Linux segmentation faults is the homebrew build bot.

The build bot making the bottles is still somehow using an older patchelf.rb which causes elf binaries to be generated with improper segments causing memory map collisions.

@iMichka
Copy link
Member

iMichka commented Feb 26, 2024

Latest patchelf updates in brew:
Homebrew/brew#16694
Homebrew/brew#14094

Most issues have been fixed with version 1.4.0 in Nov. 2022.

Why do you think it's still using an old patchelf.rb version?

In your config:

Core tap last commit: 3 months ago

This looks old. Can you update first?

@redspot
Copy link
Author

redspot commented Feb 26, 2024

Latest patchelf updates in brew: Homebrew/brew#16694 Homebrew/brew#14094

Most issues have been fixed with version 1.4.0 in Nov. 2022.

Why do you think it's still using an old patchelf.rb version?

It's not an issue with patchelf.rb. It's an issue with the pre built bottles.

See above the part with brew reinstall --force-bottle ssdeep which segfaults, but works fine when built from source.

In your config:

Core tap last commit: 3 months ago

This looks old. Can you update first?

How do I do that?

I ran brew update --auto-update with no output given.

@gromgit
Copy link
Member

gromgit commented Feb 27, 2024

How do I do that?

HOMEBREW_NO_INSTALL_FROM_API=1 brew update should do the necessary.

@iMichka
Copy link
Member

iMichka commented Feb 27, 2024

Don't worry about the 3 months old update message: you are now using the API to get the latest version, so I guess no need to update the git repo on your computer to get the latest version. I think that message is confusing, I'll have a look into it later.

I think I know what happened.

patchelf was updated in Nov 2022, but I did not realise patchelf was only updated 1 week ago in brew ...

We will have to rebuild these bottles.

I have triggered a ssdeep rebuild: https://github.com/Homebrew/homebrew-core/actions/runs/8060846410

Once this is done, you can brew update, and the brew reinstall ssdepp and try the new bottle.

@redspot
Copy link
Author

redspot commented Feb 27, 2024

Sounds great. There are several other bottles with crashes, such as golang.

@pdelre
Copy link

pdelre commented Feb 27, 2024

👋 Been quietly watching this and previous issues.

In WSL Ubuntu this is pretty wide spread from packages built with go and those that are built with gcc. I believe I saw less issues when I tested in a Ubuntu VM.

What could be an effective way forward for the project maintainers? I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful? I'd be happy to send along the packages I've had issues with and help UAT rebuilds.

@iMichka
Copy link
Member

iMichka commented Feb 27, 2024

@redspot a new ssdeep bottle has been added. Can you brew update && brew reinstall ssdeppto confirm that this fixes it for you.

I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful?

Yes, if you have a list, we can at least rebuild all of these. Most packages get updates / rebuilds at least once per year, but some others will take time to get update. Let's fix what is known to be broken.

@pdelre
Copy link

pdelre commented Feb 27, 2024

Here's a list from my bash/zsh history, though I think I had an issue with every package that uses go as a build dependency.

  • act
  • docker
  • docker-compose
  • findutils
  • fzf
  • gh
  • go
  • go-task
  • gron
  • k6
  • k9s
  • noti
  • pandoc
  • unzip
  • whalebrew
  • yq

@redspot
Copy link
Author

redspot commented Feb 28, 2024

see this comment for more potentially broken bottles #132852 (comment)

I definitely had issues with these formulas:

  • ssdeep
  • xz
  • pkg-config

but, there could have been others. It's not that I would like you to fix a specific formula. It's that you most likely have many broken bottles that need a rebuild.

I guess you could come up with a way to find bottles that are:

  • Linux x86_64
  • has an ELF64 file with:
  • a memory segment that matches '0x00000000004..... ', maybe check with readelf -l

@iMichka
Copy link
Member

iMichka commented Mar 1, 2024

Can just someone confirm that brew update && brew reinstall ssdepp fixes it before I start rebottling stuff?

Note: we probably won't rebottle everything, I'll go through the lists above, for anything else users will have to open an issue or post here.

@redspot
Copy link
Author

redspot commented Mar 1, 2024

Sure, I'll test out the new bottle on Monday.

I could always solve the problem building from source. This ticket was just to give the project and other users a heads up.

@pdelre
Copy link

pdelre commented Mar 5, 2024

@iMichka I installed ssdeep and it does not segfault as the OP describes with ssdeep -V.

Output
$ brew install ssdeep
...
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/manifests/2.14.1-1
#################################################################################################################################################################################################################################### 100.0%
==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:d62ce2005fd901e4a363b3f6592dee0388e9114d323a058499d16e8dba9c4d38
#################################################################################################################################################################################################################################### 100.0%
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.1.tar.gz
�  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 215.4KB
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ /home/linuxbrew/.linuxbrew/bin/ssdeep -h
ssdeep version 2.14.1 by Jesse Kornblum and the ssdeep Project
For copyright information, see man page or README.TXT.

Usage: ssdeep [-m file] [-k file] [-dpgvrsblcxa] [-t val] [-h|-V] [FILES]
-m - Match FILES against known hashes in file
-k - Match signatures in FILES against signatures in file
-d - Directory mode, compare all files in a directory
-p - Pretty matching mode. Similar to -d but includes all matches
-g - Cluster matches together
-v - Verbose mode. Displays filename as its being processed
-r - Recursive mode
-s - Silent mode; all errors are suppressed
-b - Uses only the bare name of files; all path information omitted
-l - Uses relative paths for filenames
-c - Prints output in CSV format
-x - Compare FILES as signature files
-a - Display all matches, regardless of score
-t - Only displays matches above the given threshold
-h - Display this help message
-V - Display version number and exit

$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1

@redspot
Copy link
Author

redspot commented Mar 5, 2024

==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:d62ce2005fd901e4a363b3f6592dee0388e9114d323a058499d16e8dba9c4d38
######################################################################################################################################### 100.0%
==> Reinstalling ssdeep 
==> Installing ssdeep
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.1.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 217.2KB
==> Running `brew cleanup ssdeep`...

$ ssdeep -V
2.14.1

Yes, it works from the bottle.

@fxcoudert
Copy link
Member

fxcoudert commented Mar 5, 2024

I've launched a couple of rebottles for formulas that were listed above and where the rebottling won't erase any old bottle (for previous macOS versions). In those cases, there is no downside to rebottling. Then, we can confirm whether the issues are all fixed, and decide how to proceed next.

I've launched: act docker docker-compose fzf gawk gh go-task k6 k9s pandoc xz yq
Once this is all done (I'd say in a few hours, but you can check progress there: https://github.com/Homebrew/homebrew-core/actions/workflows/dispatch-rebottle.yml) please reinstall, test, and report success or failure (if you were seeing the bug before).

Copy link
Contributor

github-actions bot commented Mar 5, 2024

@fxcoudert bottle request for gh failed.

Copy link
Contributor

github-actions bot commented Mar 5, 2024

@fxcoudert bottle request for k9s failed.

@pdelre
Copy link

pdelre commented Mar 5, 2024

@fxcoudert I've tried a few of these (act, k6, gh, yq) and they still segfault. Looking at the GH Actions, it appears they depend on the Homebrew go package which is affected on my system. Could an order of operation be affecting the built bottles?

$ k6 --version # previously built from source
k6 v0.49.0 (go1.21.6, linux/amd64)

$ brew update && brew reinstall k6
Already up-to-date.
==> Downloading https://ghcr.io/v2/homebrew/core/k6/manifests/0.49.0
####################################################################################################################################################################################################################### 100.0%
==> Fetching k6
==> Downloading https://ghcr.io/v2/homebrew/core/k6/blobs/sha256:2e548198d61b1f5336646ff43bf7f2cd75ed69c288920c5d47d18f3219c40e65
####################################################################################################################################################################################################################### 100.0%
==> Reinstalling k6 
==> Pouring k6--0.49.0.x86_64_linux.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
�  /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 37.6MB
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ k6 --version
zsh: segmentation fault  k6 --version

$ brew update && brew reinstall -s k6
Already up-to-date.
Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
==> Fetching k6
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/726ba7780161a49c1d6b4b202f854e13fb5dca66/Formula/k/k6.rb
####################################################################################################################################################################################################################### 100.0%
==> Downloading https://github.com/grafana/k6/archive/refs/tags/v0.49.0.tar.gz
Already downloaded: /home/pdelre/.cache/Homebrew/downloads/cb0f5adcdfa30d830e9f759aeb1266ce066200d6e60a225a1a5fb2bf7d8ca74c--k6-0.49.0.tar.gz
==> Reinstalling k6 
==> go build -ldflags=-s -w
==> Caveats
zsh completions have been installed to:
  /home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
�  /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 38MB, built in 1 minute 8 seconds
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ k6 --version
k6 v0.49.0 (go1.22.0, linux/amd64)

@fxcoudert
Copy link
Member

fxcoudert commented Mar 5, 2024

I don't know enough about how go executables are built & linked to be able to answer that. go is being rebuilt as part of this PR #165116 so we'll know soon.

@redspot
Copy link
Author

redspot commented Mar 5, 2024

@pdelre perhaps you could brew install -s golang first.

@pdelre
Copy link

pdelre commented Mar 5, 2024

@redspot Yes, that's how I resolve the builds locally.

@chenrui333
Copy link
Member

chenrui333 commented Mar 5, 2024

the root cause is with the CGO_ENABLED for go build, we should turn it all for all affected formulae.

see https://github.com/grafana/k6/blob/master/release%20notes/v0.26.0.md?plain=1#L127 (for k6 for example)

This was referenced Mar 5, 2024
Copy link
Contributor

github-actions bot commented Mar 5, 2024

@fxcoudert bottle request for go-task failed.

@chenrui333 chenrui333 mentioned this issue Mar 5, 2024
6 tasks
@pdelre
Copy link

pdelre commented Mar 19, 2024

@Bo98 That patch works on my WSL2 Ubuntu, Kernel 4.

Kernel: Linux 4.19.104-microsoft-standard x86_64 GNU/Linux
OS: Ubuntu 22.04.4 LTS (jammy)
WSL: 2

@Shaloc
Copy link

Shaloc commented Mar 19, 2024

Hi, can you try the following patch:

diff --git a/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb b/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
index 7937d14e6e..d7caf39e54 100644
--- a/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
+++ b/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
@@ -561,11 +561,12 @@ module PatchELF
       if needed_space > start_offset
         needed_space += seg_num_bytes # new load segment is required
 
-        needed_pages = Helper.alignup(needed_space - start_offset, page_size) / page_size
+        extra_bytes = needed_space - start_offset
+        needed_pages = Helper.alignup(extra_bytes, page_size) / page_size
         Logger.debug "needed pages is #{needed_pages}"
         raise PatchError, 'virtual address space underrun' if needed_pages * page_size > first_page
 
-        shift_file(needed_pages, start_offset)
+        shift_file(needed_pages, start_offset, extra_bytes)
 
         first_page -= needed_pages * page_size
         start_offset += needed_pages * page_size
@@ -776,7 +777,7 @@ module PatchELF
     end
     # rubocop:enable Metrics/PerceivedComplexity
 
-    def shift_file(extra_pages, start_offset)
+    def shift_file(extra_pages, start_offset, extra_bytes)
       raise PatchError, "start_offset(#{start_offset}) < ehdr.num_bytes" if start_offset < ehdr.num_bytes
 
       oldsz = @buffer.size
@@ -799,8 +800,8 @@ module PatchELF
         p_offset: split_phdr.p_offset - split_shift - shift,
         p_vaddr: split_phdr.p_vaddr - split_shift - shift,
         p_paddr: split_phdr.p_paddr - split_shift - shift,
-        p_filesz: split_shift + shift,
-        p_memsz: split_shift + shift,
+        p_filesz: split_shift + extra_bytes,
+        p_memsz: split_shift + extra_bytes,
         p_flags: ELFTools::Constants::PF_R | ELFTools::Constants::PF_W,
         p_align: page_size
       )

It seems to fix things for me on a x86_64 CentOS 8 machine.

This works on my aarch64, CentOS 7, linux 4.19.91.

@redspot
Copy link
Author

redspot commented Mar 25, 2024

@Bo98 Ok, I'll give that patch a try.

First, I'll make sure everything is up-to-date:

$ brew update
==> Updating Homebrew...
Already up-to-date.
$ brew upgrade
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ brew doctor
Your system is ready to brew.

$ brew config
HOMEBREW_VERSION: 4.2.15-4-g966f819
ORIGIN: https://github.com/Homebrew/brew
HEAD: 966f819deb81796a1b9ea91abc65ab433c70362b
Last commit: 2 hours ago
Core tap HEAD: f74275618b8a99822357df93e75e86934b2620df
Core tap last commit: 3 minutes ago
Core tap JSON: 25 Mar 13:39 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_EDITOR: vim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
SUDO_ASKPASS: /home/wmartin45/bin/askpass-kdewallet
Homebrew Ruby: 3.1.4 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: octa-core 64-bit skylake
Clang: 17.0.6
Git: 2.44.0 => /home/linuxbrew/.linuxbrew/bin/git
Curl: 7.61.1 => /bin/curl
Kernel: Linux 4.18.0-513.18.1.el8_9.x86_64 x86_64 GNU/Linux
OS: Red Hat Enterprise Linux release 8.9 (Ootpa)
Host glibc: 2.28
/usr/bin/gcc: 8.5.0
/usr/bin/ruby: N/A
glibc: 2.35_1
gcc@11: N/A
gcc: 13.2.0
xorg: N/A

$ cd $(brew --repo) && pwd
/home/linuxbrew/.linuxbrew/Homebrew

$ git apply --check patchelf.patch && git apply --summary patchelf.patch
$ # no output

So, it seems like the patch is already applied upstream.

Next, I'll try the steps suggested by @Shaloc :

$ brew uninstall --ignore-dependencies xz
Uninstalling /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1... (167 files, 2.8MB)

$ brew install --build-bottle xz
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Fetching xz
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/14af31c6a3799ffe63a6cd74c6f738d48f549ef4/Formula/x/xz.rb
######################################################################################################################################### 100.0%
==> Downloading https://github.com/tukaani-project/xz/releases/download/v5.6.1/xz-5.6.1.tar.gz
==> Downloading from https://objects.githubusercontent.com/github-production-release-asset-2e65be/553665726/20b30ac5-c91c-4dc7-85a1-2b41dd49d6cb
######################################################################################################################################### 100.0%
==> ./configure --disable-silent-rules --disable-nls
==> make check
==> make install
==> Not running 'post_install' as we're building a bottle
You can run it manually using:
  brew postinstall xz
🍺  /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1: 167 files, 2.7MB, built in 29 seconds
==> Running `brew cleanup xz`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ brew postinstall xz
==> Postinstalling xz
Warning: xz: no `post_install` method was defined in the formula!

Let's check xz before bottling:

$ readlink -f $(brew --prefix)/opt/xz/bin/xz
/home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1/bin/xz

$ /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1/bin/xz --version  # before brew bottle
xz (XZ Utils) 5.6.1
liblzma 5.6.1

No segfault. Now, we try bottling:

$ brew bottle xz
==> Determining xz bottle rebuild...
==> Bottling xz--5.6.1.x86_64_linux.bottle.1.tar.gz...
./xz--5.6.1.x86_64_linux.bottle.1.tar.gz
  bottle do
    rebuild 1
    sha256 cellar: :any_skip_relocation, x86_64_linux: "e519149ba6e96fc972118e41fcb49f3aabfa3b290c911fdf969044ac78ec78e8"
  end

$ /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1/bin/xz --version  # after brew bottle
xz (XZ Utils) 5.6.1
liblzma 5.6.1

Also no segfault.

And, finally let's check the virtual address maps from the ELF header:

$ readelf -l /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1/bin/xz \
| awk '/(\s0x[[:xdigit:]]{16}){3}/{print $3}'
0x0000000000000040
0x0000000000000000
0x0000000000000000
0x0000000000004000
0x000000000000f000
0x00000000000129a0
0x0000000000015230
0x0000000000015230
0x0000000000015db0
0x000000000001c000
0x000000000001c378
0x000000000001c398
0x000000000001c398
0x000000000001d000
0x000000000001e000
0x000000000001f000
0x000000000001f828

$ readelf -l /home/linuxbrew/.linuxbrew/Cellar/xz/5.6.1/bin/xz \
| awk '/(\s0x[[:xdigit:]]{16}){3}/{print $3}' \
| cut -c 12-
0000040
0000000
0000000
0004000
000f000
00129a0
0015230
0015230
0015db0
001c000
001c378
001c398
001c398
001d000
001e000
001f000
001f828

No segments match virtual address 0x400000-0x4fffff.

and, here's some kernel info:

$ dnf info kernel-core-4.18.0-513.18.1.el8_9
Not root, Subscription Management repositories not updated
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)                                                        165 kB/s | 4.5 kB     00:00    
Installed Packages
Name         : kernel-core
Version      : 4.18.0
Release      : 513.18.1.el8_9
Architecture : x86_64
Size         : 71 M
Source       : kernel-4.18.0-513.18.1.el8_9.src.rpm
Repository   : @System
From repo    : rhel-8-for-x86_64-baseos-rpms
Summary      : The Linux kernel
URL          : http://www.kernel.org/
License      : GPLv2 and Redistributable, no modification permitted
Description  : The kernel package contains the Linux kernel (vmlinuz), the core of any
             : Linux operating system.  The kernel handles the basic functions
             : of the operating system: memory allocation, process allocation, device
             : input and output, etc.

@pdelre
Copy link

pdelre commented Apr 4, 2024

@Bo98 Thank you for your patch solution as it's continued to work through multiple upgrades (including xz 😬)!

It appears an upstream contribution must be made to https://github.com/david942j/patchelf.rb and that you were the author of the NixOS alt_saver.rb bugfixes port the Ruby library. Is there anything you could use assistance with so the solution is available in the Homebrew mainline?

@Bo98
Copy link
Member

Bo98 commented Apr 4, 2024

Yes, I will be getting things moving this weekend. There was a patch for a rustfmt-specific issue that I was preparing as well.

@redspot
Copy link
Author

redspot commented Apr 9, 2024

the mingw-w64 bottle also segfaults. while trying to build mingw-w64 from source, bottle m4 also segfaulted.

$ brew info mingw-w64
==> mingw-w64: stable 11.0.1 (bottled)
Minimalist GNU for Windows and GCC cross-compilers
https://sourceforge.net/projects/mingw-w64/
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/m/mingw-w64.rb
License: ZPL-2.1
==> Dependencies
Build: texinfo ✘
Required: gmp ✔, isl ✔, libmpc ✔, mpfr ✔, gcc ✔, glibc ✔
==> Analytics
install: 3,952 (30 days), 12,579 (90 days), 67,741 (365 days)
install-on-request: 1,770 (30 days), 6,139 (90 days), 29,420 (365 days)
build-error: 71 (30 days)

$ brew install mingw-w64
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Downloading https://ghcr.io/v2/homebrew/core/mingw-w64/manifests/11.0.1
######################################################################################################################################### 100.0%
==> Fetching mingw-w64
==> Downloading https://ghcr.io/v2/homebrew/core/mingw-w64/blobs/sha256:ed0e3ad639d4c7754beb5943b5b5dc0e806005a14c2aee5230cd5d114dc4f9a8
######################################################################################################################################### 100.0%
==> Pouring mingw-w64--11.0.1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/mingw-w64/11.0.1: 7,870 files, 1GB
==> Running `brew cleanup mingw-w64`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
[551043.099975] 259846 (i686-w64-mingw3): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already
[551474.392061] 298031 (m4): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

@redspot
Copy link
Author

redspot commented Apr 14, 2024

Is there a way to trigger patchelf.rb on a precompiled bottle, like brew postinstall? @Shaloc @pdelre @Bo98

@sastorsl
Copy link

sastorsl commented Apr 22, 2024

I just migrated a Centos 8 Stream to Rocky Linux, and encountered the Segmentation fault problem.
...another problem is that I had not touched brew for a while, so knowing when the problem hit is an issue, but I removed brew entirely and re-installed, and got the same problem.

While debugging I started with the rockylinux:8 docker image to try and replicate, but was not able to replicate, akà everything works.

But I'm dropping this here in case it can help the debugging and/or reproducing efforts.

docker run \  # ...or nerdctl
  -ti \
  --rm \
  --platform amd64 \
  -v $HOME/.docker/certs.d:/etc/pki/ca-trust/source/anchors \  # My env requires a trust CA, remove at will
  rockylinux:8 bash -c '
    update-ca-trust        # My env requires a trust CA, remove at will
    dnf -y install git procps sudo
    dnf -y groupinstall "Development Tools"
    echo "brewtest ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/brewtest
    useradd -m brewtest
    sudo su - brewtest -c /bin/bash -c "NONINTERACTIVE=1 $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    (echo; echo "eval \"$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)\"") >> /home/brewtest/.bashrc
    sudo su - brewtest -c /bin/bash -c "brew install gcc"
    sudo su - brewtest -c /bin/bash -c "brew install gh"
    sudo su - brewtest -c /bin/bash -c "gh"
  '

Now, in my server env the gh command fails, but in this docker env it works.
If possible I'll update if I find a way to reproduce the error.

@redspot

This comment was marked as outdated.

@redspot
Copy link
Author

redspot commented May 15, 2024

After recently upgrading brew, gcc-14.1.0 crashes.

[627937.886892] 590828 (collect2): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

recompiling gcc from source is rather time-consuming. But, I guess that's the only fix for now.

@redspot
Copy link
Author

redspot commented May 15, 2024

After recently upgrading brew, gcc-14.1.0 crashes.

[627937.886892] 590828 (collect2): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

recompiling gcc from source is rather time-consuming. But, I guess that's the only fix for now.

==> make
Last 15 lines from ~/.cache/Homebrew/Logs/gcc/02.make:
yes
configure: updating cache ./config.cache
configure: creating ./config.status
config.status: creating Makefile
config.status: creating testsuite/Makefile
config.status: creating config.h
config.status: executing default commands
config.status: creating Makefile
config.status: creating testsuite/Makefile
config.status: creating config.h
config.status: executing default commands
make[2]: Leaving directory '/tmp/gcc-20240514-616152-651wxv/gcc-14.1.0/build'
make[1]: *** [Makefile:25452: stage1-bubble] Error 2
make[1]: Leaving directory '/tmp/gcc-20240514-616152-651wxv/gcc-14.1.0/build'
make: *** [Makefile:1100: all] Error 2

@Moroshima
Copy link

It seems to effect the treesit support (e.g. rust-ts-mode) of emacs.

$ emacs main.rs
Fatal error 11: Segmentation fault
zsh: segmentation fault  emacs main.rs
SEGV $

@redspot

This comment was marked as outdated.

This comment was marked as outdated.

@github-actions github-actions bot added the stale No recent activity label Jun 26, 2024
@redspot

This comment was marked as outdated.

@Bo98
Copy link
Member

Bo98 commented Jun 27, 2024

Issues were fixed in Homebrew 4.2.20, with success reported from multiple users above. It was released fully (after a short period on the developer stream) on 29th of April and there's been little comments here since (treesit report I believe is a separate issue).

I've tested it again under 4.3.7 with this kernel & OS:

Kernel: Linux 4.18.0-544.el8.x86_64 x86_64 GNU/Linux
OS: CentOS Stream release 8

and was successful running:

gh --version
xz --version
m4 --version
gcc --version

whereas it previously failed on 4.2.19.

This seems to be an almost identical kernel to what you are running, so there must be something very specific to that RHEL 8.9 setup that even its sister OS CentOS 8 doesn't exhibit.

@redspot
Copy link
Author

redspot commented Jun 27, 2024

I'll test it tomorrow. Thanks for the update

@redspot
Copy link
Author

redspot commented Jun 30, 2024

Here is the latest info from brew:

$ brew --version
Homebrew 4.3.7-47-g09223c7
Homebrew/homebrew-core (git revision 6bd82412ebd; last commit 2024-06-30)

$ brew outdated
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause formulae that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  moreutils

$ brew config
HOMEBREW_VERSION: 4.3.7-47-g09223c7
ORIGIN: https://github.com/Homebrew/brew
HEAD: 09223c700a35c020f9922b36dce4cf045674467f
Last commit: 16 hours ago
Core tap HEAD: 03e0e5c9f8cfc05d013b93c516c852dea5c47362
Core tap last commit: 2 minutes ago
Core tap JSON: 30 Jun 15:30 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_EDITOR: vim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
SUDO_ASKPASS: /home/wmartin45/bin/askpass-kdewallet
Homebrew Ruby: 3.3.3 => /home/linuxbrew/.linuxbrew/Cellar/ruby/3.3.3/bin/ruby
CPU: octa-core 64-bit skylake
Clang: 18.1.8
Git: 2.45.2 => /home/linuxbrew/.linuxbrew/bin/git
Curl: 7.61.1 => /bin/curl
Kernel: Linux 4.18.0-513.24.1.el8_9.x86_64 x86_64 GNU/Linux
OS: Red Hat Enterprise Linux release 8.10 (Ootpa)
Host glibc: 2.28
/usr/bin/gcc: 8.5.0
/usr/bin/ruby: N/A
glibc: 2.35_1
gcc@11: N/A
gcc: 14.1.0_1
xorg: N/A


$ brew info ssdeep
==> ssdeep: stable 2.14.1 (bottled)
Recursive piecewise hashing tool
https://ssdeep-project.github.io/ssdeep/
Installed
/home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1 (17 files, 232.3KB) *
  Poured from bottle using the formulae.brew.sh API on 2024-06-30 at 14:18:24
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/s/ssdeep.rb
License: GPL-2.0
==> Dependencies
Required: gcc ✔, glibc ✔
==> Analytics
install: 343 (30 days), 968 (90 days), 3,932 (365 days)
install-on-request: 22 (30 days), 73 (90 days), 312 (365 days)
build-error: 0 (30 days)

$ brew info gcc
==> gcc: stable 14.1.0 (bottled), HEAD
GNU compiler collection
https://gcc.gnu.org/
Installed
/home/linuxbrew/.linuxbrew/Cellar/gcc/14.1.0_1 (1,741 files, 500.4MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-06-30 at 14:18:10
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/g/gcc.rb
License: GPL-3.0-or-later with GCC-exception-3.1
==> Dependencies
Required: gmp ✔, isl ✔, libmpc ✔, mpfr ✔, zstd ✔, binutils ✔, zlib ✔, glibc ✔
==> Options
--HEAD
        Install HEAD version
==> Analytics
install: 164,368 (30 days), 440,269 (90 days), 1,374,369 (365 days)
install-on-request: 71,288 (30 days), 175,797 (90 days), 580,481 (365 days)
build-error: 2,168 (30 days)

$ brew info mingw-w64
==> mingw-w64: stable 12.0.0 (bottled)
Minimalist GNU for Windows and GCC cross-compilers
https://sourceforge.net/projects/mingw-w64/
Installed
/home/linuxbrew/.linuxbrew/Cellar/mingw-w64/12.0.0 (8,182 files, 1.4GB) *
  Poured from bottle using the formulae.brew.sh API on 2024-06-30 at 14:31:24
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/m/mingw-w64.rb
License: ZPL-2.1
==> Dependencies
Build: pkg-config ✔, texinfo ✘
Required: gmp ✔, isl ✔, libmpc ✔, mpfr ✔, zstd ✔, zlib ✔, gcc ✔, glibc ✔
==> Analytics
install: 5,542 (30 days), 15,110 (90 days), 65,036 (365 days)
install-on-request: 2,957 (30 days), 8,027 (90 days), 30,314 (365 days)
build-error: 11 (30 days)

Here is a version test for various programs:

$ /home/linuxbrew/.linuxbrew/opt/xz/bin/xz --version | head -n1
xz (XZ Utils) 5.4.6
$ /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version | head -n1
0.29.2
$ /home/linuxbrew/.linuxbrew/opt/go/bin/go version | head -n1
go version go1.22.4 linux/amd64
$ /home/linuxbrew/.linuxbrew/opt/ssdeep/bin/ssdeep -V | head -n1
2.14.1
$ /home/linuxbrew/.linuxbrew/opt/m4/bin/m4 --version | head -n1
m4 (GNU M4) 1.4.19
$ /home/linuxbrew/.linuxbrew/opt/gcc/bin/gcc-14 --version | head -n1
gcc-14 (Homebrew GCC 14.1.0_1) 14.1.0
$ /home/linuxbrew/.linuxbrew/opt/mingw-w64/bin/i686-w64-mingw32-gcc --version | head -n1
i686-w64-mingw32-gcc (GCC) 14.1.0
$ /home/linuxbrew/.linuxbrew/opt/mingw-w64/bin/x86_64-w64-mingw32-gcc --version | head -n1
x86_64-w64-mingw32-gcc (GCC) 14.1.0

Everything seems to work, including gcc-14. However, I tested a little deeper, just to be sure.

$ cat test_arg0.c 
#include <libgen.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
  printf("arg0=:%s:\n", basename(argv[0]));
  return 0;
}

$ x86_64-w64-mingw32-gcc -o test_me_pe64.exe test_arg0.c
$ i686-w64-mingw32-gcc -o test_me_pe32.exe test_arg0.c
$ file test_me_*.exe
test_me_pe32.exe: PE32 executable (console) Intel 80386, for MS Windows
test_me_pe64.exe: PE32+ executable (console) x86-64, for MS Windows
$ w64 test_me_pe64.exe
arg0=:test_me_pe64.exe:
$ w64 test_me_pe32.exe
arg0=:test_me_pe32.exe:
$ w64 --version
wine-7.0 (Staging)

$ ssdeep -b test_me_pe??.exe
ssdeep,1.1--blocksize:hash:hash,filename
1536:iq6I4eQnBSQii7gJ7CBqw4ePV1fZAK+cwzr50Qol6s/c4zfQ4Qe:iqptQ6vsn,"test_me_pe32.exe"
1536:EcKYhtkywmP1/4ggghHio9DsavWhMTQMTzRdD24L3tpo5q6uNNQD3baHlZ5k5mWz:EcKYhuLe1/lPvhVKuEf2K,"test_me_pe64.exe"

So far, ssdeep, x86_64-w64-mingw32-gcc and i686-w64-mingw32-gcc work as expected.

Compiling with gcc-14 seems to fail:

$ $(brew --prefix gcc)/bin/gcc-14 -o test_me.elf test_arg0.c
gcc-14: internal compiler error: Segmentation fault signal terminated program collect2
Please submit a full bug report, with preprocessed source (by using -freport-bug).
See <https://github.com/Homebrew/homebrew-core/issues> for instructions.

$ dmesg | grep Uhuuh | tail -n1
[3245331.812075] 2671516 (collect2): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

$ readelf -l /home/linuxbrew/.linuxbrew/opt/gcc/bin/gcc-14 | grep '0x0000000000401... '
  INTERP         0x00000000000022a0 0x00000000004012a0 0x00000000004012a0
  NOTE           0x00000000000022c8 0x00000000004012c8 0x00000000004012c8
  NOTE           0x00000000000022e8 0x00000000004012e8 0x00000000004012e8
  GNU_PROPERTY   0x00000000000022e8 0x00000000004012e8 0x00000000004012e8
$ readelf -l /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2 | grep '0x0000000000401... '
  INTERP         0x00000000000020f8 0x00000000004010f8 0x00000000004010f8
  NOTE           0x0000000000002120 0x0000000000401120 0x0000000000401120
  NOTE           0x0000000000002140 0x0000000000401140 0x0000000000401140
  GNU_PROPERTY   0x0000000000002140 0x0000000000401140 0x0000000000401140
  LOAD           0x0000000000002f30 0x0000000000401f30 0x0000000000401f30

$ stat /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2
  File: /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2
  Size: 1418656         Blocks: 2776       IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 4199100     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (702600111/wmartin45)   Gid: (702600111/wmartin45)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2024-06-30 10:18:23.483942899 -0400
Modify: 2024-06-30 10:18:14.167873379 -0400
Change: 2024-06-30 10:18:14.173873424 -0400
 Birth: 2024-06-30 10:18:10.545846350 -0400

$ sha256sum /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2 | cut -c 1-64
81cd6a9df6f4fb76eafef8ff3a70d338d93bd7dfd93330d486022dc1e21961a6

Note the extra LOAD section from collect2.
So, I would say that this issue, regarding broken pre-built bottles appears to be mostly fixed.

But, gcc-14 still seems to have a problem. Also, I did not test building anything with go.

Issue not fully fixed yet.

@patjgardner
Copy link

It would have been nice if the fixes in Homebrew 4.2.20 had been mentioned here as this is the issue thread that I'm subscribed to. The absence of any update gave the appearance that this issue was being ignored or would never get any fix.

My own issue #136432 which was closed because of similarity to this has been resolved. 'go' package no longer segfaults.

@Bo98
Copy link
Member

Bo98 commented Jul 1, 2024

While I developed the initial fix, it required a few hops to merge (the patch initially had to go upstream first) and the eventual final release (4.2.20) was made when I was holiday, I apologise for this as I should have remembered to login and close this issue.

It's good to know that the issue is fixed with exception to collect2. I'll look into the collect2 issue and post back soon.

@redspot
Copy link
Author

redspot commented Jul 2, 2024

I really have no idea how to debug which memory mappings are overlapping.

different kernels act differently.

here's 4.18.0-513.24.1.el8_9.x86_64:

$ strace --trace=%memory /home/linuxbrew/.linuxbrew/lib/ld.so /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2 --version
brk(NULL)                               = 0x5555571bd000
mmap(0x3ff000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x3ff000
mmap(0x401000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x401000
mmap(0x403000, 827392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x403000
mmap(0x4cd000, 557056, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xce000) = 0x4cd000
mmap(0x555000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x155000) = 0x555000
mmap(0x55a000, 1512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x55a000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46622c9000
mmap(NULL, 28203, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f46622c2000
mmap(NULL, 921856, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f46621e0000
mmap(0x7f46621ee000, 458752, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f46621ee000
mmap(0x7f466225e000, 401408, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7e000) = 0x7f466225e000
mmap(0x7f46622c0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xdf000) = 0x7f46622c0000
mmap(NULL, 2166320, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4661fcf000
mprotect(0x7f4661ff7000, 1925120, PROT_NONE) = 0
mmap(0x7f4661ff7000, 1466368, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f4661ff7000
mmap(0x7f466215d000, 454656, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18e000) = 0x7f466215d000
mmap(0x7f46621cd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1fd000) = 0x7f46621cd000
mmap(0x7f46621d3000, 52784, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f46621d3000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4661fcd000
mprotect(0x7f46621cd000, 16384, PROT_READ) = 0
mprotect(0x7f46622c0000, 4096, PROT_READ) = 0
mprotect(0x555000, 16384, PROT_READ)    = 0
mprotect(0x7f4662301000, 8192, PROT_READ) = 0
munmap(0x7f46622c2000, 28203)           = 0
brk(NULL)                               = 0x5555571bd000
brk(0x5555571de000)                     = 0x5555571de000
collect2 version 14.1.0
/home/linuxbrew/.linuxbrew/opt/binutils/bin/ld --version
mmap(NULL, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f4661fc4000
munmap(0x7f4661fc4000, 36864)           = 0
GNU ld (GNU Binutils) 2.42
Copyright (C) 2024 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2825911, si_uid=702600111, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++
$ strace --trace=%memory /home/linuxbrew/.linuxbrew/opt/gcc/libexec/gcc/x86_64-pc-linux-gnu/14/collect2 --version--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)

$ dmesg | tail -n1
[3367016.068513] 2825988 (collect2): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

here's 5.15.0-112-generic

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
$ strace --trace=%memory ./collect2.rhel8.segfault --version
brk(NULL)                               = 0x59e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2270251000
mmap(NULL, 6291, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f227024f000
mmap(NULL, 921856, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f227016d000
mmap(0x7f227017b000, 458752, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f227017b000
mmap(0x7f22701eb000, 401408, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7e000) = 0x7f22701eb000
mmap(0x7f227024d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xdf000) = 0x7f227024d000
mmap(NULL, 2166320, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f226ff5c000
mprotect(0x7f226ff84000, 1925120, PROT_NONE) = 0
mmap(0x7f226ff84000, 1466368, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f226ff84000
mmap(0x7f22700ea000, 454656, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18e000) = 0x7f22700ea000
mmap(0x7f227015a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1fd000) = 0x7f227015a000
mmap(0x7f2270160000, 52784, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2270160000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f226ff5a000
mprotect(0x7f227015a000, 16384, PROT_READ) = 0
mprotect(0x7f227024d000, 4096, PROT_READ) = 0
mprotect(0x555000, 16384, PROT_READ)    = 0
mprotect(0x7f2270289000, 8192, PROT_READ) = 0
munmap(0x7f227024f000, 6291)            = 0
brk(NULL)                               = 0x59e000
brk(0x5bf000)                           = 0x5bf000
collect2 version 14.1.0
/home/linuxbrew/.linuxbrew/bin/ld --version
mmap(NULL, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f226ff51000
munmap(0x7f226ff51000, 36864)           = 0
GNU ld (GNU Binutils) 2.40
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=147332, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

On the newer kernel, the fixed mappings are ignored, which is why there's no overlapping memory maps.

this is the relevant kernel commit: torvalds/linux@a4ff8e8

this shows that the flag MAP_FIXED_NOREPLACE is a real thing: https://man7.org/linux/man-pages/man2/mmap.2.html

however, man 2 mmap on the RHEL8 machine has no mention of MAP_FIXED_NOREPLACE.

so, because of https://github.com/torvalds/linux/blob/a4ff8e8620d3f4f50ac4b41e8067b7d395056843/mm/mmap.c#L1376, the program crashes during ELF loading, meaning there's no way to trace what's getting mapped.

The whole point of me investigating this is figure out how to make a test that can check to see if an ELF file would crash due to overlaps on kernel 4.18.0-513.24.1.el8_9.x86_64.

so, I don't know how to look for ELFs that would crash, other than running them, like with collect2.

Sorry, I'm stumped.

@redspot
Copy link
Author

redspot commented Jul 2, 2024

Also, can the stale label be removed from this, since it's not stale?

@ZhongRuoyu ZhongRuoyu added help wanted Task(s) needing PRs from the community or maintainers in progress Stale bot should stay away and removed stale No recent activity labels Jul 2, 2024
@redspot

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Reproducible Homebrew/homebrew-core bug help wanted Task(s) needing PRs from the community or maintainers in progress Stale bot should stay away
Projects
None yet
Development

No branches or pull requests

13 participants