codeberg projects (star smake cl-nodgui cdrtools): checksum mismatch

[mrkapqa]

Hello ,

on MacOS Monterey the following error for Cdrtools

--->  Fetching distfiles for cdrtools
--->  Attempting to fetch 2023-09-28.tar.gz from https://codeberg.org/schilytools/schilytools/archive
--->  Verifying checksums for cdrtools                                          
Error: Checksum (rmd160) mismatch for 2023-09-28.tar.gz
Error: Checksum (sha256) mismatch for 2023-09-28.tar.gz
Error: Checksum (size) mismatch for 2023-09-28.tar.gz
Error: Failed to checksum cdrtools: Unable to verify file checksums
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_sysutils_cdrtools/cdrtools/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there
is a bug.
Error: Processing of port cdrtools failed

thanks alot.

[ryandesign (Ryan Carsten Schmidt)]

Analyzing the file currently available from codeberg and the one we mirrored in November, the .tar.gz files differ but the .tar files inside are identical, so their contents are identical too.

gzip files can store the original filename and its timestamp which can lead to gzip files generated at different times differing despite identical contents, but that's not the case here. These .tar.gz files store no original filename or timestamp.

I recompressed the .tar file with various compression levels and discovered that the checksums of the file we mirrored in November match those produced by Apple gzip on macOS 12.7.2 when using compression level 6 (the default) while the checksums of the file available from codeberg now match those produced by GNU gzip when using compression level 6. So I suspect that sometime between November and now, codeberg changed their servers from using BSD gzip to using GNU gzip. This could affect every port that uses automatically-generated tarballs from codeberg. Checking…

% port file all|sort -u|xargs grep 'codeberg\.setup'
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/archivers/star/Portfile:codeberg.setup          schilytools schilytools 2023-09-28
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/audio/soundtouch/Portfile:codeberg.setup      soundtouch soundtouch 2.3.2
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/devel/mz-cmaketools/Portfile:codeberg.setup      emzeat mz-cmaketools c3852c301586c53fed76d9201b8cb62377597650
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/devel/smake/Portfile:codeberg.setup          schilytools schilytools 2023-09-28
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/lisp/cl-nodgui/Portfile:codeberg.setup      cage nodgui 96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/net/newsraft/Portfile:codeberg.setup      newsraft newsraft 0.23 newsraft-
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/net/snac/Portfile:codeberg.setup      grunfink snac2 2.47
/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/cdrtools/Portfile:codeberg.setup          schilytools schilytools 2023-09-28
% sudo port clean --all star soundtouch mz-cmaketools smake cl-nodgui newsraft snac cdrtools
Password:
--->  Cleaning star
--->  Cleaning soundtouch
--->  Cleaning mz-cmaketools
--->  Cleaning smake
--->  Cleaning cl-nodgui
--->  Cleaning newsraft
--->  Cleaning snac
--->  Cleaning cdrtools
% sudo port fetch --no-mirrors star soundtouch mz-cmaketools smake cl-nodgui newsraft snac cdrtools
--->  Fetching distfiles for star
--->  Attempting to fetch 2023-09-28.tar.gz from https://codeberg.org/schilytools/schilytools/archive
--->  Fetching distfiles for soundtouch
--->  Attempting to fetch 2.3.2.tar.gz from https://codeberg.org/soundtouch/soundtouch/archive
--->  Fetching distfiles for mz-cmaketools
--->  Attempting to fetch c3852c301586c53fed76d9201b8cb62377597650.tar.gz from https://codeberg.org/emzeat/mz-cmaketools/archive
--->  Fetching distfiles for smake
--->  Attempting to fetch 2023-09-28.tar.gz from https://codeberg.org/schilytools/schilytools/archive/
--->  Fetching distfiles for cl-nodgui
--->  Attempting to fetch 96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a.tar.gz from https://codeberg.org/cage/nodgui/archive
--->  Fetching distfiles for newsraft
--->  Attempting to fetch newsraft-0.23.tar.gz from https://codeberg.org/newsraft/newsraft/archive
--->  Fetching distfiles for snac
--->  Attempting to fetch 2.47.tar.gz from https://codeberg.org/grunfink/snac2/archive
--->  Fetching distfiles for cdrtools
--->  Attempting to fetch 2023-09-28.tar.gz from https://codeberg.org/schilytools/schilytools/archive
% sudo port -p checksum star soundtouch mz-cmaketools smake cl-nodgui newsraft snac cdrtools
--->  Verifying checksums for star
Error: Checksum (rmd160) mismatch for 2023-09-28.tar.gz
Error: Checksum (sha256) mismatch for 2023-09-28.tar.gz
Error: Checksum (size) mismatch for 2023-09-28.tar.gz
Error: Failed to checksum star: Unable to verify file checksums
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_star/star/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port star failed
--->  Verifying checksums for soundtouch
--->  Verifying checksums for mz-cmaketools
--->  Verifying checksums for smake
Error: Checksum (rmd160) mismatch for 2023-09-28.tar.gz
Error: Checksum (sha256) mismatch for 2023-09-28.tar.gz
Error: Checksum (size) mismatch for 2023-09-28.tar.gz
Error: Failed to checksum smake: Unable to verify file checksums
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_smake/smake/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port smake failed
--->  Verifying checksums for cl-nodgui
Error: Checksum (rmd160) mismatch for 96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a.tar.gz
Error: Checksum (sha256) mismatch for 96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a.tar.gz
Error: Checksum (size) mismatch for 96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a.tar.gz
Error: Failed to checksum cl-nodgui: Unable to verify file checksums
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lisp_cl-nodgui/cl-nodgui/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port cl-nodgui failed
--->  Verifying checksums for newsraft
--->  Verifying checksums for snac
--->  Verifying checksums for cdrtools
Error: Checksum (rmd160) mismatch for 2023-09-28.tar.gz
Error: Checksum (sha256) mismatch for 2023-09-28.tar.gz
Error: Checksum (size) mismatch for 2023-09-28.tar.gz
Error: Failed to checksum cdrtools: Unable to verify file checksums
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_sysutils_cdrtools/cdrtools/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port cdrtools failed
%

I am not sure why soundtouch, mz-cmaketools, newsraft, and snac appear to be unaffected. I hope it is not the case that some of codeberg's servers use GNU gzip and others use BSD gzip and that they are part of a single server pool. If that's what's happening, then we don't know which file we'll get when we fetch which would make codeberg an unsuitable choice for master_sites. We probably have to contact codeberg to find out what's going on.

[ryandesign (Ryan Carsten Schmidt)]

I am not sure why soundtouch, mz-cmaketools, newsraft, and snac appear to be unaffected.

We probably have to contact codeberg to find out what's going on.

Looks like others already did so. Here is the upstream issue:

​https://codeberg.org/Codeberg/Community/issues/1366

according to which:

• archives downloaded before November 12th 2023 (the date we deployed the new Git version which introduced the problem): They have the same checksum as today
• archives downloaded between November 12th 2023 and Jan 12 2024: The checksums might have changed between the initial download and today, because of the initial problem and our attempt to fix it; archives from older commits downloaded during this time might also have changed checksums. But they should be back to the initial checksums now.
• archives downloaded after Jan 12 2024: Our patch has made checksums deterministic depending on the date of the commit.

% for url in $(port distfiles star soundtouch mz-cmaketools smake cl-nodgui newsraft snac cdrtools | grep 'https://distfiles\.macports\.org'); do echo $url; curl -sI $url | grep -i ^last-modified; done
https://distfiles.macports.org/star/2023-09-28.tar.gz
last-modified: Mon, 13 Nov 2023 20:17:55 GMT
https://distfiles.macports.org/soundtouch/2.3.2.tar.gz
last-modified: Sun, 21 May 2023 10:54:01 GMT
https://distfiles.macports.org/mz-cmaketools/c3852c301586c53fed76d9201b8cb62377597650.tar.gz
last-modified: Sat, 23 Dec 2023 16:02:03 GMT
https://distfiles.macports.org/smake/2023-09-28.tar.gz
last-modified: Mon, 13 Nov 2023 19:43:34 GMT
https://distfiles.macports.org/nodgui/96af1af4a0205ea6f7f0af8b6d9da5f180b39d2a.tar.gz
last-modified: Tue, 19 Dec 2023 19:54:41 GMT
https://distfiles.macports.org/newsraft/newsraft-0.23.tar.gz
last-modified: Fri, 02 Feb 2024 15:42:07 GMT
https://distfiles.macports.org/snac2/2.47.tar.gz
last-modified: Mon, 12 Feb 2024 23:01:13 GMT
https://distfiles.macports.org/cdrtools/2023-09-28.tar.gz
last-modified: Mon, 13 Nov 2023 17:14:46 GMT
%

The first bullet point explains why soundtouch is unaffected: we fetched it before November 12 2023.

The third bullet point explains why newsraft and snac2 are unaffected: we fetched them after January 12 2024.

The second bullet point must be the explanation for why star, smake, cl-nodgui, and cdrtools are affected and mz-cmaketools is not: we fetched them between November 12 2023 and January 12 2024.

So we need to fix star, smake, cl-nodgui, and cdrtools either by updating them to new versions or by following the stealth update recipe.

[ryandesign (Ryan Carsten Schmidt)]

It's the same ​issue that GitHub had in early 2023 which ​affected us at the time as well.

[ryandesign (Ryan Carsten Schmidt)]

Robert, Kirill, what's the plan? Do these four ports need updates, or do we handle it as a stealth update, or do we ignore the stealth update by setting master_sites macports_distfiles?

[TonyCrawford]

Bump!
Simple user here, five weeks down the road, trying to get cdrtools.

--->  Verifying checksums for cdrtools
Error: Checksum (rmd160) mismatch for 2023-09-28.tar.gz
Error: Checksum (sha256) mismatch for 2023-09-28.tar.gz
Error: Checksum (size) mismatch for 2023-09-28.tar.gz
Error: Failed to checksum cdrtools: Unable to verify file checksums

[ryandesign (Ryan Carsten Schmidt)]

Until this is solved, you can work around the problem by downloading the right file from e.g. ​https://distfiles.macports.org/cdrtools/ and putting it where MacPorts expects it to be.

[RobK88]

In the next few days, I will fix the ports.

@Ryan - I do not think it will make any difference which way we go. But I prefer the stealth update approach.

[ryandesign (Ryan Carsten Schmidt)]

The stealth update recipe is intended for the situation where upstream has re-released a tarball containing new contents, and we have determined that those changes are good and we would like to mirror those new files and push those changes out to users.

In this case, the tarball has changed, but only due to infrastructure changes at codeberg. The contents of the archives have not changed so there is no value in mirroring the new files and pushing the (nonexistent) changes out to users.

For each of the affected ports, if there is a new version or commit available that we want to update to, we should update as usual. If there is no new version or commit that we want to update to, setting master_sites macports_distfiles is the most straightforward way to prevent users from getting the new files that have the mismatching checksums.

[RobK88]

@ryandesign -- No problem. I will use master_sites macports_distfiles in the portfile instead. I plan to work on it this weekend.

[RobK88]

@ryandesign - I did not need to use master_sites macports_distfiles afterall since there was an update to the source for smake, star, and cdrtools.

I have submitted Pull Requests (PRs):

smake: ​https://github.com/macports/macports-ports/pull/23652

star: ​https://github.com/macports/macports-ports/pull/23653

cdrtools: ​https://github.com/macports/macports-ports/pull/23654

smake passed all the tests.

star and cdrtools passed all the tests except for macOS-14.

It looks like the dependencies for star and cdrtools are not properly installed on macOS-14. I do not have a Mac running macOS-14. I would appreciate it if someone could try to build star and cdrtools on a Mac running macOS-14 and let me know what they see.

I also tried to update cl-nodgui since new updates from the developer are available. But I could not build the most recent version of cl-nodgui. I see thse errors:

40: (SB-IMPL::TOPLEVEL-INIT)
41: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
42: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
43: (SB-IMPL::%START-LISP)

unhandled condition in --disable-debugger mode, quitting
Command failed: /opt/local/bin/sbcl --no-sysinit --no-userinit --non-interactive --eval '(require "asdf")' --eval '(setf asdf:*central-registry* (list* (quote *default-pathname-defaults*) #p"/opt/local/var/macports/build/_Users_grinch_Development_MacPorts_local-repo_ports_lisp_cl-nodgui/cl-nodgui/work/build/system/" #p"/opt/local/share/common-lisp/system/" asdf:*central-registry*))' --eval '(asdf:operate (quote asdf:build-op) (quote nodgui))' 2>&1
Exit code: 1
Error: Failed to build cl-nodgui: asdf:build-op cannot be executed
Error: See /opt/local/var/macports/logs/_Users_grinch_Development_MacPorts_local-repo_ports_lisp_cl-nodgui/cl-nodgui/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port cl-nodgui failed

I am not a LISP programmer so I cannot easily fix cl-nodgui. I suspect there is a missing dependency.

@catap -- any ideas?

[catap (Kirill A. Korinsky)]

RobK88, if you scroll up to the actual issue, I may say something. Right now the error is truncated and here only the tail.

[RobK88]

@catap -- I will attach main.log for cl-nodgui

[RobK88]

main.log for cl-nodgui

[RobK88]

@catap -- I will also attach my Portfile for cl-nodgui

[RobK88]

Updated Portfile for cl-nodgui

[catap (Kirill A. Korinsky)]

1773	:info:build Unhandled ASDF/FIND-COMPONENT:MISSING-DEPENDENCY in thread #<SB-THREAD:THREAD "main thread" RUNNING
1772	:info:build                                                               {1001AC80A3}>:
1773	:info:build   Component :ZPNG not found, required by #<SYSTEM "nodgui">
1774	:info:build Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1001AC80A3}>
1775	:info:build 0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK Component :ZPNG not found, required by #<SYSTEM "nodgui"> #<unused argument> :QUIT T)
1776	:info:build 1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* Component :ZPNG not found, required by #<SYSTEM "nodgui">)

You need to add cl-zpng into dependencies

[RobK88]

In 241a6dfa6e78f28f7a1ca4126f611a6fafeea063/macports-ports (master):

smake: update to 1.7-2024-03-21

• Update codeberg.setup to version 2024-03-21
• Update checksums

See: #69395

[RobK88]

@catap - Thanks for identifying the missing dependency. I added it. But the build still fails. See attached main.log

It looks like we are missing the sdl2 component. So I tried adding cl-sdl2 as a dependency but cl-sdl2 does not exist.

Any ideas?

[RobK88]

Second attempt to build cl-nodgui after adding cl-zpng

[catap (Kirill A. Korinsky)]

You need a create it :)

[RobK88]

@catap - Thanks. Well I am not a LISP programmer. :-(

I hope someone can create cl-sdl2 port. Alternatively, we can use an earlier version of cl-nodgui that does not use sdl2.

[RobK88]

FYI -- @ryandesign pointed out that it looks like Apple forgot to add yacc in CLT 15.3 used in macOS-14. So I added bison as a build dependency for star and cdrtools. Now star and cdtools are building fine on macOS -14 (as well as on macOS 11 to macOS-13) according to the buildbots.

star and cdrtools are ready to be merged.

[RobK88]

In 8678e588b55ccf5f4538cac278c42b0abf40b0a2/macports-ports (master):

cdrtools: update to 3.02-2024-03-21

• Update codeberg.setup to version 2024-03-21
• Update checksums
• Add bison as a build dependency

See: #69395

[RobK88]

In cf4a24068f6bceb75f8773102df9c2a996bedfb1/macports-ports (master):

star: update to 1.7.0-2024-03-21

• Update codeberg.setup to version 2024-03-21
• Update checksums
• Add bison as a build dependency

See: #69395

[RobK88]

I see that there is a common LISP library for sdl2 called cl-sdl2.

See ​https://github.com/lispgames/cl-sdl2

When I have time in a couple of weeks, I will create a port for it and then add it as a dependency to cl-nodgui

If someone wants to do it sooner, please go ahead!