Significantly reduce MacPorts base install size by removing base.tar and compressing ports.tar

[esbugz]

While comparing MacPorts to Homebrew I've noticed that in some respect it's even worse in its installation size despite having an architectural advantage of not relying on a repo with a history (though Homebrew is currently even better in allowing skipping the repos altogether with HOMEBREW_INSTALL_FROM_API)

The base.tar and ports.tar files are

• ~215m uncompressed (~113m base and 101m ports) and only
• ~35m compressed (~15 and ~21 respectively), which is a 80% savings!!!

So here is an easy win — just compress the files instead of downloading the uncompressed versions!

But then what's even worse is that these files remain even after being unpacked, which is another source of wasted space

1. Base.tar should just be removed after being unpacked. And the unpacked version has way too many large tests and idoc folders to be pushed to every single MacPorts install

2. And the better way to deal with ports.tar is: instead of having a gazillion of unpacked text files (with some taking as much as 1m per port!) that you'll mostly never need (because you'd install only a handful of ports) occupying ~170m to use a few compressed database files of ~21m or at least a single compressed file per port (just like Yarn package manager does with its modern PnP ZIP APIs) and then read those compressed files having them unpacked on a hard drive

Or maybe there is a way to just go the HOMEBREW_INSTALL_FROM_API way and read the port manifests the same way you install the apps by downloading the file from ​https://packages.macports.org/app_name

Then ideally the whole MacPorts install could be a couple of dozen Mbs instead of taking half a Gigabyte!

[ryandesign (Ryan Carsten Schmidt)]

The base.tar and ports.tar files are kept on disk, and are downloaded as uncompressed files instead of compressed ones, so that rsync can produce a small diff of them so that only a small amount of network traffic is needed during updates.

Admittedly, this design decision was made decades ago when base and ports were both much smaller.

I understand where your suggestions are coming from but they represent several significant changes to how MacPorts base works. If you would like to work on one or more of these changes, I suggest you discuss them in detail on the macports-dev mailing list and gain consensus on the method of doing so before beginning to write any code. That way you can avoid unnecessary work.

[jmroot (Joshua Root)]

Overlaps with #38265.

[esbugz]

Replying to ryandesign:

The base.tar and ports.tar files are kept on disk, and are downloaded as uncompressed files instead of compressed ones, so that rsync can produce a small diff of them so that only a small amount of network traffic is needed during updates.

Sure, but there are much better ways of achieving this while also not requiring the full duplicate file to be present, you could get just the currently installed base version from the user and then send him the compressed diffs, which will be removed after having been applied to the installed base to update it to the latest version

Or for ports you might only need a small index with the latest versions

Admittedly, this design decision was made decades ago when base and ports were both much smaller.

Yeah, unfortunately, the duplicately :) unsound decision persisted for way too long :(

I understand where your suggestions are coming from but they represent several significant changes to how MacPorts base works.

Sure, while also representing a significant improvement in how it works :)

If you would like to work on one or more of these changes, I suggest you discuss them in detail on the macports-dev mailing list and gain consensus on the method of doing so before beginning to write any code. That way you can avoid unnecessary work.

Sorry wouldn't be of much help here, don't know any Tcl or much of C...

Meanwhile, at least pairing down the vendored Tcl version to the bare minimum required to run MacPorts shouldn't need any major changes, right?

[kencu (Ken)]

I think the entire base TCL installation is something like 14MB

% du -sh tcl*
292K	tcl8
2.5M	tcl8.6
8.0K	tclConfig.sh
 11M	tcllib1.21
4.0K	tclooConfig.sh
276K	tclx8.6

so not much of a burden (size of three photos), although the source code for it is about 50MB, if you have to download that for a new release.

Figuring out how to scrape that source code down by modifying the upstream build is not something I could see being maintainable over the long term. Might even break the license, have to check...

[esbugz]

Replying to kencu:

I think the entire base TCL installation is something like 14MB

% du -sh tcl*

Why would you ignore all the other junk that doesn't have tcl* prefix??? The whole base/vendor folder is ~125m

• 52m tcl8.6.12
  • ~10m of which a very helpful tests/doc/changelogs, so almost your "entire base"
• ~70m is tcllib-1.21 with similarly useful tests and docs in the modules taking most(?) of the space (7/9m for pt is tests)