Boost: refactor and enhance

[michaelld (Michael Dickens)]

Tracking ticket to cover various improvements/refactoring for our Boost ports.

[michaelld (Michael Dickens)]

​https://github.com/macports/macports-ports/pull/5996

[michaelld (Michael Dickens)]

I've closed the WIP PR for 1.72.0 because it was stale. I'll do a local update / test to work on compatibility and if things aren't horrible then I'll create another PR. Hopefully this Boost is more project-compatible than the prior 2 versions, and so we can get it in place more quickly.

[mascguy (Christopher Nielsen)]

Any progress/updates on this?

[mascguy (Christopher Nielsen)]

And is there anything I can do to help?

[michaelld (Michael Dickens)]

Thanks for the prod, @mascguy!

Boost is a challenging port to get updated. The API changes regularly which breaks backwards compatibility. It takes time even for actively-in-development ports to get working with these releases; and older ports that are not actively in development will just fail eventually because the Boost API is so different. We can't just update the "boost" port because it breaks other ports; reliably; potentially for weeks if not months until fixes are in place. Some ports will of course work without modification; but many will not. Some of those ports are not trivially updated to work with newer Boost, whether by us directly or via upstream fixes. These reasons are why Boost is stuck where it is for such a long time. Prior PRs have failed because too many ports could not be updated, and it would take too much aggregate time to do all of this work -- and so some of the ports updated in the PR get stale & need to be rebased, which takes even more time ... just not enough time in the day / week / month to get all of this done!

The best solution I've been able to come up with -- and, this is imperfect -- is to use the current "boost_169" port or create some more recent version such as "boost_172" as the dependency instead of "boost" on -all- dependent ports. Then, we can update the boost port to the current release without worrying about breaking ports that use it. Ports that are able to use the "boost" current release port can change the dependency to allow for either of those Boost ports; but, at least one version of Boost should be tested to always work ... which is why this solution is imperfect! What happens if some future port that uses Boost can't work with 1.69 or the current release? I guess we'll need intermediate ports such as "boost_174" etc ... which so long as they can be built and installed in parallel could be made to work. Not a perfect solution, but at least the vast majority of ports that depend on Boost would work regardless of the Boost release port version and compatibility.

[mascguy (Christopher Nielsen)]

Replying to michaelld:

The best solution I've been able to come up with -- and, this is imperfect -- is to use the current "boost_169" port or create some more recent version such as "boost_172" as the dependency instead of "boost" on -all- dependent ports. [...]

Makes sense Michael, thanks for the info!

Given the lack of backward-compatibility between boost releases, multiple version-specific ports make perfect sense. That's arguably better than being limited to a single, older release.

And please let me know if there's anything I can do to assist, particularly building and testing of dependent ports.

[reneeotten (Renee Otten)]

any progress on this (right now upstream is at version 1.76)? I was looking at a build failure for a port and see messages like

/opt/local/include/boost/smart_ptr/detail/sp_counted_base_clang.hpp:29:9: error: '_Atomic' is a C11 extension [-Werror,-Wc11-extensions]
typedef _Atomic( boost::int_least32_t ) atomic_int_least32_t;

that seems to be resolved by ​this commit and is, therefore, likely available for version after 1.72.

Is the plan to keep boost following the latest version and add boostXYZ to keep older ones around? To maintain the status quo in MacPorts one would need to rename the current boost port to boost172, update all ports that depend on it to that new name, and then update boost to the latest version - correct? Maintainers can then check whether "their" port builds with boost 1.76 and, if so, change the dependency. Is there at all a reason to generate ports for intermediate versions?

[michaelld (Michael Dickens)]

Yes that's still my hope ... not sure when I'll get to this, but it would be good to get it done! The first step would be to create the boost171 port & move all current ports to use it instead of boost. Then we could start adding in boostXYZ & boost (latest release) ports, which would be standalone & we could then update any port that depends on Boost to whatever boostXYZ it works with. Eventually we'd want to deprecate older boostXYZ. This will be a bit of a pain to maintain for older OSX / macOS, but it provides the most robust solution that allows us to easily update boost for development purposes.

[mascguy (Christopher Nielsen)]

Michael, I'll take a look at 1.76.0, and try to determine what patches are still needed.

[mascguy (Christopher Nielsen)]

This is my work-in-progress branch. Ultimately I'd need to add a separate segregated subport for 1.76.0, but one thing at a time. Lol

​https://github.com/macports/macports-ports/compare/master...mascguy:mascguy-boost-upgrade

[kencu (Ken)]

as you can see, updating boost itself is trivially easy.

Building and testing 12,768 ports that use boost and fixing all the issues that have arisen from the update: not so much.

[mascguy (Christopher Nielsen)]

The new version will be wholly segregated, allowing it to be installed side-by-side with our current version. It'll require a bit of work, but no big deal.

That approach will allow us to support as many versions as desired, with no conflicts. And ports can depend on whichever version they need.

[kencu (Ken)]

Yep, that is indeed the major part of the effort that needs to be done here.

[kencu (Ken)]

I don't relish the day I see coming where I might need four different boost versions installed to support different ports that may or may not be all compatible with the latest version, but haven't been moved along.

As Michael says, an ideal solution does not present itself just yet.

[mascguy (Christopher Nielsen)]

Replying to kencu:

I don't relish the day I see coming where I might need four different boost versions installed to support different ports that may or may not be all compatible with the latest version, but haven't been moved along.

As Michael says, an ideal solution does not present itself just yet.

So long as the required set of patches is manageable, I don't see an issue with supporting a few different versions. Particularly if there are at least two maintainers, which there will be soon.

[michaelld (Michael Dickens)]

Thanks for working on this @macsguy! I have no issues sharing maintainership of Boost and very likely a bunch of Boost versions. As noted: So long as they all install in parallel and we can coerce dependent ports to use the correct one, then this setup can be made to work. Very much like the Qt ports. Folks have complained over the years about the lack of default, but thus far we've been able to make everything work well enough & that's the bottom line :)

[mascguy (Christopher Nielsen)]

Sounds good. As part of the effort, I'd also like to migrate away from python variants, to separate subports. This has worked well for other cases, like our OpenCV ports, and allow users to have Boost installed for multiple Python versions simultaneously.

How does that sound? Your thoughts?

[michaelld (Michael Dickens)]

Yes please! I've wanted completely separate Boost::PythonXY installs like forever ... just never gotten around to dealing with it! If we're making port:boostXYZ installable in parallel, then I'd hope we can do the same for Boost::PythonXY!

[mascguy (Christopher Nielsen)]

The addition of a boost 1.76 port is covered by the following open PR, courtesy of @CJones. We hope to merge it within the next 48 hours:

​PR 11185 - Add boost171 and boost176 ports, with isolated builds, and a new boost PG to handle configuration

Let's leave this ticket open though, to track follow-on enhancements (particularly migration to Python subports).

[mascguy (Christopher Nielsen)]

Another good idea, suggested by @RJVB: Consider adding a dev variant, requesting headers be installed. When not requested, would eliminate installation of the many thousands of headers per Boost port installation.

[RJVB (René Bertin)]

Replying to mascguy:

Another good idea, suggested by @RJVB: Consider adding a dev variant, requesting headers be installed. When not requested, would eliminate installation of the many thousands of headers per Boost port installation.

A variant, or a subport which would have some (theoretical) advantages for those who need the headers occasionally.

[RJVB (René Bertin)]

Picking up on some criticism of the above idea, on PR 11185:

On Tuesday June 01 2021 06:00:23 Ken wrote:

boost needs all it's headers installed. In many cases the headers are the only thing needed. If people take a moment, they can see if only headers are used in a given port, and if so, list boost as a build dep only.

But installing boost without the headers would be very problematic.

Creating a whole new system of boost lib-only installations is way too confusing.

I think you and Chris are making an error one should never make: under (and over!) estimating users.

Think of what port this is, and who is likely to install it, why, and how. 1 the majority of people will probably inherit it as a dependency of a binary package. They probably couldn't care less what's in the port, aside (maybe) from how much disk space it takes 2 the next biggest group probably consists of people who install it because of a build dependency. They too probably don't care what's in it, as long as their builds succeed. Here the boost PG comes in: any port that declares a dependency on boost can automatically add a depends_build on the -dev subport in addition to the (optional) depends_lib on the main port, through the PG. Here a -dev *subport* (instead of a variant) probably makes things easier. 3 and then there will be people who use boost in their own software, i.e. developers. More likely than not they'll have at least some experience with Linux (and -dev packages), but above all I assume they are savy enough to realise that they have to install more than just the main port (if they simply haven't already, or after they discover that the main port installs just the shared libraries).

if we were ever going to get into that on MP, I guess it would have happened by now...too big of a project to ever undertake now, though, IMHO.

I see 2 reasons to spring for -dev ports : 1) size of the developer content and 2) making build conflicts a lot easier. About that: the 1st ports in which I used my devport PG in my MacPorts adaptation for Linux were gettext and openssl. They often caused conflict, gettext notably because a number of the functions in its libintl are also provided in libc but not all symbols that get referenced when you use gettext headers (or something like that). I used conflicts_build for a while, but these ports are ubiquitous enough that deactivating them makes a whole lot of applications unusable. Not such problem with a -dev port - and forgetting to reactivate it will only break subsequent builds (and even then only if the port isn't listed as a build dependency).

I have never felt the need to use this approach in every single one of my own ports, and I agree that the endeavour to apply it throughout the entire ports tree would be an impossible one. But that doesn't mean one cannot use it where it's justified - and in this instance it would be easy. Ports that decide to use the new boost ports and PG will have to make some changes anyway, but in this case that wouldn't even be required (see point 2) above).

PS: I do use a -dev port with my personal port:acl, again because its headers can cause problems (mask system headers IIRC) but disabling the entire port would render lots of my KF5 applications inoperable. But I remember having run into a few other ports where I would have appreciated this level of control.

[michaelld (Michael Dickens)]

Just merged ​the PR #11185 ... lovely piece of work (wishing I'd thought of it myself a long time ago!] ... installing Boost169 and Boost176 now & will start testing those against the primary ports I maintain: mostly GR and UHD related. I strongly doubt I'll have time to catch up with all of the comments in this issue ... hopefully anything critical has been incorporated into PRs!

[mascguy (Christopher Nielsen)]

Reconciled boost to be as consistent with boost171 as possible:

​https://github.com/macports/macports-ports/commit/c603d5c8548a56ef200071f43a09e696cdbec8a2

Ditto for boost169:

​https://github.com/macports/macports-ports/commit/3b9a3cd96cd56a29538be7a7baebcde3c7bdca9d

This is a first step toward migrating shared logic to a portgroup.

[mascguy (Christopher Nielsen)]

Replying to RJVB:

I see 2 reasons to spring for -dev ports : 1) size of the developer content and 2) making build conflicts a lot easier.

I have never felt the need to use this approach in every single one of my own ports, and I agree that the endeavour to apply it throughout the entire ports tree would be an impossible one. But that doesn't mean one cannot use it where it's justified - and in this instance it would be easy. Ports that decide to use the new boost ports and PG will have to make some changes anyway, but in this case that wouldn't even be required (see point 2) above).

Bear in mind that, a few short weeks ago, the boost ports installed 38,000-ish files each.

Since the recent introduction of our new docs variant - disabled by default - that's been reduced to 14,000-ish. That's almost a 3x improvement in file count! And it makes a huge difference, particularly when multiple versions are installed side-by-side.

So we need to keep that in perspective. And while it doesn't provide quite as much benefit as eliminating the headers entirely [via Dev ports], the improvement is still quite significant.

Nonetheless, if you still think it's worth pursuing the Dev port approach - despite the recent reduction in file count and storage footprint - I'm willing to pursue it.

Your thoughts?

[cjones051073 (Chris Jones)]

Replying to mascguy:

Replying to RJVB:

I see 2 reasons to spring for -dev ports : 1) size of the developer content and 2) making build conflicts a lot easier.

I have never felt the need to use this approach in every single one of my own ports, and I agree that the endeavour to apply it throughout the entire ports tree would be an impossible one. But that doesn't mean one cannot use it where it's justified - and in this instance it would be easy. Ports that decide to use the new boost ports and PG will have to make some changes anyway, but in this case that wouldn't even be required (see point 2) above).

Bear in mind that, a few short weeks ago, the boost ports installed 38,000-ish files each.

Since the recent introduction of our new docs variant - disabled by default - that's been reduced to 14,000-ish. That's almost a 3x improvement in file count! And it makes a huge difference, particularly when multiple versions are installed side-by-side.

So we need to keep that in perspective. And while it doesn't provide quite as much benefit as eliminating the headers entirely [via Dev ports], the improvement is significant.

Nonetheless, if you still think it's worth pursuing the Dev port approach - despite the recent reduction in file count and storage footprint - I'm willing to pursue it.

Your thoughts?

For me, its not worth it. Keep as single boost ports.

[RJVB (René Bertin)]

Replying to mascguy:

Nonetheless, if you still think it's worth pursuing the Dev port approach - despite the recent reduction in file count and storage footprint - I'm willing to pursue it.

Definitely. 14k'ish of files is still unreasonably huge in place of the 36 files only the port could have for the probably vast majority of users. Times the number of copies people will now start getting installed. I've already exposed my views on the very limited maintenance penalty of my suggested solution, and on the unlikelyhood that it will lead to user confusion.

So yeah, I am willing to contribute on this. Definitely no hurries though; let's give the boost PG a chance to stabilise first.

[eborisch (Eric A. Borisch)]

I'm late to the party; do we know what is preventing moving the base boost to 1.72.0?

FWIW, FreeBSD's Ports tree (also started by JKH, one of the MP founders) uses just one version (1.72.0) currently.

There might be less overall friction if we can follow in their wake. They split boost out into:

• boost-all (50)
• boost-docs (53)
• boost-jam (439)
• boost-libs (1,230)
• boost-python-libs (8)
• boost_build (5)

The (numbers) are from bsdstats.org, their opt-in usage tracking, so you can see the relative installs.

[cjones051073 (Chris Jones)]

Changing the version of boost the old port provides requires testing every port using it, which is a herculean task no one really wants to take on. The approach taken is that with the new isolated boost ports, and the new boost PG to help configure ports to use the isolated builds. This way ports can be migrated one at a time to one of the boost versions currently offered, 1.69 1.71 or 1.76. If there is a pressing reason for more versions they of course could be added.

[mascguy (Christopher Nielsen)]

Agreed, we don't want to change port boost. That's now considered legacy, and will be retired once everything has been migrated off of it.

But we are considering further breakout of Boost-related functionality:

• Docs have already been taken care of, via new variant 'docs'.
• Per Rene's suggestion, we're hoping to separate headers, to further reduce the footprint.

Any and all thoughts/suggestions welcome though!

[mascguy (Christopher Nielsen)]

FYI, I've started the refactoring effort for our Boost ports, with the goal of sharing as much logic as possible via a new portgroup. (As well as migrating to a containing port and subports.)

It's mostly a skeleton at this point, but hope to make progress over the next week or two:

​https://github.com/macports/macports-ports/compare/master...mascguy:mascguy-boost

And a big thanks to @CJones, who simplified our legacy Boost port via a shim. That's one less item to tackle, and it's a beautiful (and elegant) approach:

​https://github.com/macports/macports-ports/commit/fdbf9a0f4418fc8bfa119f8bbfab196fbd175169

[RJVB (René Bertin)]

Starting (finally!) to migrate to the new boost port infrastructure. 2 things I noticed:

• dist_subdir could have been used to store all distfiles in a single directory
• instead of a cmake_scripts variant, they could have been provided through a subport. That would have made the experimenting easier, and make things possible like adding a build conflict for dependents where these scripts should NOT be used.