Boost: update to 1.76.0

[michaelld (Michael Dickens)]

Tracking ticket to cover various improvements/refactoring for our Boost ports, once the following PR has been merged:

​PR 11185 - Add boost171 and boost176 ports, with isolated builds, and a new boost PG to handle configuration

[michaelld (Michael Dickens)]

​https://github.com/macports/macports-ports/pull/5996

[michaelld (Michael Dickens)]

I've closed the WIP PR for 1.72.0 because it was stale. I'll do a local update / test to work on compatibility and if things aren't horrible then I'll create another PR. Hopefully this Boost is more project-compatible than the prior 2 versions, and so we can get it in place more quickly.

[mascguy (Christopher Nielsen)]

Any progress/updates on this?

[mascguy (Christopher Nielsen)]

And is there anything I can do to help?

[michaelld (Michael Dickens)]

Thanks for the prod, @mascguy!

Boost is a challenging port to get updated. The API changes regularly which breaks backwards compatibility. It takes time even for actively-in-development ports to get working with these releases; and older ports that are not actively in development will just fail eventually because the Boost API is so different. We can't just update the "boost" port because it breaks other ports; reliably; potentially for weeks if not months until fixes are in place. Some ports will of course work without modification; but many will not. Some of those ports are not trivially updated to work with newer Boost, whether by us directly or via upstream fixes. These reasons are why Boost is stuck where it is for such a long time. Prior PRs have failed because too many ports could not be updated, and it would take too much aggregate time to do all of this work -- and so some of the ports updated in the PR get stale & need to be rebased, which takes even more time ... just not enough time in the day / week / month to get all of this done!

The best solution I've been able to come up with -- and, this is imperfect -- is to use the current "boost_169" port or create some more recent version such as "boost_172" as the dependency instead of "boost" on -all- dependent ports. Then, we can update the boost port to the current release without worrying about breaking ports that use it. Ports that are able to use the "boost" current release port can change the dependency to allow for either of those Boost ports; but, at least one version of Boost should be tested to always work ... which is why this solution is imperfect! What happens if some future port that uses Boost can't work with 1.69 or the current release? I guess we'll need intermediate ports such as "boost_174" etc ... which so long as they can be built and installed in parallel could be made to work. Not a perfect solution, but at least the vast majority of ports that depend on Boost would work regardless of the Boost release port version and compatibility.

[mascguy (Christopher Nielsen)]

Replying to michaelld:

The best solution I've been able to come up with -- and, this is imperfect -- is to use the current "boost_169" port or create some more recent version such as "boost_172" as the dependency instead of "boost" on -all- dependent ports. [...]

Makes sense Michael, thanks for the info!

Given the lack of backward-compatibility between boost releases, multiple version-specific ports make perfect sense. That's arguably better than being limited to a single, older release.

And please let me know if there's anything I can do to assist, particularly building and testing of dependent ports.

[reneeotten (Renee Otten)]

any progress on this (right now upstream is at version 1.76)? I was looking at a build failure for a port and see messages like

/opt/local/include/boost/smart_ptr/detail/sp_counted_base_clang.hpp:29:9: error: '_Atomic' is a C11 extension [-Werror,-Wc11-extensions]
typedef _Atomic( boost::int_least32_t ) atomic_int_least32_t;

that seems to be resolved by ​this commit and is, therefore, likely available for version after 1.72.

Is the plan to keep boost following the latest version and add boostXYZ to keep older ones around? To maintain the status quo in MacPorts one would need to rename the current boost port to boost172, update all ports that depend on it to that new name, and then update boost to the latest version - correct? Maintainers can then check whether "their" port builds with boost 1.76 and, if so, change the dependency. Is there at all a reason to generate ports for intermediate versions?

[michaelld (Michael Dickens)]

Yes that's still my hope ... not sure when I'll get to this, but it would be good to get it done! The first step would be to create the boost171 port & move all current ports to use it instead of boost. Then we could start adding in boostXYZ & boost (latest release) ports, which would be standalone & we could then update any port that depends on Boost to whatever boostXYZ it works with. Eventually we'd want to deprecate older boostXYZ. This will be a bit of a pain to maintain for older OSX / macOS, but it provides the most robust solution that allows us to easily update boost for development purposes.

[mascguy (Christopher Nielsen)]

Michael, I'll take a look at 1.76.0, and try to determine what patches are still needed.

[mascguy (Christopher Nielsen)]

This is my work-in-progress branch. Ultimately I'd need to add a separate segregated subport for 1.76.0, but one thing at a time. Lol

​https://github.com/macports/macports-ports/compare/master...mascguy:mascguy-boost-upgrade

[kencu (Ken)]

as you can see, updating boost itself is trivially easy.

Building and testing 12,768 ports that use boost and fixing all the issues that have arisen from the update: not so much.

[mascguy (Christopher Nielsen)]

The new version will be wholly segregated, allowing it to be installed side-by-side with our current version. It'll require a bit of work, but no big deal.

That approach will allow us to support as many versions as desired, with no conflicts. And ports can depend on whichever version they need.

[kencu (Ken)]

Yep, that is indeed the major part of the effort that needs to be done here.

[kencu (Ken)]

I don't relish the day I see coming where I might need four different boost versions installed to support different ports that may or may not be all compatible with the latest version, but haven't been moved along.

As Michael says, an ideal solution does not present itself just yet.

[mascguy (Christopher Nielsen)]

Replying to kencu:

I don't relish the day I see coming where I might need four different boost versions installed to support different ports that may or may not be all compatible with the latest version, but haven't been moved along.

As Michael says, an ideal solution does not present itself just yet.

So long as the required set of patches is manageable, I don't see an issue with supporting a few different versions. Particularly if there are at least two maintainers, which there will be soon.

[michaelld (Michael Dickens)]

Thanks for working on this @macsguy! I have no issues sharing maintainership of Boost and very likely a bunch of Boost versions. As noted: So long as they all install in parallel and we can coerce dependent ports to use the correct one, then this setup can be made to work. Very much like the Qt ports. Folks have complained over the years about the lack of default, but thus far we've been able to make everything work well enough & that's the bottom line :)

[mascguy (Christopher Nielsen)]

Sounds good. As part of the effort, I'd also like to migrate away from python variants, to separate subports. This has worked well for other cases, like our OpenCV ports, and allow users to have Boost installed for multiple Python versions simultaneously.

How does that sound? Your thoughts?

[michaelld (Michael Dickens)]

Yes please! I've wanted completely separate Boost::PythonXY installs like forever ... just never gotten around to dealing with it! If we're making port:boostXYZ installable in parallel, then I'd hope we can do the same for Boost::PythonXY!

[mascguy (Christopher Nielsen)]

The addition of a boost 1.76 port is covered by the following open PR, courtesy of @CJones. We hope to merge it within the next 48 hours:

​PR 11185 - Add boost171 and boost176 ports, with isolated builds, and a new boost PG to handle configuration

Let's leave this ticket open though, to track follow-on enhancements (particularly migration to Python subports).

[mascguy (Christopher Nielsen)]

Another good idea, suggested by @RJVB: Consider adding a dev variant, requesting headers be installed. When not requested, would eliminate installation of the many thousands of headers per Boost port installation.