libcxx: undo changes to system directories when deactivating

[ryandesign (Ryan Carsten Schmidt)]

The original purpose of the cxx11 portgroup was so that ports that require C++11 both themselves and in their dependencies can inform the user that they must perform the steps in LibcxxOnOlderSystems and reinstall all ports.

However there are other ports that only require C++11 for themselves, and do not have any C++-using dependencies. mongodb and aria2 are examples. In mongodb it was no problem to just include the cxx11 portgroup and write "configure.cxx_stdlib libc++" into the Portfile since mongodb requires OS X 10.7 and later which includes libc++. However aria2 might work on Snow Leopard, if only libc++ were there.

I would like to modify the cxx11 portgroup to accommodate this use case, by having it check if configure.cxx_stdlib is libc++, and if so, adding a library dependency on port:libcxx on OS X 10.6 and earlier. I understand that the libcxx port needs to install files into OS directories after activation, but it makes me uncomfortable that those files are not removed before deactivation. Could that be done? It seems like it should be possible to list the contents of the tar archive that was expanded on activation, and then delete every item of that list. This assumes that the libcxx port doesn't overwrite any existing OS files; I don't know if that's normally the case. I realize it's not the case if the replacemnt_libcxx variant is used on OS X 10.7 or later. In that case, the port could make a backup of the files it would overwrite, and restore those files on deactivation.

[jeremyhu (Jeremy Huddleston Sequoia)]

The reason that the port is setup this way is to specifically prevent deactivation from removing contents of system directories. What you suggest could be *extremely* dangerous, especially if some bug causes that code to accidentally remove them in the Lion+ case.

libc++ and libc++abi first appeared in Lion, so that is why the ports default to +replacemnt_libcxx on SL and earlier and -replacemnt_libcxx on Lion and later. The port never replaces system files on its own. On Lion+, we tell the users that if they want to replace the system libraries, they should use darwinup to do so, so darwinup can be used to roll back their changes. On SL-, we just update the host ourselves using the same technique (tarball root rather than destroot directly) because of this case:

1. User installs libcxx on Snow Leopard
2. User updates to Lion+
3. User uninstalls all ports in order to update them

We specifically don't want to remove the (now-system-provided) libc++abi.dylib and libc++.dylib

The disk space cost of having 2 dylibs (libc++abi and libc++) that the user might not need any more on their system is far less than the potential risk of things going wrong because we removed the dylibs.

I'd feel more comfortable giving instructions in deactivate than actually doing the work there. Would that be acceptable?

[ryandesign (Ryan Carsten Schmidt)]

Replying to jeremyhu@…:

The reason that the port is setup this way is to specifically prevent deactivation from removing contents of system directories. What you suggest could be *extremely* dangerous, especially if some bug causes that code to accidentally remove them in the Lion+ case.

libc++ and libc++abi first appeared in Lion, so that is why the ports default to +replacemnt_libcxx on SL and earlier and -replacemnt_libcxx on Lion and later. The port never replaces system files on its own. On Lion+, we tell the users that if they want to replace the system libraries, they should use darwinup to do so, so darwinup can be used to roll back their changes. On SL-, we just update the host ourselves using the same technique (tarball root rather than destroot directly) because of this case:

1. User installs libcxx on Snow Leopard
2. User updates to Lion+
3. User uninstalls all ports in order to update them

We specifically don't want to remove the (now-system-provided) libc++abi.dylib and libc++.dylib

That's an excellent point that I had not considered. But wouldn't it be a simple matter of checking the OS version to handle that? The post-activate block in the portfile that untars the tarball is already guarded by a check that os.major < 11. Wouldn't guarding my proposed pre-deactivate block in the same way be sufficient to avoid the scenario you described?

The disk space cost of having 2 dylibs (libc++abi and libc++) that the user might not need any more on their system is far less than the potential risk of things going wrong because we removed the dylibs.

I'm not concerned about the disk space; I'm concerned about unregistered components being left on the user's system which might have an effect on it. For example, software like aria2 that checks for libc++ would find it and would build against it.

I'd feel more comfortable giving instructions in deactivate than actually doing the work there. Would that be acceptable?

I'm looking for a solution that also works on the buildbot buildslaves, where builds happen automatically and nobody is reading instructions. I don't want the installation of one port that declares a dependency on libcxx (as I propose to do for aria2 via a modification to the libcxx portgroup) to cause libc++ to be permanently available on the Snow Leopard build slave and potentially cause other ports built on that build slave to build differently in the future.

Hmm... In fact, this has already occurred, of course. The moment the libcxx port was created, the buildbot built it, which caused libc++ to be installed on the Snow Leopard build slave, and since the port doesn't clean it up afterward, it's still installed. I guess the only ports that would actually have a chance of using it are those that already require a newer clang.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to ryandesign@…:

That's an excellent point that I had not considered. But wouldn't it be a simple matter of checking the OS version to handle that? The post-activate block in the portfile that untars the tarball is already guarded by a check that os.major < 11. Wouldn't guarding my proposed pre-deactivate block in the same way be sufficient to avoid the scenario you described?

Yes, but it just doesn't sit well with me.

The disk space cost of having 2 dylibs (libc++abi and libc++) that the user might not need any more on their system is far less than the potential risk of things going wrong because we removed the dylibs.

I'm not concerned about the disk space; I'm concerned about unregistered components being left on the user's system which might have an effect on it. For example, software like aria2 that checks for libc++ would find it and would build against it.

Shouldn't aria2 be honoring configure.cxx_stdlib?

I'd feel more comfortable giving instructions in deactivate than actually doing the work there. Would that be acceptable?

I'm looking for a solution that also works on the buildbot buildslaves, where builds happen automatically and nobody is reading instructions. I don't want the installation of one port that declares a dependency on libcxx (as I propose to do for aria2 via a modification to the libcxx portgroup) to cause libc++ to be permanently available on the Snow Leopard build slave and potentially cause other ports built on that build slave to build differently in the future.

Wouldn't that then be a bug in those other ports for not honoring configure.cxx_stdlib?

Hmm... In fact, this has already occurred, of course. The moment the libcxx port was created, the buildbot built it, which caused libc++ to be installed on the Snow Leopard build slave, and since the port doesn't clean it up afterward, it's still installed. I guess the only ports that would actually have a chance of using it are those that already require a newer clang.

Right.

I'm not saying no, I'm just nervous about potential problems here as this is a delicate area.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jeremyhu@…:

Replying to ryandesign@…:

I'm not concerned about the disk space; I'm concerned about unregistered components being left on the user's system which might have an effect on it. For example, software like aria2 that checks for libc++ would find it and would build against it.

Shouldn't aria2 be honoring configure.cxx_stdlib?

aria2 requires C++11, so it requires libc++. Its configure script looks for libc++ and uses it if found; otherwise, it displays an error. My intention is to set "configure.cxx_stdlib libc++" to indicate this; to include the cxx11 1.0 portgroup; and to modify the cxx11 1.0 portgroup so that when configure.cxx_stdlib is set to libc++ on darwin 9 or 10, a dependency on port:libcxx is added.

I'm looking for a solution that also works on the buildbot buildslaves, where builds happen automatically and nobody is reading instructions. I don't want the installation of one port that declares a dependency on libcxx (as I propose to do for aria2 via a modification to the libcxx portgroup) to cause libc++ to be permanently available on the Snow Leopard build slave and potentially cause other ports built on that build slave to build differently in the future.

Wouldn't that then be a bug in those other ports for not honoring configure.cxx_stdlib?

Yes, in the same way that it is a bug when ports have undeclared dependencies on optional software, but we developed trace mode to counteract that, and until trace mode can be enabled, the buildbot buildslaves are configured so that all ports are deactivated before any port build is attempted, so that such bugs do not affect the automated builds; the behavior of the libcxx port seems like it would invite problems. For example it might be the case that a build on the Snow Leopard builder would succeed, because the software in question included code in its build system to check for and use libc++ if present; that same port would fail to run on user systems unless they had also installed port:libcxx. Or, for users not using binaries, the port would fail to build.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to ryandesign@…:

Replying to jeremyhu@…:

Replying to ryandesign@…:

I'm not concerned about the disk space; I'm concerned about unregistered components being left on the user's system which might have an effect on it. For example, software like aria2 that checks for libc++ would find it and would build against it.

Shouldn't aria2 be honoring configure.cxx_stdlib?

aria2 requires C++11, so it requires libc++. Its configure script looks for libc++ and uses it if found; otherwise, it displays an error. My intention is to set "configure.cxx_stdlib libc++" to indicate this; to include the cxx11 1.0 portgroup; and to modify the cxx11 1.0 portgroup so that when configure.cxx_stdlib is set to libc++ on darwin 9 or 10, a dependency on port:libcxx is added.

In the llvm ports, we're just doing that unconditionally.

I'm looking for a solution that also works on the buildbot buildslaves, where builds happen automatically and nobody is reading instructions. I don't want the installation of one port that declares a dependency on libcxx (as I propose to do for aria2 via a modification to the libcxx portgroup) to cause libc++ to be permanently available on the Snow Leopard build slave and potentially cause other ports built on that build slave to build differently in the future.

Wouldn't that then be a bug in those other ports for not honoring configure.cxx_stdlib?

Yes, in the same way that it is a bug when ports have undeclared dependencies on optional software, but we developed trace mode to counteract that, and until trace mode can be enabled, the buildbot buildslaves are configured so that all ports are deactivated before any port build is attempted, so that such bugs do not affect the automated builds; the behavior of the libcxx port seems like it would invite problems. For example it might be the case that a build on the Snow Leopard builder would succeed, because the software in question included code in its build system to check for and use libc++ if present; that same port would fail to run on user systems unless they had also installed port:libcxx. Or, for users not using binaries, the port would fail to build.

Yes, I agree that is a problem, I'm just cautious about trying to address it. I'll have to be extra diligent in testing this change.

[ryandesign (Ryan Carsten Schmidt)]

Mojca pointed out that a user might install two copies of MacPorts, install libcxx in each of them, and then later uninstall libcxx in one of them. In that scenario, it's actually desirable that libcxx doesn't clean up after itself after it's uninstalled, otherwise the libc++-requiring ports in the remaining MacPorts installation would be hosed.

[kencu (Ken)]

It would appear, in the end, it is best to leave libc++ installed even if the libcxx port is deactivated, esp now that libc++ is the default on almost all systems now.