MacPorts should prevent overlinking due to .la files

[jeremyhu (Jeremy Huddleston Sequoia)]

base should delete ${prefix}/lib/*.la in our standard destroot phases, and we should have a verification phase which asserts that no ${prefix}/lib/*.la files are installed.

These .la files just serve to screw up builds by adding unnecessary links. They are only useful in legacy situations or when building static executables. They have no place on a modern OS.

[larryv (Lawrence Velázquez)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Here's the most recent thread that lead to this idea: ​https://lists.macosforge.org/pipermail/macports-dev/2013-February/021943.html

I think if we have a post-destroot-type thing in base that deletes .la files, then that should suffice; I don't see a need for a separate verification step. If we delete the files correctly, why would verification ever fail?

Why not delete all .la files everywhere in ${destroot}, even if they're not in ${destroot}${prefix}/lib?

[jmroot (Joshua Root)]

They're apparently used by libltdl in some situations, so always deleting them isn't correct in general. Also deleting random files that happen to be named *.la but aren't libtool archives is obviously wrong.

[jmroot (Joshua Root)]

Debian's notes on how they're solving the problem: ​http://wiki.debian.org/ReleaseGoals/LAFileRemoval

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmr@…:

They're apparently used by libltdl in some situations, so always deleting them isn't correct in general.

In what way does libltdl use them? Jeremy claims they are never useful on OS X and he has experienced no problems in removing them from his personal system, and they have already been deleted from shipping versions of OS X.

Also deleting random files that happen to be named *.la but aren't libtool archives is obviously wrong.

Obviously I mean delete all .la files that are libtool archives even if they're not in ${destroot}${prefix}/lib.

[jmroot (Joshua Root)]

A Gentoo developer's blog entry about the issues: ​http://blog.flameeyes.eu/2010/10/libtool-archives-and-their-pointless-points

[cooljeanius (Eric Gallager)]

Replying to a question from the mailing lists:

Then I'm surprised MacPorts has gone 11 years already without removing them. Do other OS X package managers remove them?

I don't know about fink or homebrew, so I can't comment there,

Yes, they both do remove them. Homebrew has a "cleaning" step after it finishes what would be the equivalent of MacPorts's destroot step, and it looks something like this:

==> Cleaning
rm: /var/root/usr/Cellar/gdal/1.9.2/lib/libgdal.la

Also, Fink has a step during its "setting up packages" phase when it does this:

Setting up libtool2 (2.4.2-3) ...
Clearing dependency_libs of .la files being installed

[cooljeanius (Eric Gallager)]

Cc Me!

[neverpanic (Clemens Lang)]

While we're at it we might add a check that walks all binaries, gets a list of symbols and checks whether there's at least one symbol being used from the linked libraries. I think Debian has something similar in place.

[cooljeanius (Eric Gallager)]

Replying to cal@…:

While we're at it we might add a check that walks all binaries, gets a list of symbols and checks whether there's at least one symbol being used from the linked libraries. I think Debian has something similar in place.

So does Fink, for that matter. Besides linked libraries, they also check headers and stuff, too. For example:

fink-package-precedence --prohibit-bdep=libxml2 .
Scanning /\.Plo,\.Po$/ dependency files...
	./.deps/c14n.Plo
	./.deps/catalog.Plo
	./.deps/chvalid.Plo
	./.deps/debugXML.Plo
	./.deps/dict.Plo
        ...

<extra lines trimmed to save space>

        ...
	./doc/examples/.deps/xpath2.Po
	./example/.deps/gjobread.Po
	./python/.deps/libxml.Plo
	./python/.deps/libxml2-py.Plo
	./python/.deps/types.Plo
Looking for incorrect headers in 87 dependency files...
Determining fink providers of 22 headers...
	1 - 22
Found use of headers from 3 fink packages:
	libiconv-dev
	liblzma5
	readline6
Scanning binaries for incorrect dyld linking...

I'm not sure how many of these steps MacPorts should copy here, but I just thought I'd bring it up for comparison's sake.

[skymoo (Adam Mercer)]

Cc Me!

[jeremyhu (Jeremy Huddleston Sequoia)]

I think we can certainly explore the other cleanup / verification options employed by other packaging systems, but I want to keep this ticket about the .la files. Please open additional tickets for the other options you would like to see.

[jeremyhu (Jeremy Huddleston Sequoia)]

Here's an initial patch to address this. It still needs to be updated to make sure the file is really a glibtool la file and not just something named *.la

[jeremyhu (Jeremy Huddleston Sequoia)]

Ok, I think this should work for ensuring that the .la file is from glibtool before we delete it.

[neverpanic (Clemens Lang)]

Replying to jeremyhu@…:

It still needs to be updated to make sure the file is really a glibtool la file and not just something named *.la

/usr/share/file/magic/misctools has a possible solution for that:

# Summary: Libtool library file
# Extension: .la
# Submitted by: Tomasz Trojanowski <tomek@uninet.com.pl>
0   search/80   .la\ -\ a\ libtool\ library\ file   libtool library file

So it seems file just searches the fist line for ".la - a libtool library file".

[jeremyhu (Jeremy Huddleston Sequoia)]

prune-la.patch

[jeremyhu (Jeremy Huddleston Sequoia)]

Yeah, that's essentially what I used.

[jmroot (Joshua Root)]

Replying to ryandesign@…:

Replying to jmr@…:

They're apparently used by libltdl in some situations, so always deleting them isn't correct in general.

In what way does libltdl use them?

When loading at runtime. Read the manual.

[jmroot (Joshua Root)]

Replying to egall@…:

Also, Fink has a step during its "setting up packages" phase when it does this:

Setting up libtool2 (2.4.2-3) ...
Clearing dependency_libs of .la files being installed

That's a better approach since it's conducive to incrementally fixing the problem, unlike deleting all the files. If A depends on B and thus lists B in its .la file in dependency_libs, and then B's .la file goes away, libtool will fail to process A's when linking.

[jmroot (Joshua Root)]

r102945

[jeremyhu (Jeremy Huddleston Sequoia)]

Almost. We should change the default to be "yes" ... reopening to track that after more testing.

[jeremyhu (Jeremy Huddleston Sequoia)]

We should also have 'sudo make install' have a hook to delete the existing .la files or you can run into an issue where an intermediate .la is removed and a dependency is still referring to it.

[jmroot (Joshua Root)]

Defaulting to deletion right away would cause tons of breakage for the reasons I mentioned above. Sometime in the future when all the dependency_libs in installed files are cleared, and everything that really needs them is explicitly setting 'no', then the default can change.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to jmr@…:

Defaulting to deletion right away would cause tons of breakage for the reasons I mentioned above.

Yes, which is why I mentioned that we should have an install hook (when installing base) that will delete any .la files that the user has installed at the time they upgrade base.

Sometime in the future when all the dependency_libs in installed files are cleared, and everything that really needs them is explicitly setting 'no', then the default can change.

I fail to see how that is relevant or will help the situation. The deployment should be:

1) Install a new base -> delete all existing .la files

2) Install new ports -> no new .la files are deployed

That's what I'm doing here, and it's working out fine. I have a couple installs going from clean and many more doing upgrades after deleting *.la.

[jmroot (Joshua Root)]

That relies on nobody ever deactivating/reactivating anything, and also assumes that nothing legitimately needs the files.

Clearing dependency_libs does fix the overlinking, so why the burning need to nuke all .la files?

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to jmr@…:

That relies on nobody ever deactivating/reactivating anything, and also assumes that nothing legitimately needs the files.

True.

Clearing dependency_libs does fix the overlinking, so why the burning need to nuke all .la files?

I guess Im just anxious. Ok, I'll take this as a win. I have the option available for me, and I'll use it ;).

[raimue (Rainer Müller)]

Cc Me!

[cooljeanius (Eric Gallager)]

Replying to jeremyhu@…:

I think we can certainly explore the other cleanup / verification options employed by other packaging systems, but I want to keep this ticket about the .la files. Please open additional tickets for the other options you would like to see.

Just to check up on this and to refresh my memory, did any such tickets for other options ever get opened? If so, could I have some links? I feel like there were some, I just forget their numbers...

[jeremyhu (Jeremy Huddleston Sequoia)]

I'm not sure what you mean by other options. IMO, the "best" option is to uninstall everything, set destroot.delete_la_files, and reinstall =)

[cooljeanius (Eric Gallager)]

Replying to jeremyhu@…:

I'm not sure what you mean by other options.

Well the original post I was quoting was referring to the following idea that cal had:

Replying to cal@…:

While we're at it we might add a check that walks all binaries, gets a list of symbols and checks whether there's at least one symbol being used from the linked libraries. I think Debian has something similar in place.

[larryv (Lawrence Velázquez)]

Replying to egall@…:

Well the original post I was quoting was referring to the following idea that cal had:

Replying to cal@…:

While we're at it we might add a check that walks all binaries, gets a list of symbols and checks whether there's at least one symbol being used from the linked libraries. I think Debian has something similar in place.

There was ​a bit of discussion about this recently.

[jeremyhu (Jeremy Huddleston Sequoia)]

Ah, those are really other alternative options. Those are other additional options. Sorry, I was confused there for a moment.

Yes, we should certainly implement those additional changes, but I don't know if there are existing tickets tracking them. Feel free to create them and mention here, so I can CC myself. Thanks. =)

[afb@…]

It seems rather wishy-washy to keep the static libraries and remove the libtool libraries... Wouldn't you need that information, in order to link anything anyway ? The dynamic libraries do record their dependencies, but the static libraries do not. And if not linking things statically, then why not remove the static libraries as well ? That is, use --disable-static etc.

[jeremyhu (Jeremy Huddleston Sequoia)]

Yeah, I'm all for nuking static libraries as well, but I was voted down when I brought that up last year.