gobject-introspection 1.60.2_5: g-ir-scanner produces incorrect result for gdk-pixbuf2 build

[VinDuv]

On my High Sierra machine, building gdk-pixbuf2 produces a .gir file (and associated .typelib file) with an incorrect library path:

$ grep shared-lib /opt/local/share/gir-1.0/GdkPixbuf-2.0.gir
             shared-library="./gdk-pixbuf/libgdk_pixbuf-2.0.0.dylib"

(instead of /opt/local/lib/libgdk_pixbuf-2.0.0.dylib). Strangely, the binary package gets it right.

I’ve investigated why that happens and found some issues in g-ir-scanner:

1. The _link method in giscanner/dumper.py (after modifications by patch patch-fix-scanner-in-build-execution.diff) merges library path lists by putting them in a set and converting them into a list. For instance:

        self._options.library_paths = list (set().union (self._options.library_paths,
                                                         this_L))

The problem with this approach is that the resulting list will be in a random order, depending on the random hash seed chosen by the Python interpreter. One solution may be to use sorted() instead of list() to ensure an order on the resulting list. Note that this does not directly affects the gdk-pixbuf2 build since the resulting path list is always ['./gdk-pixbuf'] in that case (the other path specified on the g-ir-scanner command line is in LIBRARY_PATH so it is removed).

2. The same _link method directly modifies options.library_paths, which affects other parts of the program. I’m actually not sure if that’s by design or not. The consequence is that giscanner/shlibs.py code executes with options.library_paths = ['./gdk-pixbuf'] instead of the original ['./gdk-pixbuf', '/opt/local/lib', '/opt/local/lib'].

3. In giscanner/shlibs.py, _resolve_non_libtool (as modified by patch patch-fix-rpath-gir-typelib.diff) tries to find the first path in options.library_paths that is not a build path, and assumes that it is the path where the libraries will be installed.

Due to the previous issues, looking for the first path is fragile since options.library_paths will be in an unpredictable order; but in the case of the gdk-pixbuf2build, the issue is that this method does not handle relative paths: ./gdk-pixbuf will be considered a non-build path, which is incorrect. The method should either normalize the library paths before before checking if they start with os.getcwd(), or assume that any relative path is a build path and skip it.

[michaelld (Michael Dickens)]

Let me boot a buildbot into High Sierra (10.13) & see what my installs show.

[michaelld (Michael Dickens)]

Ug ... have to update & reinstall MacPorts & all ports ... it'll take a bit but I'll get there ...

[VinDuv]

Thanks!

I did a bit more testing on my end: I reinstalled from source a bunch of packages that were providing .gir files (67 packages total) and wrote a script to check if the installed .gir files were correct (i.e. shared-libraries point to files that actually exists).

I got the following errors (the other .gir files were apparently fine):

/opt/local/share/gir-1.0/PeasGtk-1.0.gir (package libpeas): ./libpeas-gtk/libpeas-gtk-1.0.0.dylib
/opt/local/share/gir-1.0/GrlNet-0.3.gir (package grilo): ./libs/net/libgrlnet-0.3.0.dylib
/opt/local/share/gir-1.0/Atk-1.0.gir (package atk): ./atk/libatk-1.0.0.dylib
/opt/local/share/gir-1.0/GrlPls-0.3.gir (package grilo): ./libs/pls/libgrlpls-0.3.0.dylib
/opt/local/share/gir-1.0/Atspi-2.0.gir (package at-spi2-core): ./atspi/libatspi.0.dylib
/opt/local/share/gir-1.0/GWeather-3.0.gir (package libgweather): ./libgweather/libgweather-3.16.dylib
/opt/local/share/gir-1.0/Graphene-1.0.gir (package graphene): ./src/libgraphene-1.0.0.dylib
/opt/local/share/gir-1.0/JavaScriptCore-4.0.gir (package webkit2-gtk): /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_www_webkit2-gtk/webkit2-gtk/work/build/lib/libjavascriptcoregtk-4.0.18.dylib
/opt/local/share/gir-1.0/Gegl-0.4.gir (package gegl): ./gegl/libgegl-0.4.0.dylib
/opt/local/share/gir-1.0/AppStreamGlib-1.0.gir (package appstream-glib): ./libappstream-glib/libappstream-glib.8.dylib
/opt/local/share/gir-1.0/GdkPixdata-2.0.gir (package gdk-pixbuf2): ./gdk-pixbuf/libgdk_pixbuf-2.0.0.dylib
/opt/local/share/gir-1.0/Notify-0.7.gir (package libnotify): ./libnotify/libnotify.4.dylib
/opt/local/share/gir-1.0/GtkChamplain-0.12.gir (package libchamplain): ./champlain-gtk/libchamplain-gtk-0.12.0.dylib
/opt/local/share/gir-1.0/Babl-0.1.gir (package babl): ./babl/libbabl-0.1.0.dylib
/opt/local/share/gir-1.0/GExiv2-0.10.gir (package gexiv2): ./gexiv2/libgexiv2.2.dylib
/opt/local/share/gir-1.0/GCab-1.0.gir (package gcab): ./libgcab/libgcab-1.0.0.dylib
/opt/local/share/gir-1.0/GData-0.0.gir (package libgdata): ./gdata/libgdata.22.dylib
/opt/local/share/gir-1.0/Soup-2.4.gir (package libsoup): ./libsoup/libsoup-2.4.1.dylib
/opt/local/share/gir-1.0/Dazzle-1.0.gir (package libdazzle): ./src/libdazzle-1.0.0.dylib
/opt/local/share/gir-1.0/GnomeDesktop-3.0.gir (package gnome-desktop): ./libgnome-desktop/libgnome-desktop-3.19.dylib
/opt/local/share/gir-1.0/GdkPixbuf-2.0.gir (package gdk-pixbuf2): ./gdk-pixbuf/libgdk_pixbuf-2.0.0.dylib
/opt/local/share/gir-1.0/Totem-1.0.gir (package totem): ./src/libtotem.0.dylib
/opt/local/share/gir-1.0/Grl-0.3.gir (package grilo): ./src/libgrilo-0.3.0.dylib
/opt/local/share/gir-1.0/TotemPlParser-1.0.gir (package totem-pl-parser): ./plparse/libtotem-plparser.18.dylib
/opt/local/share/gir-1.0/WebKit2WebExtension-4.0.gir (package webkit2-gtk): /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_www_webkit2-gtk/webkit2-gtk/work/build/lib/libwebkit2gtk-4.0.37.dylib
/opt/local/share/gir-1.0/WebKit2WebExtension-4.0.gir (package webkit2-gtk): /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_www_webkit2-gtk/webkit2-gtk/work/build/lib/libjavascriptcoregtk-4.0.18.dylib
/opt/local/share/gir-1.0/GtkSource-4.gir (package gtksourceview4): ./gtksourceview/libgtksourceview-4.0.dylib
/opt/local/share/gir-1.0/Gucharmap-2.90.gir (package gucharmap): ./gucharmap/libgucharmap_2_90.7.dylib
/opt/local/share/gedit/gir-1.0/Gedit-3.0.gir (package gedit): ./gedit/libgedit-3.36.dylib
/opt/local/share/eog/gir-1.0/Eog-3.0.gir (package eog): ./src/libeog.dylib
/opt/local/share/gir-1.0/WebKit2-4.0.gir (package webkit2-gtk): /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_www_webkit2-gtk/webkit2-gtk/work/build/lib/libwebkit2gtk-4.0.37.dylib
/opt/local/share/gir-1.0/WebKit2-4.0.gir (package webkit2-gtk): /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_www_webkit2-gtk/webkit2-gtk/work/build/lib/libjavascriptcoregtk-4.0.18.dylib
/opt/local/share/gir-1.0/AvahiCore-0.6.gir (package avahi): libavahi-core.so.7
/opt/local/share/gir-1.0/Ggit-1.0.gir (package libgit2-glib): ./libgit2-glib/libgit2-glib-1.0.0.dylib
/opt/local/share/gir-1.0/Handy-0.0.gir (package libhandy): ./src/libhandy-0.0.0.dylib
/opt/local/share/gir-1.0/Peas-1.0.gir (package libpeas): ./libpeas/libpeas-1.0.0.dylib
/opt/local/share/gir-1.0/GeocodeGlib-1.0.gir (package geocode-glib): ./geocode-glib/libgeocode-glib.0.dylib
/opt/local/share/gir-1.0/SoupGNOME-2.4.gir (package libsoup): ./libsoup/libsoup-gnome-2.4.1.dylib
/opt/local/share/gir-1.0/Devhelp-3.0.gir (package devhelp): ./devhelp/libdevhelp-3.6.dylib
/opt/local/share/gir-1.0/Champlain-0.12.gir (package libchamplain): ./champlain/libchamplain-0.12.0.dylib

So GdkPixbuf-2.0.gir is not the only affected gir file, and some of them have other issues (webkit2-gtk seems to leave a build path in the .gir file, and avahi doesn’t have a path at all).

I’ll try to modify g-ir-scanner to see if I can fix those issues on my machine.

Update: I tried the following changes (patches attached):

• Modify dumper.py so it sorts the library paths after uniquing them (1) and does not modify the global options.library_paths (2)
• Modify shlibs.py so a path that does not start with / is considered a build path, and ignored (3)

I then rebuilt from source all the packages, with the following results:

• g-ir-scanner executes successfully on all packages
• It generates the same files as before, except on shared-library attributes, as expected
• All the incorrect relative paths ./directory/libsomething.dylib are now /opt/local/lib/libsomething.dylib and the libraries actually exist at the specified paths.

The only two remaining issues are:

• The incorrect path libavahi-core.so.7 in the AvahiCore-0.6.gir from package avahi. Looking at the package source, it seems that this .gir file is not generated by g-ir-scanner but is part of the sources, so this can only be fixed by modifying the avahi package.
• The incorrect .gir files from the webkit2-gtk package come from the fact that the library path given to g-ir-scanner is

  /opt/local/var/macports/build/.../webkit2-gtk/work/build/lib
  

  but the current working directory is

  /opt/local/var/macports/build/.../webkit2-gtk/work/build/Source/WebKit
  

Since it’s not a prefix of the library path, the library path is not detected as a build path and is used as the final library path in the .gir file. This could probably be fixed by changing the heuristic used by g-ir-scanner (use the last library path instead of the first? It seems to be /opt/local/lib more consistently) but could break other packages.

[kencu (Ken)]

What you probably don't know VinDuv is Michael spend about 500 hours trying to get this issue with g-ir-scanner sorted out a couple of months ago, and we thought he had finally kicked this particular can off the road.

So -- everyone is suitably disappointed to see it is apparently still broken.

Michael has good insight into the issues, and once he can reproduce it, we'll see what is up. It's frustrating, because the way it is all designed to work is not working very well, and yet there are about 100 linux variants to be supported, etc, etc, etc.

[michaelld (Michael Dickens)]

Thank you @VinDuv for the continued sleuthing and information. I will get there sooner or later and work through what you're found & the current patches, and we can work out a better solution. Thanks @kencu for keeping the discussion going!

Sorry this is taking so long! My actual W2 "work" is keeping me quite busy at the moment. The 10.13 boot is still updating ... not the fastest Mac in the world & all those !@#$ GCC and Clang compilers take quite some time to build. Thanks for your patience!

[michaelld (Michael Dickens)]

OK finally got to the point of being able to recreate this issue. Let me cross-check on more recent OSX.

[michaelld (Michael Dickens)]

Sorry for the very long delay. I will strive to get to this issue and review the ​explanation and patches from comment:4 "soonish" ... my W2 work continues to just inundate me!

[evanmiller (Evan Miller)]

In 8f58afc2f37e944f1b96b2bbdf21bbdf67b10258/macports-ports (master):

meson: Fix dylib install paths for typelib files

See: #62391
See: #61792
See: #61934

[mascguy (Christopher Nielsen)]

Let us know if this is still an issue, after updating to the latest version (1.70.0).