atk @2.22.0_0: does not activate following install/upgrade due to false positive file exists error with gtk-docs on OS X 10.9 (Mavericks)

[Hasimir (Ben McGinnes)]

OS version: 10.9.5 uname -a: Darwin redacted.example.org 13.4.0 Darwin Kernel Version 13.4.0: Mon Jan 11 18:17:34 PST 2016; root:xnu-2422.115.15~1/RELEASE_X86_64 x86_64

The build for atk 2.22.0 fails to complete as either stand alone (64-bit) or universal due to what appears to be a false positive error regarding moving (renaming) the documentation files to the live directory. Everything else appears fine with the build so it should work if only it gets past this little thing and then allows the registry to update.

The major problem, however, is the flow on effect through preventing installation of at-spi2-atk and upgrading of, well, basically all of gnome, bash, cmake, wine, various libraries and all text processing tools. Probably a good deal more than that.

The actual error is this:

:error:activate org.macports.activate for port atk returned: error renaming "/opt/local/var/macports/software/atk/mpextract0k6HadQ1/opt/local/share/gtk-doc/html/atk/atkobject.html" to "/opt/local/share/gtk-doc/html/atk/atkobject.html": file already exists
:debug:activate Error code: POSIX EEXIST {file already exists}
:debug:activate Backtrace: error renaming "/opt/local/var/macports/software/atk/mpextract0k6HadQ1/opt/local/share/gtk-doc/html/atk/atkobject.html" to "/opt/local/share/gtk-doc/html/atk/atkobject.html": file already exists

Full details are in the log file attached.

Obviously, I checked to make sure that the file wasn't there and it wasn't. The same is true of /opt/local/share/gtk-doc/html/atk/AtkObject.html and running the attempts to activate or install with force flags, binary installations, no dependencies or the opposite makes no difference. Installations are cleaned and usually force deactivated between attempts.

I've also tried renaming /opt/local/share/gtk-doc/html/ and creating an empty html directory there, the only difference in that case was the new directory was deleted when the activation failed. So leaving some other file in the /opt/local/share/gtk-doc/html/atk/ directory was attempted to see if it left any other relics of its attempt behind, but no, there weren't any.

I do have the built tarball in /opt/local/var/macports/software/atk/ still and could run through it manually, but require advice on which parts of the registry the install and upgrade or activation commands modify to make it live. I am aware that this is a sub-optimal solution (and that most people should never try it), but too many things depend on this package and that's still preferable to a scorched earth response. I have over 800 active ports and many of them are essential, whereas the scorched earth solution takes days or weeks to complete with full system and network resources tied up which is unacceptable at this time (over a holiday period, sure, but not now).

Alternatively I can custom compile the damn thing over in /usr/local (or some alternative location) if it is possible to force at-spi2-atk to use that path and ignore the ports failure, then everything else shoud return to normal.

If you really want, I can also provide the build directory contents for the atk builds, but that hardly seems likely to help. Still, if needed it can be provided. The commands attempted include:

sudo port -v -d -n install -b -p -f atk
sudo port -v -d -n install -k -p -f atk
sudo port -v -d -n install -p -f atk
sudo port -v -d -n install -f atk
sudo port -v -d install -f atk
sudo port -v -d -n install -b -p -f atk +universal
sudo port -v -d -n install -k -p -f atk +universal
sudo port -v -d -n install -p -f atk +universal
sudo port -v -d -n install -f atk +universal
sudo port -v -d install -f atk +universal

Along with the less verbose options preceding those.

Force deactivation works, but force activation produces the same error as installation and upgrade attempts. Which makes sense since it is the point of activating any installations or upgrades which fails.

Leaving the force (-f) and the no dependency (-n) flags out makes no difference, though in some cases with attempts to upgrade other outdated packages it will affect how much of the upgrade is attempted and depending entirely on how much of the process can be done before the atk dependency is mandatory. None of those outdated packages successfully upgrade either way.

There is currently approximately thirty outdated packages affected by this, including gtk3, but that number will grow.

[Hasimir (Ben McGinnes)]

main.log for atk installation and activation attempt.

[ryandesign (Ryan Carsten Schmidt)]

Please, do not use the -p flag. Please also refrain from using the -f or -n flags as much as possible. Use of these flags can result in you building ports in the wrong order, which can in some cases defeat the purpose of rebuilding the ports in the first place, and leave you with a set of ports which appears to be up to date, but is not.

Note that single-letter flags only have effect when placed between the command port and the subcommand (like install); they don't have any effect when placed after the subcommand.

The log you attached shows only the activate phase, and shows that you were using the force flag. It would help if you sudo port clean atk, then sudo port -f uninstall atk (you'll need to force here, if you have other ports installed that depend on atk, which you said you did), then sudo port install atk and see what happens.

Your log mentioned both AtkObject.html and atkobject.html. Do you have a case-sensitive or case-insensitive filesystem? Either is supposed to work, I'm just gathering data.

[Hasimir (Ben McGinnes)]

The filesystem is Mac OS Extended (Journaled) which, IIRC, is HFS+ and is case sensitive. As for the discrepancy in the log, I'm not sure why the lowercase only filename is reported, but the file in the extracted docs is the camel case one (AtkObject.html).

Uninstalling is fine, reinstalling with no extra flags produces the same errors as above. Oh, I normally don't use any of the flags anyway, except when a problem is encountered and rarely modify the port files for anything (the exceptions being for the ports of a couple of packages I'm involved in the dev for and I know their innards a little better).

[larryv (Lawrence Velázquez)]

Replying to ben@…:

The filesystem is Mac OS Extended (Journaled) which, IIRC, is HFS+ and is case sensitive.

No, that is case-insensitive.

% diskutil listFilesystems
Formattable file systems

These file system personalities can be used for erasing and partitioning.
When specifying a personality as a parameter to a verb, case is not considered.
Certain common aliases (also case-insensitive) are listed below as well.

-------------------------------------------------------------------------------
PERSONALITY                     USER VISIBLE NAME                               
-------------------------------------------------------------------------------
ExFAT                           ExFAT                                           
Free Space                      Free Space                                      
  (or) free
MS-DOS                          MS-DOS (FAT)                                    
MS-DOS FAT12                    MS-DOS (FAT12)                                  
MS-DOS FAT16                    MS-DOS (FAT16)                                  
MS-DOS FAT32                    MS-DOS (FAT32)                                  
  (or) fat32
HFS+                            Mac OS Extended                                 
Case-sensitive HFS+             Mac OS Extended (Case-sensitive)                
  (or) hfsx
Case-sensitive Journaled HFS+   Mac OS Extended (Case-sensitive, Journaled)     
  (or) jhfsx
Journaled HFS+                  Mac OS Extended (Journaled)                     
  (or) jhfs+

[ryandesign (Ryan Carsten Schmidt)]

Replying to ben@…:

The filesystem is Mac OS Extended (Journaled) which, IIRC, is HFS+ and is case sensitive. As for the discrepancy in the log, I'm not sure why the lowercase only filename is reported, but the file in the extracted docs is the camel case one (AtkObject.html).

If you're on a case-insensitive filesystem (which you seem to be), and are trying to extract a file containing two files whose names differ only by case (which I've verified is indeed the case for the precompiled atk packages we're distributing from our packages server), then it's normal that one of those will overwrite the other one.

Uninstalling is fine, reinstalling with no extra flags produces the same errors as above.

May we see the new main.log file then?

[Hasimir (Ben McGinnes)]

Oh damn ... I guess it's not right to go strangle someone at Apple or beat them over the head with the ZFS specs ... *sigh* ...

Anyway, you most certainly can see the new main file. One moment ...

[ryandesign (Ryan Carsten Schmidt)]

Replying to ben@…:

Oh damn ... I guess it's not right to go strangle someone at Apple or beat them over the head with the ZFS specs ... *sigh* ...

What are you talking about? macOS has no support for ZFS...

Anyway, you most certainly can see the new main file. One moment ...

Thanks. There are a number of things in this log I don't understand.

730	:debug:main Executing org.macports.main (atk)
731	:debug:main changing euid/egid - current euid: 0 - current egid: 0
732	:debug:main egid changed to: 501
733	:debug:main euid changed to: 504
734	:debug:main Skipping completed org.macports.archivefetch (atk)
735	:debug:main Privilege de-escalation not attempted as not running as root.
736	:debug:main Skipping completed org.macports.fetch (atk)
737	:debug:main Privilege de-escalation not attempted as not running as root.
738	:debug:main Skipping completed org.macports.checksum (atk)
739	:debug:main Privilege de-escalation not attempted as not running as root.
740	:debug:main Skipping completed org.macports.extract (atk)
741	:debug:main Privilege de-escalation not attempted as not running as root.
742	:debug:main Skipping completed org.macports.patch (atk)
743	:debug:main Privilege de-escalation not attempted as not running as root.
744	:debug:main Skipping completed org.macports.configure (atk)
745	:debug:main Privilege de-escalation not attempted as not running as root.
746	:debug:main Skipping completed org.macports.build (atk)
747	:debug:main Privilege de-escalation not attempted as not running as root.
748	:debug:main Skipping completed org.macports.destroot (atk)
749	:debug:main Privilege de-escalation not attempted as not running as root.

"Skipping completed" means this was not a clean log. I would like to see the contents of these skipped phases, though we can probably make some headway without them.

959	:debug:install Creating atk-2.22.0_0.darwin_13.x86_64.tgz

Why .tgz? Our default archive suffix is .tbz2. Have you deliberately changed this? I think this would prevent you from being able to use our precompiled binaries.

966	:debug:install Assembled command: 'cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_atk/atk/work/destroot" && /usr/bin/tar -cvf - . | /usr/bin/gzip -c9 > /opt/local/var/macports/software/atk/atk-2.22.0_0.darwin_13.x86_64.tgz'
967	:debug:install Executing command line:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_atk/atk/work/destroot" && /usr/bin/tar -cvf - . | /usr/bin/gzip -c9 > /opt/local/var/macports/software/atk/atk-2.22.0_0.darwin_13.x86_64.tgz
...
1289	:info:install a ./opt/local/share/gtk-doc/html/atk/AtkObject.html
...
1326	:info:install a ./opt/local/share/gtk-doc/html/atk/atkobject.html
...
1386	:debug:install Archive atk-2.22.0_0.darwin_13.x86_64.tgz packaged

You were able to create the archive, containing two files whose names differ only by case. This implies the disk this was happening on is case sensitive, else those two files couldn't have existed at the same time.

1394	:msg:activate --->  Activating atk @2.22.0_0
...
1718	:info:activate x ./opt/local/share/gtk-doc/html/atk/AtkObject.html
...
1755	:info:activate x ./opt/local/share/gtk-doc/html/atk/atkobject.html
...
1880	:debug:activate activating file: /opt/local/share/gtk-doc/html/atk/AtkObject.html
...
1917	:debug:activate activating file: /opt/local/share/gtk-doc/html/atk/atkobject.html
1918	:debug:activate Activation failed, rolling back.
...
2021	:error:activate org.macports.activate for port atk returned: error renaming "/opt/local/var/macports/software/atk/mpextractRNhdWJUE/opt/local/share/gtk-doc/html/atk/atkobject.html" to "/opt/local/share/gtk-doc/html/atk/atkobject.html": file already exists
2022	:debug:activate Error code: POSIX EEXIST {file already exists}
2023	:debug:activate Backtrace: error renaming "/opt/local/var/macports/software/atk/mpextractRNhdWJUE/opt/local/share/gtk-doc/html/atk/atkobject.html" to "/opt/local/share/gtk-doc/html/atk/atkobject.html": file already exists

You were unable to extract atkobject.html from the archive presumably because AtkObject.html had been previously extracted and it thinks they're the same file, which implies this is happening on a case insensitive filesystem.

Do you have more than one filesystem? Is part of your MacPorts installation on one filesystem and part on another filesystem, and one of them is case sensitive and the other is case insensitive?

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

trying to extract a file containing two files whose names differ only by case (which I've verified is indeed the case for the precompiled atk packages we're distributing from our packages server),

I've filed an atk bug about this: ​https://bugzilla.gnome.org/show_bug.cgi?id=772236

[Hasimir (Ben McGinnes)]

It could be, a different filesystem actually. Currently /opt/local/var/macports/build is actually a sym link to an external drive in order to keep sometimes disk hungry builds from interfering with other things on the laptop. I'll check ...

Okay, yes, you're right. The laptop's SSD is case insensitive, but the external drive is Mac OS Extended (Case-sensitive, Journaled). So that's why it extracts in build, but then the copy and activation fails. Excuse me a moment while I LART myself.

Still, with the fs difference in those volumes, restoring the build dir to the laptop won't help, the failure will just occur earlier in the process. If they were the other way around it'd be different, of course, but I've yet to see a technical fault that could be solved with wishing it were so. ;)

Thanks for lodging the gnome bug because obviously it's only a matter of time before it hits someone else. In the mean time, is it possible to change the macports copy function to automatically rename files when copying and attempt to continue, even if that is restricted to certain file types where a rename won't hurt (as in the case of docs where at worst there's a URI link failure, unlike with libs or executables)?

Obviously that shouldn't be default behaviour, but there's a pretty good case for it being a desirable option when a couple of HTML files are enough to kill the upgrade of everything from bash to wine-devel.

[ryandesign (Ryan Carsten Schmidt)]

Ok, so the problem was caused by having part of MacPorts on a case-sensitive volume and another part on a case-insensitive volume. That is not a configuration many users use, and as you see, that can cause problems, so we don't support that, and don't recommend that. You're welcome to use a case-sensitive or case-insensitive filesystem for MacPorts but the entire MacPorts installation must be on a filesystem of that type; don't mix and match.

The MacPorts copy proc just calls the Tcl file copy proc and is not involved in the error you encountered. The error you saw was from the activation code, specifically this code​ which uses Tcl's file rename proc. We could add the -force flag there, which would cause case collisions to be silently ignored by the second file overwriting the first file. That's similar to what already happens if MacPorts downloads a precompiled binary from our server, which was built on a case-sensitive filesystem, and activates it on a case-insensitive filesystem, but I'm not 100% certain whether the result is the same. We would need to check both scenarios to see if the first or the last file is the one that wins in both scenarios, or whether it differs.

The real bug is that atk has a case collision. They've proposed a fix for it. I've attached the fix to this ticket; you can use it locally if you like. I expect it will be fixed in the next version of atk. I don't intend to commit it to MacPorts because it introduces a dependency on gtk-doc, but if Dave wants to commit it he can; in that case, the revision should be increased because it does change the files installed by the port.

[dbevans (David B. Evans)]

Thanks for figuring this out, Ryan. I think I'll go with your recommendation and wait for the next revision of atk to make any changes. Folks that are bitten by this issue in the meantime can apply your patches locally. Thanks, Ben, for submitting this ticket. If you find any similar errors as you proceed please let us know.

[Hasimir (Ben McGinnes)]

Will do and thanks for all the work which is indeed working (so far; fingers would be crossed, but it's too annoying to type that way)

If there are any repeats with docs (or whatever) I'll at least know to go a-hunting for fs issues first. Though I have removed the symlink and restored the build/ dir as normal (I'll just have to remember to purge the directory after each update/upgrade instead, but that's what scripts are for). At some point I'll make a little ports specific partition on the external drive with a filesystem to match the laptop so I can get the external space and without screwing it up and needing to self-LART (again).

I guess I probably can't consider the fact that gnome still uses Docbook 4.1.2 for its documentation a bug, though, just charitably call it "retro" instead. XML docs, definitely; HTML transformations, of course; but there are better ways now. Oh well, their choice I guess.

Anyway, thanks again and well spotted with those filenames too.

[dbevans (David B. Evans)]

Seems to work with the current version.