binary nepomuk-core package corruption due to case-insensitive filesystem

[RJVB (René Bertin)]

This is a follow-up on ticket:37968 , because I have bumped into the same build failure mentioned there.

As mentioned in the original issue, installing nepomuk-core on a case-insensitive filesystem mixes up the (C++) Nepomuk2 and nepomuk2 (C?) header directories. HFS+ being case-preserving this indicates an issue in the install or in the packaging procedure. I build nepomuk-core from source on a case-sensitive FS and the resulting header tree was as expected and intended, which makes me suspect packaging procedure.

I don't know why the original ticket was closed as "invalid", but I'd really like to make a case for the use of case-sensitive filesystems on the buildbots even if the default on OS X is case-insensitive. This maps better to the origin of the majority of projects. Any adaptations required to allow installation on case-insensitive filesystems will work fine on a case-sensitive filesystem while mix-ups of the sort described in the original issue are prevented automatically.

[mf2k (Frank Schima)]

In the future, please Cc the port maintainers (port info --maintainers nepomuk-core).

[ryandesign (Ryan Carsten Schmidt)]

The buildslaves do already use case-sensitive filesystems.

[RJVB (René Bertin)]

Ah? That's not what I understood from the ticket I referred to, but if true, so much the better. But then I don't understand what went wrong with the binary package of nepomuk-core that I installed today on 10.9; as I wrote the build from source went fine on 10.6 (about a week ago).

[ryandesign (Ryan Carsten Schmidt)]

There simply seems to be a bug in the nepomuk-core software, or in software that uses it (I'm not sure which), with regard to whether the nepomuk2 directory name should be capitalized or not. The package from the buildbots has the directory name in lowercase. The kde4-runtime port has #include statements referencing the directory name capitalized. This does not seem to be a MacPorts-specific issue. You should probably report it to the developers of kde4-runtime and see what they say the solution is. Either they should change their includes to use the lowercased directory name that nepomuk-core provides, or if they say the directory name should be capitalized, then report that to the developers of nepomuk-core and see if they agree and will make that change.

[RJVB (René Bertin)]

Replying to ryandesign@…:

There simply seems to be a bug in the nepomuk-core software, or in software that uses it (I'm not sure which), with regard to whether the nepomuk2 directory name should be capitalized or not.

No, I don't think there's a bug. Remember that this software is developed on case-sensitive systems, so a mistake in case in an include file directive is detected immediately.

No, what I think is that the nepomuk2 directory contains "C-style" header files with a .h extension, while the Nepomuk2 directory contains "C++-style" headers, with extension.

I've made similar distinctions myself in the past, playing with filename case to introduce subtle variations. Not a good idea as I learned.

The package from the buildbots has the directory name in lowercase.

And that's the issue. There has to be a nepomuk2 directory, but also a Nepomuk2 directory. If case isn't preserved, these two can end up merged, and that won't be detected as long as case is also not used for file name resolution.

In installed most all of KDE from source (using MacPorts) on my 10.6 system, building and installing to a case-sensitive partition (one more reason to install MacPorts off the boot partition, which shouldn't be case-sensitive AFAIK). This worked fine, and gave me the 2 expected header file directories.

So I don't think there's an error neither in the source code nor in the Portfile/build system ... except possibly when the build/packaging chain contains a case-insensitive (potentially non-preserving) filesystem somewhere.

The question is indeed if and how the devs support installation on ... MS Windows (with its filesystems that are by default case insensitive and non preserving)!

[ryandesign (Ryan Carsten Schmidt)]

Replying to rjvbertin@…:

Replying to ryandesign@…:

There simply seems to be a bug in the nepomuk-core software, or in software that uses it (I'm not sure which), with regard to whether the nepomuk2 directory name should be capitalized or not.

No, I don't think there's a bug.

Well, you claim there is a bug: you've filed this ticket. And I'm claiming it's not a MacPorts bug.

Remember that this software is developed on case-sensitive systems, so a mistake in case in an include file directive is detected immediately.

Ok.

No, what I think is that the nepomuk2 directory contains "C-style" header files with a .h extension, while the Nepomuk2 directory contains "C++-style" headers, with extension.

If that is what they've done -- if they intend for there to be two different items in the same directory whose names differ only by case -- then that is unsupportable and that is the bug you should report to them. However, thus far, I see no evidence that that is the case.

The package from the buildbots has the directory name in lowercase.

And that's the issue. There has to be a nepomuk2 directory, but also a Nepomuk2 directory. If case isn't preserved, these two can end up merged, and that won't be detected as long as case is also not used for file name resolution.

The buildslaves have case-sensitive filesystems, so the case is preserved.

You can examine ​the packages the buildslaves produced using a command like:

bzcat /tmp/nepomuk-core-4.12.2_0.darwin_10.x86_64.tbz2 | tar tf -

As you can see, there is no directory Nepomuk2. If there should be one, then the build system is to blame. As you can see by looking at the Portfile​, we're not doing anything special.

In installed most all of KDE from source (using MacPorts) on my 10.6 system, building and installing to a case-sensitive partition (one more reason to install MacPorts off the boot partition, which shouldn't be case-sensitive AFAIK).

I know of no reason why the boot partition couldn't be case-sensitive, only that using a case-sensitive filesystem on OS X is nonstandard and you may therefore be more likely to run into problems because of developers who did not test under that configuration.

This worked fine, and gave me the 2 expected header file directories.

If that's indeed true, it would be interesting to know how your build differs from ours. Are you supplying additional configure flags, environment variables, etc?

[RJVB (René Bertin)]

Replying to ryandesign@…:

No, I don't think there's a bug.

Well, you claim there is a bug: you've filed this ticket. And I'm claiming it's not a MacPorts bug.

When I filed the ticked I thought that the buildsystem was probably not case-sensitive, as that would explain the issue, and made a case for using case-sensitive filesystems.

If that is what they've done -- if they intend for there to be two different items in the same directory whose names differ only by case -- then that is unsupportable and that is the bug you should report to them. However, thus far, I see no evidence that that is the case.

It really looks like that. On my Linux boxes I have /usr/include/nepomuk2 and /usr/include/Nepomuk2 with different, non-overlapping contents.

As you can see, there is no directory Nepomuk2. If there should be one, then the build system is to blame. As you can see by looking at

the Portfile​, we're not doing anything special.

Hmmm, and elsewhere? Is there something we could learn from doing a command like

port -k destroot -s nepomuk-core

?

I know of no reason why the boot partition couldn't be case-sensitive, only that using a case-sensitive filesystem on OS X is nonstandard and you

may therefore be more likely to run into problems because of developers who did not test under that configuration.

The OS itself isn't made for it, and is likely to run into "file not found" errors because of it. Very likely because of those last few words above ...

If that's indeed true, it would be interesting to know how your build differs from ours. Are you supplying additional configure flags, environment variables, etc?

From what I remember, I simply did a port install kde4-runtime . When I set out it was really to see how far I'd get, so I certainly didn't start doing special things!

R.

[ryandesign (Ryan Carsten Schmidt)]

Replying to rjvbertin@…:

If that is what they've done -- if they intend for there to be two different items in the same directory whose names differ only by case -- then that is unsupportable and that is the bug you should report to them. However, thus far, I see no evidence that that is the case.

It really looks like that. On my Linux boxes I have /usr/include/nepomuk2 and /usr/include/Nepomuk2 with different, non-overlapping contents.

If that's true, then please report that design error to the developers so that they can fix it. We obviously cannot accommodate such a scenario in a package manager designed for use on an OS where case-insensitive filesystems are the norm.

As you can see, there is no directory Nepomuk2. If there should be one, then the build system is to blame. As you can see by looking at

the Portfile​, we're not doing anything special.

Hmmm, and elsewhere? Is there something we could learn from doing a command like

port -k destroot -s nepomuk-core

?

I don't think so. You can run that on your case-sensitive filesystem if you like and examine the contents of the work/destroot directory, but it should be identical to what gets installed by the port and reported by "port contents nepomuk-core", since the process of getting the contents of the destroot into the archive is handled by MacPorts base automatically and individual Portfiles don't have any say in it.

I know of no reason why the boot partition couldn't be case-sensitive, only that using a case-sensitive filesystem on OS X is nonstandard and you

may therefore be more likely to run into problems because of developers who did not test under that configuration.

The OS itself isn't made for it,

Sure it is.

and is likely to run into "file not found" errors because of it. Very likely because of those last few words above ...

Only in poorly-constructed third-party software.

If that's indeed true, it would be interesting to know how your build differs from ours. Are you supplying additional configure flags, environment variables, etc?

From what I remember, I simply did a port install kde4-runtime . When I set out it was really to see how far I'd get, so I certainly didn't start doing special things!

You said you built from source, and I assumed you meant by hand, so I was wondering what configure flags and environment variables you had given when doing that.

Or did you mean that you had built from source using MacPorts? If so, again we need to figure out what's different between your system and the buildbot's system to account for the difference.

[RJVB (René Bertin)]

If that's true, then please report that design error to the developers so that they can fix it. We obviously cannot accommodate such a scenario in a package manager designed for use on an OS where case-insensitive filesystems are the norm.

I think one could also argue the other way around, that nepomuk has been conceived for a *nix desktop, and that it's not up to them but up those who want to use the software on a case-insensitive system to make that work... I'll see if there's conscious choice behind what you call a design error, but do realise that we're not talking about a set of project-internal headerfiles here, but headers included probably by anything using nepomuk ...

should be identical to what gets installed by the port and reported by "port contents nepomuk-core", since the process of getting the contents of the destroot into the archive is handled by MacPorts base automatically and individual Portfiles don't have any say in it.

I think that in fact the explanation was staring in our faces. You told me that the binary tarballs on packages.macports.org had all headers in a single nepomuk2 directory. I double-checked that's indeed the case, but now look at what I have in the binary tarball that must have been the result of the install-from-source build (one can wonder why the file wasn't regenerated today, but that's a different question):

drwxr-xr-x root/admin        0 2014-03-10 15:54 ./opt/local/include/Nepomuk2/
drwxr-xr-x root/admin        0 2014-03-10 15:54 ./opt/local/include/nepomuk2/
[...]
-rw-r--r-- root/admin     2583 2013-12-10 19:10 ./opt/local/include/nepomuk2/createresourcejob.h
-rw-r--r-- root/admin       43 2013-12-10 19:10 ./opt/local/include/Nepomuk2/CreateResourceJob

so, this tarball was likely created by simply archiving everything in the destroot folder.

Now if I unpack this tarball on a case-insensitive filesystem, all my nepomuk2 and Nepomuk2 headers end up in a single directory — in this case in the Nepomuk2 directory.

And that leads me to wonder if the tarballs over on packages.macports.org have not by any chance been unpacked and then regenerated on a case-insensitive filesystem. You claim that the buildslaves use case-sensitive filesystems ... but does that include the root, and thus places like /tmp and $TMPDIR ?

[ryandesign (Ryan Carsten Schmidt)]

Replying to rjvbertin@…:

And that leads me to wonder if the tarballs over on packages.macports.org have not by any chance been unpacked and then regenerated on a case-insensitive filesystem. You claim that the buildslaves use case-sensitive filesystems ... but does that include the root, and thus places like /tmp and $TMPDIR ?

To the best of my knowledge, the buildslaves use a case-sensitive filesystem for everything. The Mac OS Forge admins would know for sure.

[RJVB (René Bertin)]

How would one bring this issue to their attention?

[larryv (Lawrence Velázquez)]

Like this.

[RJVB (René Bertin)]

What do you know about the use of nepomuk *outside* of KDE's semantic desktop blabla? I understand that it's being phased out for that, starting with the upcoming KDE 13.1 release. If MacPorts will provide 13.1 as quickly as 12.4 appeared, then it may not be required to spend a lot of time on this particular issue.

Can MacPorts do file/directory presence checks after a binary install? If so, something like the following would be a workaround:

# test -d Nepomuk2 will fail on case-sensitive, succeed on case-insensitive (be it case-preserving or not)
if [ ! -d /opt/local/include/Nepomuk2 ] ;then
    # workaround so that #include <Nepomuk2/SomeHeader> will succeed on case-sensitive filesystems
    ln -s nepomuk2 /opt/local/include/Nepomuk2
fi

In the meantime that does NOT remove the need for someone to try and trace the reason binary packages can do case-folding!

[ryandesign (Ryan Carsten Schmidt)]

Bill or Shree, could we please get an answer to the question: do all the buildslaves still use case-sensitive filesystems? If the buildslaves use multiple partitions, do all partitions use case-sensitive filesystems, or are some case-insensitive, and if so, what's the breakdown, and what are the different volumes used for? I don't mind if the buildslaves use case-sensitive filesystems, since it helps us find situations where there's a case-only error, but if there's a mix of case-sensitive and case-insensitive filesystems, that could be a problem if MacPorts is working on both of them (for example, if MacPorts is extracting and building on one and installing to the other).

[mkae (Marko Käning)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

The old buildworkers only had a single volume. They were supposed to be case-sensitive, but I don't know if they actually were.

The new buildbot workers definitely have case-sensitive filesystems, and don't have a problem building nepomuk-core correctly. Revbumped in r153339 to force a rebuild.

See also #49559.