legacy-support missing futimens() (and utimensat)

[RJVB (René Bertin)]

From man futimens on Linux:

NAME
       utimensat, futimens - change file timestamps with nanosecond precision

SYNOPSIS
       #include <fcntl.h> /* Definition of AT_* constants */
       #include <sys/stat.h>

       int utimensat(int dirfd, const char *pathname,
                     const struct timespec times[2], int flags);

       int futimens(int fd, const struct timespec times[2]);

   Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       utimensat():
           Since glibc 2.10:
               _XOPEN_SOURCE >= 700 || _POSIX_C_SOURCE >= 200809L
           Before glibc 2.10:
               _ATFILE_SOURCE
       futimens():
           Since glibc 2.10:
                  _XOPEN_SOURCE >= 700 || _POSIX_C_SOURCE >= 200809L
           Before glibc 2.10:
                  _GNU_SOURCE

I just encountered futimens() trying to build a new kf5-kio version. It was easy enough to emulate using futimes().

What I do not know is whether this function is available on any Darwin version, nor what the project policy is for such situations (so feel free to close this if the function isn't acceptable).

[cjones051073 (Chris Jones)]

If you are suggesting an addition to the legacy-support package, then please file an PR implementing your changes at ​https://github.com/macports/macports-legacy-support

[kencu (Ken)]

thanks for suggestion. will see if darwin has this. if so, we should add it. if you get a moment, please paste up your replacement.

[cjones051073 (Chris Jones)]

Prior to APFS, does macOS have nano-second precision on timestamps ? I believe it is not supported at all in HFS+ etc.

I believe these methods are available in mac OS 10.13+ Prior to that, if the filesystem in use cannot support nano-second precision, is it even sensible to try and emulate it ?

[kencu (Ken)]

That's true enough, and I also wondered if older systems had the chops to implement it properly; I suspect not, as I believe I read somewhere that level of fs resolution was added to APFS.

Perhaps we might consider a "best approximation" so that software builds at least, and works as best the OS can support. I am thinking that is (or should be) what people are hoping for by extending the use of their older systems with our methods.

[cjones051073 (Chris Jones)]

I am not conviced, in this case, that emulating the methods on older systems is necessarily a good idea. The problem is if people are using these methods then presumably they want/need (or at least think they do) nanosecnd precision. So if the underlying file system simply cannot do this, if we pretend otherwise this could lead to all sorts of issues. Worst case, file corruption.... So in this case I remind unconvinced emulating these methods is actually doing anyone a favour...

[kencu (Ken)]

I'm coming across software that doesn't specifically need nanosecond timespec resolution for setting file access times, but is using these functions as they are assumed to exist, eg dosbox-x.

I think that adding compatibility functions for utimensat and futimens would be acceptable for the vast majority of these cases, rather than putting #ifdefs into the source to use the older microsecond resolution functions.

[Ionic (Mihai Moldovan)]

Emulating that should be safe.

POSIX/SUS specifies:

For futimens() and utimensat(), the times argument is an array of two timespec structures. The first array member represents the date and time of last access, and the second member represents the date and time of last modification. The times in the timespec structure are measured in seconds and nanoseconds since the Epoch. The file's relevant timestamp shall be set to the greatest value supported by the file system that is not greater than the specified time.

If the tv_nsec field of a timespec structure has the special value UTIME_NOW, the file's relevant timestamp shall be set to the greatest value supported by the file system that is not greater than the current time. If the tv_nsec field has the special value UTIME_OMIT, the file's relevant timestamp shall not be changed. In either case, the tv_sec field shall be ignored.

If the times argument is a null pointer, both the access and modification timestamps shall be set to the greatest value supported by the file system that is not greater than the current time.

Thus, SUS does have provisions for file systems not supporting such low resolutions. That argument is moot.

I've looked up the relevant source code in 10.13. That would be ​include/sys/stat.h and ​utimensat.c.

Ironically, if the comment is right and I read the code correctly, macOS 10.13 does support nanosecond-based time resolution for futimens/utimensat only if passed-in directly, but not for the special NOW value, since gettimeofday cannot return those. Given that, I think that emulating it with microseconds isn't a big deal.

Further, we don't even really need to emulate it. We can take the code verbatim for 10.6 - 10.12. That'll be APSL 2.0, but I guess that's fine.

For 10.6 and below, things are a bit more complicated.

utimensat looks less problematic, since it uses setattrlistat, which should be supported even by older operating systems.

futimens uses fsetattrlist, which is only available on 10.6+. This said, I guess we could likewise just copy its code, since it's likely based on setattrlistat.

I'd have to look deeper at older systems to find out if all support ATTR_BIT_MAP_COUNT, but I cannot imagine that they don't.

All in all, it sounds very doable.

Thoughts?

[kencu (Ken)]

I have some of this done, for 10.6 and up at least, but I was having troubles getting the test suite to run correctly (even on 10.13+) so I got sidetracked.

I can post up what I have so far.

[kencu (Ken)]

I left it here:

<​https://github.com/kencu/macports-legacy-support/commits/utimensat>

[Ionic (Mihai Moldovan)]

For 10.6 and up, it really should just be a copy-paste of Apple's code, with the commpage and gettimeofday stuff replaced with the user-space gettimeofday() version.

... oh, and that's mostly what you've done, I see.

[kencu (Ken)]

I was really hoping to get a proper test suite going to prove it worked right -- but it didn't work right -- and then the test suite didn't work right even on systems that were using Apple's implementation. So I got distracted a bit.

[Ionic (Mihai Moldovan)]

Thanks, I reworked your state considerably and also cleaned it up a lot.

Progress here: ​https://github.com/Ionic/macports-legacy-support/commits/utimensat

My other PRs are included there, so rebasing will be easy for me.

It builds, but currently (naturally) fails the tests, since I haven't worked on that yet. That's next.

[kencu (Ken)]

You can test back as far as 10.9, right?

[Ionic (Mihai Moldovan)]

On 10.9, yes. I do have access to some 10.11 and 10.13 VMs on which I can test, too, if necessary. Nothing below that, even though I have a 10.6 VM somewhere, but it's hopelessly outdated and depends on a machine that I wouldn't want to change till its hard drive issue is fixed.

[Ionic (Mihai Moldovan)]

Yeah, my modified test also fails on 10.13 natively, without our utimensat implementation.

test/test_utimensat
Testing: === {305419896, 987654321} {354826056, 123456789} ===
assert EQ failed {0, 987654321} post stat vs. utimensat Atime nanoseconds (utimensat Atime explicit)

The problem, though, is that the 10.13 machine I can test on doesn't use APFS as its root file system, but the older HFS+.

I created a new APFS container, mounted that:

hdiutil create -size 1.1m -fs APFS apfs.dmg
hdiutil attach apfs.dmg

and changed to that mount point and ran the test:

vm1013:~ root#  cd /Volumes/untitled/
vm1013:untitled root#  $(port work legacy-support)/macports-legacy-support-0.13-x86_64/test/test_utimensat
Warning: port definitions are more than two weeks old, consider updating them by running 'port selfupdate'.
Testing: === {305419896, 987654321} {354826056, 123456789} ===
Testing: === {0, -1} {354826056, 123456789} ===
Testing: === {305419896, 987654321} {0, -1} ===
Testing: === {0, -1} {0, -1} ===
Testing: === {0, -2} {354826056, 123456789} ===
Testing: === {305419896, 987654321} {0, -2} ===
Testing: === {0, -2} {0, -2} ===
Testing: === {0, -1} {0, -2} ===
Testing: === {0, -2} {0, -1} ===

So... the test case we have seems to hold for Apple's native utimensat implementation. But only on APFS. That's good to know.

Sadly, we'll never really be able to test it properly on any system older than 10.13, since APFS is only available, tada, on 10.13 and higher.

I'll rewrite the test to do other sanity checking on non-APFS file systems, though. We *can* test if the a/mtimes set by our version of utimensat is within seconds and microseconds range. That should be good enough for things like HFS+ and, frankly, Apple's native implementation probably won't behave differently on HFS+ either. I'm going to make sure that that's the case.

So... still a bit of work to do on my end, but it's coming along, finally.

[Ionic (Mihai Moldovan)]

Replying to Ionic:

I'll rewrite the test to do other sanity checking on non-APFS file systems, though. We *can* test if the a/mtimes set by our version of utimensat is within seconds and microseconds range. That should be good enough for things like HFS+ and, frankly, Apple's native implementation probably won't behave differently on HFS+ either. I'm going to make sure that that's the case.

Or, actually, it turns out that HFS+ doesn't even support microseconds.

That actually leads to an interesting observation: no file system I know of that is writable on OS X actually supports sub-second time precision (other, than, MAYBE... NFS and SMB?)

If that's true, utimensat could essentially just be glorified wrapper for utime (1-second precision) or utimes (microsecond precision, in theory, if the file system supports that...)

So, essentially we could have just provided a wrapper called utimensat, but really executing utimes, because that's all we'd never need there. It probably doesn't hurt to backport and hurt the "proper" code from XNU, but it also won't bring us any benefit.

Anyway, regarding the test cases: for non-APFS file systems, I will check if our utimensat implementation behaves the same as utimes. That should be sane enough. Apple's implementation does, too, of course.

[kencu (Ken)]

Yeah, we're not really trying to replicate the actual functionality.

we're just trying to make software compile.

Software authours are using those functions because they exist - but almost never because they actually NEED that functionality, I've found.

[RJVB (René Bertin)]

MacPorts wrote on 20210202::00:44:12 re: "Re: [MacPorts] #58729: legacy-support missing futimens() (and utimensat)"

Sadly, we'll never really be able to test it properly on any system older than 10.13, since APFS is only available, tada, on 10.13 and higher.

There's ​https://github.com/sgan81/apfs-fuse which may or may not work on Mac, and which may or may not have enough functionality to run the tests.

Also, if you are checking different filesystems, don't forget ZFS. I would expect it supports sub-second timestamp precision.

[Ionic (Mihai Moldovan)]

The APFS retrofit FUSE driver is read-only, so timestamp updates won't work anyway.

ZFS is third-party, so not really relevant.

However, I've since updated the test cases to check for utimes compatibility (only in case of non-APFS file systems, of course) and found an interesting bug.

On both 10.13 natively, as well as with our copied implementation on older system, the test fails when the mtime is set to UTIME_OMIT:

test/test_utimensat
Testing: === {305419896, 987654321} {354826056, 123456789} ===
Testing: === {0, -1} {354826056, 123456789} ===
Testing: === {305419896, 987654321} {0, -1} ===
Testing: === {0, -1} {0, -1} ===
Testing: === {0, -2} {354826056, 123456789} ===
Testing: === {305419896, 987654321} {0, -2} ===
assert EQ failed {1612234261, 305419896} post stat vs. utimensat Atime seconds (utimensat Atime explicit)

Obviously, utimensat doesn't update the atime correctly if the mtime is set to UTIME_OMIT, but only on non-APFS (or even just on HFS+) file systems.

That's not a problem as such, because we're essentially bug-for-bug compatible, but I'll have to change the test case to accept the failure.

[RJVB (René Bertin)]

Replying to Ionic:

ZFS is third-party, so not really relevant.

Maybe, but it's native and standards compliant, so a proper test platform for the implementation you're working on.

[Ionic (Mihai Moldovan)]

Feel free to run the test case of my branch (or, hopefully soon master) on a ZFS volume and report if it works or fails. I'll adjust the test as needed, but cannot test ZFS myself.

[kencu (Ken)]

[e34009691ebc7cebc0b8e07e9f6ffbb93cb1c573/macports-legacy-support]