if an install is interrupted, a later install of the same package can fail if version number has since changed.

[rogerdpack@…]

kind of explained here:

​https://gist.github.com/raw/933741/93ded0007937f70df19aeb3d77f523a9dbd5639c/gistfile1.txt

What happened is I did a $ sudo port install mplayer one day. I think I did a control+c to cancel it sometime in the middle. Next day, I did

$ sudo port install mplayer

again.

m4 is a dependency of mplayer. With the first one, the m4 install was interrupted. Now, however, you can see from the gist that it was saying

:debug:main Skipping completed org.macports.fetch (m4)

then attempting to build from:

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_m4/work/m4-1.4.16

$ ls /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_m4/work/
.CC_PRINT_OPTIONS   .macports.m4.state  m4-1.4.15/  

So basically it appears to be skipping the fetch phase because 1.4.15 had already been fetched (and unpacked), then trying to go ahead and build 1.4.16, which of course fails. Less than ideal, wouldn't you agree?

10.6.7, xcode 3.2.6 Thanks. -r

[ryandesign (Ryan Carsten Schmidt)]

Replying to rogerdpack@…:

Less than ideal, wouldn't you agree?

Of course, but I'm not seeing how it was possible for you to get into this situation. If, between the first and second attempt, you ran "sudo port sync", and the version of m4 was changed by this sync, then the m4 portfile was updated. Next time you tried to build m4, MacPorts would have noticed the portfile was newer than your build attempt, and thrown out the previous build attempt. So I cannot explain why it did not do so.

To fix the problem, clean m4 and try again ("sudo port clean m4").

[rogerdpack@…]

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I believe I did do a port selfupdate between installs, but that's it.

-r

[ryandesign (Ryan Carsten Schmidt)]

Replying to rogerdpack@…:

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I'm telling you that MacPorts already does this, by comparing the date the portfile was modified with the date the work directory was created.

I believe I did do a port selfupdate between installs, but that's it.

selfupdate includes sync.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

Replying to rogerdpack@…:

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I'm telling you that MacPorts already does this, by comparing the date the portfile was modified with the date the work directory was created.

Unfortunately, I've just realized at least one way this problem can occur, and have been able to reproduce a failure like yours on my own system.

It hinges on the following sequence of events (using m4 as the example port, though it would apply to any port):

1. At time T1, you sync or selfupdate, getting a current set of ports (for example including m4 1.4.15)
2. At time T2, we update a port (for example we update m4 to 1.4.16)
3. At time T3, without selfupdating or syncing again, you begin installing e.g. m4 (1.4.15) and cancel the installation before it finishes, or it stops due to an error
4. At time T4, you sync or selfupdate, which downloads our current set of portfiles, preserving the modification date/time of those changes. Now you have the new m4 1.4.16 Portfile, but its modification date/time is T2, whereas the creation date of your m4 port's work directory is T3
5. At time T5, you try installing m4 again, which, since the work directory seems to be newer than the portfile, is retained instead of being cleaned up

We're going to have to think about the proper way of fixing this.

[ryandesign (Ryan Carsten Schmidt)]

I looked at the code and it's actually using the mtime of the statefile, and not looking at the work directory itself, but that doesn't change the point I'm making.

One could say that the problem is that we preserve the mtime when we sync. When using rsync (the default for MacPorts), it's the -t flag that preserves mtime; the default rsync_options in macports.conf do include -t (they're -rtzv --delete-after). If the user has configured to sync with Subversion instead, Subversion by default does not preserve mtime, but the user can edit the Subversion configuration file to make it do so. (See ​use-commit-times in the Subversion book.)

One solution would therefore be to not preserve mtimes when we sync, by removing -t from the rsync_options, and for Subversion, by enforcing a config file that doesn't specify use-commit-times. However, I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What we really want to know, in the case of resuming an interrupted install, is: is the portfile we're now using newer than the portfile we were using when we started? In order to answer that question, we need to keep track of the mtime of the portfile we were using when we started. I'd put that in a line in the statefile. Then we can check if the mtime of the portfile is newer (or, heck: is different at all -- newer or older) than the mtime previously recorded in the statefile; if so, delete the work directory and start over.

[jmroot (Joshua Root)]

Should probably just put the version and revision in the statefile as well.

[ryandesign (Ryan Carsten Schmidt)]

Even better than mtimes, we could store / compare checksums (sha1?)

What would having the version or revision in the statefile do for us?

[raimue (Rainer Müller)]

Replying to ryandesign@…:

One solution would therefore be to not preserve mtimes when we sync, by removing -t from the rsync_options, and for Subversion, by enforcing a config file that doesn't specify use-commit-times. However, I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What other reasons do you refer to? We would still have the $Id$ line indicating the time of commit at the top.

[jmroot (Joshua Root)]

Replying to ryandesign@…:

What would having the version or revision in the statefile do for us?

It would tell us when we're building a different version than what's in the existing work dir.

[ryandesign (Ryan Carsten Schmidt)]

Replying to raimue@…:

Replying to ryandesign@…:

I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What other reasons do you refer to? We would still have the $Id$ line indicating the time of commit at the top.

I just feel that it's a non-pointless piece of metadata and it would be nice for us to not destroy it. It is nice to be able to look at a file in a file browser and see when it was last modified, and to know that it was actually modified by someone at that time, and not merely downloaded from the rsync server.

The $Id$ line exists for Portfiles, but not necessarily for other files, like those in the files directory. Portfiles don't necessarily have the Id line either (or the svn:keywords property that keeps it updated), though they should.

Conversely, while I support preserving mtimes, they are also unreliable and might be destroyed by other processes (e.g. some kinds of backups...). Therefore it would be great not to rely on them.

So while I wanted to begin by pointing out that the way we're handling mtimes now is not sufficient for identifying a changed file, it is not necessary either; a checksum is the correct way to solve this problem. I don't see disadvantage to using a checksum; if you see any, please point them out.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmr@…:

Replying to ryandesign@…:

What would having the version or revision in the statefile do for us?

It would tell us when we're building a different version than what's in the existing work dir.

Right, but the message we print is "Portfile changed since last build; discarding previous state." That is the condition we want to capture -- that the portfile has changed -- and I think using a checksum is the simplest way to check for that. We don't need to care whether the portfile's mtime or version or revision or anything else specifically in it has changed; it doesn't matter.

[ryandesign (Ryan Carsten Schmidt)]

#29741 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#25864 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#15517 might have been caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#28088 might have been caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#16196 might have been caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#15476 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#4598 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#19838 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#20159 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#29508 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#27821 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#26668 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#24780 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

comment:ticket:25872:5 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

The last portion of the description of #16785 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

comment:ticket:29805:6 was probably caused by this problem.

[nerdling (Jeremy Lavergne)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

comment:ticket:30056:2 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#29014 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#29843 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#30484 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#30606 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#30789 may have been caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#28699 may have been caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#31532 was probably caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#31258 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#31868 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#32255 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#32349 was caused by this problem.

[drkp (Dan Ports)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

comment:ticket:31652:3 was probably caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

​This post on the mailing list was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#33579 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#33582 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#33605 was caused by this problem.

[nonstop.server@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

comment:ticket:34292:3 and the original error message reported in #34291 were caused by this problem.

[nerdling (Jeremy Lavergne)]

Replying to ryandesign@…:

Of course, but I'm not seeing how it was possible for you to get into this situation. If, between the first and second attempt, you ran "sudo port sync", and the version of m4 was changed by this sync, then the m4 portfile was updated. Next time you tried to build m4, MacPorts would have noticed the portfile was newer than your build attempt, and thrown out the previous build attempt. So I cannot explain why it did not do so.

Does this problem manifest only when using the ports tarball and not Portfiles? I'm wondering if the timestamps only get checked properly in one case and not the other.

[ryandesign (Ryan Carsten Schmidt)]

To my knowledge, it doesn't matter how you got the portfile. As discussed above, the solution is to remove the timestamp checking, and to use checksum checking instead. I haven't looked into where in the code we'd need to do that exactly. But as the history of this ticket hopefully shows, fixing this issue would save us many tickets and much user time and aggravation.

[ryandesign (Ryan Carsten Schmidt)]

Looks like #34410 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#34428 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#34446 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#34557 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#34936 may have been caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35189 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35280 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35289 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35315 was caused by this problem.

[ryandesign (Ryan Carsten Schmidt)]

#29696 was probably caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35373 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35376 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35446 was caused by this issue.

[neverpanic (Clemens Lang)]

r96136.

[ryandesign (Ryan Carsten Schmidt)]

#35558 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#35578 was caused by this issue.

[neverpanic (Clemens Lang)]

Dropping support for the old statefile format would cause some undesired effects in the transition phase: Archives previously built (or downloaded from the buildbot) still contain an old statefile and would thus not get used. Change r96399 fixes this by re-introducing support for reading older statefiles (where it uses the old mechanism to check whether the partial build is outdated).

r96400 contains a number of testcases to verify correct behavior of the statefile handling.

[neverpanic (Clemens Lang)]

Merged into release_2_1 in r96466.

[neverpanic (Clemens Lang)]

The current state of the statefile handling is still buggy:

• port fetch $someport, followed by
• port build $someport +variant

doesn't correctly restart fetch phase. This is because the variants check relied on the statefile being empty when no variants have been selected and implicitly assuming the empty variant was selected if it is not. Starting fetch without specifying a variant doesn't write a "variant: " line into the statefile, which is what the code currently checks for.

So, either, we always write a "variant: " line and ignore it when checking the selected variants, or we assume variants to be "" as soon as there's a "target: " line in the statefile.

[neverpanic (Clemens Lang)]

This should be fixed in r96538 by assuming variants to be "" if there's a line matching "target: .*" in the statefile but no lines matching "variant: .*" have been found.

ChangeLog updated in r96539, merged into release_2_1 in 96540.

[ryandesign (Ryan Carsten Schmidt)]

#36012 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36136 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36300 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36309 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36407 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36436 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#36471 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#34949 seems to have been caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#37243 was caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#37760 was caused by this issue.

[cooljeanius (Eric Gallager)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

#38246 seems to have been caused by this issue.

[ryandesign (Ryan Carsten Schmidt)]

#38860 seems to have been caused by this issue.