py-bob @2.0.7 New release

[tiagofrepereira2012 (Tiago Freitas Pereira)]

Hi,

Follow attached a set of diff files for the bob@2.0.7 release. These diffs will update the py-bob port and the following related ports

• py-bob-core
• py-bob-learn-em
• py-bob-io-video

Thanks for the publication

[dbevans (David B. Evans)]

After applying patches, I did a test build on 10.10 (yosemite). py-bob-core and py-bob-io-video built without error but py-bob-learn-em failed during setup.py with a segmentation fault as follows

:notice:build --->  Building py27-bob-learn-em
:debug:build Executing proc-pre-org.macports.build-build-0
:debug:build Executing org.macports.build (py27-bob-learn-em)
:debug:build Environment:
CC='/usr/bin/clang'
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_macports_trunk_dports_python_py-bob-learn-em/py27-bob-learn-em/work/.CC_PRINT_OPTIONS'
CFLAGS='-arch x86_64'
CPATH='/opt/local/include'
CXX='/usr/bin/clang++'
CXXFLAGS='-arch x86_64'
F90FLAGS='-m64'
FCFLAGS='-m64'
FFLAGS='-m64'
LDFLAGS='-arch x86_64'
LIBRARY_PATH='/opt/local/lib'
MACOSX_DEPLOYMENT_TARGET='10.10'
OBJC='/usr/bin/clang'
OBJCFLAGS='-arch x86_64'
:debug:build Assembled command: 'cd "/opt/local/var/macports/build/_opt_macports_trunk_dports_python_py-bob-learn-em/py27-bob-learn-em/work/bob.learn.em-2.0.7" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build'
:debug:build Executing command line:  cd "/opt/local/var/macports/build/_opt_macports_trunk_dports_python_py-bob-learn-em/py27-bob-learn-em/work/bob.learn.em-2.0.7" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:info:build sh: line 1:  2999 Segmentation fault: 11  /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_macports_trunk_dports_python_py-bob-learn-em/py27-bob-learn-em/work/bob.learn.em-2.0.7" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:info:build Exit code: 139
:error:build org.macports.build for port py27-bob-learn-em returned: command execution failed

Not sure what's going on here so I'm holding off committing these updates until I hear more from you. Build log attached.

[dbevans (David B. Evans)]

Build log showing build failure on Yosemite

[andre.dos.anjos@…]

The output is not very enlightening... We're building our nightlies on 10.9 and 10.11, which seems to work fine.

Could you confirm this is specific to 10.10? If so, does that relate to your previous patch?

[tiagofrepereira2012 (Tiago Freitas Pereira)]

Dear Dave,

Indeed, the log doesn't say much. I will have a look, but the build and tests are running fine with our Yosemite.

Thanks for the report

Tiago

[dbevans (David B. Evans)]

Will test further here and report. Should these updates be committed as a group or can I go ahead and commit them individually?

[tiagofrepereira2012 (Tiago Freitas Pereira)]

Yes, go ahead.

Thanks

[dbevans (David B. Evans)]

Some more information. I tried building on 10.11 and got the same error as on 10.10. So I don't believethe problem is OS version specific. I also believe it is unrelated to the previous patch (boost 0.59 issue) as that only comes into play when libstdc++ is being used. That only occurs on 10.8 and earlier so not an issue here.

I ran setup.py, as it is invoked by py27-bob-learn-em, in pdb. Here's the last few lines before the segfault:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bob/extension/__init__.py", line 291, in __init__
    bob_includes, bob_libraries, bob_library_dirs = get_bob_libraries(self.bob_packages)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bob/extension/__init__.py", line 185, in get_bob_libraries
    pkg = importlib.import_module(package)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bob/learn/activation/__init__.py", line 8, in <module>
    from ._library import *
ImportError: bob.core._logging import error: you compiled against API version 0x200, but are now importing an API with version 0x201 which is not compatible - check your Python runtime environment for errors
Uncaught exception. Entering post mortem debugging

The module being imported is bob.learn.activation.

What am I doing wrong?

[dbevans (David B. Evans)]

Given the evidence of modules built with mismatched API version, I uninstalled all py-bob* ports and rebuilt everything from scratch. This resulted in a successful build. The implication is that even though only a few modules are being updated, if a change in API version is involved, all modules need to be rebuilt. The MacPorts way to do this is to increment the revision number for all ports whose version does not change. Otherwise, ports that were built with the previous API version and are not being updated will not be rebuilt and can possibly cause this type of failure.

[andre.dos.anjos@…]

Hello Dave,

Indeed, this commit has changed the API version for bob.core on Nov 28th last year:

​https://github.com/bioidiap/bob.core/commit/57dd5a01c451c7c90d51a3f0c77138f3a7bc4488

Version 2.0.5 of bob.core was shipped with API 0x200 (Nov. 26th). Version 2.0.6 was shipped with API 0x201.

According to our own rules, we should have shipped version bob.core-2.1.0 instead of 2.0.6. We'll fix it.

Meanwhile, I'm not sure I understand the issue: if package B depends on package A and package A is version upgraded, shouldn't package B be re-compiled? How to mark API changes that may require recompilation of dependencies?

Best, A

[dbevans (David B. Evans)]

MacPorts only rebuilds a port if there is a change of version and/or revision number. So in your example package B (whose version number hasn't changed) needs to have it's revision incremented to force the necessary rebuild with package A.

I've taken the liberty of making these changes to the remaining py-bob ports and tested (successfully) overnight on 10.11, 10.10 and 10.9. Revised patch for the 2.0.6 update attached. I'll go ahead and commit this, if you agree.

[andre.dos.anjos@…]

Hello Dave, yes please go ahead with the fix, increasing the revision version of the package. Thanks a lot for your work and this help!

Nevertheless, I should note it is quite unintuitive that MacPorts does not compute dependencies when deciding for a rebuild. Do you happen to understand the reason behind this design decision? Let me develop this:

Well, my understanding is MacPorts is a rolling revision system, so it is always being updated. At a certain point, there will be API/ABI incompatibilities so are those handled by resetting the revisions of all possible dependencies?

If package A is bob.core, we're still talking about tens of packages. If it is boost, then it is probably on the hundreds. If it is libc++ or libgcc++, this is an enormous set to bump revision numbers, right?

Best, André

[dbevans (David B. Evans)]

OK, André, I've committed the updates in r144969 and the ports are in the process of rebuilding on our buildbots. Looks like things are going well but issues remain on 10.8, 10.6 and presumeably 10.7. I'll update the various tickets after the builds complete.

Concerning MacPorts' rebuild policy, everything you say is correct. I believe the decision to require the port maintainer to decide when to rebuild a port stems from a desire to minimize the number of rebuilds to just those that are absolutely necessary. Until rather recently, doing a rebuild meant that every MacPorts user would have to do the rebuild on his own machine. This is still the case for 10.11 since we don't have a buildbot for that version commissioned as yet. The problem is now mitigated to some degree by our ability to provide prebuilt binaries for the various ports where licensing allows but we do receive reports periodically concerning the need to constantly rebuild.

Experience has shown, that, more often than not, the fact that a port has been updated does not automatically mean that its dependents need to be rebuilt. This is generally necessary only when there are ABI/API changes or when dependencies themselves change. In addition, you need to consider that we are dealing with a great number of ports from a great number of upstream developers who, by large, do not agree on when versioning changes indicate the need for a rebuild. Some ports almost never need their dependencies updated on version change (glib2 for instance) while others change their binary versioning on every minor update (poppler).

Thus, rather than trying to come up with an intricate algorithm for automating this problem (and probably getting it wrong), the current policy is for each port's maintainer to understand his port and the other ports that depend upon it and manually force a rebuild when necessary by incrementing the revision number.

I should point out that the port program itself maintains a local database of binary versions on a file by file basis and checks for binary mismatches at the end each build. The default behavior is to force a rebuild if a mismatch exists. Of course, this is not always the correct action and the behavior can be configured to just report if desired.

Hope this helps clarify the issue for you. If not, please feel free to contact me directly or to raise a discussion on the developer's email list.

Thanks for your update. Sorry it took so long to complete.

Dave

[andre.dos.anjos@…]

Hello Dave, thanks a lot for the update and explanations. We will try to be attentive on this fact for future releases. Best, A