openmpi/mpich: prune target compiler list; refactor duplicated logic, and simplify ports, via new portgroup

[mascguy (Christopher Nielsen)]

I envision this being a very iterative process, with the initial portgroup being something as simple as a few procs to eliminate copy-paste logic.

Also prune the target compiler list significantly, whilst still supporting MacOS releases back to 10.6.

[mascguy (Christopher Nielsen)]

The first two candidates are as follows. I'm not married to the proc names, but the point is to eliminate copy-paste logic:

proc mpi_add_compiler_depends_lib {cname} {
    set cport_name ""

    # As we are making wrappers, we depend on the compilers to exist.
    # Add them to depends_lib, not just depends_build.
    if {[regexp {clang[3-9]\d} ${cname}] == 1} {
        # Ports for Clang versions < 10 are named: clang-<major>.<minor>
        set cport_name          [regsub {(\d)(\d)} ${cname} {-\1.\2}]
    } elseif {[regexp {clang\d\d} ${cname}] == 1} {
        # Ports for Clang version >= 10 are named: clang-<major><minor>
        set cport_name          [regsub {(\d)(\d)} ${cname} {-\1\2}]
    } elseif {([regexp {gcc\d} ${cname}] == 1) || ([regexp {gcc\d\d} ${cname}] == 1)} {
        # Ports for GCC have names exactly matching our subports, so use as-is
        set cport_name          ${cname}
    }

    if {${cport_name} ne ""} {
        ui_debug "mpi_add_compiler_depends_lib: Adding compiler to depends_lib: ${cport_name}"
        depends_lib-append      port:${cport_name}
    } else {
        ui_debug "mpi_add_compiler_depends_lib: Not adding compiler to depends_lib; cname: ${cname}"
    }

    return 0
}

proc mpi_set_binary_eligibility {subport cname} {
    if {[lsearch -exact {mp llvm clang} ${cname}] != -1} {
        # Force local builds with Xcode-provided compilers (avoid issues with
        # different Xcode versions on buildbot and user machines)
        ui_debug "mpi_set_binary_eligibility: Disabling binary use for subport: ${subport}"
        archive_sites
    } else {
        ui_debug "mpi_set_binary_eligibility: Not disabling binary use for subport: ${subport}"
    }

    return 0
}

[mascguy (Christopher Nielsen)]

As for other shared logic, the two ports have so many differences - some subtle, and many not-so-subtle - that we might be limited in the refactoring effort.

Although the clist handling - including generation of subports, logic to exclude/fail based on compiler version and macOS release, etc - would be nice to refactor out.

If both ports supported exactly the same set of target compilers, it would be easier...

[eborisch (Eric A. Borisch)]

I'm all for them targeting the same set of compilers; what's the current delta?

Another thought: I've been toying with the idea of having all the non-default (-clang*, etc) flavors use the default libraries and headers rather than build their own, with a defaulted-on +wrapperonly variant.

In this mode, we would just make copies of mpicc, mpicxx, etc. with the appropriate changes to CC/CXX etc. in the script (and continue to depend upon the compilers to make sure they are installed.) Executables like mpiexec could just be a symlink to the -default mpiexec (likely easier to maintain than having a separate set of select files for the variants).

It's typically rare interested in seeing how the different compilers do at generating the MPI libraries themselves, and you're more interested in using the features of a particular compiler for your (or another port's) code. This would drastically cut down the overhead for having multiple flavors of the wrappers around, while still leaving the options of disabling the variant if you want a full build of MPI libraries by that compiler.

It looks like openmpi uses a binary mpicc (symlinked to a binary, actually), so I'm not sure how easy this would be to implement on openmpi, but it should be fairly easy on mpich.

[mascguy (Christopher Nielsen)]

This is the current list we support:

openmpi-clang                  @4.1.1          science/openmpi
openmpi-clang10                @4.1.1          science/openmpi
openmpi-clang11                @4.1.1          science/openmpi
openmpi-clang33                @4.1.1          science/openmpi
openmpi-clang34                @4.1.1          science/openmpi
openmpi-clang37                @4.1.1          science/openmpi
openmpi-clang50                @4.1.1          science/openmpi
openmpi-clang60                @4.1.1          science/openmpi
openmpi-clang70                @4.1.1          science/openmpi
openmpi-clang80                @4.1.1          science/openmpi
openmpi-clang90                @4.1.1          science/openmpi
openmpi-default                @4.1.1          science/openmpi
openmpi-gcc5                   @4.1.1          science/openmpi
openmpi-gcc6                   @4.1.1          science/openmpi
openmpi-gcc7                   @4.1.1          science/openmpi
openmpi-gcc8                   @4.1.1          science/openmpi
openmpi-gcc9                   @4.1.1          science/openmpi
openmpi-gcc10                  @4.1.1          science/openmpi
openmpi-gcc49                  @4.1.1          science/openmpi
mpich-clang                    @3.4.1          science/mpich
mpich-clang10                  @3.4.1          science/mpich
mpich-clang11                  @3.4.1          science/mpich
mpich-clang50                  @3.4.1          science/mpich
mpich-clang60                  @3.4.1          science/mpich
mpich-clang70                  @3.4.1          science/mpich
mpich-clang80                  @3.4.1          science/mpich
mpich-clang90                  @3.4.1          science/mpich
mpich-default                  @3.4.1          science/mpich
mpich-gcc5                     @3.4.1          science/mpich
mpich-gcc6                     @3.4.1          science/mpich
mpich-gcc7                     @3.4.1          science/mpich
mpich-gcc8                     @3.4.1          science/mpich
mpich-gcc9                     @3.4.1          science/mpich
mpich-gcc10                    @3.4.1          science/mpich
mpich-gcc49                    @3.4.1          science/mpich
mpich-gccdevel                 @3.4.1          science/mpich

[mascguy (Christopher Nielsen)]

Per the earlier discussion on the macports-users list, should we remove all GCC versions earlier than 7?

As for Clang, what's the oldest version we need to support?

[kencu (Ken)]

I am not aware of any reason to keep support for gcc older than gcc7 in the mpi family.

Likewise, anyone using a macports-clang probably wants at least macports-clang-9.0 by this point, but I don't know if there might be some niche user with an old code base that uses some older macports-clang for something.

maybe deprecate them in some way? Or just bin them?

[kencu (Ken)]

we do need to keep the older gcc versions for other purposes, and probably the older clangs too for similar reasons (bootstrapping or certain older code that needs it and can't/won't be updated).

[mascguy (Christopher Nielsen)]

Replying to kencu:

we do need to keep the older gcc versions for other purposes, and probably the older clangs too for similar reasons (bootstrapping or certain older code that needs it and can't/won't be updated).

Ken, are you referring to MacPorts as a whole, or just these two ports? I ask, as I'm speaking just for the latter...

[cjones051073 (Chris Jones)]

yes, we are not suggesting removing the primary clang/gcc ports here Ken, just the mpich/openmpi versions.

[kencu (Ken)]

exactly

[mascguy (Christopher Nielsen)]

So, would the following set of compilers be sufficient?

• Clang 9 - 11
• GCC 7 - 10

[mascguy (Christopher Nielsen)]

(Plus the host/Xcode versions, aka, variations "-clang" and "-default")

[mascguy (Christopher Nielsen)]

Replying to eborisch:

Another thought: I've been toying with the idea of having all the non-default (-clang*, etc) flavors use the default libraries and headers rather than build their own, with a defaulted-on +wrapperonly variant.

In this mode, we would just make copies of mpicc, mpicxx, etc. with the appropriate changes to CC/CXX etc. in the script (and continue to depend upon the compilers to make sure they are installed.) Executables like mpiexec could just be a symlink to the -default mpiexec (likely easier to maintain than having a separate set of select files for the variants).

It's typically rare interested in seeing how the different compilers do at generating the MPI libraries themselves, and you're more interested in using the features of a particular compiler for your (or another port's) code. This would drastically cut down the overhead for having multiple flavors of the wrappers around, while still leaving the options of disabling the variant if you want a full build of MPI libraries by that compiler.

It looks like openmpi uses a binary mpicc (symlinked to a binary, actually), so I'm not sure how easy this would be to implement on openmpi, but it should be fairly easy on mpich.

Interesting idea!

Since the default flavor doesn't utilize the buildbot binaries, does that mean an install would always cause a build from source (at least for the default libs)?

[mascguy (Christopher Nielsen)]

Of note, we also need to add subports for GCC 11 to both openmpi and mpich. While not a top priority, we'll try to do that as part of this effort, if possible.

[Christopher Nielsen <mascguy@…>]

In f13f6ed289d17e27b4421f704cdc2086cd1a0e86/macports-ports (master):

openmpi: add gcc11 subport

See: #62807

[Christopher Nielsen <mascguy@…>]

In 5cf91d1ecf9dbb3d04615960418388f25ee73374/macports-ports (master):

mpich: add subport for gcc11

See: #62807

[Christopher Nielsen <mascguy@…>]

In 1648a218814369dd892a9a23e45116e9a2879df9/macports-ports (master):

mpich: disable gcc11 for now, as builds fail across-the-board

See: #62807

[mascguy (Christopher Nielsen)]

In terms of pruning the target compiler list, my latest thought is that we go with the following.

For support of older MacOS releases, back to 10.6:

• gcc7
• clang90

Otherwise, support the two latest compiler releases:

• gcc10
• gcc11
• clang10
• clang11

Once clang12 is available - and our MacPorts version has stabilized in terms of patches, etc - it'll be added. At that point, the clock starts, and we then obsolete clang10 after 12 months.

If we can limit our current list of MacPorts-provided compilers to six, that will make these ports much easier to test and maintain. Whilst still supporting users on older MacOS releases.

Thoughts?

[mascguy (Christopher Nielsen)]

Sadly, it looks like the subports for gcc9 and later are failing, on MacOS 10.7 and earlier. The AVX instruction use issue is resolved, but now other Intel extensions are being balked at.

It might be possible to fix this - maybe - but it's probably easier to simply set those to known_fail. Users still have options like gcc7 and clang90, among others, which do work with those older MacOS releases.

[kencu (Ken)]

another good option is to just leave it until someone who wants it fixed fixes it (usually me).

If you mark it as "known_fail" without really knowing if it's fixable, then that is often it, nobody will ever look again, assuming that someone who knew what they were doing has concluded it is unfixable.

IMHO.

[mascguy (Christopher Nielsen)]

Replying to kencu:

another good option is to just leave it until someone who wants it fixed fixes it (usually me).

If you mark it as "known_fail" without really knowing if it's fixable, then that is often it, nobody will ever look again, assuming that someone who knew what they were doing has concluded it is unfixable.

Well, you're welcome to help fix these on 10.7. But ultimately users still have options like gcc7.

[Christopher Nielsen <mascguy@…>]

In 853f6645e607df0420bbda32af4da22ffeb34867/macports-ports (master):

openmpi/mpich: prune target compiler list

See: #62807

[Christopher Nielsen <mascguy@…>]

In bb7a8976f9e3887c5de7b889d3e3e351860731e2/macports-ports (master):

openmpi/mpich: disable clang9 and clang10 for ARM

See: #62807

[Christopher Nielsen <mascguy@…>]

In ef73316c06040ba68fc0b43d3a5701894d648c96/macports-ports (master):

mpich: disable gcc11 across-the-board

See: #62807

[mascguy (Christopher Nielsen)]

FYI, the effort to prune the target compiler list for openmpi/mpich is complete.

For now, all retired subports are still defined, albeit marked as obsolete/replaced_by. Files related to port select functionality for retired subports still exist, and will remain until the subports are completely removed. (Removal is slated for early-to-mid 2022.)

Logic related to subport creation and such is now as similar as possible between openmpi/mpich, supporting further refactoring via portgroup mpiutil. And now that pruning is finished, I'll start looking at the refactoring effort.

[Christopher Nielsen <mascguy@…>]

In 1b8e9a45e286a0fc0b00375de771da7234ee4a8c/macports-ports (master):

openmpi/mpich: refactor subport creation and validation to pg 'mpiutil'

See: #62807

[mascguy (Christopher Nielsen)]

FYI, refactoring is mostly complete. While there is still some copy-pasta between openmpi/mpich, most of the duplicated logic related to subport creation, validation, etc, has been refactored to pg 'mpiutil'.

Perhaps more can be done, but I'm much happier with the result.

[Christopher Nielsen <mascguy@…>]

In b6b39336e3250b51d5ca88921de607a53fe12910/macports-ports (master):

openmpi: cleanup: remove defunct port-select files

See: #62807

[mascguy (Christopher Nielsen)]

A significant amount of refactoring and cleanup has been done for openmpi/mpich over the past few days, so let's call this done.

If anyone has any additional thoughts/ideas, let me know; that can feed into a Phase 2 effort.