Opened 4 years ago

Closed 4 years ago

#61256 closed enhancement (fixed)

Multicore support for math/R

Reported by: webbp (Webb Phillips) Owned by: kjellpk (Kjell Konis)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: i0ntempest, chrstphrchvz (Christopher Chavez)
Port: R

Description

Currently, math/R is built (by default) using Apple's clang. This causes both R itself and any packages installed by R to be single-core because Apple clang doesn't support OpenMP. This very much sucks and is ultimately Apple's fault. However, there is an easy solution, which is to build R with Macports clang. More specifically, when I install macports clang-10 llvm-10 and libomp, port select clang mp-clang-10 and llvm mp-llvm-10, and macports PATH is prefixed to my PATH, sudo port install R installs a multicore R capable of building multicore R packages. Anecdotally, this R on my 9-year-old MacBook Air is faster for many common operations than the default Macports R on my 2020 MacBook Air.

The vast majority of users will not know how to do this. Therefore, please update the default R build to build a multicore-capable R.

Change History (12)

comment:1 Changed 4 years ago by mf2k (Frank Schima)

Note that a "request" ticket type is only for requesting a new port.

In the future, please add the port maintainer(s) to Cc (port info --maintainers R), if any.

comment:2 Changed 4 years ago by mf2k (Frank Schima)

Cc: i0ntempest added
Owner: set to kjellpk
Port: R added; math/R removed
Status: newassigned
Type: requestenhancement

comment:3 Changed 4 years ago by i0ntempest

How do I know if R on my system has multicore support? If it's that easy as you said, I might just blacklist apple clang.

comment:4 Changed 4 years ago by webbp (Webb Phillips)

My description of the solution isn't quite right. The way I solved this was with:

~/.R/Makevars:
LLVM_LOC = /opt/local/libexec/llvm-10
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/opt/local/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I/opt/local/include -I$(LLVM_LOC)/include

That alone on my new machine breaks installing R packages, so there are some missing steps, and some of the other steps I took above may have been irrelevant.

comment:5 Changed 4 years ago by i0ntempest

Are those just for compiling R packages? I'm not very experienced in manipulating complex builds (like R) yet, so can you try building R with multicore support on your new machine and tell me exactly what change did you make? Then I could try putting them into the portfile.

comment:6 Changed 4 years ago by chrstphrchvz (Christopher Chavez)

compiler.openmp_version should be used to specify OpenMP requirement; an OpenMP-capable compiler will be chosen and libomp dependency added automatically. See CompilerSelection#GettingtheRightCompilerinthePortfile

comment:7 Changed 4 years ago by chrstphrchvz (Christopher Chavez)

Cc: chrstphrchvz added

comment:8 Changed 4 years ago by webbp (Webb Phillips)

compiler.openmp_version 4.5 did not enable R to build packages with OpenMP, e.g., with compiler.openmp_version 4.5:

clang: error: unknown argument: '-pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64'
*** OpenMP not supported! data.table uses OpenMP to automatically
***   parallelize operations like sorting, grouping, file reading, etc.

I've investigated the problem a bit more. The issue mostly or entirely affects packages installed by R rather than the R binary itself, that is, there are many R packages which benefit greatly from being built with openmp, but the R binary may not. The above .R/Makevars cause R to build packages with OpenMP in Mac OS 10.11, whereas these below work for 10.15. (Those are the only two Mac OS versions I have readily available.)

CC=/opt/local/bin/clang -fopenmp
CXX=/opt/local/bin/clang++ -fopenmp
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/opt/local/lib -Wl,-rpath,/opt/local/lib
CPPFLAGS=-I/opt/local/include

Source: https://github.com/Rdatatable/data.table/wiki/Installation

Also, I believe that R uses the flags with which it was built as the default flags for building packages, so specifying these flags the Portfile should solve the problem. However, the specific flags that work differ for different Mac OS versions, and they are also known to be incompatible with some R packages. So maybe the best thing to do here is just to add some lines to the Portfile long_description, e.g.: to install multicore packages in R, do sudo port install clang-10 pkgconfig && sudo port select clang mp-clang-10 and create ~/.R/Makevars as above.

comment:9 Changed 4 years ago by i0ntempest

It's working for me:

* installing *source* package ‘data.table’ ...
** package ‘data.table’ successfully unpacked and MD5 sums checked
** using staged installation
zlib 1.2.11 is available ok
OpenMP supported
** libs
/opt/local/bin/clang-mp-10 -fopenmp -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/local/include  -fopenmp -fPIC  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
In file included from assign.c:1:
In file included from ./data.table.h:13:
In file included from ./myomp.h:2:
/opt/local/include/libomp/omp.h:53:9: warning: ISO C restricts enumerator values to range of 'int' (2147483648 is too large) [-Wpedantic]
        omp_sched_monotonic = 0x80000000
        ^                     ~~~~~~~~~~
...

My Makevar file is same as the one you have except I explicitly specified /opt/local/bin/clang-mp-10 and /opt/local/bin/clang++-mp-10. I've added openmp in as a variant, do you wanna try my portfile?

comment:10 Changed 4 years ago by i0ntempest

Also I just tried with the default macports R, it can build openmp packages despite itself is not built explicitly with openmp support (maybe it's automatically enabled). But it's still better to add a variant since it'll ensure one has the correct compiler and openmp library installed.

comment:12 Changed 4 years ago by i0ntempest

Resolution: fixed
Status: assignedclosed

In 71899c63ad7c5953b9313aeca35bd7aa44c019c9/macports-ports (master):

R: Add openmp variant & change openblas variant to lowercase (https://github.com/macports/macports-ports/pull/8614)

This new variant makes sure the user has OpenMP library and a OpenMP supported compiler installed to build and use OpenMP enabled R packages. OpenMP support is now explicitly disabled before build if the variant is not selected to avoid undeclared dependency.
Closes: #61256

Note: See TracTickets for help on using tickets.