nco/ncks and netcdf

[basmac]

The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files.

The linux setup is

/usr/local/bin/ncks -r
NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root
ncks version 4.4.5
Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option:	Active?	Meaning or Reference:
Check _FillValue	Yes	http://nco.sf.net/nco.html#mss_val
Check missing_value	No	http://nco.sf.net/nco.html#mss_val
DAP clients		Yes	http://nco.sf.net/nco.html#dap
Debugging: Custom	No	Pedantic, bounds checking (slowest execution)
Debugging: Symbols	No	Produce symbols for debuggers (e.g., dbx, gdb)
GNU Scientific Library	Yes	http://nco.sf.net/nco.html#gsl
HDF4 support		Unknown	http://nco.sf.net/nco.html#hdf4
Internationalization	No	http://nco.sf.net/nco.html#i18n (pre-alpha)
MPI parallelization	No	http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files	Yes	http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available	Yes	http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled	Yes	http://nco.sf.net/nco.html#nco4
OpenMP SMP threading	No	http://nco.sf.net/nco.html#omp
Optimization: run-time	No	Fastest execution possible (slowest compilation)
Parallel netCDF3	No	http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
Regular Expressions	Yes	http://nco.sf.net/nco.html#rx
Shared libraries built	No	Small, dynamically linked executables
Static libraries built	Yes	Large executables with private namespaces
UDUnits conversions	Yes	http://nco.sf.net/nco.html#udunits
UDUnits2 conversions	Yes	http://nco.sf.net/nco.html#udunits

The macports is

ncks -r
NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan  6 2015 on mac73.psd.esrl.noaa.gov by root
ncks version 4.4.7
Linked to netCDF library version 4.3.2, compiled Feb  3 2015 08:18:38
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option:	Active?	Meaning or Reference:
Check _FillValue	Yes	http://nco.sf.net/nco.html#mss_val
Check missing_value	No	http://nco.sf.net/nco.html#mss_val
DAP clients		Yes	http://nco.sf.net/nco.html#dap
Debugging: Custom	No	Pedantic, bounds checking (slowest execution)
Debugging: Symbols	No	Produce symbols for debuggers (e.g., dbx, gdb)
GNU Scientific Library	Yes	http://nco.sf.net/nco.html#gsl
HDF4 support		Unknown	http://nco.sf.net/nco.html#hdf4
Internationalization	No	http://nco.sf.net/nco.html#i18n (pre-alpha)
MPI parallelization	No	http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files	Yes	http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available	Yes	http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled	Yes	http://nco.sf.net/nco.html#nco4
OpenMP SMP threading	No	http://nco.sf.net/nco.html#omp
Optimization: run-time	No	Fastest execution possible (slowest compilation)
Parallel netCDF3	No	http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
Regular Expressions	Yes	http://nco.sf.net/nco.html#rx
Shared libraries built	No	Small, dynamically linked executables
Static libraries built	Yes	Large executables with private namespaces
UDUnits conversions	Yes	http://nco.sf.net/nco.html#udunits
UDUnits2 conversions	Yes	http://nco.sf.net/nco.html#udunits

The command is

ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite
huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
/Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc 

The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect.

[tenomoto (Takeshi Enomoto)]

netcdf has been updated to 4.3.3.1 in r133622 (See #47040). nco has been updated to 4.4.8 in r133623. Do these updates help in anyway?

[basmac]

Thanks for the updates - after many tests on linux and macs it seems the chunking behavior since 4.4.5 is causing the problem. To get it to convert in 10 seconds in 4.4.7 and 4.4.8 we have to add --cnk_plc=r1d otherwise ncks runs for hours. In 4.4.5 it runs as expected without the extra parameter. We have logged it with no.

Thanks for your help and the updates, which can be used now.

[Dave-Allured (Dave Allured)]

The reporter posted a working solution. Also, NCO and related dependencies have since been updated many times, with no further reports like this. Please close this ticket as resolved.

[Dave-Allured (Dave Allured)]

Thanks, Ryan. For the record, and loosely speaking, chunking controls were mysterious and awkward in the early days. This took some study and experimentation on users' part to get it right. Also the "chunk policy" controls and advice have evolved since then. This here looks like a case of incorrect user setting after new feature, rather than version defect, causing that bad performance.

The reporter was a co-worker and friend of mine, now retired.