Opened 4 years ago

Closed 4 years ago

#61912 closed defect (fixed)

opencv @3.4.13: only 17% of test suite passes

Reported by: kencu (Ken) Owned by: Christopher Nielsen <mascguy@…>
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: mascguy (Christopher Nielsen)
Port: opencv

Description

Quite pathetic performance on opencv's test suite.

Running on Catalina, Xcode 12.3, only 17% of tests pass. I'm not sure at this moment what portion of this might relate to the MacPorts environment, and what part of it is just opencv itself.

Anyone try to actually use opencv for anything on MacOS should be aware of this issue.

To enable testing:

% git diff
diff --git a/graphics/opencv/Portfile b/graphics/opencv/Portfile
index 936a6c28422..da4afa75993 100644
--- a/graphics/opencv/Portfile
+++ b/graphics/opencv/Portfile
@@ -147,8 +147,8 @@ configure.args-append \
                     -DBUILD_DOCS=OFF \
                     -DBUILD_EXAMPLES=OFF \
                     -DBUILD_PACKAGE=OFF \
-                    -DBUILD_PERF_TESTS=OFF \
-                    -DBUILD_TESTS=OFF \
+                    -DBUILD_PERF_TESTS=ON \
+                    -DBUILD_TESTS=ON \
                     -DBUILD_WITH_DYNAMIC_IPP=OFF \
                     \
                     -DBUILD_ZLIB=OFF \
@@ -514,6 +514,9 @@ post-destroot {
     }
 }
 
+test.run yes
+test.target test
+
 livecheck.type      regex
 livecheck.url       https://opencv.org/releases/
 livecheck.regex     {archive/(3\.[0-9.]+)?\.[tz]}

results

17% tests passed, 34 tests failed out of 41

Label Time Summary:
Accuracy             = 357.03 sec*proc (17 tests)
Main                 = 1315.69 sec*proc (41 tests)
Performance          = 875.31 sec*proc (12 tests)
Sanity               =  83.35 sec*proc (12 tests)
opencv_calib3d       =  98.62 sec*proc (3 tests)
opencv_core          = 427.26 sec*proc (3 tests)
opencv_dnn           =  26.62 sec*proc (3 tests)
opencv_features2d    =  62.15 sec*proc (3 tests)
opencv_flann         =   0.50 sec*proc (1 test)
opencv_highgui       =   0.84 sec*proc (1 test)
opencv_imgcodecs     =  23.89 sec*proc (3 tests)
opencv_imgproc       = 530.95 sec*proc (3 tests)
opencv_ml            =   3.44 sec*proc (1 test)
opencv_objdetect     =  57.65 sec*proc (3 tests)
opencv_photo         =   0.63 sec*proc (3 tests)
opencv_shape         =   0.31 sec*proc (1 test)
opencv_stitching     =   0.60 sec*proc (3 tests)
opencv_superres      =  12.64 sec*proc (3 tests)
opencv_video         =   1.54 sec*proc (3 tests)
opencv_videoio       =  64.17 sec*proc (3 tests)
opencv_videostab     =   3.89 sec*proc (1 test)

Total Test time (real) = 1315.97 sec

The following tests FAILED:
	  1 - opencv_test_core (Failed)
	  3 - opencv_sanity_core (Failed)
	  5 - opencv_test_imgproc (Failed)
	  6 - opencv_perf_imgproc (Failed)
	  7 - opencv_sanity_imgproc (Failed)
	  8 - opencv_test_ml (Failed)
	  9 - opencv_test_photo (Failed)
	 10 - opencv_perf_photo (Failed)
	 11 - opencv_sanity_photo (Failed)
	 12 - opencv_test_video (Failed)
	 13 - opencv_perf_video (Failed)
	 14 - opencv_sanity_video (Failed)
	 15 - opencv_test_dnn (Failed)
	 16 - opencv_perf_dnn (Failed)
	 17 - opencv_sanity_dnn (Failed)
	 18 - opencv_test_features2d (Failed)
	 19 - opencv_perf_features2d (Failed)
	 20 - opencv_sanity_features2d (Failed)
	 21 - opencv_test_imgcodecs (Failed)
	 24 - opencv_test_shape (Failed)
	 25 - opencv_test_videoio (Failed)
	 26 - opencv_perf_videoio (Failed)
	 27 - opencv_sanity_videoio (Failed)
	 28 - opencv_test_calib3d (Failed)
	 29 - opencv_perf_calib3d (Failed)
	 30 - opencv_sanity_calib3d (Failed)
	 32 - opencv_test_objdetect (Failed)
	 33 - opencv_perf_objdetect (Failed)
	 34 - opencv_sanity_objdetect (Failed)
	 35 - opencv_test_stitching (Failed)
	 36 - opencv_perf_stitching (Failed)
	 37 - opencv_sanity_stitching (Failed)
	 38 - opencv_test_superres (Failed)
	 40 - opencv_sanity_superres (Failed)
Errors while running CTest

Change History (5)

comment:1 Changed 4 years ago by mascguy (Christopher Nielsen)

Cc: mascguy added

comment:2 Changed 4 years ago by mascguy (Christopher Nielsen)

Ken, I'll add a tests variant, for both opencv and opencv4. Thanks!

comment:3 Changed 4 years ago by kencu (Ken)

Well, turns out opencv does not do a whole lot better on Ubuntu 20.10 either:

24% tests passed, 31 tests failed out of 41

Label Time Summary:
Accuracy             = 614.58 sec*proc (17 tests)
Main                 = 8285.79 sec*proc (41 tests)
Performance          = 6942.99 sec*proc (12 tests)
Sanity               = 728.22 sec*proc (12 tests)
opencv_calib3d       = 688.18 sec*proc (3 tests)
opencv_core          = 3091.53 sec*proc (3 tests)
opencv_dnn           = 110.29 sec*proc (3 tests)
opencv_features2d    = 294.62 sec*proc (3 tests)
opencv_flann         =   0.07 sec*proc (1 test)
opencv_highgui       =   1.21 sec*proc (1 test)
opencv_imgcodecs     =  39.52 sec*proc (3 tests)
opencv_imgproc       = 3693.58 sec*proc (3 tests)
opencv_ml            =   6.73 sec*proc (1 test)
opencv_objdetect     = 124.57 sec*proc (3 tests)
opencv_photo         =   0.91 sec*proc (3 tests)
opencv_shape         =   0.16 sec*proc (1 test)
opencv_stitching     =  91.91 sec*proc (3 tests)
opencv_superres      = 126.13 sec*proc (3 tests)
opencv_video         =   2.53 sec*proc (3 tests)
opencv_videoio       =   6.24 sec*proc (3 tests)
opencv_videostab     =   7.62 sec*proc (1 test)

Total Test time (real) = 8286.56 sec

The following tests FAILED:
	  1 - opencv_test_core (Failed)
	  3 - opencv_sanity_core (Failed)
	  5 - opencv_test_imgproc (Failed)
	  6 - opencv_perf_imgproc (Failed)
	  7 - opencv_sanity_imgproc (Failed)
	  8 - opencv_test_ml (Failed)
	  9 - opencv_test_photo (Failed)
	 10 - opencv_perf_photo (Failed)
	 11 - opencv_sanity_photo (Failed)
	 12 - opencv_test_video (Failed)
	 13 - opencv_perf_video (Failed)
	 14 - opencv_sanity_video (Failed)
	 15 - opencv_test_dnn (Failed)
	 16 - opencv_perf_dnn (Failed)
	 17 - opencv_sanity_dnn (Failed)
	 18 - opencv_test_features2d (Failed)
	 19 - opencv_perf_features2d (Failed)
	 20 - opencv_sanity_features2d (Failed)
	 21 - opencv_test_imgcodecs (Failed)
	 24 - opencv_test_shape (Failed)
	 25 - opencv_test_videoio (Failed)
	 28 - opencv_test_calib3d (Failed)
	 29 - opencv_perf_calib3d (Failed)
	 30 - opencv_sanity_calib3d (Failed)
	 32 - opencv_test_objdetect (Failed)
	 33 - opencv_perf_objdetect (Failed)
	 34 - opencv_sanity_objdetect (Failed)
	 35 - opencv_test_stitching (Failed)
	 36 - opencv_perf_stitching (Failed)
	 37 - opencv_sanity_stitching (Failed)
	 40 - opencv_sanity_superres (Failed)
Errors while running CTest
make: *** [Makefile:130: test] Error 8

so perhaps the test suite needs some work for this software...I have not as yet tried the 4.x releases.

Last edited 4 years ago by kencu (Ken) (previous) (diff)

comment:4 Changed 4 years ago by mascguy (Christopher Nielsen)

Optional variant +tests added via pull request, for both opencv and opencv4:

https://github.com/macports/macports-ports/pull/9555

comment:5 Changed 4 years ago by Christopher Nielsen <mascguy@…>

Owner: set to Christopher Nielsen <mascguy@…>
Resolution: fixed
Status: newclosed

In ca9ffa681f453399942385f9ef34cf1c124b6ea4/macports-ports (master):

opencv/opencv4: eliminate conflicts, improve consistency, add variants; update gerbil and py-pytorch for opencv4 changes

opencv: eliminate conflict with opencv4; add variants debug, nonfree, tests
opencv4: eliminate conflict with opencv; switch to shared libs; rename variant opencv_contrib to contrib, to match opencv; add variants debug, java, python35..39, qt4, qt5, tbb, tests, vtk
gerbil: update for opencv4 library move
py-pytorch: update for opencv4 library move

Fixes: #61912
Fixes: #61801
Fixes: #60118
Fixes: #58845
Fixes: #57640
Fixes: #51734
Fixes: #48218
Fixes: #32528

Note: See TracTickets for help on using tickets.