samba3 @3.6.5 fails to initialize on startup

[jim@…]

The latest SAMBA fails to initialize on startup. This seems to be related to the low value of NAX_NGROUPS in the Darwin Kernel.

Here is the link that I found describing the problem:

​http://www.sand4.info/index.php?q=aHR0cHM6Ly9naXN0LmdpdGh1Yi5jb20vMTg4ODc3OA%3D%3D

In it they show the hack:

--- system.c.orig	2012-02-22 22:46:14.000000000 -0200
+++ system.c	2012-02-22 22:47:51.000000000 -0200
@@ -1161,7 +1161,14 @@
 
 int groups_max(void)
 {
-#if defined(SYSCONF_SC_NGROUPS_MAX)
+#if defined(DARWINOS)
+	/* On OS X, sysconf(_SC_NGROUPS_MAX) returns 16
+	 * due to OS X's group nesting and getgrouplist
+	 * will return a flat list; users can exceed the
+	 * maximum of 16 groups. And easily will.
+	 */
+	return 32; // NGROUPS_MAX is defined, hence the define above is void.
+#elif defined(SYSCONF_SC_NGROUPS_MAX)
 	int ret = sysconf(_SC_NGROUPS_MAX);
 	return (ret == -1) ? NGROUPS_MAX : ret;
 #else

I haven't looked into this in any detail. Maybe someone else is more familiar with the code.

[ryandesign (Ryan Carsten Schmidt)]

Please remember to use WikiFormatting, to fill in the Port field, and to Cc the maintainer of the port.

[eduo@…]

This patch doesn't fix the problem. Macports has deployed this fix and updated the port only to have it coredump on everyone who has more than 15 groups on a user.

[eduo@…]

Reviewing my comment: I don't criticize macports adding the patch. But now there's a non-working samba port for a lot of Lion users (most, probably). I think 3.6.5 should be rolled back to the previously-working version (this is a 3.6-specific issue in Lion).

[ayearout@…]

The problem with the original patch, as far as I can tell, is that the SYS_initgroups() system call in smbd/sec_ctx.c requires that groups_max() return the correct value. If it is greater than the system defined NGROUPS_MAX, SYS_initgroups() throws an "Invalid argument" error and returns -1, causing the smbd process to bail.

The below patch solves the problem by going a different route:

--- lib/system_smbd.c.orig	2012-07-29 19:48:57.000000000 -0700
+++ lib/system_smbd.c	2012-07-29 19:49:09.000000000 -0700
@@ -210,7 +210,11 @@
 	gid_t *groups;
 	int i;
 
+#if defined(DARWINOS)
+	max_grp = 128;
+#else
 	max_grp = MIN(128, groups_max());
+#endif
 	temp_groups = SMB_MALLOC_ARRAY(gid_t, max_grp);
 	if (! temp_groups) {
 		return False;

I modified the getgroups_unix_user() call in lib/system_smbd.c to allow users to be in more than the 16 group maximum if compiled on Darwin platforms. The patch specifies a static value of 128, which should work in most cases. This still functions correctly because Darwin's getgroups() call will happily return more groups than what is defined in NGROUPS_MAX if the correct macros are defined. It's only when using setgroups() or the SYS_initgroups() system call that you run into the limit problems.

I've tested the patch on my system using Samba 3.6.6 with no issues. A user in 20 groups correctly resolves all groups. I also configured a test share with 20 individual folders, each with different group permissions, and Samba correctly resolved the permissions for all folders.

I hope this helps others! I've attached the patch as a file as well.

[ayearout@…]

Cc Me!

[thomas-corte]

Cc Me!

[eduo@…]

Cc Me!

[flexcode@…]

Cc Me!

[flexcode@…]

Replying to ayearout@…:

The problem with the original patch, as far as I can tell, is that the SYS_initgroups() system call in smbd/sec_ctx.c requires that groups_max() return the correct value. If it is greater than the system defined NGROUPS_MAX, SYS_initgroups() throws an "Invalid argument" error and returns -1, causing the smbd process to bail.

The below patch solves the problem by going a different route:

--- lib/system_smbd.c.orig	2012-07-29 19:48:57.000000000 -0700
+++ lib/system_smbd.c	2012-07-29 19:49:09.000000000 -0700
@@ -210,7 +210,11 @@
 	gid_t *groups;
 	int i;
 
+#if defined(DARWINOS)
+	max_grp = 128;
+#else
 	max_grp = MIN(128, groups_max());
+#endif
 	temp_groups = SMB_MALLOC_ARRAY(gid_t, max_grp);
 	if (! temp_groups) {
 		return False;

I modified the getgroups_unix_user() call in lib/system_smbd.c to allow users to be in more than the 16 group maximum if compiled on Darwin platforms. The patch specifies a static value of 128, which should work in most cases. This still functions correctly because Darwin's getgroups() call will happily return more groups than what is defined in NGROUPS_MAX if the correct macros are defined. It's only when using setgroups() or the SYS_initgroups() system call that you run into the limit problems.

I've tested the patch on my system using Samba 3.6.6 with no issues. A user in 20 groups correctly resolves all groups. I also configured a test share with 20 individual folders, each with different group permissions, and Samba correctly resolved the permissions for all folders.

I hope this helps others! I've attached the patch as a file as well.

Is there something else needed to patch with? I've tried following the instructions but obviously I'm missing something here

osx:samba3 rick$ patch -p0 < ~/Downloads/patch-source_lib_system_smbd_c.diff
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- lib/system_smbd.c.orig     2012-07-29 19:48:57.000000000 -0700
|+++ lib/system_smbd.c  2012-07-29 19:49:09.000000000 -0700
--------------------------
File to patch:

It just asks me what to patch and if I enter Portfile it fails

[jmroot (Joshua Root)]

-> nomaintainer

[jmroot (Joshua Root)]

Upstream report: ​https://bugzilla.samba.org/show_bug.cgi?id=8773 (which has the same patch attached as here, and says it is incorrect)

[jeremyhu (Jeremy Huddleston Sequoia)]

Dupe of #30951

[cooljeanius (Eric Gallager)]

Cc Me!