iptables: set errno correctly in iptcc_chain_index_alloc

As reported by Robert Barnhardt, iptcc_chain_index_alloc does not populate errno with the appropriate ENOMEM on allocation failures. This causes incorrect error messages to be passed back to user such as "can't initialize iptables table 'X'" even if the issue was caused by OOM condition. Fix this by passing back ENOMEM if allocation failure occurs.

This closes bugzilla #619.

Phil

index f0f7815..004b0ec 100644
@@ -502,7 +502,8 @@ static int iptcc_chain_index_alloc(struct xtc_handle *h)
 	h->chain_index = malloc(array_mem);
 	if (h->chain_index == NULL && array_mem > 0) {
 		h->chain_index_sz = 0;
-		return -ENOMEM;
+		errno = ENOMEM;
+		return -1;
 	}
 	memset(h->chain_index, 0, array_mem);
 	h->chain_index_sz = array_elems;

Personally I think libraries should not change errno at all.

int main(void) { errno = EINVAL;
	        void *v = malloc(0xffffffffffffffff);
		        if (v == 0) perror("malloc"); }

OK, but then we output misleading error messages.

Simulate an out of memory condition with this patch

@@ -500,9 +500,11 @@ static int iptcc_chain_index_alloc(struct xtc_handle *h)

              array_elems, array_mem);

        h->chain_index = malloc(array_mem);
-       if (h->chain_index == NULL && array_mem > 0) {
+       //if (h->chain_index == NULL && array_mem > 0) {
+       if (1) {
                h->chain_index_sz = 0;

With the patch, the error message returned to user:

...can't initialize iptables table `filter': Memory allocation problem

without the patch:

...can't initialize iptables table `filter': Incompatible with this kernel

The former seems better, no?

Phil

Yes, but malloc didn't fail, so malloc didn't set errno.

My point is, that we should not muck with errno, especially after libc functions that usually already set it on error.

  # grep -c 'errno = ' libiptc/libiptc.c 
  52

But ok, we can avoid adding yet another instance and drop this patch.

Phil