possible bug when scaling MTT table size with system ram
Roland,
Yishai Hadas from Mellanox noted that commit db5a7a65c05 "mlx4_core: Scale size of MTT table with system RAM" seems to introduce a bug, where if request->num_mtt becomes 2^25 or higher
Quoted 17 lines
(quoted diff) +17
-0+ /* + * We want to scale the number of MTTs with the size of the + * system memory, since it makes sense to register a lot of + * memory on a system with a lot of memory. As a heuristic, + * make sure we have enough MTTs to cover twice the system + * memory (with PAGE_SIZE entries). + * + * This number has to be a power of two and fit into 32 bits + * due to device limitations, so cap this at 2^31 as well. + * That limits us to 8TB of memory registration per HCA with + * 4KB pages, which is probably OK for the next few months. + */ + si_meminfo(&si); + request->num_mtt = + roundup_pow_of_two(max_t(unsigned, request->num_mtt, + min(1UL << 31, + si.totalram >>
Quoted 2 lines
(
log_mtts_per_seg- 1))));
+
we are somehow getting into a situation where mlx4_buddy_init needs to allocate > 128KB using kmalloc, which is impossible... he was suggesting to replace kmalloc/kfree with vmalloc/vfree (see below). What's your thinking here? should we go to get_free_pages? limit by 2^25? something else?
Or.
Quoted 31 lines
drivers/net/ethernet/mellanox/mlx4/mr.c +1
-1@@ -129,7 +129,7 @@ static int mlx4_buddy_init(struct mlx4_buddy *buddy, int max_order)for (i =0 ; i <= buddy->max_order; ++i) { s =BITS_TO_LONGS (1 << (buddy->max_order - i)); - buddy->bits[i] =kmalloc (s *sizeof (long ),GFP_KERNEL ); + buddy->bits[i] =vmalloc (s *sizeof (long ));if (!buddy->bits[i]) goto err_out_free;bitmap_zero(buddy->bits[i], 1 << (buddy->max_order - i));drivers/net/ethernet/mellanox/mlx4/mr.c +1
-1@@ -142,7 +142,7 @@ static int mlx4_buddy_init(struct mlx4_buddy *buddy, int max_order) err_out_free:for (i =0 ; i <= buddy->max_order; ++i) -kfree (buddy->bits[i]); +vfree (buddy->bits[i]); err_out:kfree(buddy->bits);drivers/net/ethernet/mellanox/mlx4/mr.c +1
-1@@ -156,7 +156,7 @@ static void mlx4_buddy_cleanup(struct mlx4_buddy *buddy)int i;for (i =0 ; i <= buddy->max_order; ++i) -kfree (buddy->bits[i]); +vfree (buddy->bits[i]);kfree (buddy->bits);
Quoted 1 line
kfree(buddy->num_free);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html