sgx_reclaimer_write() refinement

Hi

I noticed that sgx_reclaimer_write() does not destory enclave albeit it can fail. It should have a proper fallback mechanism.

I would take a long path to fix this by replacing sgx_reclaim_pages() with

struct sgx_epc_page *sgx_reclaim_page(void)

It would greatly reduce the complexity of the flow. Would be much easier to do fallback code paths. We have three trial EWB approach, which should prevent already unnecesaary ETRACK's and IPI's.

The reclaimer thread would loop this until high watermark is reached.

The alloc function would change as:

struct sgx_epc_page *sgx_alloc_page(void *owner, bool reclaim)

{
	struct sgx_epc_page *entry;

	entry = sgx_try_alloc_page(owner);
	if (entry)
		break;

	if (list_empty(&sgx_active_page_list)) {
		entry = ERR_PTR(-ENOMEM);
		goto out;
	}

	if (!reclaim) {
		entry = ERR_PTR(-EBUSY);
		goto out;
	}

	entry = sgx_reclaim_page();

out:

	if (sgx_calc_free_cnt() < SGX_NR_LOW_PAGES)
		wake_up(&ksgxswapd_waitq);

	return entry;
}

We get rid of the loop because the reclaimer function will return the EPC page. The reclaimer thread will add the page to the active page list.

/Jarkko

I recall that I used a cluster ages ago to do minimize ETRACK's and IPI's. I implemented three trial EWB during the Spring 2018 when the reclaimer code was heavily refined but failed to notice that it rendered cluster's meaningless...

/Jarkko

One potential caveat here: EBLOCK's could never sequence before ETRACK with a single thread.

However, the existing code does work in parallel already i.e. there can be multiple sgx_alloc_page() calls on different going on. This change would make sgx_alloc_page() success or fail (in the rare case of no pages to swap) always in constant time and these would group naturally behind ETRACK's.

What I would do for the reclaimer thread would be to convert it from a kthread to a multi-threaded work queue (albeit even current single threaded approach might work reasonably well as it goes in parallel with the sgx_alloc_page() calls).

/Jarkko

on differen threads

/Jarkko