Keeping the queue full

[andyg24]

One possible use for io_uring might be to process data in a file with the following loop:

for (;;) {
reap_io_completions();
submit_new_io_requests();
process_completions(); // this takes a bit of time
}

If the goal is to have a constant number of requests in flight (e.g. because the SSD works best at a certain QD), this loop isn't optimal. While initially the queue is full, the number of requests in flight will decrease while process_completions() is running. Additionally, in the general case this loop requires two syscalls per iteration -- one to reap completions, and then another to submit new requests, based on the number of reaped completions.

Would it make sense to decouple the length of the sqe queue from the internal submission queue? The idea would be to over-provision the sqe queue, and submit more requests than can be in flight at any given time. Then, whenever completions came in, these overflow requests would be automatically submitted. So even if process_completions() took some time, the data would continue to flow optimally.

Additionally, if a version of io_uring_enter() first reaped all completed events and then submitted pending sqes to refill the queue to capacity, the loop above could have only one syscall per iteration:

for (;;) {
add_io_requests_until_sqe_queue_is_full();
reap_and_submit();
process_completions();
}

Does this make sense? Is there another way to keep the number of requests in flight constant as completions arrive?

[AmkG]

My understanding is that the completion queue is always filled by the kernel asynchronously (compared to the submission queue, which is only unloaded asynchronously if and only if you use IORING_SETUP_SQPOLL). You can just look at the CQ-side head and tail at any time, which are just memory operations (though you should read both with acquire memory ordering; the kernel might not write to the head but it does write to the actual CQE it points to if you are starting from the empty-queue state). If my understanding of your overall design is correct, you dequeue completions from the CQ and put them into your own structures, then submit requests, then process all completions; you can make the "dequeue completions from the CQ" not do any syscalls by just using memory operations.

liburing has __io_uring_peek_cqe which does only memory operations and no syscalls, though as the __ suggests it is internal to the library and may disappear in future versions, so you might instead want to copy its code and use the appropriate C11 or C++11 atomic operations depending on what exactly you are using. Then you can make a cheap "dequeue completions from CQ" function that just calls the __my_io_uring_peek_cqe copy you made repeatedly (and then calling into io_uring_cqe_seen for each CQE you got) until it returns that there are no CQEs in the queue. A more efficient version can just keep reading CQEs from head to tail then write (with release semantics) what you got from the tail to head (which iis what io_uring_cqe_seen does, it calculates the new head from computing the cqe address minus the start of the cqe array), then re-read the CQ head and tail again to check if the kernel added more tail items while it was reading items from the CQ. Because it's so cheap, you can call this all over the main loop, such as between "submit" and "process completions", or between some number of completions during "process completions" sub-loop, or if the submit operation fails with EAGAIN or if the SQ is full (modern kernels will backpressure if your CQ is full, and will not take items from the SQ until you free up slots in the CQ).

In addition, if, before submitting new requests, you run "dequeue completions from CQ" and you end up with no completions you can process, you can use submit-and-wait, then also run "dequeue completions from CQ" afterward. liburing has io_uring_submit_and_wait, or you can just call into io_uring_enter with GETEVENTS flag and non-zero min_complete. So

for (;;) {
    memory_dequeue_completions();
    if (completions_to_process.empty()) {
        submit_and_wait();
        memory_dequeue_completions();
    } else {
        submit_only();
    }
    process_completions();
}

[andyg24]

Thank you for your reply, AmkG. I believe you still need a syscall to reap completions if you use flags like IORING_SETUP_DEFER_TASKRUN; just checking the cqe ring buffer from userspace is not enough.

Again, the main issue is the process_completions() call in my example above. The number of requests in flights could probably be kept constant if an entire thread was dedicated to babysitting liburing completions / submissions. But as soon as you start mixing async io with other (blocking) code, it's easy to starve the queue of requests. On a fast NVME, it only takes ~100 microseconds to complete all submitted sqes at QD = 128.

What I was suggesting was having another parameter in io_uring_params that specifies the desired number of in-flight requests and is different from the sqe ring size. This way I could queue a bunch of requests, specify that I only want 64 to be in-flight at any time, and then call my process_completions(). Even if that call takes a while to complete, the kernel could automatically refill the submission queue whenever completions come in. I suppose a possible complication with this approach is that unlike completions, submissions can take a while to execute, and therefore IRQs / kernel transitions could become unpredictably expensive.

[AmkG]

The number of requests in flights could probably be kept constant if an entire thread was dedicated to babysitting liburing completions / submissions. But as soon as you start mixing async io with other (blocking) code, it's easy to starve the queue of requests.

So why not? You could maintain a separate queue of completions that another thread or an entire threadpool could process, with just that one thread babysitting the uring. The babysitter can even use futex requests in its uring if there is contention on the queue of tasks for the other threads and go back to babysitting the completion uring (putting the completions into a local queue) until it sees the contention has been unblocked (and then batch-send the local queue to the threadpool queue for further processing).

I mean if you are using COOP_TASKRUN or DEFER_TASKRUN you are taking responsibility for managing even the loading and unloading of the uring, you might as well dedicate an entire core for that, and leave the heavy CPU work on a threadpool affinitied to other cores.