[perfgao]

I hereby granted the copyright of the changes in this pull request
to the authors of this lua-nginx-module project.

With a connection pool backlog, a queued connect() can wait until it times out while the pool has free slots.

ngx_http_lua_socket_tcp_resume_conn_op() runs each time a slot is freed and posts the event of the operation at the head of wait_connect_op. That operation stays in the queue until its event runs, where ngx_http_lua_socket_tcp_conn_op_resume_handler() removes the queue head. When two slots are freed in the same cycle, for example two setkeepalive() calls in a row, both calls post the same event and ngx_post_event() ignores the second one. The second slot is handed to no waiter. Since spool->connections counts queued operations too, the pool still counts that slot as busy, so new connect() calls queue behind it instead of taking it. Each lost resumption shrinks the pool until the queue drains.

The smallest case: pool_size = 2, backlog = 2, two connected sockets, two waiters queued with a 500ms connect timeout, then both sockets released in the same cycle:

w1: connected
w2: timeout
[error] lua tcp socket queued connect timed out, when trying to connect to 127.0.0.1:1984

Under load this is not a corner case. With 9 workers, pool_size = 100 and backlog = 500, and a burst of 300 concurrent requests that each run 7 concurrent cosocket queries against a local backend over a unix domain socket, the queries of one request failed in every run after the full connect timeout, about 1s with a 1s timeout and 3.45s with a 3s timeout, while the other requests went through. With a warm pool, or without a backlog, there were no failures.

This change resumes the first queued operation whose event is not posted yet, so every freed slot resumes its own waiter, still in queue order, and the resume handler removes its own operation instead of the queue head. As several operations can now be posted at the same time, ngx_http_lua_socket_shutdown_pool_helper() also deletes a posted event before it frees the operation; before, only the head could be posted.

TEST 56 in t/068-socket-keepalive.t is the case above and expects both waiters to connect. The first waiter holds its connection for 1s, longer than the second waiter's connect timeout, so the second one can only connect if it is resumed. Without the fix it fails on every run with w2: timeout; with it, it passes over HTTP/1.1 and HTTP/2. The other tests in the file fail or pass the same with and without the change in my environment; the failures come from my local setup (some tests need services it does not have, and 127.0.0.2:12345 is refused at once instead of timing out). Tested against nginx 1.31.1 in an OpenResty build; the changed functions and the test file are the same as on master.

The same code is in stream-lua-nginx-module: openresty/stream-lua-nginx-module#415

Contributed by AxisNow.