Improve Performance by Pre-Initializing Docker Containers for Faster /eval Requests

[EmmanuelStan12]

Currently, for each /eval request, a new Docker container is created, initialized, and assigned to the session. This process involves running startup scripts, which can significantly impact performance due to the time required to initialize and start the container. To improve performance and reduce latency during /eval requests, the following enhancements are proposed:

1. Create a buffer or pool of pre-initialized Docker containers, each with a unique session ID, that are ready to be used for future /eval requests.
2. For every new /eval request, allocate a pre-initialized container from the buffer and assign it to the session. This will significantly reduce the time it takes to respond to requests since the container will already be running.
3. The container should be initialized with the necessary startup scripts beforehand so that each session has the environment pre-configured and ready for code evaluation.

Benefits:

1. Reduced Latency: By having containers pre-initialized, the time to respond to /eval requests will be reduced.
2. Improved Performance: Sessions can begin processing requests almost immediately, reducing the need to create and initialize new containers on demand.