Add python bindings to the global thread pool functionality

[khoover]

Description

Allows users to configure and enable the global thread pool via Python, and have inference sessions use it instead of session-local thread pools.

Motivation and Context

Forked off of #23495 to take over implementation, see issue #23523.

Our particular use case involves a single service instance serving thousands of individual models, each relatively small (e.g. small decision trees). Creating individual services for each model is too much overhead, and attempting to start several thousand thread-pools is a non-starter. We could possibly have each session be single-threaded, but we would like to be able to separate the request handler thread count from the compute thread count (e.g. 2 handler threads but 4 intra-op ones).

[snnn]

We don't have a good way to shutdown the thread pool. It should be done before the python process started to exit and unload DLLs. So this doc: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices#deadlocks-caused-by-lock-order-inversion
So, generally speaking, if you have a thread pool and you declared it as a global var, then you must manually shutdown it, otherwise it will cause a deadlock. The issue was highly reproducible. However, recently probably the restriction is gone? I am not sure. But the Windows doc is not updated. So I am not very sure if this will work.

[snnn]

Like this: apache/mxnet#11163

[khoover]

@microsoft-github-policy-service agree company="Instacart"

[khoover]

We don't have a good way to shutdown the thread pool. It should be done before the python process started to exit and unload DLLs. So this doc: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices#deadlocks-caused-by-lock-order-inversion So, generally speaking, if you have a thread pool and you declared it as a global var, then you must manually shutdown it, otherwise it will cause a deadlock. The issue was highly reproducible. However, recently probably the restriction is gone? I am not sure. But the Windows doc is not updated. So I am not very sure if this will work.

Would adding onnxruntime::Environment::ShutdownGlobalThreadPools and then hooking it into atexit in Python work? I imagine the existing C++ API has examples of needing to do this as well, no?

EDIT: I see, the native implementation would just let Environment destruct at the end of the program, and that handles shutting down. But the Python bindings have a static shared_ptr<Environment>, and it's really just the thread pools that need to go.

[snnn]

On Windows when a thread shuts down, the OS also needs to notify all loaded DLLs in case if they need to do any cleanup work. It cannot be too late. So, in general all user created threads should be shutdown before unloading DLLs. Theoretically speaking, there is another way of doing this: the cleanup functions should test if the currently the process is shutting down, if yes, giving up doing any clean up. We do not have such logics in ORT yet. We may consider doing it later.