Add session config to return an error if model needs to be compiled

[adrianlizarraga]

Description

Adds session config option ("session.disable_model_compile") that disables model compilation during session initialization.

If this option is set to "1", inference session creation will fail with error code ORT_MODEL_REQUIRES_COMPILATION if compilation is required to run the model on any Execution Provider added to the session. Only the following kinds of models are valid when this option is set to "1":

• Pre-compiled models that have EPContext nodes for the compiling Execution Providers in the session.
• Non-compiled models that run only on non-compiling Execution Providers, like CPU EP.

Example usage

The following example (taken from a unit test) tries to load a model that requires compilation with a session that disables compilation. The session creation fails with error code ORT_MODEL_REQUIRES_COMPILATION. Then, the example compiles the model and loads the compiled model successfully.

  // Taken from a unit test ...

  // Initialize session options with QNN EP
  Ort::SessionOptions session_options;
  ProviderOptions provider_options;
  provider_options["backend_type"] = "htp";
  provider_options["offload_graph_io_quantization"] = "0";

  session_options.AppendExecutionProvider("QNN", provider_options);
  session_options.AddConfigEntry(kOrtSessionOptionsDisableEpCompile, "1");  // Disable model compilation!

  // Create an inference session that fails with error ORT_MODEL_REQUIRES_COMPILATION
  try {
    Ort::Session session(*ort_env, input_model_file, session_options);
    FAIL() << "Expected Session creation to fail but it succeeded";  // Should not get here!
  } catch (const Ort::Exception& excpt) {
    OrtErrorCode error_code = excpt.GetOrtErrorCode();
    std::string_view error_msg = excpt.what();
    ASSERT_EQ(error_code, ORT_MODEL_REQUIRES_COMPILATION);
    ASSERT_THAT(error_msg, testing::HasSubstr(kQnnExecutionProvider));
  }

  // Session creation failed because the model was not pre-compiled.
  // Try to compile it now.

  // Create model compilation options from the session options.
  Ort::ModelCompilationOptions compile_options(*ort_env, session_options);
  compile_options.SetInputModelPath(input_model_file);
  compile_options.SetOutputModelPath(output_model_file);

  // Compile the model.
  Ort::Status status = Ort::CompileModel(*ort_env, compile_options);
  ASSERT_TRUE(status.IsOK()) << status.GetErrorMessage();

  // Should be able to create a session with the compiled model and the original session options.
  Ort::Session session(*ort_env, output_model_file, session_options);

Motivation and Context

Compiling models can take a very long time. Want to have a session option that requires input models that do not need to be compiled.

[adrianlizarraga]

Hi @adrastogi, can you please take a look when able?

[adrastogi]