in cmake/CMakeList.txt all avx related option all set off, do we need do anything to use avx features?

[mmxuan18]

in cmake/CMakeList.txt all avx related option are set OFF as follow, and can't find any place to set to ON, do we need to care about these options? in my product env have very different machine types, which have different cpu arch, is onnxruntime auto select right avx type to use for acc?

i add a message command at the end of CMakeFile.txt to display CMAKE_CXX_FLAGS and only output "-ffunction-sections -fdata-sections -Wno-error=attributes -DCPUINFO_SUPPORTED" , which do'nt has any infomation about avx arch

[yuslepukhin]

@chenfucn Mind taking a look?

[yufenglee]

For the heaviest operators, like MatMul, Conv and etc., ORT automatically selects to run the code that fit the arch best.
Those options are for other operators. if you know the exact arch of your target hardware, you can try turning them on and build from source. Usually there is little impact.

[SuperLee188]

For the heaviest operators, like MatMul, Conv and etc., ORT automatically selects to run the code that fit the arch best. Those options are for other operators. if you know the exact arch of your target hardware, you can try turning them on and build from source. Usually there is little impact.

Can this be understood that for CPUs that support the AVX instruction set, OnnxRuntime will use AVX acceleration by default?

[chenfucn]

Yes

[LY000001]

For the heaviest operators, like MatMul, Conv and etc., ORT automatically selects to run the code that fit the arch best. Those options are for other operators. if you know the exact arch of your target hardware, you can try turning them on and build from source. Usually there is little impact.

I turned the avx, avx2, avx512 off, then compile the onnxruntime.But I find there are still avx instructions.Why?

[ogencoglu]

Can this be understood that for CPUs that support the AVX instruction set, OnnxRuntime will use AVX acceleration by default?

Yes

@chenfucn is it so that onnxruntime should be compiled specifically for AVX2 etc to benefit from them or simple pip install is enough?

[sikandermukaram]

Is onnxruntime using avx512 by default? If yes how to confirm this? like oprnvino has benchmark that shows which instruction set is being used for dfifferent layers/ops, is there something similar for onnxruntime? Also how to make sure onnxruntime doesn't use avx512 or use avx2?

[chenfucn]

#11833 (comment)
at startup, mlas detect underlying platform, and based on instruction sets available, hookup the best available kernels.

hi @jywu-msft would you help with future mlas questions as I moved my focus to other areas? Thanks!

[sikandermukaram]

#11833 (comment) at startup, mlas detect underlying platform, and based on instruction sets available, hookup the best available kernels.

hi @jywu-msft would you help with future mlas questions as I moved my focus to other areas? Thanks!

So I tested a quantised model on an E2 instance which doesn't support avx512 vs N1 instance that has avx512 support. The scores for the model changes a lot. I don't want to move to E2 instance as I do development in N1 but deployment on E2, so is there a way for mlas to not use avx512 on supported cpu? and I still couldn't find a way to know which kernels it used (as done in openvino benchmark)