ONNX Runtime
onnxruntime.ai
- Free plan
- Windows
- macOS
- Linux
- Web
- iPhone and iPad
- Android
- Self-hosted
What ONNX Runtime is
ONNX Runtime is a free engine for running machine-learning models and training them within existing technology stacks. It supports models from frameworks including PyTorch, TensorFlow/Keras, TFLite, and scikit-learn. For inference, it optimizes latency, throughput, memory use, and binary size through graph optimizations, accelerator-aware graph partitioning, and optimized computation kernels. Its Execution Providers connect ONNX models to hardware-specific acceleration on CPUs, GPUs, FPGAs, and specialized NPUs. Listed options include CUDA and TensorRT, OpenVINO, DirectML, QNN, NNAPI, CoreML, and WebGPU. Deployments can run on cloud servers, edge or mobile devices, and browsers; dedicated options include ONNX Runtime Web and ONNX Runtime Mobile for Android and iOS. The project lists languages including Python, C#, C++, Java, JavaScript, and Rust. Its generative AI support covers text, image, and audio models such as Llama, Mistral, Phi, Stable Diffusion, and Whisper. Developers can create custom web or mobile packages with only the operators and opsets their models require. The runtime is available for Android, iOS, Linux, macOS, self-hosted environments, web, and Windows.
Who it suits
It suits developers who need to deploy or train machine-learning models across different hardware and environments. It may also suit teams building browser or mobile inference, or generative AI applications.
Good to know, and what to watch
- Free under the MIT license.
- Supports models from several machine-learning frameworks.
- Execution Providers connect models to diverse hardware accelerators.
- Custom builds can reduce web or mobile package size.
- Offers browser, mobile, edge, and cloud deployment options.
- Nightly builds have limited support and are discouraged for production.
- DirectML is in sustained engineering; new Windows projects are advised to use WinML.
- Untrusted models may consume excessive memory or compute resources.
Plans
Every plan and priceOpen source
Free plan
Getting started
- Open https://onnxruntime.ai/
- Choose the runtime for Android, iOS, Linux, macOS, web, Windows, or self-hosted deployment
- Select a supported programming language and model format
- Install or build a package for the target environment
- Choose an Execution Provider for available hardware acceleration
- Deploy inference or training on cloud, edge, mobile, or browser targets
Limits to check
Nightly builds are intended for testing, have limited support, and are strongly discouraged for production workloads. The DirectML provider is in sustained engineering, with new Windows projects advised to use WinML instead.
The Club's review
Overview
ONNX Runtime is a free execution engine for machine-learning inference and training inside existing software stacks. It is best suited to developers and IT teams that need to deploy models across varied hardware and environments, rather than build models in a single end-to-end framework. Its breadth is a compelling reason to choose it, provided teams plan around platform-specific guidance and package size.
Key features
Models and execution
The runtime can execute models originating in PyTorch, TensorFlow and Keras, TFLite, scikit-learn, Hugging Face, and other frameworks; its listed model formats are ONNX and ORT. Graph optimization, optimized computation kernels, and graph partitioning help target inference latency, throughput, memory use, and binary size. This makes ONNX Runtime a practical bridge between model development and applications that should not depend on the original training framework.
That flexibility is useful, not automatic: teams still need to select and validate the execution path for their model and target hardware. If a prebuilt web or mobile package is too large, developers can create a custom build with only the operators and opsets the model needs.
Hardware acceleration and deployment
Execution Providers connect ONNX models to hardware-specific acceleration libraries across CPUs, GPUs, FPGAs, and specialized NPUs. Integrations include NVIDIA CUDA and TensorRT, Intel OpenVINO, AMD MIGraphX, Windows DirectML, Qualcomm QNN, Android NNAPI, Apple CoreML, and WebGPU, among others. The range gives teams options across heterogeneous fleets, though the provider and platform must fit the project: DirectML is in sustained engineering, and new Windows projects are advised to use WinML instead.
Deployment spans cloud servers, edge and mobile devices, and browsers. ONNX Runtime Web runs models in browsers; ONNX Runtime Mobile supports Android and iOS applications. The project also documents integrations with Azure Machine Learning, Azure Custom Vision, Azure SQL Edge, Azure Synapse Analytics, ML.NET, and NVIDIA Triton Inference Server, making it relevant to teams fitting inference into existing services as well as standalone applications.
Languages, generative AI, and training
Supported languages include Python, C, C++, C#, Java, JavaScript, TypeScript, Kotlin, Objective-C, and Rust. The generative AI deployment material covers text, image, and audio models such as Llama, Mistral, Phi, Stable Diffusion, and Whisper. Running inference on-device can keep it private and reduce costs; on-device training is also supported, and the project says it can reduce costs for large-model training. Those are useful options when workloads need to run near users or data, but teams should weigh them against their own device and deployment constraints.
Pricing
Open source: free. The plan is cross-platform and licensed under MIT, with a local training mode, multiple deployment targets, GPU acceleration, and no paid tier identified. That makes it a straightforward option for teams seeking a runtime without a software subscription. Costs associated with the hardware and infrastructure used to train or serve models are separate.
Platforms
ONNX Runtime supports Android, iOS, Linux, macOS, self-hosted environments, web, and Windows. That spread is valuable when teams need one execution layer across server, browser, and mobile targets, while the DirectML guidance is a reason for Windows developers to consider WinML for new projects.
Who it's for
Choose ONNX Runtime when you need to bring models from several frameworks into existing applications, use hardware acceleration across different vendors, or deploy inference to browsers and mobile devices. It is also a fit for teams exploring local inference or on-device training. It is less suitable as a complete model-development environment: its core role is model execution and optimization, and deployment still requires choosing compatible providers and packaging.
Nightly builds are intended for testing, have limited support, and are strongly discouraged for production workloads. Models from untrusted sources may consume excessive memory or compute, so inspect and test them safely before deployment. For support, the project directs documentation questions to issue filing and welcomes bug reports, feature suggestions, and pull requests on GitHub; non-trivial vulnerabilities can be reported through GitHub Security Advisories for coordinated fixes and disclosure.
Pros and cons
- Pros: Broad framework compatibility lets teams execute models from multiple development ecosystems through a common runtime.
- Pros: Providers for a wide range of CPUs, GPUs, and specialized accelerators give deployment teams hardware choices across platforms.
- Pros: Browser, mobile, cloud, and edge deployment options support varied application targets, with custom builds available to reduce web and mobile package size.
- Cons: DirectML is in sustained engineering, so it is not the recommended starting point for new Windows projects.
- Cons: Nightly builds have limited support and are discouraged for production, ruling them out as a dependable production channel.
- Cons: Untrusted models can consume excessive resources, so model intake requires inspection and safe testing.
Alternatives
Choose NVIDIA Triton Inference Server when you want a free, open-source server option with Linux, Windows, API, and self-hosted platforms. Apache TVM is another free, open-source choice, under Apache License 2.0, for readers comparing runtime and deployment tools across a similarly broad set of platforms.
Lightning AI Studios is worth considering if you want a hosted web and API workspace with a free tier that includes one active Studio, 50 GB persistent storage, and up to two concurrent GPUs. Paperspace Gradient offers a free individual plan with 5 GB storage, while its Pro individual plan is $8 a month plus utilization costs on paid instances.
Amazon SageMaker Autopilot is a pay-as-you-go alternative with no minimum fees or upfront commitments. MATLAB Grader is free with a MATLAB license current under maintenance, and LMS integration requires a qualifying academic license. Deeplearning4j is an open-source library option under Apache 2.0 and Eclipse Foundation governance; Ludwig is a free open-source alternative under Apache 2.0.
Verdict
ONNX Runtime is a strong choice for teams that need to execute and optimize models from different frameworks across varied hardware, languages, and deployment targets without paying for the runtime. Its main advantage is that breadth; look elsewhere if you need a higher-level model-building workspace, or if your Windows project depends on starting with DirectML.
The verdict
ONNX Runtime brings model inference and training to a broad range of platforms, languages, and accelerators. Check the deployment guidance for nightly builds and Windows projects, and inspect models from untrusted sources before use.
Questions people ask
Is ONNX Runtime free?
Yes. The Open source plan is free and uses the MIT license.
Which model frameworks can it run?
It supports models from PyTorch, Hugging Face, TensorFlow and Keras, TFLite, scikit-learn, and other frameworks.
Where can it deploy models?
Deployment targets include cloud servers, edge and mobile devices, web browsers, and self-hosted environments.
What hardware acceleration is available?
Execution Providers support CPUs, GPUs, FPGAs, and specialized NPUs, including CUDA, TensorRT, OpenVINO, QNN, CoreML, NNAPI, DirectML, and WebGPU.
Which languages are supported?
Listed languages include Python, C, C++, C#, Java, JavaScript, TypeScript, Kotlin, and Objective-C.
Can it run generative AI models on devices?
Yes. Its generative AI documentation covers text, image, and audio models, and says on-device inference can run privately and save costs.
What the maker publishes
- Deployment
- Inference is described for cloud servers, edge and mobile devices, and web browsers. source
- DirectML status
- The DirectML execution provider is in sustained engineering, and new Windows projects are advised to use WinML instead. source
- Execution providers
- Execution providers include NVIDIA CUDA and TensorRT, DirectML, Intel OpenVINO, AMD MIGraphX, Qualcomm QNN, CoreML, NNAPI, and others. source
- Framework support
- It can run models from PyTorch, TensorFlow/Keras, TFLite, scikit-learn, and other frameworks. source
- Generative AI
- The generative AI page describes deploying text, image, and audio models, including Llama, Mistral, Phi, Stable Diffusion, and Whisper. source
- Hardware acceleration
- Its extensible Execution Providers framework lets ONNX models use hardware-specific acceleration libraries across CPUs, GPUs, FPGAs, and specialized NPUs. source
- Inference optimization
- ONNX Runtime applies graph optimizations, partitions graphs for available accelerators, and uses optimized computation kernels. source
- Integrations
- The ecosystem documentation lists integrations with Azure Machine Learning, Azure Custom Vision, Azure SQL Edge, Azure Synapse Analytics, ML.NET, and NVIDIA Triton Inference Server. source
- Languages
- The site lists support for Python, C#, C++, Java, JavaScript, and Rust, among other languages. source
- Maker
- The site identifies Microsoft in its copyright notice; the pages reviewed do not state headquarters or a founding date. source
- Model frameworks
- Inference supports models from PyTorch, Hugging Face, and TensorFlow across different software and hardware stacks. source
- Nightly builds
- Nightly builds are available for testing but have limited support and are strongly discouraged for production workloads. source
- Nightly build support
- The install page warns that nightly builds have limited support and advises against deploying them to production workloads. source
- On-device privacy
- The generative AI page says on-device models can run inference privately and save costs. source
- Package sizing
- If a prebuilt web or mobile package is too large, developers can make a custom build containing only the operators and opsets their models need. source
- Performance
- It provides optimizations for inference latency, throughput, memory utilization, and binary size. source
- Performance
- The runtime optimizes latency, throughput, memory utilization, and binary size across CPU, GPU, and NPU hardware. source
- Platforms
- The site says ONNX Runtime runs on Linux, Windows, Mac, iOS, Android, and web browsers. source
- Provider integrations
- Listed providers include NVIDIA CUDA and TensorRT, Intel OpenVINO, Windows DirectML, Qualcomm QNN, Android NNAPI, Apple CoreML, Azure, and WebGPU. source
- Purpose
- ONNX Runtime is a production-grade engine for accelerating machine-learning training and inference in existing technology stacks. source
- Purpose
- ONNX Runtime is a cross-platform machine-learning model accelerator with interfaces for hardware-specific libraries. source
- Security guidance
- The documentation warns that models from untrusted sources may consume excessive memory or compute resources and recommends inspection and safe testing. source
- Security reporting
- The project accepts non-trivial vulnerability reports through GitHub Security Advisories and coordinates fixes and disclosure. source
- Support
- Documentation questions are directed to issue filing, and the project invites users to report bugs, suggest features, and submit pull requests on GitHub. source
Checked against the maker's pages on 1 Oct 2026.