Notes on Amazon AWS
AWS Trainium
AWS Trainium is a purpose-built AI chip designed for one goal: the best economics for high performance AI training and inference at scale. Today's AI builders face two pressures: train models faster to keep pace with innovation, and serve end users at the performance they expect. Lower the cost of both, and you unlock more experiments, faster iteration, and broader reach. The infrastructure behind every foundation model, every agent, every real-time AI experience determines training speed, inference performance, and the cost of both. The system that delivers the best training and token economics wins. Achieving that requires higher speed, lower cost, accessibility, scale, fault tolerance, and developer velocity. Trainium is at the center of a fully integrated system: chip, server, network, software, and services - purpose-built at every layer and co-designed to work as one.
AWS Inferentia
The first-generation AWS Inferentia chip powers Amazon Elastic Compute Cloud (Amazon EC2) Inf1 instances, which deliver up to 2.3x higher throughput and up to 70% lower cost per inference than comparable Amazon EC2 instances. Many customers, including Finch AI, Sprinklr, Money Forward, and Amazon Alexa, have adopted Inf1 instances and realized its performance and cost benefits.
AWS Inferentia2 chip delivers up to 4x higher throughput and up to 10x lower latency compared to Inferentia. Inferentia2-based Amazon EC2 Inf2 instances are optimized to deploy increasingly complex models, such as large language models (LLM) and latent diffusion models, at scale. Inf2 instances are the first inference-optimized instances in Amazon EC2 to support scale-out distributed inference with ultra-high-speed connectivity between chips. Many customers, including Leonardo.ai, Deutsche Telekom, and Qualtrics have adopted Inf2 instances for their DL and generative AI applications.
AWS Neuron SDK helps developers deploy models on the AWS Inferentia chips (and train them on AWS Trainium chips). It integrates natively with popular frameworks, such as PyTorch and TensorFlow, so that you can continue to use your existing code and workflows and run on Inferentia chips.
AWS Neuron - the SDK
AWS Neuron is the developer stack for running deep learning and generative AI workloads on AWS Trainium and AWS Inferentia. Built on an open-source foundation, Neuron enables developers to build, deploy and explore natively with PyTorch and JAX frameworks and with ML libraries such as HuggingFace, vLLM, PyTorch Lightning, and others without modifying your code. It includes a compiler, runtime, training and inference libraries, and developer tools for monitoring, profiling, and debugging. Neuron supports your end-to-end machine learning (ML) development lifecycle from building and deploying deep learning and AI models, optimizing to achieve highest performance and lowest cost, and getting deeper insights into model behavior.
Neuron enables rapid experimentation, production scale training of frontier models, low level performance optimization through the Neuron Kernel Interface (NKI) for custom kernels, cost optimized inference deployment for agentic AI and reinforcement learning workloads, and comprehensive profiling and debugging with Neuron Explorer.
AWS EC2
What is EC2 in AWS?
Amazon EC2, or Amazon Elastic Compute Cloud, is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. With EC2, you can launch as many or as few virtual servers as you need, configure security and networking, and manage storage.
AWS Trainium
The offerings from Amazon AWS are quite extensive
Amazon say that AWS Trainium are "Purpose-built AI chips to deliver high performance and the best economics for AI training and inference at scale"
AWS Inferentia
Get high performance at the lowest cost in Amazon EC2 for deep learning and generative AI inference
AWS Neuron
SDK to optimize AI and deep learning on AWS Trainium and AWS Inferentia
Amazon Elastic Compute Cloud - EC2
Amazon EC2, or Amazon Elastic Compute Cloud is a way that web designers can host their offering on the cloud.