Skip to content

Benefits of AIPC's

Key Factors:

  • Time Saved = more iterations possible in a shorter time frame
  • Quality = more iterations in the same time frame
  • Focus = AI takes care of repetitive tasks so you can focus on what matters

NPUs vs GPUs

NPUs typically deliver 10× or more performance per mm² for AI inference compared to GPUs because:

  • Specifically designed for low-precision compute (INT8, INT4) which AI most commonly uses. GPUs traditionally focus on FP32 which is more compute intensive and not required for AI.
  • They minimize data movement with on-chip SRAM. GPUs rely heavily on external memory (GDDR), which introduces latency and higher power consumption.

Info

Closer memory to compute = more bandwidth = lower latency = faster compute

From worst to best: 

graph LR
  A[Socketed DRAM/DIMMs] --> 
  B[Soldered DRAM/LPDDR] --> 
  C[VRAM/GDDR/HBMs] --> 
  D[On-chip SRAM];
  • They have hardwired pipelines for common AI ops. They integrate specialized memory hierarchies (scratchpads, tiling strategies) to reduce DRAM access and exploit reuse in convolution layers.
  • This leads to higher TOPS/Watt because they eliminate unnecessary logic for non-AI tasks and optimize for reduced data movement.

Note: Unified Memory is a memory architecture, not a physical memory type (however it is most commonly Soldered on DRAM/LPDDR). It means CPU and GPU (or NPU) share the same physical memory space, eliminating the need for explicit data copies between them. Unified memory is generally better for mobile SoCs because it simplifies design and saves power, even though dedicated VRAM would give higher bandwidth for GPUs in high-end scenarios. Unified memory still has higher latency than on-chip SRAM or HBM-based VRAM

Radeon Open Compute (ROCm, pronounced “Rock-em”) is AMD’s open-source software stack for GPU and AI acceleration, similar to NVIDIA’s CUDA ecosystem.

Microsoft ML

Microsoft’s ecosystem of tools, frameworks, and services for machine learning and AI development. It’s not a single product but a collection of technologies designed to make building, training, and deploying ML models easier across different environments.

Key Components

  1. ML.NET

  2. An open-source, cross-platform machine learning framework for .NET developers.

  3. Allows you to build custom ML models using C# or F# without needing Python or R.

  4. ONNX Runtime

  5. A high-performance inference engine for running models in the Open Neural Network Exchange (ONNX) format.

  6. Supports multiple hardware backends (CPU, GPU, accelerators).

  7. Azure Machine Learning

  8. A cloud-based platform for training, deploying, and managing ML models at scale.

  9. Includes automated ML, pipelines, and integration with popular frameworks like TensorFlow and PyTorch.

  10. Integration with Microsoft Products

  11. ML capabilities embedded in tools like Power BI, Excel, and Dynamics 365 for predictive analytics and AI-driven insights.

Purpose
  • Provide a standardized API layer for ML workloads (similar to how DirectX standardizes graphics).
  • Enable developers to train, deploy, and run models efficiently across different platforms and hardware.
  • Support interoperability between frameworks via ONNX.