Groq

What is Groq?

Groq is on a mission to set the standard for GenAI inference speed, helping real-time AI applications come to life today.Groq utilizes a technology known as LPU.An LPU Inference Engine, with LPU standing for Language Processing Unit™, is a new type of end-to-end processing unit system that provides the fastest inference for computationally intensive applications with a sequential component to them, such as AI language applications (LLMs).

The LPU is designed to overcome the two LLM bottlenecks: compute density and memory bandwidth.An LPU has greater compute capacity than a GPU and CPU in regards to LLMs.This reduces the amount of time per word calculated, allowing sequences of text to be generated much faster.

Additionally, eliminating external memory bottlenecks enables the LPU Inference Engine to deliver orders of magnitude better performance on LLMs compared to GPUs.To start using Groq, request API access to run LLM applications in a token-based pricing model.

You can also purchase the hardware for on-premise LLM inference using LPUs.