Prerequisites

Before starting, ensure you have:

Node.js 20+ installed
A Next.js 15 project (or create one with npx create-next-app@latest)
A Groq Cloud account and API key from console.groq.com
Basic knowledge of TypeScript and React

What You'll Build

You will build a production-ready streaming AI chat application using Groq's inference API with Llama 4 Scout. By the end, you'll have:

A Next.js API route that streams Groq completions to the browser
A React chat UI with real-time token streaming and speed metrics
Tool use (function calling) with proper argument validation
Model selection across Groq's available models
Robust error handling with exponential backoff

Groq achieves over 800 tokens per second on Llama 4 Scout — roughly 10 times faster than most hosted inference providers. This makes it ideal for latency-sensitive applications like coding assistants, real-time Q&A, and agentic workflows where multiple sequential LLM calls happen.

Step 1: Set Up Your Groq Account

Go to console.groq.com and create a free account.
Navigate to API Keys and click Create API Key.
Copy your key — it starts with gsk_.
Add it to .env.local in your project root.

The free tier includes generous limits for development — 30 requests per minute on most models. Paid plans offer higher rate limits and dedicated capacity for production workloads.

Step 2: Install the Groq SDK

In your Next.js project, install the official Groq TypeScript SDK:

pnpm add groq-sdk

Add your API key to .env.local:

GROQ_API_KEY=gsk_your_key_here

Security: Never expose your GROQ_API_KEY on the client side. Always call Groq from server-side API routes or Server Actions. Next.js automatically excludes env vars without the NEXT_PUBLIC_ prefix from the client bundle.

Step 3: Initialize the Groq Client

Create lib/groq.ts to initialize the Groq client as a singleton:

import Groq from 'groq-sdk';
 
export const groq = new Groq({
  apiKey: process.env.GROQ_API_KEY,
});

Test a simple completion to confirm your setup works:

// scripts/test-groq.ts
import { groq } from '../lib/groq';
 
const completion = await groq.chat.completions.create({
  model: 'llama-4-scout-17b-16e-instruct',
  messages: [
    {
      role: 'user',
      content: 'Explain what makes Groq different from other AI inference providers in two sentences.',
    },
  ],
  max_tokens: 256,
});
 
console.log(completion.choices[0].message.content);
console.log('Usage:', completion.usage);

Run it:

npx tsx scripts/test-groq.ts

You'll see a response in under 500 milliseconds, plus usage metadata showing token counts and inference time.

Step 4: Choose the Right Model

Groq supports multiple open models in 2026. Choose based on your use case:

Model ID	Context	Best For
`llama-4-scout-17b-16e-instruct`	131k	Fast chat, coding, Q&A
`llama-4-maverick-17b-128e-instruct`	131k	Complex reasoning, long documents
`llama-3.3-70b-versatile`	128k	High-accuracy tasks
`mixtral-8x7b-32768`	32k	Balanced speed and quality
`gemma2-9b-it`	8k	Lightweight or budget deployments

Recommendation: For most chat and agentic applications, start with llama-4-scout-17b-16e-instruct. It runs at over 800 tokens per second with a 131k token context window — enough for most real-world tasks, and free tier limits are generous.

Step 5: Implement Streaming in a Next.js API Route

Streaming is essential for a good chat UX — users see tokens appear immediately rather than waiting for the full response. Create app/api/chat/route.ts:

import { groq } from '@/lib/groq';
import { NextRequest } from 'next/server';
 
export const runtime = 'nodejs';
 
export async function POST(req: NextRequest) {
  const { messages, model = 'llama-4-scout-17b-16e-instruct' } = await req.json();
 
  const stream = await groq.chat.completions.create({
    model,
    messages,
    stream: true,
    max_tokens: 1024,
    temperature: 0.7,
  });
 
  const encoder = new TextEncoder();
 
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const delta = chunk.choices[0]?.delta?.content ?? '';
        if (delta) {
          controller.enqueue(encoder.encode(delta));
        }
      }
      controller.close();
    },
  });
 
  return new Response(readable, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
      'X-Content-Type-Options': 'nosniff',
    },
  });
}

This route:

Accepts a POST request with messages and optional model
Opens a streaming connection to Groq
Forwards each token chunk to the browser the moment Groq produces it
Closes the stream cleanly when done

Step 6: Build the Streaming Chat UI

Create components/GroqChat.tsx with real-time streaming and speed metrics:

'use client';
 
import { useState, useRef } from 'react';
 
interface Message {
  role: 'user' | 'assistant';
  content: string;
}
 
const MODELS = [
  { value: 'llama-4-scout-17b-16e-instruct', label: 'Llama 4 Scout (fastest)' },
  { value: 'llama-4-maverick-17b-128e-instruct', label: 'Llama 4 Maverick' },
  { value: 'llama-3.3-70b-versatile', label: 'Llama 3.3 70B' },
  { value: 'mixtral-8x7b-32768', label: 'Mixtral 8x7B' },
];
 
export function GroqChat() {
  const [messages, setMessages] = useState<Message[]>([]);
  const [input, setInput] = useState('');
  const [model, setModel] = useState(MODELS[0].value);
  const [loading, setLoading] = useState(false);
  const [tokensPerSec, setTokensPerSec] = useState<number | null>(null);
  const abortRef = useRef<AbortController | null>(null);
 
  async function sendMessage() {
    if (!input.trim() || loading) return;
 
    const userMessage: Message = { role: 'user', content: input };
    const newMessages = [...messages, userMessage];
    setMessages(newMessages);
    setInput('');
    setLoading(true);
    setTokensPerSec(null);
 
    const assistantMessage: Message = { role: 'assistant', content: '' };
    setMessages([...newMessages, assistantMessage]);
 
    abortRef.current = new AbortController();
    const startTime = performance.now();
    let tokenCount = 0;
 
    try {
      const res = await fetch('/api/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ messages: newMessages, model }),
        signal: abortRef.current.signal,
      });
 
      const reader = res.body!.getReader();
      const decoder = new TextDecoder();
      let accumulated = '';
 
      while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        const chunk = decoder.decode(value, { stream: true });
        accumulated += chunk;
        tokenCount += chunk.split(' ').length;
        setMessages(prev => [
          ...prev.slice(0, -1),
          { role: 'assistant', content: accumulated },
        ]);
      }
 
      const elapsed = (performance.now() - startTime) / 1000;
      setTokensPerSec(Math.round(tokenCount / elapsed));
    } catch (err: unknown) {
      if (err instanceof Error && err.name !== 'AbortError') {
        console.error('Groq stream error:', err);
      }
    } finally {
      setLoading(false);
    }
  }
 
  return (
    <div className="max-w-2xl mx-auto p-4 flex flex-col gap-4">
      <div className="flex items-center gap-2">
        <select
          value={model}
          onChange={e => setModel(e.target.value)}
          className="border rounded px-2 py-1 text-sm"
        >
          {MODELS.map(m => (
            <option key={m.value} value={m.value}>{m.label}</option>
          ))}
        </select>
        {tokensPerSec !== null && (
          <span className="text-sm text-green-600 font-mono">
            {tokensPerSec} tok/s
          </span>
        )}
      </div>
 
      <div className="flex flex-col gap-2 min-h-64 border rounded p-3 bg-gray-50">
        {messages.map((msg, i) => (
          <div
            key={i}
            className={`rounded p-2 text-sm whitespace-pre-wrap ${
              msg.role === 'user' ? 'bg-blue-100 self-end' : 'bg-white self-start'
            }`}
          >
            {msg.content}
          </div>
        ))}
        {loading && messages.at(-1)?.content === '' && (
          <div className="text-gray-400 text-sm animate-pulse">Thinking…</div>
        )}
      </div>
 
      <div className="flex gap-2">
        <input
          value={input}
          onChange={e => setInput(e.target.value)}
          onKeyDown={e => e.key === 'Enter' && !e.shiftKey && sendMessage()}
          placeholder="Type a message…"
          className="flex-1 border rounded px-3 py-2 text-sm"
          disabled={loading}
        />
        <button
          onClick={loading ? () => abortRef.current?.abort() : sendMessage}
          className="px-4 py-2 bg-blue-600 text-white rounded text-sm"
        >
          {loading ? 'Stop' : 'Send'}
        </button>
      </div>
    </div>
  );
}

Add it to a page:

// app/chat/page.tsx
import { GroqChat } from '@/components/GroqChat';
 
export default function ChatPage() {
  return (
    <main className="py-12">
      <h1 className="text-2xl font-bold text-center mb-8">Groq AI Chat</h1>
      <GroqChat />
    </main>
  );
}

Step 7: Add Tool Use (Function Calling)

Groq supports OpenAI-compatible tool use on Llama 4 and Llama 3.3 models. Here's a complete example with a weather tool:

import Groq from 'groq-sdk';
import { z } from 'zod';
import { groq } from '@/lib/groq';
 
const tools: Groq.Chat.Completions.ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get the current weather for a city',
      parameters: {
        type: 'object',
        properties: {
          city: { type: 'string', description: 'City name' },
          unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
        required: ['city'],
      },
    },
  },
];
 
const WeatherArgs = z.object({
  city: z.string(),
  unit: z.enum(['celsius', 'fahrenheit']).default('celsius'),
});
 
async function getWeather(city: string, unit: string) {
  // Replace with a real weather API in production
  return { city, temperature: unit === 'celsius' ? 22 : 72, condition: 'sunny' };
}
 
export async function chatWithTools(userMessage: string) {
  const messages: Groq.Chat.Completions.ChatCompletionMessageParam[] = [
    { role: 'user', content: userMessage },
  ];
 
  const response = await groq.chat.completions.create({
    model: 'llama-4-scout-17b-16e-instruct',
    messages,
    tools,
    tool_choice: 'auto',
  });
 
  const toolCalls = response.choices[0].message.tool_calls;
  if (!toolCalls?.length) {
    return response.choices[0].message.content;
  }
 
  const toolResults = await Promise.all(
    toolCalls.map(async tc => {
      const args = WeatherArgs.parse(JSON.parse(tc.function.arguments));
      const result = await getWeather(args.city, args.unit);
      return {
        tool_call_id: tc.id,
        role: 'tool' as const,
        content: JSON.stringify(result),
      };
    })
  );
 
  const finalResponse = await groq.chat.completions.create({
    model: 'llama-4-scout-17b-16e-instruct',
    messages: [
      ...messages,
      response.choices[0].message,
      ...toolResults,
    ],
  });
 
  return finalResponse.choices[0].message.content;
}

Validation tip: Always parse tool call arguments with Zod or a similar schema library before executing them. The model can occasionally produce malformed JSON or unexpected field types — validation catches these before they cause runtime errors.

Step 8: System Prompts for Specialized Assistants

System prompts shape assistant behavior and personality. Here is a coding assistant configuration:

const CODING_ASSISTANT_PROMPT = `You are an expert TypeScript and React developer.
- Always provide complete, working code examples
- Explain the reasoning behind architectural decisions  
- Point out potential performance issues or security concerns
- Keep explanations concise but thorough`;
 
const completion = await groq.chat.completions.create({
  model: 'llama-4-maverick-17b-128e-instruct',
  messages: [
    { role: 'system', content: CODING_ASSISTANT_PROMPT },
    { role: 'user', content: 'How do I implement optimistic updates in React Query v5?' },
  ],
  max_tokens: 2048,
  temperature: 0.3,
});

Temperature guide:

0.1–0.3 — factual, consistent responses (technical Q&A, code generation)
0.5–0.7 — balanced creativity and coherence (general chat)
0.8–1.0 — more varied, creative responses (brainstorming, writing)

Step 9: Error Handling with Exponential Backoff

The free tier has rate limits. Implement robust error handling:

import Groq from 'groq-sdk';
import { groq } from '@/lib/groq';
 
export async function safeGroqCompletion(
  messages: Groq.Chat.Completions.ChatCompletionMessageParam[],
  retries = 3
): Promise<string | null> {
  for (let attempt = 0; attempt < retries; attempt++) {
    try {
      const completion = await groq.chat.completions.create({
        model: 'llama-4-scout-17b-16e-instruct',
        messages,
        max_tokens: 1024,
      });
      return completion.choices[0].message.content;
    } catch (error) {
      if (error instanceof Groq.RateLimitError) {
        const delay = Math.pow(2, attempt) * 1000;
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }
      if (error instanceof Groq.APIError) {
        console.error(`Groq API error ${error.status}:`, error.message);
        return null;
      }
      throw error;
    }
  }
  return null;
}

Production tip: Cache repeated queries with Upstash Redis or a similar TTL-based cache. Many users ask similar questions, and caching reduces both costs and rate-limit pressure. A 5-minute TTL on deterministic prompts (temperature 0) is a safe starting point.

Step 10: Measure and Compare Performance

One of Groq's key advantages is raw inference speed. Track it alongside your application metrics:

export async function completionWithMetrics(prompt: string) {
  const start = Date.now();
 
  const completion = await groq.chat.completions.create({
    model: 'llama-4-scout-17b-16e-instruct',
    messages: [{ role: 'user', content: prompt }],
    max_tokens: 512,
  });
 
  const elapsed = (Date.now() - start) / 1000;
  const usage = completion.usage!;
 
  return {
    content: completion.choices[0].message.content,
    metrics: {
      totalTokens: usage.total_tokens,
      completionTokens: usage.completion_tokens,
      promptTokens: usage.prompt_tokens,
      elapsedSeconds: elapsed,
      tokensPerSecond: Math.round(usage.completion_tokens / elapsed),
    },
  };
}

Groq consistently delivers over 800 tokens per second on Llama 4 Scout. For comparison, most hosted inference providers deliver 30–80 tokens per second. This 10x speed advantage matters most in agentic loops where 5–10 sequential LLM calls happen per user request.

Troubleshooting

"Invalid API Key" error Make sure your key starts with gsk_ and is set in .env.local, not .env. Next.js only loads .env.local automatically in development.

Rate limit errors in development The free tier allows 30 requests per minute on most models. Use exponential backoff (Step 9) and consider switching to gemma2-9b-it during development — it has a separate, less-contested rate limit bucket.

Streaming not working in production Ensure your deployment platform supports streaming responses. Vercel, Cloudflare Workers, and Railway all support streaming. Add export const runtime = 'nodejs' to your API route if you encounter issues on edge runtimes.

Tool call parsing errors Always validate tool arguments with Zod before executing (Step 7). The LLM can occasionally produce malformed JSON — runtime validation prevents those from propagating.

Context window exceeded Llama 4 Scout has a 131k context window. If you hit limits, trim the oldest messages from history while keeping the system prompt and the last N exchanges. A sliding window of 20 messages is a practical default.

Next Steps

Explore Groq's vision endpoints for multimodal image understanding tasks
Try Groq's audio transcription API (Whisper v3 Large) — it processes audio at 189x realtime speed
Build a multi-agent workflow where Groq handles fast-path classification and Claude handles complex reasoning
Add Langfuse for LLM observability and prompt versioning across model providers
Implement Cloudflare Workers as an edge proxy in front of Groq for global low-latency inference

Conclusion

Groq's inference speed fundamentally changes what is possible in AI applications. At over 800 tokens per second, you can build truly responsive AI experiences — coding assistants that feel instant, agentic workflows that complete in seconds instead of minutes, and real-time AI features that do not compromise UX.

The Groq SDK's OpenAI-compatible API design means you can migrate existing OpenAI projects with minimal code changes. Combined with Llama 4 Scout's 131k context window, strong coding and reasoning capabilities, and a generous free tier, Groq is one of the most practical choices for production AI applications in 2026.