Perplexity's Ivy: The HTTP Gateway Behind Its AI Serving Stack

Key Info

Perplexity describes Ivy, its HTTP gateway that handles CPU-side request preparation—parsing, tokenization, templating, and large-batch splitting—before forwarding requests to its Tulip inference servers over gRPC.

Highlights

• Ivy offloads request formatting and tokenization from the heavier inference servers.
• It splits large batches before sending them to Tulip via gRPC.
• This design lets Perplexity tune request handling and tokenization without touching inference infrastructure.