Ploy
Ploy
Features

AI Integration

Build AI-powered applications with OpenAI-compatible models in your Ploy workers.

AI Integration

Ploy provides built-in AI integration for workers, allowing you to build AI-powered applications with env.AI.run() and with direct OpenAI-compatible SDK calls when needed.

Enabling AI

To use AI in your worker, add ai: true to your ploy.yaml:

ploy.yaml
kind: dynamic
ai: true

This injects an env.AI binding plus the raw escape-hatch environment variables:

  • AI - AI binding with env.AI.run(model, inputs, options?)
  • PLOY_AI_URL - OpenAI-compatible base URL for direct SDK or HTTP requests
  • PLOY_AI_TOKEN - Authentication token (automatically managed per deployment)

Local Development

When you run ploy dev, remote AI bindings are attributed to an organization-scoped local development AI sandbox in Ploy.

  • If your checkout matches a deployed project, Ploy uses that project to choose the organization, then routes local AI usage through the organization's sandbox.
  • If your checkout does not match a deployed project and you belong to multiple organizations, set PLOY_DEV_AI_PROJECT_ID to any project in the target organization to choose the correct sandbox.
  • For full manual control, set both PLOY_DEV_AI_URL and PLOY_DEV_AI_TOKEN.

Example:

PLOY_DEV_AI_PROJECT_ID=proj_123 pnpm exec ploy dev

Basic AI Example

Here's a simple worker that calls an AI model:

src/index.ts
interface Env {
	AI: Ai;
	PLOY_AI_URL: string;
	PLOY_AI_TOKEN: string;
}

export default {
	async fetch(request, env) {
		const url = new URL(request.url);

		// Health check endpoint
		if (url.pathname === "/health") {
			return new Response("ok");
		}

		// AI endpoint
		if (url.pathname === "/ai") {
			try {
				// Get parameters from query string
				const model = url.searchParams.get("model") || "glm-4.6v-flash";
				const prompt = url.searchParams.get("prompt") || "just reply 'OK'";

				const data = await env.AI.run(model, { prompt });
				return new Response(JSON.stringify(data), {
					status: 200,
					headers: { "Content-Type": "application/json" },
				});
			} catch (error) {
				return new Response(
					JSON.stringify({
						error: error instanceof Error ? error.message : String(error),
					}),
					{
						status: 500,
						headers: { "Content-Type": "application/json" },
					},
				);
			}
		}

		return new Response("hi!");
	},
};

Usage

# Basic request
curl https://your-deployment.ploy.app/ai

# Custom prompt
curl "https://your-deployment.ploy.app/ai?prompt=What is TypeScript?"

# Different model
curl "https://your-deployment.ploy.app/ai?model=gpt-4&prompt=Hello"

env.AI.run()

Ploy exposes env.AI.run(model, inputs, options?):

const response = await env.AI.run("auto", {
	prompt: "Hello, World",
});

env.AI.run() is the primary worker API. The raw PLOY_AI_URL and PLOY_AI_TOKEN variables remain available when you want to call the OpenAI-compatible gateway directly with your own client.

Examples

  • examples/ai-simple uses env.AI.run() for the basic request path.
  • examples/ai-streaming uses env.AI.run() with stream: true and returns SSE.
  • examples/ai-openai uses the OpenAI SDK with PLOY_AI_URL and PLOY_AI_TOKEN.

OpenAI-Compatible API

Ploy's AI integration uses the OpenAI chat completions format:

Request Format

{
  model: string;           // Model name (e.g., "glm-4.6v-flash", "gpt-4")
  messages: Array<{        // Conversation messages
    role: "system" | "user" | "assistant";
    content: string;
  }>;
  temperature?: number;    // Randomness (0-2, default: 1)
  max_tokens?: number;     // Maximum response length
  top_p?: number;         // Nucleus sampling (0-1, default: 1)
  stream?: boolean;       // Enable streaming responses
}

Response Format

{
	id: string;
	object: "chat.completion";
	created: number;
	model: string;
	choices: Array<{
		index: number;
		message: {
			role: "assistant";
			content: string;
		};
		finish_reason: string;
	}>;
	usage: {
		prompt_tokens: number;
		completion_tokens: number;
		total_tokens: number;
	}
}

Conversation Context

Build conversational applications by maintaining message history:

interface Message {
	role: "system" | "user" | "assistant";
	content: string;
}

export default {
	async fetch(request, env) {
		if (request.method !== "POST") {
			return new Response("Method Not Allowed", { status: 405 });
		}

		try {
			const { messages, model = "glm-4.6v-flash" } = await request.json();

			// Add system prompt
			const fullMessages: Message[] = [
				{
					role: "system",
					content: "You are a helpful assistant that provides concise answers.",
				},
				...messages,
			];

			const data = await env.AI.run(model, {
				messages: fullMessages,
			});
			return new Response(JSON.stringify(data), {
				headers: { "Content-Type": "application/json" },
			});
		} catch (error) {
			return new Response(
				JSON.stringify({
					error: error instanceof Error ? error.message : String(error),
				}),
				{
					status: 500,
					headers: { "Content-Type": "application/json" },
				},
			);
		}
	},
};

Direct SDK Usage

If you need a library that expects an OpenAI-compatible baseURL and apiKey, use the injected raw variables:

import OpenAI from "openai";

const client = new OpenAI({
	apiKey: env.PLOY_AI_TOKEN,
	baseURL: env.PLOY_AI_URL,
});

const response = await client.chat.completions.create({
	model: "auto",
	messages: [{ role: "user", content: "Hello" }],
});

Vercel AI SDK

Next.js projects with ai: true get Ploy registered as the AI SDK's default provider, so a bare model id resolves through Ploy instead of the Vercel AI Gateway:

app/api/chat/route.ts
import { convertToModelMessages, streamText } from "ai";

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = streamText({
		model: "anthropic/claude-sonnet-5",
		messages: await convertToModelMessages(messages),
	});

	return result.toUIMessageStreamResponse();
}

No provider setup, no AI_GATEWAY_API_KEY, and no code change when moving an existing AI Gateway app onto Ploy — model ids keep the same provider/model format. Ploy wires this up during the build (and during ploy dev) by generating two files next to your app directory:

  • ploy-ai-provider.js — registers globalThis.AI_SDK_DEFAULT_PROVIDER
  • instrumentation.js — imports it through the Next.js instrumentation hook

If you already have an instrumentation file, Ploy adds the import to it and leaves the rest alone. Both files are regenerated on every build, so you can commit them or add them to .gitignore.

This needs ai v5 or later in your project. To opt out, set globalThis.AI_SDK_DEFAULT_PROVIDER yourself or pass a model instance instead of a string — Ploy never overwrites a provider that is already registered.

Set PLOY_AI_GATEWAY_URL to point the AI SDK at a specific gateway URL. By default it is derived from PLOY_AI_URL.

Streaming

env.AI.run() also supports streaming:

const stream = await env.AI.run("auto", {
	prompt: "Write one short sentence.",
	stream: true,
});

return new Response(stream, {
	headers: {
		"Content-Type": "text/event-stream; charset=utf-8",
	},
});

Usage

curl -X POST https://your-deployment.ploy.app/ai \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is JavaScript?"},
      {"role": "assistant", "content": "JavaScript is a programming language..."},
      {"role": "user", "content": "How do I use async/await?"}
    ]
  }'

LangChain Integration

Use LangChain for advanced AI features like agents and tools:

Installation

package.json
{
	"dependencies": {
		"@langchain/core": "^1.1.0",
		"@langchain/openai": "^1.1.3",
		"langchain": "^1.1.1",
		"zod": "^3.24.1"
	}
}

Creating Tools

Define tools that the AI can use:

src/index.ts
import { ChatOpenAI } from "@langchain/openai";
import { createAgent, tool } from "langchain";
import * as z from "zod";

interface Env {
	PLOY_AI_URL: string;
	PLOY_AI_TOKEN: string;
}

// Define a tool for getting weather information
const getWeather = tool((input) => `It's always sunny in ${input.city}!`, {
	name: "get_weather",
	description: "Get the weather for a given city",
	schema: z.object({
		city: z.string().describe("The city to get the weather for"),
	}),
});

export default {
	async fetch(request, env) {
		const url = new URL(request.url);

		if (url.pathname === "/health") {
			return new Response("ok");
		}

		if (url.pathname === "/ai") {
			try {
				const prompt = url.searchParams.get("prompt") || "just reply 'OK'";

				// Initialize OpenAI-compatible model
				const model = new ChatOpenAI({
					model: "glm-4.6v-flash",
					apiKey: env.PLOY_AI_TOKEN,
					configuration: {
						baseURL: env.PLOY_AI_URL,
					},
				});

				// Create agent with tools
				const agent = createAgent({
					model,
					tools: [getWeather],
				});

				// Invoke agent
				const result = await agent.invoke({
					messages: [{ role: "user", content: prompt }],
				});

				// Extract response
				const lastMessage = result.messages[result.messages.length - 1];
				const content =
					typeof lastMessage.content === "string"
						? lastMessage.content
						: JSON.stringify(lastMessage.content);

				// Return in OpenAI format
				return new Response(
					JSON.stringify({
						choices: [
							{
								message: {
									role: "assistant",
									content: content,
								},
							},
						],
					}),
					{
						status: 200,
						headers: { "Content-Type": "application/json" },
					},
				);
			} catch (error) {
				return new Response(
					JSON.stringify({
						error: error instanceof Error ? error.message : String(error),
					}),
					{
						status: 500,
						headers: { "Content-Type": "application/json" },
					},
				);
			}
		}

		return new Response("LangChain agent example!");
	},
};

Token Usage and Billing

Monitor token consumption in API responses:

const data = await response.json();

// Extract token usage
const usage = data.usage;
console.log(`Prompt tokens: ${usage.prompt_tokens}`);
console.log(`Completion tokens: ${usage.completion_tokens}`);
console.log(`Total tokens: ${usage.total_tokens}`);

// Track costs (example rates)
const cost = (
	usage.prompt_tokens * 0.00001 +
	usage.completion_tokens * 0.00002
).toFixed(4);
console.log(`Estimated cost: $${cost}`);

Token usage varies by model. Larger models (like GPT-5) cost more per token than smaller models (like GPT-4o-mini).

Examples

Summarization Service

export default {
	async fetch(request, env) {
		if (request.method !== "POST") {
			return new Response("Method Not Allowed", { status: 405 });
		}

		const { text } = await request.json();

		const response = await fetch(`${env.PLOY_AI_URL}/chat/completions`, {
			method: "POST",
			headers: {
				"Content-Type": "application/json",
				Authorization: `Bearer ${env.PLOY_AI_TOKEN}`,
			},
			body: JSON.stringify({
				model: "glm-4.6v-flash",
				messages: [
					{
						role: "system",
						content:
							"You are a summarization assistant. Provide concise summaries.",
					},
					{
						role: "user",
						content: `Summarize the following text:\n\n${text}`,
					},
				],
				max_tokens: 150,
			}),
		});

		const data = await response.json();
		const summary = data.choices[0].message.content;

		return new Response(JSON.stringify({ summary }), {
			headers: { "Content-Type": "application/json" },
		});
	},
};

Next Steps

How is this guide?

Last updated on

Timers

Schedule durable one-shot and recurring timers that fire your worker at a specified time.

Skew Protection

Pin a client's requests to the deployment that served its page, avoiding version-skew errors after a new deployment goes live.

On this page

AI Integration
Enabling AI
Local Development
Basic AI Example
Usage
env.AI.run()
Examples
OpenAI-Compatible API
Request Format
Response Format
Conversation Context
Direct SDK Usage
Vercel AI SDK
Streaming
Usage
LangChain Integration
Installation
Creating Tools
Token Usage and Billing
Examples
Summarization Service
Next Steps