LangSmith

LangSmith is an AI agent evaluation and observability platform by LangChain. Features offline/online evaluations, automated evaluators, expert annotation queues, prompt iteration tools, and scalable pricing by seats and traces.

About LangSmith

LangSmith is a comprehensive evaluation and observability platform designed for continuously improving AI agent quality. Built by LangChain, it supports both offline evaluations on datasets for benchmarking and regression testing, as well as online evaluations on production traffic to monitor real world performance.

The platform enables teams to score agent runs using multiple evaluator types: LLM-as-judge, heuristic checks, pairwise comparisons, and human review. LangSmith's annotation queues allow subject-matter experts to review and label runs systematically, providing structured feedback for improving prompts, evaluators, and datasets.

LangSmith includes a Prompt Playground for experimenting with models and prompts, comparing outputs across versions or providers, and using the Prompt Canvas UI to auto improve prompts. The platform is designed to work across frameworks and providers, not just LangChain, and uses asynchronous trace collection to avoid adding latency to applications.

The platform offers flexible deployment options including managed cloud, bring your own cloud (BYOC), and self hosted for enterprises with strict data residency requirements.

Key Features

Offline Dataset Evals: Run benchmarking and regression tests on curated datasets.
Online Production Evals: Evaluate real traffic in near real time to monitor deployed agents.
Multiple Evaluator Types: LLM-as-judge, heuristic, pairwise, and human evaluation methods.
Annotation Queues: Collect expert feedback via structured review workflows and labeling.
Prompt Iteration Tools: Playground and Canvas for experimenting with and improving prompts.
Zero-Latency Tracing: Asynchronous trace collection that doesn't impact application performance.

Pricing

Developer: Free (1 seat) Access to LangSmith with base trace allowance, pay as you go beyond included usage, 1 free dev deployment with unlimited runs.
Plus: $39 per seat/month Collaboration features, 10k base traces/month included, pay as you go beyond included usage, 50 Agent Builder agents/month.
Enterprise: Custom pricing Advanced hosting options (BYOC, self hosted), enhanced security, custom seats/workspaces, dedicated support.

Additional usage: Deployment runs at $0.005/run, dev deployments at $0.0007/min, production deployments at $0.0036/min. Agent Builder runs at $0.05/run beyond included limits.

Pricing last updated: February 22, 2026 at 9:03 AM

Use Cases

Run regression tests on agents before and after prompt/model changes
Monitor production quality with online evaluations on real traffic
Collect SME feedback through annotation queues to improve quality
Compare outputs across prompt versions and providers systematically
Iterate on prompts using Playground and auto improvement tools
Maintain data residency with self hosted or BYOC deployment options

Pros & Cons

Pros:

Supports both offline benchmarking and online production monitoring
Multiple evaluation modes including expert feedback workflows
Works across frameworks and providers (not LangChain locked)
Flexible deployment: cloud, BYOC, and self hosted options
Zero latency tracing design doesn't impact application performance
Clear free tier with predictable per seat pricing

Cons:

Usage costs scale with trace volume and retention choices
Per seat pricing can become expensive for larger teams
Some advanced features limited to Enterprise tier
Agent Builder usage incurs additional costs beyond base plans

Integrations

OpenAI, Anthropic, CrewAI, Vercel AI SDK, Pydantic AI, LangChain, LlamaIndex, Hugging Face

FAQ

Can I use LangSmith without using LangChain?

Yes, LangSmith works with any framework or provider. It is not locked to LangChain and supports OpenAI, Anthropic, CrewAI, Vercel AI SDK, and others.

Does LangSmith add latency to my application?

No, LangSmith uses asynchronous trace collection via a callback handler. Traces are sent to a distributed collector without blocking your application.

What is the difference between offline and online evaluations?

Offline evaluations run on datasets for benchmarking and regression testing. Online evaluations run on production traffic in near real time to monitor deployed agents.