Skip to content

Prompt Studio

The Prompt Studio is a powerful environment for testing and evaluating prompts across multiple test cases with automated LLM evaluation. Unlike traditional prompt testing tools, Coherence's Prompt Studio works seamlessly with our synthetic data generation to automatically create comprehensive test cases - eliminating the need for manual test case creation.

Overview

Prompt Studio allows you to:

  • Test your system/user prompts against automatically generated test cases
  • Leverage Coherence's synthetic data generation for comprehensive testing
  • Compare outputs across different models
  • Automatically evaluate results using LLM judges
  • Iterate and refine prompts based on performance

Components

Prompt Configuration

  • Define your system and user prompts
  • Variables in prompts are automatically populated from test case inputs
  • Select which model to use for each test run

Integrated Test Cases

  • Automatically generate diverse test cases using Coherence's synthetic data engine
  • Input variables from test cases are intelligently injected into prompts
  • Test against edge cases and scenarios you might not think to create manually
  • Results are stored for comparison across iterations

Test Case Generation

  • Define your test case requirements and constraints
  • Let Coherence automatically generate diverse, realistic test scenarios
  • Save time by eliminating manual test case creation
  • Ensure comprehensive coverage of edge cases and variations

Automated Evaluation

Two LLM evaluators assess each result:

  1. Format Evaluator

    • Checks if the output follows the expected structure
    • Customizable evaluation prompt
    • Configurable scoring: pass/fail, traffic light (1/2/3), or numeric (1-10)

  2. Faithfulness Evaluator

    • Assesses accuracy and reliability of the content
    • Customizable evaluation prompt
    • Same scoring options as format evaluator

Evaluation Prompts

Each evaluator can be configured with custom prompts that receive:

  • Original prompt
  • Actual result
  • Expected output
  • Additional context as needed

Best Practices

  • Let Coherence generate a diverse set of test cases for you
  • Use the generated test cases to discover edge cases you hadn't considered
  • Iterate on prompts based on evaluation results

  • Use different scoring types based on your needs:

  • Pass/fail for binary requirements

  • Traffic light for quick visual assessment
  • Numeric for detailed performance tracking

Integration with Synthetic Data Generation

The true power of Coherence's Prompt Studio comes from its integration with our synthetic data generation engine. While other prompt testing tools require manual creation of test cases, Coherence automatically:

  • Generates diverse, realistic test scenarios
  • Creates edge cases that humans might miss
  • Produces structured test data that matches your schema
  • Scales test case generation to ensure comprehensive coverage

This integration means you can focus on refining your prompts while Coherence handles the complex task of creating comprehensive test scenarios.