Giskard

Summary

Giskard is an open-source Python library, licensed under Apache 2.0, for testing, red-teaming, and vulnerability scanning of LLM-based agents. You install it with pip and import it into your own code, either as the base `giskard` package for checks and suites or with the `scan` extra for its vulnerability and quality scanner; it targets teams building and evaluating agentic systems rather than traditional ML models, which the now-unmaintained v2 line handled. It generates adversarial scenarios such as prompt injection, crescendo-style multi-turn attacks, and GCG suffix attacks, and can pull in third-party scan providers like DeepTeam and LiDAR for extra coverage, positioning it as a working layer over agent evaluation rather than a single benchmark. Development is active, with 106 contributors, 451 commits in the past year, and a release 14 days ago.

Open-Source Evaluation & Testing library for LLM Agents

Command line

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

1. giskard-checks/v1.0.3 Aug 26, 2026 · issue 008

   giskard-checks v1.0.3 lets set_default_generator accept model name strings in addition to generator objects.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch giskard-checks/v1.0.3 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout giskard-checks/v1.0.3

   └──▷ USE IT
   Set the default generator for LLM-as-judge checks using a model name string instead of constructing a generator object.
   python copy

   from giskard.checks import set_default_generator
   
   set_default_generator('openai/gpt-4o-mini')

   • ›Extends set_default_generator to accept plain model name strings (e.g. 'openai/gpt-4o') in addition to generator objects, reducing boilerplate when configuring the default LLM judge or generator.
2. giskard-checks/v1.0.3 Aug 26, 2026 · issue 002

   giskard-checks v1.0.3 lets set_default_generator accept model name strings in addition to generator objects.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch giskard-checks/v1.0.3 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout giskard-checks/v1.0.3

   └──▷ USE IT
   Set the default generator for LLM-as-judge checks using a model name string instead of constructing a generator object.
   python copy

   from giskard.checks import set_default_generator
   
   set_default_generator('openai/gpt-4o-mini')

   • ›Extends set_default_generator to accept plain model name strings (e.g. 'openai/gpt-4o') in addition to generator objects, reducing boilerplate when configuring the default LLM judge or generator.
3. v3.0.0 Aug 26, 2026 · issue 008

   Giskard v3 rewrites as a modular monorepo with new LLM judge checks, OWASP scanning, Suite batching, regex/composition operators, and JUnit XML export.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v3.0.0 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout v3.0.0

   └──▷ USE IT
   Run a grounded-answer eval on your agent and print a human-readable report to spot hallucinations immediately.
   python copy

   import asyncio
   from giskard.checks import Scenario, Groundedness
   
   def get_answer(inputs: str) -> str:
       return my_agent(inputs)  # replace with your model/agent
   
   async def main() -> None:
       scenario = (
           Scenario("test_capital")
           .interact(inputs="What is the capital of France?", outputs=get_answer)
           .check(
               Groundedness(
                   name="answer is grounded",
                   context="France is in Western Europe. Its capital is Paris.",
               )
           )
       )
       result = await scenario.run()
       result.print_report()
   
   asyncio.run(main())

   Automatically red-team your agent across OWASP LLM Top-10 categories, including prompt injection probes, without writing test cases manually.
   python copy

   import asyncio
   from giskard.scan import vulnerability_scan
   
   async def my_agent(inputs: str) -> str:
       return f"Echo: {inputs}"  # replace with your real agent
   
   async def main() -> None:
       await vulnerability_scan(
           target=my_agent,
           description="A customer support chatbot for an e-commerce platform.",
           languages=["en"],
       )
   
   asyncio.run(main())

   • ›Rewrites Giskard as a v3 monorepo structured around three installable packages: giskard-checks, giskard-agents, and giskard-core, with giskard-scan available via pip install 'giskard[scan]'.
   • ›Adds opt-out telemetry via DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1 environment variables, collected through optional PostHog analytics in giskard-core.
   • ›Adds RegexMatching check to giskard-checks for asserting outputs match a regular expression, with ReDoS mitigation via regex timeout.
   • ›Adds AllOf, AnyOf, and Not check composition operators to giskard-checks for combining existing checks with boolean logic.
   • ›Adds AnswerRelevance LLM judge check to giskard-checks.
   +18 moreshow less

   • ›Adds Toxicity LLM judge check to giskard-checks.
   • ›Adds annotations support to Scenario/trace objects for attaching metadata to eval steps.
   • ›Introduces Suite for running batches of scenarios with dynamic binding; Suite.append() is chainable.
   • ›Allows scenarios to run multiple times within a suite.
   • ›Adds print_report() to result models for human-readable check output.
   • ›Adds scenario and check error details to suite reports.
   • ›Adds JUnit XML export for SuiteResult, enabling CI integration.
   • ›Adds JSON validity check to giskard-checks.
   • ›Adds vulnerability_scan function in giskard-scan for automated red-teaming across OWASP LLM Top-10 threat categories including prompt injection, harmful content, stereotypes, and misinformation.
   • ›Adds minimal OWASP LLM suite generator covering LLM01 indirect injection.
   • ›Introduces giskard-llm as a lean, provider-agnostic LLM routing layer replacing litellm, supporting provider extras (openai, anthropic, etc.) via pip install 'giskard[openai]'.
   • ›Introduces personas and extended context support for UserSimulator in generators.
   • ›Adds metadata parameter to the generator completion pipeline.
   • ›Adds generator retry and timeout policies to giskard-agents.
   • ›Adds step-level type discriminator and tool input coercion with output serialization to giskard-agents.
   • ›Adds generator-as-protocol-adapter pattern to giskard-agents for wrapping arbitrary LLM backends.
   • ›Supports pydantic-compatible input types in checks.
   • ›Introduces a judges directory for LLM and prompt-reliant checks in giskard-checks.
   └──▷ BREAKING ON UPGRADE

   • !Giskard v3 is a full rewrite; v2 APIs (giskard.Model, giskard.Dataset, giskard.testing, Giskard Hub) are not available in v3. Install pip install 'giskard[llm]>2,<3' to keep v2.
   • !Requires Python 3.12+; Python versions below 3.12 are no longer supported.
   • !Scenario.from_sequence is removed; use the step-based Scenario API instead.
   • !Templating in conformity rules is no longer supported; configurations relying on template syntax in conformity rules will break.
   • !Jinja parsing in Workflow.chat is now opt-in; workflows that relied on Jinja template rendering by default will no longer render templates unless explicitly enabled.
   • !The Interact injection is now name-based; code using positional injection patterns will break.
   • !The scenario() factory is removed; use the mutable Scenario constructor directly.
4. v3.0.0 Aug 26, 2026 · issue 002

   Giskard v3 rewrites as a modular monorepo with new LLM judge checks, OWASP scanning, Suite batching, regex/composition operators, and JUnit XML export.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v3.0.0 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout v3.0.0

   └──▷ USE IT
   Run a grounded-answer eval on your agent and print a human-readable report to spot hallucinations immediately.
   python copy

   import asyncio
   from giskard.checks import Scenario, Groundedness
   
   def get_answer(inputs: str) -> str:
       return my_agent(inputs)  # replace with your model/agent
   
   async def main() -> None:
       scenario = (
           Scenario("test_capital")
           .interact(inputs="What is the capital of France?", outputs=get_answer)
           .check(
               Groundedness(
                   name="answer is grounded",
                   context="France is in Western Europe. Its capital is Paris.",
               )
           )
       )
       result = await scenario.run()
       result.print_report()
   
   asyncio.run(main())

   Automatically red-team your agent across OWASP LLM Top-10 categories, including prompt injection probes, without writing test cases manually.
   python copy

   import asyncio
   from giskard.scan import vulnerability_scan
   
   async def my_agent(inputs: str) -> str:
       return f"Echo: {inputs}"  # replace with your real agent
   
   async def main() -> None:
       await vulnerability_scan(
           target=my_agent,
           description="A customer support chatbot for an e-commerce platform.",
           languages=["en"],
       )
   
   asyncio.run(main())

   • ›Rewrites Giskard as a v3 monorepo structured around three installable packages: giskard-checks, giskard-agents, and giskard-core, with giskard-scan available via pip install 'giskard[scan]'.
   • ›Adds opt-out telemetry via DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1 environment variables, collected through optional PostHog analytics in giskard-core.
   • ›Adds RegexMatching check to giskard-checks for asserting outputs match a regular expression, with ReDoS mitigation via regex timeout.
   • ›Adds AllOf, AnyOf, and Not check composition operators to giskard-checks for combining existing checks with boolean logic.
   • ›Adds AnswerRelevance LLM judge check to giskard-checks.
   +18 moreshow less

   • ›Adds Toxicity LLM judge check to giskard-checks.
   • ›Adds annotations support to Scenario/trace objects for attaching metadata to eval steps.
   • ›Introduces Suite for running batches of scenarios with dynamic binding; Suite.append() is chainable.
   • ›Allows scenarios to run multiple times within a suite.
   • ›Adds print_report() to result models for human-readable check output.
   • ›Adds scenario and check error details to suite reports.
   • ›Adds JUnit XML export for SuiteResult, enabling CI integration.
   • ›Adds JSON validity check to giskard-checks.
   • ›Adds vulnerability_scan function in giskard-scan for automated red-teaming across OWASP LLM Top-10 threat categories including prompt injection, harmful content, stereotypes, and misinformation.
   • ›Adds minimal OWASP LLM suite generator covering LLM01 indirect injection.
   • ›Introduces giskard-llm as a lean, provider-agnostic LLM routing layer replacing litellm, supporting provider extras (openai, anthropic, etc.) via pip install 'giskard[openai]'.
   • ›Introduces personas and extended context support for UserSimulator in generators.
   • ›Adds metadata parameter to the generator completion pipeline.
   • ›Adds generator retry and timeout policies to giskard-agents.
   • ›Adds step-level type discriminator and tool input coercion with output serialization to giskard-agents.
   • ›Adds generator-as-protocol-adapter pattern to giskard-agents for wrapping arbitrary LLM backends.
   • ›Supports pydantic-compatible input types in checks.
   • ›Introduces a judges directory for LLM and prompt-reliant checks in giskard-checks.
   └──▷ BREAKING ON UPGRADE

   • !Giskard v3 is a full rewrite; v2 APIs (giskard.Model, giskard.Dataset, giskard.testing, Giskard Hub) are not available in v3. Install pip install 'giskard[llm]>2,<3' to keep v2.
   • !Requires Python 3.12+; Python versions below 3.12 are no longer supported.
   • !Scenario.from_sequence is removed; use the step-based Scenario API instead.
   • !Templating in conformity rules is no longer supported; configurations relying on template syntax in conformity rules will break.
   • !Jinja parsing in Workflow.chat is now opt-in; workflows that relied on Jinja template rendering by default will no longer render templates unless explicitly enabled.
   • !The Interact injection is now name-based; code using positional injection patterns will break.
   • !The scenario() factory is removed; use the mutable Scenario constructor directly.
5. giskard-core/v1.0.1rc1 Aug 18, 2026 · issue 005

   Giskard v1.0.1rc1 adds PEP 561 type stubs, tighter public Literal/status types, and expanded prompt-injection scenarios for LLM agents.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch giskard-core/v1.0.1rc1 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout giskard-core/v1.0.1rc1

   • ›Ships PEP 561 py.typed marker files for giskard-core, giskard-llm, giskard-checks, and giskard-scan, enabling full static-type-checking support in downstream projects.
   • ›Exports and tightens public Literal/status types from giskard.types, giving typed Python codebases a stable surface to import against.
   • ›Expands the built-in prompt-injection dataset with additional scenarios and templates, broadening adversarial coverage in vulnerability_scan.
   └──▷ BREAKING ON UPGRADE

   • !Unknown fields anywhere in the persisted-scenario tree are now rejected (strict validation); saved scenarios containing unrecognised fields will fail to load after upgrade.
   • !The field holding the value under test on every check is renamed to target_key; any code that referenced the previous field name will break.
6. giskard-core/v1.0.1rc1 Aug 18, 2026 · issue 002

   Giskard v1.0.1rc1 adds PEP 561 type stubs, tighter public Literal/status types, and expanded prompt-injection scenarios for LLM agents.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch giskard-core/v1.0.1rc1 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout giskard-core/v1.0.1rc1

   • ›Ships PEP 561 py.typed marker files for giskard-core, giskard-llm, giskard-checks, and giskard-scan, enabling full static-type-checking support in downstream projects.
   • ›Exports and tightens public Literal/status types from giskard.types, giving typed Python codebases a stable surface to import against.
   • ›Expands the built-in prompt-injection dataset with additional scenarios and templates, broadening adversarial coverage in vulnerability_scan.
   └──▷ BREAKING ON UPGRADE

   • !Unknown fields anywhere in the persisted-scenario tree are now rejected (strict validation); saved scenarios containing unrecognised fields will fail to load after upgrade.
   • !The field holding the value under test on every check is renamed to target_key; any code that referenced the previous field name will break.
7. v3.0.0b3 Aug 14, 2026 · issue 005

   Giskard v3.0.0b3 ships a modular async-first eval and red-teaming library for agentic systems with vulnerability_scan, quality_scan, and LLM-as-judge checks.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v3.0.0b3 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout v3.0.0b3

   └──▷ USE IT
   Red-team a customer-facing chatbot against prompt injection and harmful content probes without writing any test cases manually.
   python copy

   import asyncio
   from giskard.scan import vulnerability_scan
   
   async def my_agent(inputs: str) -> str:
       return f"Echo: {inputs}"  # replace with your real agent
   
   async def main() -> None:
       await vulnerability_scan(
           target=my_agent,
           description="A customer support chatbot for an e-commerce platform.",
           languages=["en"],
       )
   
   asyncio.run(main())

   Verify a RAG answer is grounded in its retrieved context using the built-in LLM-as-judge check, catching hallucinations in CI.
   python copy

   import asyncio
   from giskard.checks import Scenario, Groundedness
   
   def get_answer(inputs: str) -> str:
       return "Paris"  # replace with your RAG pipeline
   
   async def main() -> None:
       scenario = (
           Scenario("test_capital_grounded")
           .interact(inputs="What is the capital of France?", outputs=get_answer)
           .check(
               Groundedness(
                   name="answer is grounded",
                   context="France is in Western Europe. Its capital is Paris.",
               )
           )
       )
       result = await scenario.run()
       result.print_report()
   
   asyncio.run(main())

   • ›Adds vulnerability_scan function in giskard.scan for automated red-teaming of agents across OWASP LLM Top-10 threat categories including prompt injection, harmful content, stereotypes, and misinformation — accepts target, description, and languages parameters.
   • ›Adds quality_scan with KnowledgeBase support in giskard-scan for RAG knowledge-base quality evaluation, replacing v2 RAGET.
   • ›Adds generate_suite function accepting custom ScenarioGenerator instances, with a vulnerability_suite_generator_registry for registering domain-specific generators.
   • ›Introduces Scenario API in giskard.checks for composing multi-turn eval interactions via .interact() and .check() chaining, with a .run() async entrypoint and .print_report() output.
   • ›Ships built-in LLM-as-judge checks — Groundedness, Conformity, and LLMJudge — in giskard.checks, with default model openai/gpt-4o-mini.
   +3 moreshow less

   • ›Adds a Suite abstraction in giskard-checks for running multiple Scenario instances together.
   • ›New giskard[scan], giskard[openai], and giskard[anthropic] install extras wire in provider SDKs and scanner dependencies independently.
   • ›Supports opting out of telemetry via DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1 environment variables set before import.
   └──▷ BREAKING ON UPGRADE

   • !Giskard v2 is no longer actively maintained; the v2 automatic tabular scan (giskard.Model + giskard.Dataset), giskard.testing ML test suite, and Giskard Hub are not present in v3 — install pip install 'giskard[llm]>2,<3' to retain v2 behavior.
   • !Requires Python 3.12+; earlier Python versions are no longer supported.
8. v3.0.0b3 Aug 14, 2026 · issue 002

   Giskard v3.0.0b3 ships a modular async-first eval and red-teaming library for agentic systems with vulnerability_scan, quality_scan, and LLM-as-judge checks.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v3.0.0b3 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout v3.0.0b3

   └──▷ USE IT
   Red-team a customer-facing chatbot against prompt injection and harmful content probes without writing any test cases manually.
   python copy

   import asyncio
   from giskard.scan import vulnerability_scan
   
   async def my_agent(inputs: str) -> str:
       return f"Echo: {inputs}"  # replace with your real agent
   
   async def main() -> None:
       await vulnerability_scan(
           target=my_agent,
           description="A customer support chatbot for an e-commerce platform.",
           languages=["en"],
       )
   
   asyncio.run(main())

   Verify a RAG answer is grounded in its retrieved context using the built-in LLM-as-judge check, catching hallucinations in CI.
   python copy

   import asyncio
   from giskard.checks import Scenario, Groundedness
   
   def get_answer(inputs: str) -> str:
       return "Paris"  # replace with your RAG pipeline
   
   async def main() -> None:
       scenario = (
           Scenario("test_capital_grounded")
           .interact(inputs="What is the capital of France?", outputs=get_answer)
           .check(
               Groundedness(
                   name="answer is grounded",
                   context="France is in Western Europe. Its capital is Paris.",
               )
           )
       )
       result = await scenario.run()
       result.print_report()
   
   asyncio.run(main())

   • ›Adds vulnerability_scan function in giskard.scan for automated red-teaming of agents across OWASP LLM Top-10 threat categories including prompt injection, harmful content, stereotypes, and misinformation — accepts target, description, and languages parameters.
   • ›Adds quality_scan with KnowledgeBase support in giskard-scan for RAG knowledge-base quality evaluation, replacing v2 RAGET.
   • ›Adds generate_suite function accepting custom ScenarioGenerator instances, with a vulnerability_suite_generator_registry for registering domain-specific generators.
   • ›Introduces Scenario API in giskard.checks for composing multi-turn eval interactions via .interact() and .check() chaining, with a .run() async entrypoint and .print_report() output.
   • ›Ships built-in LLM-as-judge checks — Groundedness, Conformity, and LLMJudge — in giskard.checks, with default model openai/gpt-4o-mini.
   +3 moreshow less

   • ›Adds a Suite abstraction in giskard-checks for running multiple Scenario instances together.
   • ›New giskard[scan], giskard[openai], and giskard[anthropic] install extras wire in provider SDKs and scanner dependencies independently.
   • ›Supports opting out of telemetry via DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1 environment variables set before import.
   └──▷ BREAKING ON UPGRADE

   • !Giskard v2 is no longer actively maintained; the v2 automatic tabular scan (giskard.Model + giskard.Dataset), giskard.testing ML test suite, and Giskard Hub are not present in v3 — install pip install 'giskard[llm]>2,<3' to retain v2 behavior.
   • !Requires Python 3.12+; earlier Python versions are no longer supported.
9. giskard-core/v1.0.1b6 Aug 6, 2026 · issue 005

   Giskard v1.0.1b6 adds third-party scan integration, exposes default registry generators on the public API, and enforces non-blank judge reasons.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch giskard-core/v1.0.1b6 https://github.com/Giskard-AI/giskard-oss.git
   # already have the repo? check out this version:
   $ git checkout giskard-core/v1.0.1b6

   • ›Exposes default registry generators on the public API via vulnerability_suite_generator_registry, allowing third-party generators to be registered and consumed by vulnerability_scan and generate_suite.
   • ›Integrates third-party scan support, enabling external ScenarioGenerator instances to plug into the scan pipeline.
   • ›Enforces non-blank judge reasons in LLM-as-judge checks, ensuring Groundedness, Conformity, and LLMJudge results always carry an explanatory rationale.
10. giskard-core/v1.0.1b6 Aug 6, 2026 · issue 002

    Giskard v1.0.1b6 adds third-party scan integration, exposes default registry generators on the public API, and enforces non-blank judge reasons.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch giskard-core/v1.0.1b6 https://github.com/Giskard-AI/giskard-oss.git
    # already have the repo? check out this version:
    $ git checkout giskard-core/v1.0.1b6

    • ›Exposes default registry generators on the public API via vulnerability_suite_generator_registry, allowing third-party generators to be registered and consumed by vulnerability_scan and generate_suite.
    • ›Integrates third-party scan support, enabling external ScenarioGenerator instances to plug into the scan pipeline.
    • ›Enforces non-blank judge reasons in LLM-as-judge checks, ensuring Groundedness, Conformity, and LLMJudge results always carry an explanatory rationale.