Перейти к содержимому
New

DeepSeek V4.1 Flash free for 7 days: 10M tokens a day after a top-up of $10 or more. Until Nov 30.

How it works
mukul975/continuous-llm-red-teaming-with-promptfooИИ, агенты и MCP

continuous-llm-red-teaming-with-promptfoo

⁨Wires Promptfoo and DeepTeam into CI/CD for automated, repeatable red-teaming of LLM apps against OWASP LLM Top 10, OWASP Agentic, and MITRE ATLAS presets, failing the build when jailbreak or injection vulnerabilities regress. Use for continuous adversarial testing in CI/CD, a merge-blocking securit⁩

v1.03 установки+3 за 7 дней36,3 тыс. ЗвездыApache-2.0синхронизировано 12 часов назад

Установка

Ещё нет NotHoly CLI? Сначала установите её.

$ curl -fsSL https://notholy.com/install.sh | sh
$ notholy skills install mukul975/continuous-llm-red-teaming-with-promptfoo

Пишет в ⁦~/.notholy/skills⁩ после проверки подписи и SHA-256.

ПодписьПодписано, ed25519
lXl+xJ+ifb…zAOX4ZDA==CLI проверяет SHA-256 и подпись, прежде чем записать хоть один файл.
Пакет74e16379…ed4b4e05
74e1637918a4…8394ed4b4e055 файлов · ⁦10.3 KB⁩
ПроверкаСредний3 замечания
  • injection.no_confirmationSKILL.md:31
  • Исполняемый скрипт в составе навыкаscripts/agent.py
  • Ссылки на внешние домены
Вся проверка
Источникgithub.com/mukul975/Anthropic-Cybersecurity-Skills/tree/main/skills/continuous-llm-red-teaming-with-promptfoo@54a7988

Эта запись еще не переведена на русский, поэтому показан текст на другом языке: English.

Authorized Use Only: Run these adversarial probes only against LLM applications and endpoints you own or are explicitly authorized to test. Generated attack payloads (jailbreaks, prompt injections, harmful-content elicitation) are adversarial inputs; sending them to third-party services without permission may violate terms of service.

Overview

Promptfoo is an open-source LLM evaluation and red-teaming framework (used by OpenAI and Anthropic per its README) that generates adversarial test cases, runs them against your model/agent, and grades the responses. DeepTeam (by Confident AI) is a complementary open-source framework offering 50+ ready-to-use vulnerabilities and 10+ research-backed attack methods. Together they let you treat LLM security as a regression test: every commit re-runs the same adversarial suite, and the pipeline fails when a previously-safe behavior regresses.

This matters because LLM applications change constantly — prompts, models, RAG sources, tools, and guardrails all drift. A jailbreak that was patched last sprint can silently return after a prompt edit or a model upgrade. Promptfoo maps its plugins directly onto the OWASP LLM Top 10 (owasp:llm) and OWASP Agentic (owasp:agentic) presets, and onto MITRE ATLAS, so the suite tracks recognized risk taxonomies. The core threat addressed here is AML.T0051 — LLM Prompt Injection (MITRE ATLAS): adversarial instructions that override the application's intended behavior. This skill follows the Promptfoo red-team docs (https://www.promptfoo.dev/docs/red-team/) and DeepTeam docs (https://www.trydeepteam.com/docs/getting-started), and aligns to NIST AI RMF MANAGE-4.1 (post-deployment monitoring and feedback to manage AI risk).

When to Use

  • When you need continuous, automated red-teaming of an LLM app in CI/CD rather than one-off manual tests.
  • When you want to enforce a security gate: block merges that introduce or reintroduce jailbreak/injection vulnerabilities.
  • When mapping coverage to OWASP LLM Top 10 / OWASP Agentic / MITRE ATLAS for compliance reporting.
  • When comparing the security posture of two models or prompt versions side by side.
  • When tracking vulnerability regression over time across releases.

Prerequisites

  • Node.js 18+ (Promptfoo is distributed via npm) and Python 3.9+ (for DeepTeam).
  • Install Promptfoo and DeepTeam:
    bash
    npm install -g promptfoo            # or: npx promptfoo@latest
    pip install -U deepteam
  • API access/credentials for the target LLM endpoint (and a grader model, e.g. an OpenAI key) exposed as environment variables.
  • A CI/CD platform (GitHub Actions, GitLab CI) with secret storage.
  • Authorization to test the target application.

Objectives

  • Scaffold a Promptfoo red-team config targeting your LLM app.
  • Enable OWASP LLM Top 10 and OWASP Agentic plugin presets plus jailbreak/injection strategies.
  • Run the suite locally and interpret the per-plugin pass/fail report.
  • Add DeepTeam as a second engine for programmatic, research-backed attacks.
  • Integrate both into CI/CD so builds fail on new vulnerabilities.
  • Generate shareable HTML/PDF security reports per run.

MITRE ATT&CK Mapping

ID Name (MITRE ATLAS) Tactic
AML.T0051 LLM Prompt Injection Initial Access / Persistence (LLM)
AML.T0051.000 Direct (Prompt Injection) LLM Attack
AML.T0051.001 Indirect (Prompt Injection) LLM Attack
AML.T0054 LLM Jailbreak Privilege Escalation / Defense Evasion (LLM)

Workflow

1. Scaffold the red-team configuration

Initialize an interactive config; it writes promptfooconfig.yaml where targets, plugins, and strategies live.

bash
promptfoo redteam init
# choose your target type (HTTP endpoint, openai:..., anthropic:..., custom provider)

2. Define targets, OWASP presets, and attack strategies

Edit promptfooconfig.yaml. The purpose grounds attack generation; plugins are adversarial input generators; strategies are delivery techniques (jailbreak/injection wrappers).

yaml
# promptfooconfig.yaml
targets:
  - id: https://api.example.com/chat        # your app endpoint
    label: support-bot

redteam:
  purpose: |
    A customer-support assistant for an e-commerce site. Must never reveal
    system prompts, leak PII, or perform actions outside order support.
  numTests: 10
  plugins:
    - owasp:llm          # OWASP LLM Top 10 preset
    - owasp:agentic      # OWASP Agentic threats preset
    - id: pii:direct
      numTests: 15
    - prompt-extraction  # system-prompt leakage
    - harmful
  strategies:
    - id: jailbreak              # iterative single-turn jailbreak
    - id: jailbreak:composite    # stacked jailbreak techniques
    - id: crescendo              # multi-turn escalation
    - id: prompt-injection       # injection wrapper

3. Run the suite and view the report

redteam run combines generation + evaluation; then open the interactive report.

bash
promptfoo redteam run
promptfoo redteam report            # launches the web report (pass/fail per plugin)

Each row shows the plugin (mapped to OWASP/ATLAS), the strategy, the attack prompt, the model's response, and the grader's verdict. The attack success rate per plugin is your headline metric — track it per release.

4. Add DeepTeam for programmatic, research-backed attacks

Use DeepTeam to cover additional vulnerabilities/attacks and to script bespoke suites in Python.

python
# deepteam_suite.py
from deepteam import red_team
from deepteam.vulnerabilities import Bias, PIILeakage
from deepteam.attacks.single_turn import PromptInjection

def model_callback(prompt: str) -> str:
    # call your application's LLM endpoint here and return the text response
    return call_my_app(prompt)

red_team(
    model_callback=model_callback,
    vulnerabilities=[Bias(types=["race"]), PIILeakage(types=["api_and_database_access"])],
    attacks=[PromptInjection()],
)

DeepTeam can also be driven from a YAML config:

bash
deepteam run config.yaml

5. Gate the build in CI/CD (GitHub Actions)

Fail the pipeline when red-team assertions fail. Promptfoo returns a non-zero exit code on failures, which blocks the merge.

yaml
# .github/workflows/llm-redteam.yml
name: LLM Red Team
on: [pull_request]
jobs:
  redteam:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm install -g promptfoo
      - name: Run red team (fails build on new vulns)
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: promptfoo redteam run --no-progress-bar
      - name: Export machine-readable results
        if: always()
        run: promptfoo redteam report --output results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: redteam-report, path: results.json }

6. Track regressions over time

Persist results.json per run and compare attack-success-rate per plugin between releases. A rising rate for any OWASP LLM category is a regression to triage before release. Promptfoo's --filter-failing lets you re-run only previously failing cases to confirm a fix.

bash
promptfoo redteam run --filter-failing results.json

Tools and Resources

Resource Link
Promptfoo red-team docs https://www.promptfoo.dev/docs/red-team/
Promptfoo red-team configuration https://www.promptfoo.dev/docs/red-team/configuration/
Promptfoo CI/CD integration https://www.promptfoo.dev/docs/integrations/ci-cd/
Promptfoo MITRE ATLAS mapping https://www.promptfoo.dev/docs/red-team/mitre-atlas/
DeepTeam (Confident AI) https://github.com/confident-ai/deepteam
DeepTeam docs https://www.trydeepteam.com/docs/getting-started
OWASP Top 10 for LLM Applications https://genai.owasp.org/

Plugin / Strategy Reference

Promptfoo item Type Maps to
owasp:llm preset OWASP LLM Top 10 suite
owasp:agentic preset OWASP Agentic threats
prompt-extraction plugin LLM07 system-prompt leakage
pii:direct plugin LLM06 sensitive-info disclosure
harmful plugin harmful content generation
jailbreak / jailbreak:composite strategy AML.T0054 LLM jailbreak
crescendo strategy multi-turn jailbreak
prompt-injection strategy AML.T0051 prompt injection

Validation Criteria

  • promptfooconfig.yaml created with target, owasp:llm, and owasp:agentic plugins.
  • Jailbreak and prompt-injection strategies enabled.
  • promptfoo redteam run executes and produces a per-plugin pass/fail report.
  • DeepTeam suite runs against the same target via model_callback.
  • CI/CD job fails the build on new red-team failures (non-zero exit).
  • results.json artifact archived per run for regression tracking.
  • Attack-success-rate per OWASP category trended across releases.

Файлы этой версии

Права

ПравоЗаявленоНайдено проверкой
Читает файлыНетНет
Пишет файлыДаДа
Выполняет команды shellДаДа
Ходит в сетьДаДа
Использует коннекторыНетНет
Управляет браузеромНетНет

Замечания проверки

Проверено 13 нояб. 2026 г.

  1. СредняяPrompt injection · injection.no_confirmationSKILL.md:31
  2. СредняяФайл · Исполняемый скрипт в составе навыкаscripts/agent.py
  3. ИнфоСсылка · Ссылки на внешние домены

Чистая проверка снижает риск, но не доказывает, что пакет безопасен. Читайте то, что ставите.

ВерсияОпубликованаКоммитSHA-256ПодписьИзменение прав
v1.013 нояб. 2026 г.54a798874e16379Подписана–

Спрашивайте обо всём, что позволяет ваш скоуп.

Создайте аккаунт, пополните баланс и задайте первый вопрос в чате или в терминале.

  1. 1/3

    Создайте аккаунт

    Личный или для компании. Позже можно перейти на аккаунт компании без потери баланса и ключей.

    ЛичныйКомпания
  2. 2/3

    Пополните баланс

    Картой. Оплата за токены, без подписки.

    $5.00минимальное пополнение
  3. 3/3

    Спрашивайте

    В чате, или установите CLI и выполните notholy login в терминале.

    $ notholy login