Using an LLM Safely in Security Work

An AI assistant can summarize a CVE record in seconds. It can also name the wrong fixed version in the same calm tone it uses for the right one. This article covers that gap: the four risks that matter most in security work, a method for keeping them small, and a lab where you check a sample AI-written summary against the NVD record, the CISA Known Exploited Vulnerabilities (KEV) catalog and the Apache Log4j security page.

The short answer: use an assistant for work you can check, keep sensitive data and real actions out of its reach unless someone approved them, and make a person decide every step that changes something.

#What the risks are

An assistant is a language model with a text box, and often some tools attached: a search function, a file reader, a shell. Four risks show up again and again.

#Confident wrong facts

A language model produces the most plausible next words. Often the plausible answer is also true. Sometimes it is a version number that sounds right and is not. NIST's Generative AI Profile, NIST AI 600-1, calls this confabulation: "the production of confidently stated but erroneous or false content." The same profile notes that confabulation is a result of how these models are built, not a rare malfunction.

Identifiers are the most exposed. CVE numbers, package versions, dates, function names, RFC numbers and port numbers all look structured, so a wrong one looks checked. It is easy to copy into a ticket and hard to notice later.

#Sensitive data in prompts

Anything you paste into a prompt leaves your control when it reaches a service you do not run. What happens next depends on the service, its settings and its terms. The text may be stored, logged or reviewed. For security work, the risky material is familiar: client names, internal hostnames and IP ranges, tokens that appear in logs, unpublished findings, and personal data from an incident. OWASP lists this under LLM02:2025 Sensitive Information Disclosure, which covers both what the model reveals and what the surrounding application exposes.

#Unsafe tool actions

An assistant with only a text box can give you a bad answer. An assistant with a shell, an API token or write access to a repository can do a bad thing. OWASP's LLM06:2025 Excessive Agency describes the problem: the system has more functions, permissions or autonomy than the task needs. When the model misreads a request, the damage depends on what it was allowed to touch.

#Prompt injection from untrusted content

Prompt injection happens when content the system reads as data contains instructions that the model then follows. The content can be a web page you asked it to summarize, an email in your queue, a log line with a crafted user agent string, or a README in a repository you asked it to review. OWASP covers this as LLM01:2025 Prompt Injection. NIST's profile describes the indirect version, where an attacker plants instructions in data the assistant is likely to retrieve. The assistant may not tell the difference between your request and the text it just read.

#Why it matters for analysts

Analysts are the people who copy AI output into reports, detection rules, tickets and messages to customers. The error travels with the copy. A wrong fixed version in your summary becomes a wrong patch decision in someone else's change window.

NIST's AI Risk Management Framework, released on January 26, 2023, organizes the work into four functions: GOVERN, MAP, MEASURE and MANAGE. You do not need to run an AI program to borrow the idea. For one analyst it means knowing what the tool is for, what data it can see, how often it is wrong on your own kind of task, and who approves an action. The framework is voluntary, and that is fine. The questions still apply.

#The method

Five rules, applied in order every time.

#1. Decide what may enter the prompt

Before you paste anything, sort it into three groups:

• Public: vendor advisories, published CVE records, public documentation.
• Internal but not sensitive: a generic procedure, a sanitized code pattern.
• Restricted: client names, hostnames, IP addresses, credentials, unpublished findings, personal data.

Restricted data stays out unless the data owner approved this use and your organization approved the tool for that class of data. If you need the assistant to reason about a log, redact it first. A redaction pass with sed is enough for a quick case:

sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/10.x.x.x/g; s/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+/user@example/g' incident.log > redacted.log

Run it on a copy, read the output, and check for the cases the regex misses: hostnames, usernames, tokens and internal domains. A regex does not know what your client calls their servers.

#2. Ask for sources, and separate claims from guesses

Tell the assistant to list each factual claim with the source it came from, and to write "no source" when it has none. Then treat every unsourced claim as a question for you to answer.

A citation in the answer proves nothing by itself. A model can produce a plausible link or a plausible quote from a document it never opened. Open the link. Read the paragraph. If the paragraph does not say what the answer says, the answer is wrong.

#3. Verify against primary data

Primary data is the record the claim is about: the NVD entry for a CVE, the vendor advisory, the CISA KEV entry, the text of the standard, the output of your own test. A second AI answer does not count, because the second assistant can repeat the same mistake with the same confidence. Use the assistant to find where to look. Use the source to decide.

#4. Limit what the tool can do

Start with read-only access. If the tool can run commands, give it an account with the minimum rights, a disposable virtual machine or container, and an allowlist of commands. Any step that writes, sends, deletes or calls a live system should wait for a confirmation. Keep the log of what it ran. You will want that log the first time something unexpected happens.

#5. Keep the decision human

The tool proposes. A person with context and authority decides, and that person writes down what they approved and the evidence they used. This matters most for actions that cannot be undone: a blocked IP range, a closed ticket, a patch deployed to production, a message sent to a customer.

#Hands-on: check an AI-written CVE summary

This lab uses only public, read-only sources. You will not run anything against a live system. The three requests go to NVD, CISA and the Apache Software Foundation, and nothing else.

What you need: a terminal (Git Bash on Windows works), curl, and jq. If jq is not installed, install it from the jq project site. The commands below were run on Windows 11 with Git Bash. jq was not installed on that machine, so the outputs shown come from a Node.js command that reads the same JSON fields. The jq commands are the ones to use.

#The sample summary

Here is a summary an assistant wrote about CVE-2021-44228, the flaw known as Log4Shell. Read it before you check anything, and mark what you think is wrong.

CVE-2021-44228 is a remote code execution flaw in Apache Log4j2. An attacker who controls a logged value can make the server load code from an LDAP server. NVD published the record on 2021-12-09. The flaw affects Log4j 2.13.0 through 2.14.1, and upgrading to 2.14.1 fixes it. The CVSS v3.1 base score is 10.0. CISA added the flaw to the KEV catalog on 2021-12-10. Attackers began scanning within hours of disclosure. The flaw is in log4j-core only.

The summary contains two deliberate errors. Find them with the sources below.

#Step 1: pull the NVD record

NVD's API returns the CVE record as JSON. Run this command:

curl -s 'https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2021-44228' | jq '.vulnerabilities[0].cve | {published, descriptions: .descriptions[0].value}'

Equivalent check without jq, which is the output shown below:

curl -s 'https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2021-44228' -o nvd.json
node -e "const c=JSON.parse(require('fs').readFileSync('nvd.json','utf8')).vulnerabilities[0].cve;console.log(JSON.stringify({published:c.published,descriptions:c.descriptions[0].value},null,2))"

Observed output:

{
  "published": "2021-12-10T10:15:09.143",
  "descriptions": "Apache Log4j2 2.0-beta9 through 2.15.0 (excluding security releases 2.12.2, 2.12.3, and 2.3.1) JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. From version 2.16.0 (along with 2.12.2, 2.12.3, and 2.3.1), this functionality has been completely removed. Note that this vulnerability is specific to log4j-core and does not affect log4net, log4cxx, or other Apache Logging Services projects."
}

Look at the published field. The sample said 2021-12-09. The record says 2021-12-10. That is the first error.

The description also adds a condition the sample dropped: the code loading happens when message lookup substitution is enabled. Leaving that out makes the flaw sound like it hits every Log4j installation, and the record does not say that.

#Step 2: pull the KEV entry

CISA publishes the KEV catalog as one JSON file. Download it and filter for the CVE:

curl -s -L 'https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json' -o kev.json
jq '.vulnerabilities[] | select(.cveID=="CVE-2021-44228")' kev.json

Observed output, from a catalog with version 2026.11.06 and 1769 entries, downloaded on 2026-11-08 (the jq filter was run through Node.js, as above):

{
  "cveID": "CVE-2021-44228",
  "vendorProject": "Apache",
  "product": "Log4j2",
  "vulnerabilityName": "Apache Log4j2 Remote Code Execution Vulnerability",
  "dateAdded": "2021-12-10",
  "shortDescription": "Apache Log4j2 contains a vulnerability where JNDI features do not protect against attacker-controlled JNDI-related endpoints, allowing for remote code execution.",
  "requiredAction": "For all affected software assets for which updates exist, the only acceptable remediation actions are: 1) Apply updates; OR 2) remove affected assets from agency networks. Temporary mitigations using one of the measures provided at https://www.cisa.gov/uscert/ed-22-02-apache-log4j-recommended-mitigation-measures are only acceptable until updates are available.",
  "dueDate": "2021-12-24",
  "knownRansomwareCampaignUse": "Known",
  "forensicTriage": "No",
  "notes": "https://nvd.nist.gov/vuln/detail/CVE-2021-44228",
  "cwes": [
    "CWE-20",
    "CWE-400",
    "CWE-502"
  ]
}

The entry confirms the KEV listing and the date added. It says nothing about when scanning started, and it does not name a fixed version. Use it for the listing, the due date and the required action, and nothing more.

#Step 3: open the Apache page and mark each claim

The NVD record links to the Apache Log4j security page. Open it and find the section for CVE-2021-44228. Read four fields: Versions affected, Versions fixed, Components affected and the description. In the version of the page fetched on 2026-11-08, the section listed these values:

• Versions affected: [2.0-beta9, 2.3.1), [2.4, 2.12.2) and [2.13.0, 2.15.0). The brackets are interval notation: the left end is included and the right end is not, so 2.15.0 itself is outside the range.
• Versions fixed: 2.3.1 for Java 6, 2.12.2 for Java 7, and 2.15.0 for Java 8 and later.
• Components affected: log4j-core.

Read the page yourself, because it changes. The page also has a separate section for CVE-2021-45046, a follow-up issue, which says the fix in 2.15.0 was incomplete in some configurations and points to 2.16.0 for Java 8 and later.

#The verdicts

Mark each claim in the sample as confirmed, wrong or unverifiable. Here is the result of that check.

Two claims are wrong, one is unverifiable, and the rest check out. The two wrong claims are the two deliberate errors in the sample.

#What the check teaches

The NVD prose and the version ranges disagree in a small way. The description says "through 2.15.0", while the machine-readable range ends before 2.15.0. Both the NVD configuration and the Apache page say 2.15.0 is not affected for this CVE on Java 8 and later. When sources disagree, record the disagreement and cite the one that defines the range. Do not pick the version that reads better.

The fixed version depends on the Java version and on the CVE. A single "fixed in" number hides both. Ask for the exact CVE, the product, the component and the platform, and read the vendor's table.

Unverifiable is a useful verdict. The sample states a fact about attacker timing that none of your sources gives. That claim may be true, and it may be in another source you have not read. In a report, it needs its own citation or it needs to go.

#Common mistakes

• Trusting the shape of a citation. A tidy link with a plausible title can point to a page that says something else. Open it.
• Pasting a whole incident to get a "quick look". Redact first, and paste only the fragment the question needs.
• Asking for the latest version without a date. A model can be unaware of releases after its training data ends. Give it the release notes or the advisory, and ask it to read them.
• Accepting a generated patch without running the tests or reading the diff.
• Testing how the assistant handles real client data "just to see". The test itself is the leak.
• Giving an agent production credentials because the demo needed them.
• Summarizing a web page without thinking about who wrote the page. The page can carry instructions. Treat its text as data to read, not as orders to follow.

#How to go further

Build a small evaluation for your own work. Pick twenty CVEs where you already know the answer, ask your assistant for a summary of each, and count the errors by type: dates, versions, components, scores. A small check like this tells you more about your own tasks than a vendor's demo does. Repeat it when the tool or your task changes.

Read the OWASP entries for LLM01, LLM02, LLM06 and LLM09. They give a shared vocabulary for prompt injection, data exposure, excessive permissions and misinformation. Then read section 2 of NIST AI 600-1, which defines the risks the profile covers.

If you want the basics before this material, read What Is a Vulnerability? and What Is CVSS?. To practice writing the findings you verified, read How to Write Your First Security Report. If you send work to a model through the NotHoly API, the same five rules apply to what you send.

#Sources

• NIST AI Risk Management Framework
• NIST AI 600-1, Generative AI Profile (July 2024)
• OWASP Top 10 for LLM and GenAI Applications, 2025
• NVD Vulnerability API documentation
• NVD record for CVE-2021-44228
• CISA Known Exploited Vulnerabilities catalog (JSON feed)
• Apache Log4j Security Vulnerabilities