What Is a SIEM? Logs, Queries and Alerting

When a security team says it "saw the attack," it usually means the attack left records in several places, and a system pulled those records together and raised a flag. That system is most often a SIEM, short for security information and event management. Much of a SOC's daily work starts there: an analyst opens an alert, checks the events behind it, and decides whether someone has to act.

This article explains what a SIEM does, step by step. It covers where the data comes from, why logs have to be reshaped before they are useful, how correlation rules turn events into alerts, and what happens after an alert fires. It also shows where a SIEM sits next to EDR and SOAR tools, and it maps one sample alert to a MITRE ATT&CK technique so you can see how a detection connects to attacker behavior.

You do not need a lab for this one. The hands-on part is a small Python script that runs on any machine with Python 3.12, using synthetic logs. It produces a real alert from raw lines, and you can change the data and watch the result change. If you want a full stack, the last sections name two free options you can install yourself.

#What a SIEM is

A SIEM does three jobs on one platform. It collects log and event data from many sources. It stores that data so you can search it later. And it runs rules over the data that raise alerts when something matches a pattern.

NIST's guide to computer security log management, SP 800-92, describes practical guidance for developing, implementing and maintaining log management across an organization. The guide treats log management as an ongoing practice that a team has to plan and keep running. A SIEM is one common way to run that practice at scale, not the only one. A small team can get a lot done with a central log server and a few well-written searches.

The word "security" in the name matters less than the word "event." A SIEM does not know what a bad event is by default. It knows the events you send it and the rules you write. Its quality depends on the data going in and the detections running on top.

#Why it matters

Attackers rarely produce one clear signal. A failed login, a new scheduled task and an outbound connection to an unfamiliar host are each ordinary on their own. Together they can describe an intrusion. A SIEM is useful because it can place those records side by side, across hosts, accounts and time. A person reading individual files on individual machines cannot do that at any scale.

Volume is the other reason. An organization with a few hundred machines can generate millions of log lines in a day. No analyst reads them. The SIEM's job is to reduce that volume to a short list of things a human should look at, with enough context to decide quickly.

#The pipeline: sources, collection, normalization

A SIEM works in stages. Each stage can break, and many detection problems trace back to one of them.

#Log sources

A log source is anything that writes a record about something that happened. In most environments the common ones are:

• Operating systems: authentication events, process creation, service changes, and startup and shutdown.
• Identity systems: directory sign-ins, multi-factor prompts and account changes.
• Network devices: firewall allow and deny records, VPN sessions and DNS queries.
• Applications: web server access logs, application errors and admin actions.
• Cloud platforms: control-plane activity, such as API calls that create, change or delete resources.
• Endpoint tools: EDR telemetry, when you have it.

Each source describes the world from its own point of view. A firewall sees a connection. A Windows host sees a logon. Neither sees the whole story.

#Collection

Collection moves records from the source to the SIEM. The usual methods are an agent installed on the host, a forwarder that reads log files, syslog messages sent over the network, or an API that pulls events from a cloud service. Agents can capture more detail from the host. Syslog is simple and widely supported, but it can lose messages under load if nothing buffers them.

Collection failures are quiet. If a server stops sending logs, the SIEM does not show an error by default. It has nothing to say about that server, and the silence can look like calm. Good teams watch the event volume per source and alert when a source goes quiet.

#Normalization

Each source uses its own format. One system writes the username as user, another as account, a third as subject.name inside nested JSON. Normalization maps those fields onto one shared schema, so a rule can say "the user field" and mean the same thing for every source.

Most SIEM products do this for you, with built-in parsers or a published schema. Whatever the product, the goal is the same: after normalization, a search for failed logons looks the same whether the events came from Linux, Windows or a cloud identity provider.

The sample script below does this on a small scale. It takes one raw SSH line and produces a dictionary with the same field names every time.

#Retention

Retention is how long you keep logs before deleting them. It sounds administrative, but it decides what you can investigate. An intrusion discovered three months after the fact needs logs from three months ago. If retention is two weeks, the evidence is gone before anyone looks.

The right period depends on your policy, legal and regulatory duties, and the cost of storage. Log management guidance like NIST SP 800-92 covers the whole life of log data, so retention belongs in the plan from the start rather than appearing when the disk fills up. Write down what you keep, for how long, and who can read it. Then check that deletion actually happens.

Recent data usually sits in fast, expensive storage. Older data can move to cheaper storage that is slower to search. The trade-off is simple: the older the event, the longer it takes to find it.

#Correlation rules, alerts and triage

Correlation is the step where the SIEM looks across events instead of inside one. A correlation rule describes a pattern: a condition, a time window and a grouping key. When the pattern appears, the rule creates an alert.

Here is a plain-language rule for password spraying. We will test it in the hands-on section.

Group events by source IP over a 10-minute window.
Count distinct user accounts with a failed login from that source.
If the count is 4 or more, and the same source later has a successful login,
create an alert with the source IP, the accounts and the successful login.

Two things make a rule like this useful. It uses more than one event, so a single failed login stays quiet. It also groups by a key that fits the attack. Attackers who guess passwords often try one password against many accounts from one place, and that shape shows up clearly when you group by source.

An alert is a claim that something needs attention. It is not a conclusion that an attack happened. Triage is the first decision after the alert fires. The analyst asks four questions: is the alert based on real data, does the account or host matter, what happened before and after, and what is the next step. Many alerts close here as benign, for example a vulnerability scan that you ran yourself, or a user who mistyped a password after a change.

The output of good triage is a decision and a note. The note records what the analyst checked, so the next person does not repeat the work. Teams that skip the note end up re-investigating the same benign alert every week.

#Where a SIEM fits next to EDR and SOAR

Three terms come up together in most SOCs, and their scopes overlap.

A SIEM collects and correlates records from many sources across the environment. Its strength is breadth and history. Its limit is that it only sees what each source reports.

EDR, endpoint detection and response, watches endpoints in detail. It records process activity, file changes and network connections on each host, and it often can isolate a machine. Its strength is depth on the endpoint. It does not see a firewall log or a cloud audit trail unless you connect it to them.

SOAR, security orchestration, automation and response, runs playbooks. A playbook might enrich an alert with a reputation lookup, open a ticket, disable an account after approval, or notify the person on call. Its strength is repeatable response. Its results depend on the quality of the alerts it receives.

#A hands-on exercise with synthetic logs

This exercise needs Python 3.12 and nothing else. The logs are synthetic and use addresses from 192.0.2.0/24, a range reserved for documentation. You are not touching any system. The script reads the lines, normalizes them, applies the password spraying rule and prints the result.

Save this as siem_demo.py:

# Synthetic logs only. 192.0.2.0/24 is a documentation range (TEST-NET-1).
import re
from collections import defaultdict

RAW = """\
2026-11-07T10:00:01Z sshd[811]: Failed password for alice from 192.0.2.44 port 50111
2026-11-07T10:00:03Z sshd[812]: Failed password for bob from 192.0.2.44 port 50112
2026-11-07T10:00:05Z sshd[813]: Failed password for carol from 192.0.2.44 port 50113
2026-11-07T10:00:07Z sshd[814]: Failed password for dave from 192.0.2.44 port 50114
2026-11-07T10:00:09Z sshd[815]: Failed password for erin from 192.0.2.44 port 50115
2026-11-07T10:00:11Z sshd[816]: Accepted password for erin from 192.0.2.44 port 50116
2026-11-07T10:02:00Z sshd[820]: Accepted password for frank from 192.0.2.10 port 50200
2026-11-07T10:03:00Z sshd[821]: Failed password for frank from 192.0.2.10 port 50201
"""

PATTERN = re.compile(
    r"^(?P<ts>\S+) sshd\[\d+\]: (?P<outcome>Failed|Accepted) password for "
    r"(?P<user>\S+) from (?P<src>\S+) port \d+$"
)

def normalize(line):
    m = PATTERN.match(line)
    if not m:
        return None
    return {
        "timestamp": m["ts"],
        "event_type": "authentication",
        "outcome": "success" if m["outcome"] == "Accepted" else "failure",
        "user": m["user"],
        "source_ip": m["src"],
    }

events = [e for e in (normalize(l) for l in RAW.splitlines()) if e]
print(f"normalized {len(events)} events")

# Rule: one source IP fails against 4+ different accounts, then gets a success from that IP.
fails = defaultdict(set)
successes = defaultdict(list)
for e in events:
    if e["outcome"] == "failure":
        fails[e["source_ip"]].add(e["user"])
    else:
        successes[e["source_ip"]].append(e)

for ip, users in fails.items():
    if len(users) >= 4 and successes.get(ip):
        s = successes[ip][0]
        print(f"ALERT src={ip} failed_accounts={len(users)} then_success_user={s['user']} at={s['timestamp']}")
        print("  mapped: T1110 Brute Force, sub-technique T1110.003 Password Spraying (candidate)")

Run it from the folder where you saved it:

python -I siem_demo.py

The -I flag runs Python in isolated mode. It ignores environment variables and keeps the script's folder off the import path. It is a good habit when you run a script you did not write yourself.

On the Windows 11 machine where this article was checked, with Python 3.12.10, the output was:

normalized 8 events
ALERT src=192.0.2.44 failed_accounts=5 then_success_user=erin at=2026-11-07T10:00:11Z
  mapped: T1110 Brute Force, sub-technique T1110.003 Password Spraying (candidate)

The script keeps the rule short on purpose. It does not enforce the 10-minute window from the plain-language version, and it does not check that the success came after the failures. A production rule would do both.

#Read the output

Five accounts failed from one address within about ten seconds. Then erin logged in successfully from the same address. The rule fired because both conditions held: many accounts, few attempts each, and a success afterward. A typo or a saved password rarely looks like this.

Now look at the second source, 192.0.2.10. Frank logged in successfully, and a failure appeared later. The rule stayed quiet, because only one account failed from that address. This is the point of the grouping key. One failed login is noise. Five different accounts failing from one place is a pattern.

Change the data to see the trade-offs. If you delete the success line for erin, the alert disappears even though the failures remain, because the rule needs both. If you raise the threshold to 6, the alert also disappears, since only five accounts failed. In production, a stricter rule misses slower attacks, and a looser rule fires on more ordinary behavior.

#Map the alert to ATT&CK

MITRE ATT&CK is a public knowledge base of adversary behavior. Each technique has an ID, a tactic and a description, and the detection guidance for some techniques lists the data sources that reveal them.

The alert fits T1110, Brute Force, under the Credential Access tactic. ATT&CK describes brute force as repeated attempts to access accounts when the attacker does not know the password. It lists four sub-techniques. Password Spraying (T1110.003) means trying one or a few passwords against many accounts, which avoids locking out any single account. Password Guessing (T1110.001) tries many passwords against one account. Credential Stuffing (T1110.004) reuses passwords leaked from other services. The page also points to a detection strategy that correlates authentication failures across platforms, which is the same idea as our small rule.

Notice the word "candidate" in the output. The rule matches a pattern, and the pattern is consistent with spraying. It is also consistent with a misconfigured service that tries several stored credentials, or with an internal test. Triage has to check which one it is. The ATT&CK mapping tells you where to look and which related behavior to check next, such as use of valid accounts (T1078) after a successful login. It does not prove the attack happened.

#Free ways to practise

You can learn most of this without buying anything. Two open options cover the path from collection to alert. Run either one on a spare machine or in a virtual machine.

#Wazuh

Wazuh's documentation describes it as a free and open source security platform that unifies XDR and SIEM capabilities. It has three central components, the Wazuh indexer, the Wazuh server and the Wazuh dashboard, plus a universal agent installed on the systems you monitor. The getting-started guide offers a quickstart that installs the stack in a few minutes. It also documents step-by-step installs, virtual machines, Docker and Kubernetes. Check the current docs for the version and exact commands, because they change between releases.

For practice, install an agent on a Linux virtual machine. From a second virtual machine you control, make a few failed SSH logins, then watch the events arrive in the dashboard. That covers collection, normalization and alerting on a real pipeline.

#Elastic

Elastic Security documents a SIEM built on the Elastic Stack. Its overview describes data collection through integrations and the Elastic Agent, detection with prebuilt rules, custom rules and machine learning, alert triage, Timeline for hunting, and Cases for recording investigations. Which of these features a self-managed setup includes depends on the license tier, so read Elastic's subscription page before you deploy. The overview does not list which SIEM features a self-managed setup gets at no cost, so do not assume a free tier covers any particular detection feature.

#Common mistakes

• Treating every alert as an incident. An alert is a reason to look. Most alerts close as benign, and the triage note is what makes that result useful.
• Collecting everything and reading nothing. Volume is not coverage. Start with a few sources you understand and add more once you can say what each one tells you.
• Forgetting silent sources. A source that stops sending looks the same as a quiet network. Alert on missing data.
• Setting retention by accident. If nobody chose the period, the storage system chose it, usually at the wrong length.
• Writing rules nobody can explain. A rule the team does not understand gets disabled the first time it is noisy.

#How to go further

Start with the NIST SP 800-92 guide. It is organizational and high level, so read it for the questions it asks, such as who owns each log source and how long you keep it. Then open the ATT&CK page for T1110 and read the detection notes next to the sub-techniques.

Next, install Wazuh or Elastic on a virtual machine and send logs from one source you control. Write one rule that matches a pattern you can trigger on purpose, then write the triage note as if a colleague had to pick it up tomorrow.

The next article in this series covers the Windows event log and the event IDs that many correlation rules depend on, which makes those rules easier to read: Windows event logs and event IDs. If you want a sense of how a single detection gets written in practice, writing a YARA rule against a benign test file shows the same idea of testing a rule on known data before trusting it.

#Sources

• NIST SP 800-92, Guide to Computer Security Log Management
• MITRE ATT&CK
• Wazuh documentation: getting started
• Elastic Security documentation: Elastic Security overview