## Feeding the Beast: Log Sources & Ingestion

A SIEM is only as good as the data you feed it. As the saying goes: **"Garbage In, Garbage Out."** (Or worse: "Nothing In, Blindness Out").

### Types of Log Data
Not all logs are created equal. A SOC Analyst deals with three primary categories:

#### 1. Event Logs (Structured)
These are logs generated by the Operating System. They are highly structured and rich in metadata.
*   **Windows Event Logs**: The gold standard for endpoint visibility. Stored in binary `.evtx` format.
    *   *Security*: Logins (`4624`), Process Creation (`4688`), Account Management.
    *   *System*: Service failures, startup/shutdown.
    *   *Application*: Errors from SQL Server, IIS, Chrome.
*   **Linux Syslog**: The standard for Unix-based systems. Usually text-based but follows a facility/severity structure. `/var/log/auth.log` (authentication), `/var/log/syslog` (general).

#### 2. Network Logs (Traffic)
These show the "flow" of data.
*   **Firewall Logs**: Allow/Deny decisions. Crucial for perimeter defense.
*   **NetFlow/IPFIX**: Metadata about traffic (Who, What, Where, When, How much) without the actual packet payload. "Phone bill" style records.
*   **Proxy/DNS Logs**: Provide Layer 7 visibility. URLs visited (`evil.com`), DNS queries made.

#### 3. Security Alerts (Context)
*   Antivirus detections, IDS/IPS signatures (`ET TROJAN...`), DLP violations.

### Ingestion Methods: Push vs. Pull

#### The PUSH Method (Agents/Forwarders)
This is the most common and robust method for servers and endpoints. A small software agent is installed on the device.
*   **How it works**: The agent watches log files/channels in real-time. When a new line is written, it immediately encrypts and sends it to the SIEM.
*   **Examples**:
    *   **Splunk Universal Forwarder (UF)**: Lightweight, reliable.
    *   **Elastic Winlogbeat**: Sends Windows logs to Elasticsearch.
    *   **Wazuh Agent**: Performs log collection + FIM + Config assessment.
*   **Pros**: Real-time, encrypted, can buffer logs if the network goes down (caching).
*   **Cons**: requires installing software on every endpoint (management overhead).

#### The PULL Method (Agentless)
Used for devices where you cannot install software (Routers, Switches, Printers, Appliances).
*   **WMI (Windows Management Instrumentation)**: The SIEM remotely queries the Windows server to ask for logs. *High overhead, rarely used for logs now.*
*   **Log Collectors/Concentrators**: A dedicated server sits in the middle. Network devices verify syslog to the Collector. The Collector forwards to the SIEM.
*   **API Fetching**: Modern Cloud method. The SIEM runs a script every 5 minutes to fetch logs from AWS CloudTrail, Office 365, or Okta APIs.

### The Challenges of Ingestion

#### 1. Time Synchronization (NTP)
If your Firewall thinks it is 2:00 PM and your Web Server thinks it is 1:00 PM, correlation is impossible. **NTP (Network Time Protocol)** is critical. All devices must sync to a central, reliable time source.

#### 2. EPS (Events Per Second) & Licensing
SIEMs are often expensive. Licensing is frequently based on:
*   **Volume**: GBs of logs ingested per day.
*   **EPS**: Average events per second.
A noisy firewall in "Debug" mode can bankrupt a SOC by generating millions of useless events. Analysts must tune "logging levels" to capture *security-relevant* info without the noise.

#### 3. Bandwidth
Sending full packet captures or debug logs from a satellite office over a T1 line will crash the network. Compression and filtering at the source (Agent) are vital.

### Analyst Tip
When investigating an incident, always ask: **"What is NOT logging?"** Attackers frequently disable logging (clearing Event Logs) or stop the Splunk Forwarder service as their first move. A "Gap in Logs" is often the biggest indicator of compromise.