Detection Engineering

• Logs

  • Logs provide visibility, context, and carry evidence value in security operations, incident response, threat hunting, compliance, cyber resilience, and cyber maturity.
  • Most importantly, the ability to correlate simultaneous and distinct events among distributed networks and systems is also achieved with the visibility and context gained by logs. No matter what kind of security operation is being implemented, the logs are one of the essential components.
• Log Data Specifications

  Common base qualifications are listed below:

  • The system which created the log.
  • The log creation time (date, time and timezone).
  • The event that caused the log to be created (message).
  • The severity of the log.
  • The source associated with the log (IP and port, MAC address, username, or system name).
• Logs

  1. Apache Access Log Files: Only Apache access log analysis.
  2. Linux System Log Files: Only Linux system log analysis.
  3. Raw Text In File: Cleartext log analysis.
  4. Windows Event Log Files: Only Windows log analysis.
• Windows Event Log Anatomy

  • Windows event logs provide in-depth footprint information on the system, security, and applications installed on a Windows operating system. Windows provides a generous amount of logs, and you will need to activate them according to your visibility needs and capacity
  • Remember, the logging scope is fully configurable, and the default settings are not enough for the current state of the threats. Being comfortable with logs is a vital skill, but it is also important to have the general characteristics before deep diving into each log source's details. 
• Windows Event Log Anatomy

  • Default format/extension:
  • .evt
  • .evtx
  • The log files' directory can differ according to the Windows OS version. MS Windows changed the event log file directory/location with Windows Vista.
  • 2003 and earlier versions:
  • C:\WINDOWS\system32\config
  • Vista and newer:
  • C:\WINDOWS\System32\winevt\Logs
  • Default:
  • Event Viewer
• Windows Logs

  • Built-in Windows event logs.
  • The main focus is system-level activities.
  • Not customisable.
  • Gives insights into the overall system.

  Applications and Services Logs

  • More specific logs on individual applications and services.
  • Applications and services can create their custom log formats.
  • Gives insights on particular applications and services.
• Windows Logs

  • Application: Application and software logs.
  • System: OS system and component logs.
  • Security: Local and group policy audit logs.
  • Setup: Application installation logs.
  • Forwarded Events: Logs sent from hosts residing in the same network.
• Windows Logs

  Microsoft/Windows/PowerShell/Operational:

  • Detailed PowerShell logs focused on security, audit, and compliance.
  • Detectebilities:
  • Script block executions.
  • Module loads.
  • Administrative actions.
• Windows Logs

  Windows PowerShell

  • PowerShell logs focused on debugging and scripts for diagnosing script issues.
  • Detectabilities:
  • Script block executions.
  • Errors and script info.
• Windows Logs

  • The severity of the event
  • Information: Events without issues (success).
  • Error: Issues in the system or service.
  • Warning: Potential problem.
  • Critical: Significant issue.
  • Verbose: Progress or success messages.
  • The type of the event log that highlights the purpose of the log.
  • Common categories:
  • Process Creation
  • Service Creation
  • Log Clear
• Windows Logs

  • PID and TID data are beneficial for process tracing, correlation, and understanding the natural flow of events during log analysis.
  • The timestamp also plays an essential role in determining which processes and threads were running at a particular time or during the workflow and, if applicable, the call times of the child processes and threads created by the user or execution flow.
• Windows Event IDs

  • 4720: User account creation.
  • 4722: User account enabled.
  • 4723: Attempt to change an account password.
  • The user attempts to change their password.
  • 4724: Attempt to reset the account password.
  • The user attempts to reset the password of another account.
  • 4725: Account disable.
  • 4726: Account removal.
  • 4624: Successful logon.
  • 4625: Failed logon.
  • 4634 and 4647: Logoff.
  • 4779: Session disconnect.
• Windows Event IDs

  • 4698: Scheduled task creation.
  • 4702: Scheduled task updated.
  • 4699: Scheduled task deletion.
  • 1100: Logging service disabled.
  • 1102: Log deletion.
  • 1116: Malware detection.
• Linux Logs

  • Default format/extension: .log
  • Syslog stores logs in cleartext format.
  • Systemd and Journald stores logs in binary format.
  • The directory of the log files can differ according to the used services and configuration file. Still, most Unix-like operating systems share the same directory to store the log files.
  • /var/log
• Linux Cleartext Logs

  General system logs

  • /var/log/syslog
  • /var/log/messages

  System boot logs

  • /var/log/boot.log

  Background daemon logs

  • /var/log/daemon.log

  Authentication logs

  • /var/log/auth.log
  • /var/log/secure
• Linux Cleartext Logs

  Kernel logs

  • /var/log/kern.log

  Audit logs

  • /var/log/audit

  Cron daemon logs

  • /var/log/cron.log

  Debug logs

  • /var/log/debug

  Apt command logs

  • /car/log/apt/
• Linux Binary Format Logs

  • Failed logins only: /var/log/faillog.Command: faillog
  • Active login records: /var/log/wtmp. Command: who
  • Last logins: /var/log/lastlog. Command: lastlog
  • Kernel ring buffer logs: /var/log/dmesg. Command: dmesg
• Linux Logs: Construct

  • timestamp hostname process[pid]: message

  E.g. Log:

  • 2023-09-28T15:05:55.333333Z TempServer sudo[2345]: [AUTH] User alice executed command '/usr/bin/apt-get update'.

  The above log entry showcases an event where the user "Alice" executed the command '/usr/bin/apt-get update' using the 'sudo' command on the server "TempServer". This log entry provides valuable insights into user actions, including the executed command and authentication details, which can aid in auditing and monitoring system activities.
• SysLog: Restart Due to Insufficient RAM

  • The above log entries showcase events on "TempServer" that include crucial memory-related events. It shows that processes with IDs "5678" (myapp), "9876" (database), and "1234" (myapp) caused memory problems and resulted in process terminations, memory limit violations, swap space exhaustion, and low memory warnings, ultimately requiring reboots for recovery and stability.

  
• Misc Logs: Application Logs

  • Misc logs provide in-depth footprint information on application-based events, giving more insights on application and process-based details that will help analysts in security operations, including monitoring, threat hunting, and incident response.
• Apache Logs

  • Default Format: .log
  • Main directory: /var/log/apache2/
  • Access logs: /var/log/apache2/access.log
  • Error logs: /var/log/apache2/error.log
  • Alternative tools and utilities can be used to view cleartext log files.
  • lnav
  • cat, tail, more, less, grep, awk
• Apache Logs: Access.log

  • Access logs are invaluable records generated by web servers, containing essential attributes that form the backbone of effective log analysis.
  • These attributes, including IP addresses, timestamps, HTTP methods, URLs, status codes, and user agent information, play a vital role in web server management and security.
  • These attributes enable administrators and analysts to ensure server health, diagnose problems, detect security threats, and optimise web services for a seamless user experience.
• Apache Logs: Structure:

  • %h: IP address of the client (requester).
  • %l: It represents the "hyphen" (-). Appears when the requested information is not available
  • %u: User id information.
  • %t: Time information when the request is received.
  • \"%r\": The request.
  • %>s: Status code that is sent back to the client.
  • %b: Size of the returned object.
  • \"%{Referer}i\": HTTP referer field that identifies the previous resource used to come to the current page.
• Apache Logs: Combined Logs

  • LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

  Case: Using HTTP POST Method

  • 192.168.1.100 - THMjohn [15/Nov/2025:10:30:45 -0500] "POST /api/data HTTP/1.1" 404 1234 "https://www.LogUniverse-THM.com/page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.0.0 Safari/537.36"
• Apache Logs: Error.log

  • Error logs are an essential component of web server management, providing critical insight into system health and potential issues that can impact server performance and user experience.
  • These logs capture essential attributes such as timestamps, error messages, file paths, and originating IP addresses.
  • These attributes are essential for administrators and analysts to diagnose and resolve errors, identify vulnerabilities, and maintain server security
• Apache Error Logs: Structure

  • %t: Time information when the request is received.
  • %l: Log level of the message.
  • %P: Process ID if the current process.
  • %F: Source file and line number of the log call
  • %E: Error status code.
  • %a: Client's IP and port info.
  • %M: The actual log message.
• Apache Error Logs: Structure

  • ErrorLogFormat "[%t] [%l] [pid %P] %F: %E: [client %a] %M"

  Resource Availability

  • [Thu May 12 08:28:57.652118 2011] [core:error] [pid 8777] [client ::1] File does not exist: /usr/local/apache2/htdocs/favicon.ico
• Elasticsearch

  • As the core component of the Elastic Stack, Elasticsearch is a distributed search and analytics engine. It is designed to handle large volumes of data by indexing, searching, and analyzing it in near real-time
  • In a Security Operations Center (SOC), analysts can use Elasticsearch to store and index security logs (e.g., firewall logs, intrusion detection system logs) for fast searching and analysis, enabling quick identification and response to security incidents.
• Logstash

  Logstash is a flexible data ingestion and processing tool that collects, processes, and transforms data from various sources, preparing it for storage in Elasticsearch or other destinations.

  Key Features:

  • Data Ingestion: Supports multiple input sources, including logs, metrics, events, and other structured or unstructured data.
  • Data Transformation: Offers a wide range of filters for parsing and transforming data.
  • Extensive Plugin Ecosystem: Includes numerous input, filter, and output plugins for flexible data handling.
• Kibana

  Kibana is the interactive visualization and analytics platform for the Elastic Stack. It provides a user-friendly interface for creating dashboards, visualizations, and reports based on the data stored in Elasticsearch.

  Key Features:

  • Interactive Dashboards: Allows users to create and customize dashboards for data visualization.
  • Data Visualization: Supports a variety of charts, including line charts, bar charts, pie charts, heat maps, etc.
  • Search and Filter Capabilities: Enables users to search and filter log events for detailed analysis.
• Beats

  Beats are lightweight data shippers that collect data from various sources (endpoints) and forward it to Elasticsearch or Logstash for processing and analysis.

  Key Features:

  • Efficiency: Beats have a small footprint, making them suitable for resource-constrained environments such as IoT devices or small systems.
  • Specific Use Cases: Each Beat is designed for a specific type of data, such as system logs (Filebeat), network data (Packetbeat), metrics (Metricbeat), etc.
• Detecting Incidents

  • In the case of Bill's incident, we know that a malicious attacker potentially gained remote access to his workstation.
  • Through this context, we can infer that the attacker likely ran commands on the system, attempted to elevate privileges, accessed files, attempted to maintain persistence, and more.
• Elastic

  • The Top values feature is often a quick way to infer interesting patterns or trends in the logs. For example, we can quickly determine if any IP addresses stand out, indicating the potential use of a command-and-control (C2) server. We can also learn which processes run most frequently on the compromised workstation or which user accounts are being accessed or created.
• Detecting Incidents

  • By looking at the file's path, it is clear that Bill downloaded a PowerShell script disguised as a PDF invoice from the phishing email he received. We can determine that this file was a PowerShell script from the actual ".ps1" file extension.
  • Attackers commonly use a technique called "file extension spoofing" to disguise malicious files as harmless documents to trick users into downloading and executing them.
• Discovery

  • In the context of MITRE ATT&CK, "Discovery" refers to the techniques and methods used by adversaries to gather information about the target environment, such as its network, systems, and user accounts.
  • It involves activities that help the attackers better understand the victim's infrastructure and identify potential vulnerabilities or opportunities for further exploitation.
• Discovery

  • Like running PowerShell, system administrators and legitimate users commonly use the above commands to manage and troubleshoot Windows systems.
  • However, combined with other suspicious activity, these logs appear to be indicators of compromise (IOCs).
  • The attacker is likely using these commands to gather information about the system and identify potential vulnerabilities or targets. It's doubtful that Bill from Finance would have run these commands in the command line.
• Discovery: Powershell

  • This is very common in the "enumeration phase" of an attack on an internal network.
  • The information provided by running these commands can help the attacker identify the type of system they have gained access to and determine what kind of access they have to mount further attacks, gain persistence, or move laterally through the network.
• Discovery: Powershell Attacker

  • From this output, it appears the attacker used the Invoke-WebRequest cmdlet to download the "winPEASany.exe" file from the attacker's server, saving it as winPEAS.exe.
  • A script that searches for possible paths to escalate privileges on Windows hosts.
  • It can quickly identify potential privilege escalation paths by looking for misconfigured scheduled tasks, insecure registry keys, and other common vulnerabilities.
• Discovery: Powershell Attacker

  • Specifically, the attacker is querying the registry to see if the AlwaysInstallElevated entry is set. This entry determines whether users can install programs or updates on the system with elevated privileges, which is a security risk if set without proper controls
  • If the attacker managed to gain higher levels of access to Bill's system, they could perform more malicious activities, such as creating backdoors for persistent access or modifying system configurations.
• IoCs: Privilege Escalation

  • In the context of MITRE ATT&CK, "Privilege Escalation" refers to a tactic used by adversaries to obtain higher access privileges within a compromised system or network.
  • It involves using techniques and exploiting misconfigurations to elevate privileges from the initial compromised state to gain more control and access to sensitive resources, data, or functionality.