Logging

• Log Operations

  • Log configuration is an essential aspect of managing systems and involves various purposes, including security, operational stability, regulatory compliance, and debugging.
  • Properly configured logs are vital for cybersecurity, operational efficiency, compliance with regulations, and software development, providing organizations with detailed information about their systems, assets, and resources.
• Log Configuration

  Security Purposes

  Log configuration for security focuses on detecting and responding to anomalies and security issues. The goal is to ensure user activity is legitimate, maintain authorization control, and detect unauthorized access promptly. Key areas include:

  • Anomaly and threat detection
  • Logging user authentication data
  • Ensuring system integrity and data confidentiality
• Log Configuration

  Operational Purposes

  Operational logging and configuration aim to detect and address system errors, enhance performance, and ensure system reliability. This involves:

  • Generating reports and notifications on system and component status
  • Troubleshooting
  • Capacity planning
  • Service billing
• Log Configuration

  Legal Purposes

  Logging for legal purposes focuses on maintaining compliance with regulations and obligations. Organizations must adhere to a set of responsibilities and guidelines, such as:

  • Aligning with standards and regulations (e.g., ISO 27001, COBIT, GDPR, PCI DSS, HIPAA, FISMA)
  • Maintaining a central log management system and adequate log configuration
  • Ensuring 12-month log retention with readily searchable recent data
  • Conducting system security checks and yearly audits for compliance (e.g., PCI logging compliance)
• Log Configuration

  Debug Purposes

  Debug logging is primarily used to improve system reliability and feature enhancement by identifying bugs and potential faults. This configuration is often not used in production but in testing and development environments. Key focuses include:

  • Increasing visibility for application debugging
  • Enhancing efficiency
  • Speeding up the development process
• Deciding Logging Purposes

  • What will you log, and for what (asset scope and logging purpose)?
  • How much are you going to log (detail scope)?
  • How much do you need to log?
  • How are you going to log (collection)?
  • How are you going to store collected logs?
  • How are you going to protect the logs?
  • How are you going to analyse collected logs?
  • Do you have enough resources and workforce to do logging?
  • Do you have enough budget to plan, implement and maintain logging?
• Log Configuration Dilemma: Requirements, Aspirations, Resources, and Investment

  • The configuration dilemma highlights the challenges associated with implementing effective log configurations.
  • Each scope of log configuration carries specific responsibilities, guidelines, and challenges, making the process more complex than merely enabling logging on assets.
  • This complexity arises from the need to balance various factors, including requirements, scope, detail, and cost, which encompass financial and labor expenses, risks, and investments.
• Log Configurations: Balancing Requirements and Aspirations

  • Log configuration plans are crafted through a detailed analysis of the scope, assets, objectives, requirements, and expectations. This process involves system administrators, legal and financial advisors, and managers to ensure all aspects are considered.
  • The primary challenge is balancing non-negotiable operational and security requirements with aspirations for enhanced capabilities through additional data and insights.
  • Comprehensive risk assessments that prioritize security, compliance, and legal needs are essential
• Logging

  Base Requirements:

  • What happened?
  • When did it happen? (With time data, if possible)
  • Where did it happen? (Network, system, folder, path, interface)
  • Who/What caused it to occur? (From which log source)
  • Is it possible to have more data?

  The base requirements focus on an incident detection mindset, providing a solid framework for logging and analysis that is primarily reactive. This approach is effective against known threats but may not address more sophisticated challenges.
• Logging

  Aspirations for Better Insights:

  • More details
  • How sure can I be that this is true?
  • What is affected?
  • What will happen next?
  • Is there anything else that requires attention?
  • What should I do about the incident?

  Aspirations represent a threat-hunting mindset, which is proactive and resource-intensive. This approach is geared towards addressing advanced and sophisticated threats by going beyond basic requirements.
• Logging Principles

  Collection

  • Define the logging purpose.
  • Collect what you will need and use.
  • Do not collect irrelevant data.
  • Avoid log noise.

  Format

  • Log at the correct level and detail.
  • Implement a consistent log format.
  • Ensure that timestamps in logs are accurate and synchronised.

  Archiving and Accessibility

  • Define log retention policies and implement them.
  • Store log data and make sure the important part is available for analysis. 
  • Create backups of stored log data and used systems.
• Logging Principles

  Monitoring and Alerting

  • Create alerts and notifications for important and noteworthy cases.
  • Focus on actionable alerts and avoid noise.

  Security

  • Protect logs by implementing access controls.
  • Implement encryption if required.
  • Use a dedicated log management solution.

  Continuous Change

  • Logging sources, types, and messages are constantly changing and being updated.
  • Be open to continuous change.
  • Train your personnel.
• Logging: Challenges

  Data Volume and Noise

  • Having multiple data sources to deal with.
  • Differences in the log volumes created by applications.
  • Some applications generate an insufficient amount of logs.
  • Large-scale applicants could generate massive log volumes
  • Some logs can provide non-essential data and make the identifying process difficult.

  System Performance and Collection

  • Log collection can slow down the system's performance.
  • Systems are not always "state of the art".
  • Managing system and agent version updates and synchronisation in large-scale networks is overwhelming.
• Logging Challenges

  Process and Archive

  • Having multiple data formats to deal with it.
  • Balancing the log retention can be challenging. Especially when dealing with many compliance regulations and standards.

  Security

  • Ensuring data security is a task/challenge in itself.

  Analysis

  • Combining, correlating, and analysing data from multiple sources to understand the context of an incident is a time-consuming process that requires significant computing resources and expertise.
  • Achieving this in real-time is also another challenge in the same scope.
  • Avoiding false positives/negatives is overwhelming.
• Logging Challenges

  Misc

  • Lack of planning and roadmap.
  • Lack of financial resources/budget.
  • Lack of implementation scenarios, playbooks, and exercises.
  • Lack of technical skills to implement, maintain, and analyse.
  • Focusing on log collection instead of the analysis phase.
  • Ignoring human factors and potential system errors.
• Logging: Common Mistakes and Best Practices

  • Logging sensitive information!
  • Creating logs by yourself.
  • Having uncollected logs.
  • Collecting everything but not analysing.
  • Collecting logs without proper planning and configuration.
  • Having systems that lack the planned/required log configuration.
  • Skipping the scale, testing, and functionality analysis.
  • Focusing on edges and skipping the internal systems in analysis.
  • "Searching for what you want to find" and "Not investigating what you see".
  • Forgetting that the process takes the form of proper planning, management, and analysis.
• Logging: Common Mistakes and Best Practices:

  Do's

  • Create a suitable log configuration and plan according to your systems.
  • Implement testing on scale, functionality, and operational stability.
  • Exclude logging sensitive information!
  • Secure your logs.
  • Create meaningful alerts/notifications.
  • Focus on having insights on actionable and impactful results.
  • Train your analysts and enhance their skills.
  • Update/maintain your operation plans and components/assets as needed.