Part 5: Microsoft XDR Advanced Hunting and Automating Security Workflows
Utilizing Microsoft XDR to automate repetitive security tasks and workflows, improving efficiency and response times.
In this fifth blog post I will explore how to use Microsoft XDR to automate security workflows and improve efficiency and response times. One of the key features of Microsoft XDR is advanced hunting, which allows you to run complex queries across various data sources and hunt for threats and anomalies. Advanced hunting can be combined with custom detection rules, which are a way to create custom alerts and trigger automated responses based on advanced hunting queries.
Introduction
What are Advanced Hunting and Custom detection rules?
How to Automate Security Workflows with Advanced Hunting and Custom Detection Rules
My recommendations
Conclusion
Introduction
Microsoft XDR leverages the power of artificial intelligence, machine learning, and behavioral analytics to identify and respond to threats in real time. One of the key features of Microsoft XDR is advanced hunting, which allows security analysts to create custom queries and rules to hunt for threats across the entire environment. Advanced hunting can also be used to automate security workflows, such as alert triage, investigation and remediation. In this blog post, I will explain what advanced hunting and custom detection rules are, how they can help automate security workflows, and share some of my recommendations for using them effectively.
What are Advanced Hunting and Custom Detection Rules?
Advanced hunting is a feature of Microsoft XDR that enables security analysts to perform proactive and reactive threat hunting across the entire environment. It allows analysts to use a query language called Kusto Query Language (KQL) to search for indicators of compromise, suspicious activities, and anomalies across different data sources, such as endpoints, network devices, cloud services, and identity events. Advanced hunting also provides a rich set of operators, functions, and visualizations to help analysts filter, aggregate, and analyze the data. Advanced hunting can be accessed from the Microsoft Defender XDR portal.
Advanced hunting is part of the security.microsoft.com portal and uses the open-source Kusto Query Language (KQL). Microsoft made KQL to query large data sets stored in the Azure Cloud. The language is used in Log Analytics, Sentinel, Defender for Endpoint, Azure Data Explorer amongst other Microsoft resources.
Since most of the Microsoft security products share one portal, you can search for data and do unified hunting steps across all signals. advanced hunting is also good for checking the state of Defender, not just for threat/ security hunting. For example:
Show all devices where ASR is blocking a file
Show all devices with EOL versions of Windows
Show all user logins from RU
Show blocked network connections via Web content Filtering
There are two options for advanced hunting; you can use a guided or an advanced mode. The guided mode is helpful if you are not very familiar with KQL. I strongly advise you to learn KQL and use the advanced mode.
Advanced mode = Query in Editor
Guided mode = Query in builder
With the query builder in guided mode, you can make new queries visually without Kusto Query Language(KQL) knowledge. The query builder and data filters work for all experience levels. But you should learn KQL and the advanced editor. KQL is very useful and important for future detections and hunting.
Microsoft information Microsoft: Kusto Query Language (KQL) overview
Data Sources
These are the applications/ services that belong to the advanced hunting platform in Microsoft Defender XDR/ security.microsoft.com at the moment:
Microsoft Defender for Endpoint (MDE)
Microsoft Defender for Cloud Apps (MDA)
Microsoft Defender for Office (MDO)
Microsoft Defender for Identity (MDI)
Azure AD Identity Protection (AADIP)
Azure AD user/ sign-in information (AAD)
Important knowledge
Data coverage: Each query can access data from up to the last 30 days. The data timeframe is consistent for hunting and custom detections.
Result limit: Each query can show up to 10,000 records. Use filtering whenever possible to narrow down results.
Max duration: Each query can run for up to 10 minutes. If it takes longer than 10 minutes, there will be a service error.
CPU resources: When queries use more than 10% of the allocated resources, there will be an error. Queries are blocked if the tenant has reached 100% until the next 15-minute cycle. Always optimize queries.
Custom detection rules
A feature that I really like in Defender for Endpoint is the ability to create detection rules based on advanced hunting queries. Every environment needs custom detections for specific apps, behaviors, and more. Microsoft does not alert for every “potentially malicious” activity, there is much more data available directly in Defender for creating custom detection rules. Some examples of custom detection rules are:
Forbidden software installed
Remote tools installed and connected outbound (TeamViewer)
Local account created
Hacker tools downloaded or installed (mimikatz)
An alert with a specific title or severity
What is it custom detection rules
With custom detection rules, you can proactively monitor different events and system states. You can set each rule to run regularly and alert you and take "automated" actions when there is a match.
You need the Security administrator/ Security operator roles or custom roles in MDE to create rules.
Custom detection rules require different fields. Each detection rule should have the following output:
Timestamp
ReportId
Accepted columns that identify specific devices, users, or mailboxes
Why do proactive hunting?
Alerts are not the only way to detect threats. Proactive hunting can provide more value than just protection. Each organization has different use cases. Advanced hunting allows getting more information on entities and IOCs. For example; display all activity for website xx, or display all activity from a specific IP address.
How to create detection rules?
In the query editor, write the KQL query and choose Create detection rule to fill in all necessary information including the general details. Create a short and clear title with the key information, and if a vendor assists you, be aware of their work and yours.
Provide all necessary information. Each rule looks for matches from the last 30 days. The rule can have more than one frequency interval. You can use this schema:
· Every 24 hours—runs every 24 hours, checking data from the past 30 days
· Every 12 hours—runs every 12 hours, checking data from the past 48 hours
· Every 3 hours—runs every 3 hours, checking data from the past 12 hours
· Every hour—runs hourly, checking data from the past 4 hours
The Continuous NRT frequency rules have been available from Microsoft for a while. These rules allow users to create new custom detections near real-time and detect threats more quickly.
At the moment, NRT works for queries that use a single table and do not have join, unions, or external data operator in the query. NRT works for these tables now:
AlertEvidence
DeviceEvents
DeviceFileCertificateInfo
DeviceFileEvents
DeviceImageLoadEvents
DeviceLogonEvents
DeviceNetworkEvents
DeviceNetworkInfo
DeviceInfo
DeviceProcessEvents
DeviceRegistryEvents
EmailAttachmentInfo
EmailEvents
EmailPostDeliveryEvents
EmailUrlInfo
UrlClickEvents
Impacted entities
To query and set up the right impacted entities, you need to know what they are. They can be a device, a user, or another entity. You can map a device with its DeviceId or DeviceName. You can use the AccountSid for a user. The wizard only shows the columns that the query returns.
Actions
You can set up custom detections to take automatic actions on devices, users, files, or emails. For devices, these actions involve separating, collecting investigation packages, scanning for antivirus, initiating investigations, or restricting app execution.
There are different actions that can be done on files, users, and emails. Some fields are suggested for each action. Tables are needed for each type of action (Device, User, Email, File). For example; for device actions, the table with DeviceId is needed, and for user actions, one of the tables (AccountObjectID, InitiatingProcessAccountObjectId, RecipientObjectId) is needed.
How to Automate Security Workflows with Advanced Hunting and Custom Detection Rules
Advanced hunting and custom detection rules can help automate security workflows in various ways, such as:
Alert triage: Custom detection rules can help filter out false positives and reduce alert fatigue by creating alerts only for the most relevant and critical events. For example, a custom detection rule can create an alert only when a suspicious process is executed by a user who is not part of the IT department, or when a network connection is made to a known malicious domain.
Investigation: Advanced hunting can help speed up the investigation process by providing contextual and historical data for the alerts. For example, an analyst can use advanced hunting to query for related events, such as other devices that have executed the same file, or other users who have logged in from the same IP address. Advanced hunting can also help identify the root cause and scope of the incident by tracing the attack chain and finding common patterns or indicators.
Remediation: Custom detection rules can help automate the remediation process by triggering actions that can mitigate the impact of the attack. For example, a custom detection rule can isolate a compromised device, block a malicious file, revoke an access token, or reset a password. These actions can help contain the threat and prevent further damage or data loss.
Reporting: Advanced hunting can help generate reports and dashboards that can provide insights and trends on the security posture and performance. For example, an analyst can use advanced hunting to create charts and tables that show the number and types of alerts, the response times and actions taken, the affected devices and users, and the common attack vectors and techniques.
My Recommendations
Advanced hunting and custom detection rules are powerful tools that can help security analysts automate repetitive security tasks and workflows, improving efficiency and response times. However, they also require careful planning, testing, and maintenance to ensure they are accurate, relevant, and secure. Here are some of my recommendations for using them effectively:
Define your use cases and objectives. Before creating advanced hunting queries and custom detection rules, you should have a clear idea of what you are trying to achieve, what data sources you need, what indicators of compromise you are looking for, and what actions you want to take. This will help you design your queries and rules more efficiently and avoid unnecessary complexity and false positives.
Build your queries in the query editor. Microsoft helps you learn KQL so you can make advanced hunting queries and custom detection rules with KQL code. It simplifies selecting data sources, fields, filters, and aggregations.
Test and validate your queries and rules. Before deploying your advanced hunting queries and custom detection rules, you should test and validate them to ensure they are working as expected and producing the desired results. You can use the test query and the test rule features in Microsoft XDR to run your queries and rules on historical data and see the results. You can also use the preview alert and the preview response features to see how your rules will generate alerts and trigger actions. You should also review your queries and rules for any potential security or privacy issues, such as exposing sensitive data or performing unauthorized actions.
Monitor and update your queries and rules. After deploying your advanced hunting queries and custom detection rules, you should monitor and update them regularly to ensure they are still effective and relevant. You can use the query performance and the rule performance features in Microsoft XDR to see the statistics and trends of your queries and rules, such as the number of executions, the number of results, the number of alerts, and the number of responses. You should also keep track of the changes and updates in the data sources, the threat landscape, and the best practices, and adjust your queries and rules accordingly.
Use the built-in queries and templates as a starting point: Microsoft XDR provides a library of built-in queries and templates that cover common scenarios and use cases. These queries and templates can help analysts get started with advanced hunting and custom detection rules, and can be modified or customized according to the specific needs and environment of the organization.
Test and validate the queries and rules before deploying them: It is important to test and validate the queries and rules before deploying them to avoid errors, false positives, or unintended consequences. Analysts can use the query editor and the rule preview features to run the queries and rules on historical data and check the results and accuracy. Analysts can also use the> operator to limit the scope of the query or rule to a specific device or user for testing purposes.
Review and update the queries and rules regularly: As the threat landscape and the environment evolve, the queries and rules may need to be updated or adjusted to keep up with the changes. Analysts should review and update the queries and rules regularly to ensure that they are still relevant, effective, and accurate. Analysts should also monitor the performance and impact of the queries and rules by checking the alerts and actions generated, and the feedback and comments from other analysts or stakeholders.
Conclusion
Advanced hunting and custom detection rules are key features of Microsoft XDR that allow security analysts to create custom queries and rules to hunt for threats across the entire environment and automate security workflows. In this blog post, I explained what advanced hunting and custom detection rules are, how they can help automate security workflows, and shared some of my recommendations for using them effectively. I hope you found this blog post useful and informative.