Welcome to part II of the series dedicated to troubleshooting Windows account lockouts using Splunk. In part I (http://securitysynapse.com/2018/08/troubleshooting-windows-account-lockout-part-i.html) of the series, we highlighted and examined a 4740 event pulled from a domain controller. This 4740 event contained the following information:
- The domain controller that handled the authentication request and reported the lockout
- Domain name
- Account name
- The original host where the account attempted authentication
In this article we will look at a 4625 event from the originating host because it will contain further authentication details such as the reason for failure and the application that is attempting to authenticate. Our dashboard provided at the end of the article with also include searches for Event ID 529 to include Windows operating systems that are end of life (EOL).
 |
| Figure 1: Combining Event ID 4740 and Event ID 4625 to gain more insight into account lockout |
Examine the Problem
As we did with the 4740 event, we will now examine a fictional 4625 event and we will highlight and summarize the key points below. This fictional 4625 event was pulled from a host indicated by the 4740 event pulled from the domain controller.LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4625
EventType=0
Type=Information
ComputerName=WIN-R9H5Y.MYFAKEDOMAIN.COM
TaskCategory=Logon
OpCode=Info
RecordNumber=267153
Keywords=Audit Failure
Message=An account failed to log on.
Subject:
Security ID: S-1-5-18
Account Name: WIN-R9H5Y$
Account Domain: MYFAKEDOMAIN
Logon ID: 0x3E7
Logon Type: 8
Account For Which Logon Failed:
Security ID: S-1-0-0
Account Name: John
Account Domain: MyFakeDomain.com
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006D
Sub Status: 0xC000006A
Process Information:
Caller Process ID: 0x5aac
Caller Process Name: C:\Windows\System32\inetsrv\w3wp.exe
Network Information:
Workstation Name: WIN-R9H5Y
Source Network Address: 192.1.1.100
Source Port: 49770
Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
The most important takeaways from this event are:
- Failure Reason: In this case it was an unknown username or password. We know the username is correct, so it must be a bad password
- Caller Process Name: A quick Google search for w3wp.exe shows that it is most likely associated with an Exchange server running IIS.
After pulling a few more events, we see several more bad passwords and then the eventual lockout. Common causes for account lockouts indicated by this process are mobile devices (phone or tablet) that contain stale credentials. The mobile device continues to attempt to authenticate until it locks out the account. Mystery solved!
Conclusion
Even though we presented fictional event logs, this example is based on real situations. Fortunately we had the 4740 events from the domain controllers and we were collecting the 4625 logs from the rest of the servers (and some workstations). It would be very difficult and time consuming to perform this sort of correlation without a central point of aggregation such as Splunk. Even if you were to do this manually for one or two instances, you would not want to do it for the entire enterprise. To make your life easier, we are including dashboard code in the section below to display the 4625 events. We eventually added some workflow integration between the 4740 dashboard provided in the previous article and the 4625 dashboard below, but we will leave that exercise up to the reader. Have fun and happy Splunking.Dashboard Code
The following dashboard code relies on the index name of wineventlog. If this is not your Windows event log index, just change it to suit your needs. Also, the past few cases we worked had either a Qualys on Nessus scanner generating some noise. We left the Qualys filter in but disabled it. Feel free to also tweak that as needed.<form>
<label>Auth Examination - 4625</label>
<description>Event ID 4625 or 529</description>
<fieldset submitButton="true">
<input type="time" token="time" searchWhenChanged="true">
<label>Time Range</label>
<default>
<earliest>-4h@h</earliest>
<latest>now</latest>
</default>
</input>
<input type="text" token="wild" searchWhenChanged="true">
<label>Wildcard Search</label>
<default>*</default>
</input>
<input type="radio" token="notqualys" searchWhenChanged="true">
<label>Exclude Qualys</label>
<choice value="NOT Qualys">Yes</choice>
<choice value="*">No</choice>
<default>*</default>
<initialValue>*</initialValue>
</input>
</fieldset>
<row>
<panel>
<table>
<title>Top Failure_Reason</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 Failure_Reason</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
<panel>
<table>
<title>Top Domain</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 Account_Domain</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
<panel>
<table>
<title>Top User</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 user</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
<panel>
<table>
<title>Top src</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 src</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
</row>
<row>
<panel>
<table>
<title>Top Process</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 Caller_Process_Name</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
<panel>
<table>
<title>Top Status</title>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name | top limit=0 Status</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
</row>
<row>
<panel>
<title>Timechart by Account_Name</title>
<chart>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529"| timechart count by user</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">0</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">area</option>
<option name="charting.chart.bubbleMaximumSize">50</option>
<option name="charting.chart.bubbleMinimumSize">10</option>
<option name="charting.chart.bubbleSizeBy">area</option>
<option name="charting.chart.nullValueMode">gaps</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
</chart>
</panel>
<panel>
<title>Timechart by reporting host</title>
<chart>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529"| timechart count by dvc</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">0</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">area</option>
<option name="charting.chart.bubbleMaximumSize">50</option>
<option name="charting.chart.bubbleMinimumSize">10</option>
<option name="charting.chart.bubbleSizeBy">area</option>
<option name="charting.chart.nullValueMode">gaps</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Timechart by Account_Domain</title>
<chart>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529"| timechart count by Account_Domain</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">0</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">area</option>
<option name="charting.chart.bubbleMaximumSize">50</option>
<option name="charting.chart.bubbleMinimumSize">10</option>
<option name="charting.chart.bubbleSizeBy">area</option>
<option name="charting.chart.nullValueMode">gaps</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
</chart>
</panel>
<panel>
<title>Timechart by src</title>
<chart>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529"| timechart count by src</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">0</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">area</option>
<option name="charting.chart.bubbleMaximumSize">50</option>
<option name="charting.chart.bubbleMinimumSize">10</option>
<option name="charting.chart.bubbleSizeBy">area</option>
<option name="charting.chart.nullValueMode">gaps</option>
<option name="charting.chart.showDataLabels">none</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Details</title>
<table>
<search>
<query>index=wineventlog $wild$ $notqualys$ source="WinEventLog:Security" EventCode="4625" OR EventCode="529" | table _time, EventCode, Logon_Type, Status, Failure_Reason, Account_Domain, Account_Name, user, dvc, src, src_ip, Caller_Process_Name</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="drilldown">cell</option>
</table>
</panel>
</row>
</form>
