A question arrives from a faculty executive after an incident. Which staff sent material relating to this project to an external AI service in the last six months, and what did they send?
The security team has a mature deployment: several years of data, a well-run detection practice, a platform that ingests from everything anyone has thought to connect. They can answer part of the question in an afternoon. No amount of skill or budget will produce the rest, and the reason is not that the query is hard to write.
Two structural reasons, and they are different. Some of the events that would answer the second half of the question were never produced by anything, so there is nothing to find. And among the events that were produced, some fell outside a retention window that a budget decision set eighteen months ago, when nobody knew this question would be asked.
Everything upstream in Part Four produces events. This chapter is about where they land, what can be done with them once they are there, and why the container is so often mistaken for the contents.
Why centralise at all
Events are produced by systems that do not know about each other. A sign-in is recorded by the identity layer. A network connection is recorded by an endpoint agent. A policy match is recorded by the compliance stack. A consent grant is recorded in the tenant audit log.
Each of those is a fact. Only together are they a narrative, and the narrative is what an investigation needs: this person signed in from an unfamiliar address, then this device opened a connection to this service, then this policy matched on this content. Centralisation exists so that the join is possible. That is the reason for the category, and everything else is implementation.
Normalisation, and the join that fails silently
Sources disagree about everything. Field names, timestamp formats, time zones, whether a user is identified by a principal name, a display name, an object identifier, a device account or an address. Normalising all of it to a common schema is unglamorous work and it is what makes correlation possible at all.
The practical consequence is worth dwelling on because it produces the most dangerous failure in this chapter. Identity resolution is the hard part. The same human appears differently in every source, and joining across them requires a mapping that is correct. When that mapping fails, the query does not error. It returns nothing, or returns less than it should, and an empty result set reads exactly like a clean bill of health.
A defender should treat a null result as a claim requiring evidence: run the query against a case known to exist, and confirm the join works, before reporting that nothing was found.
Correlation, detection engineering and hunting
A correlation rule is a hypothesis about what a pattern means, expressed so that a machine can evaluate it continuously. Writing rules, testing them against history, tuning them and retiring them when they stop earning their place is an ongoing engineering practice rather than a configuration step. Organisations that treat detection content as something purchased once end up with a rule set that reflects the threat landscape of whenever they bought it.
Hunting is the complement: a person asking a question the rules did not anticipate, over the same data. Rules answer the questions somebody thought of. Hunting is how new rules get discovered.
Behavioural analytics, usually sold under the initials UEBA, sits alongside both. Microsoft's implementation builds dynamic profiles for users, hosts, addresses and applications, and flags deviation from a baseline drawn from the entity's own history, from its peer group, and from the organisation as a whole (high). Peer groups are computed rather than declared, ranked from group memberships and similar associations (high).
The technique is powerful and the assumption underneath it is worth naming, because Part Five will need it. A baseline is a model of normal, and flagging deviation is flagging abnormality relative to a machine's judgement of what this person usually does and what people like them usually do. In a university, where a researcher's working pattern may legitimately look nothing like anyone else's, the assumption is weaker than it is in a call centre. That is a limitation of the technique and also the beginning of an ethical question.
Retention, tiering and the cost model
Ingestion is priced by volume and retention by volume and time. That single sentence determines more about what an organisation can investigate than any product capability.
The shape of the current model is worth understanding because it has moved. Microsoft Sentinel keeps data in an analytics tier, interactive and queryable at high performance, with ninety days the stated default and extension available up to two years. Microsoft's own pages are not consistent on that default, and data originating in the extended detection and response product defaults to thirty days, so confirm the figure for the specific table before relying on it. Beneath that sits a data lake tier, optimised for retaining large volumes cheaply for long periods, where storage is charged per gigabyte per month against a uniform compression assumption, and where queries are charged per gigabyte of uncompressed data scanned (high). The pattern generalises well beyond one vendor: hot data costs a lot and is free to query, cold data costs little and is charged to read.
Two consequences follow. The horizon of what can be investigated is set by a finance decision made in advance of the incident. And the data lake pattern changes where the cost falls rather than removing it, which is genuinely useful for exactly the case in this chapter's opening scene, where the question is rare, specific and historical.
SIEM and XDR distinguished
The two terms are used interchangeably and are not the same thing.
Extended detection and response correlates one vendor's own signals across endpoint, identity, mail and cloud, with the correlation logic built and maintained by that vendor. High fidelity, low effort, bounded to what that vendor sees.
Security information and event management ingests anything and requires the organisation to build the correlation. Unbounded scope, considerable effort, and the quality is whatever the organisation's own engineering makes it.
Most estates run both, with the vendor-integrated product as a high-fidelity core and the broader platform as the wide net. The distinction to hold onto is simply who does the integration work, and therefore who is accountable when a source is not integrated at all.
The shadow AI lens: the query and its four joins
Take the executive's question and express it as a query, and watch where each part comes from.
Start with sign-in records for the AI service, which exist if and only if the service federates to the tenant and the staff member used their work account. Join to discovery records for the destination, which exist for onboarded devices anywhere and for any device on a logged campus egress point, the latter usually without a name attached. Join to policy match events for the paste or upload, which exist only where a policy was in place and a sensor produced the match. Join to the device inventory, to establish whether the machine was onboarded at all, which determines whether the absence of endpoint events means 'nothing happened' or 'nothing was watching'.
That last join is the one defenders forget and the one that changes the interpretation of everything else.
What the platform can see, source by source. The structural finding from the internal device-state report is that these platforms reflect the blind spots of their connectors exactly. Endpoint events come only from onboarded devices. Discovery comes only from endpoint-integrated devices or from log collectors. Sign-in records cover every authentication the tenant brokered, regardless of device. Audit records cover tenant activity. Policy match events exist only where both a policy and a sensor were present (high).
The consequence, which is the sentence to carry into a governance meeting: off-campus personal-device use of a consumer AI service with a personal account appears in none of these, and off-campus personal-device use with a work account appears only as a sign-in record and audit metadata (high). The platform is not failing. It is reporting faithfully on a population that excludes the case of most concern.
Retention against the discovery interval. Incidents involving material sent to an AI service are usually discovered late, and usually through a channel other than an alert: a journalist, a complaint, a colleague, a vendor breach notification. The operative question is therefore not 'do we log this' but 'do we still have it'. Note the specific case: audit records for interactions with non-Microsoft AI applications are retained for 180 days under the pay-as-you-go model of Chapter 22 (high). Six months is a plausible discovery interval for this class of incident, which makes 180 days a boundary worth knowing rather than an abstraction.
The reporting temptation. A dashboard aggregating AI usage by named user is easy to build once the data is in one place, and it is the point at which a monitoring programme quietly changes character: from measuring a phenomenon to watching individuals. Nothing in the technology marks the transition, and the same data supports both. Flagging it here as a design decision with legal weight, and handing it to Part Five, where notice, proportionality and the Australian workplace surveillance position are dealt with properly.
The container is not the contents. A mature platform with poor connector coverage produces confident, fast, empty answers, and confidence is the dangerous part.
Identity resolution failures produce silent nulls. Test the join before trusting the absence.
Cost pressure truncates retention, and the truncation is decided long before the incident that needs the data. Nobody experiences the trade at the moment they make it.
Behavioural baselines encode assumptions about normal work. In a research institution the variance is genuine, and false positives here fall on individuals rather than on systems.
Correlation cannot manufacture a signal that no sensor produced. If the prompt was never observed, no query recovers it, and no amount of platform investment changes that. This is the single most important limit in the chapter and the one most often obscured by a good demonstration.
Alert volume competes with everything else in the queue. A shadow AI detection that fires often and resolves benign will be tuned out, exactly as in Chapter 25, and for the same reasons.
- Explain to an executive why the platform can answer 'who signed in to the AI service' but not 'what did they send'.
- For a four-part shadow AI query, name the source that answers each part and say what each source cannot contribute.
- Describe what a silent identity-join failure looks like in a report, and how you would detect one before publishing.
- Name the two decisions that set the investigation horizon, say when each is made, and say who makes them.
- Identify the point at which usage reporting becomes individual monitoring, and name two things you would want in place before crossing it.
Glossary terms used in this chapter
analytics tier · connector · correlation rule · data lake · Defender XDR · Defined from earlier chapters and reused here · detection engineering · device onboarding · ingestion · Insider Risk Management · normalisation · peer group analysis · retention tier · Sentinel · SIEM · SOAR · threat hunting · UEBA · unified audit log · XDR
Sources
Every Microsoft Learn page below was verified against the Microsoft Learn documentation service on 9 August 2026.
- Microsoft Learn, 'Log retention tiers in Microsoft Sentinel'. learn.microsoft.com The source for the analytics tier and data lake tier distinction, the ninety-day default interactive retention extensible to two years, and the framing of the trade between coverage and cost. Also the source for the statement that Microsoft Sentinel in the Azure portal will not be supported after 31 March 2027. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Plan costs and understand Microsoft Sentinel pricing and billing'. learn.microsoft.com Supports the cost model described here: ingestion charged per gigabyte, data lake storage charged per gigabyte per month against a uniform 6:1 compression assumption, and queries charged per gigabyte of uncompressed data scanned. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Manage data tiers and retention in Microsoft Sentinel'. learn.microsoft.com Supports the worked retention and cost examples, including that extending total retention beyond the analytics tier incurs data lake storage costs. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Azure Monitor Logs cost calculations and options'. learn.microsoft.com Supports the general statement that ingestion and retention are the dominant charges in a log platform of this kind. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Advanced threat detection with User and Entity Behavior Analytics (UEBA) in Microsoft Sentinel'. learn.microsoft.com The source for baselines drawn from an entity's own history, its peer group and the organisation, and for peer groups being computed from group memberships and similar associations rather than declared. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Use customizable anomalies to detect threats in Microsoft Sentinel'. learn.microsoft.com Supports the description of anomaly detection as deviation from a baseline across attributes including location, device, resource and network provider. Last checked 9 August 2026; (high).
- Microsoft Learn, 'Audit logs for Copilot and AI applications'. learn.microsoft.com Supports the 180-day retention of audit records for interactions with non-Microsoft AI applications under pay-as-you-go billing. Last checked 9 August 2026; (high).
- Shadow AI Controls in Microsoft 365 E5, Behaviour by Windows Device State (this project's internal source report), section 10. The source of the structural finding that these platforms reflect the blind spots of their connectors, and of the specific statement about what remains visible for off-campus personal-device use with a work account.
Open questions
Microsoft Sentinel's tiering and pricing model has changed more than once, including the replacement of the archive tier by the data lake tier and the introduction of query-time charging. The structure described here is current as at the check date; the figures and tier names should be verified before being quoted in a costing paper (medium). The ninety-day analytics tier default in particular is stated inconsistently across Microsoft's own documentation, with thirty days appearing for extended detection and response data, so read the retention setting for the table in question rather than the general figure (medium).
Microsoft has published that Sentinel in the Azure portal will not be supported after 31 March 2027 and that the Defender portal becomes the single interface. This affects tooling and runbooks rather than the concepts in this chapter, and organisations planning multi-year detection work should account for it (high).
Whether behavioural analytics is proportionate for staff monitoring in an Australian public-sector institution is a question this chapter raises and Part Five answers. I am not aware of published Australian sector guidance specific to entity behaviour analytics applied to staff (low).
Last updated 9 August 2026