Most web development material stops at "and now it is live".
That was a reasonable place to stop when the only audiences were people. It is not any more. A site published in 2026 immediately raises two questions the old curriculum never had to answer: who is allowed to reach this content, and what happens when a machine reads it, summarises it, and answers somebody's question with it.
This course starts where the tutorials finish. It assumes you can get a static site live from a repository, and it picks up the two things that happen next. The first stream is about access: what a gate actually is, why the one an AI assistant writes for you protects nothing, and what the real options cost in money and effort. The second is about discoverability in a world where the destination is often a citation rather than a click, and where a lot of what is sold as the answer has no evidence behind it.
Both streams are built on the same distinction, which is the reason they belong in one course: where the decision is made. A gate is real when the decision happens somewhere the visitor cannot reach. A signal is worth publishing when somebody who matters has decided to read it. Get that distinction straight once and both halves of this become much simpler.
No rankings, no citation counts, no traffic numbers. The honest state of this field is that much AI influence is currently unmeasurable, and that the tactics with evidence behind them are research-benchmark findings rather than guarantees. What you get instead is the ability to build correctly, to check a claim against its primary source, and to report uncertainty rather than paper over it. Those outlast any tactic list.
- What actually happens between a request and a response, and where code runs
- Why a passcode checked in the browser protects nothing, demonstrated rather than asserted
- The three genuine options for one shared passcode, and what each costs
- Where secrets belong, and why a repository holding gate logic should be private
- How to tell a ratified standard from one party's proposal
- The layered model of machine readability, from robots.txt down to unproven conventions
- Which AI crawlers to allow and which to block, and why blocking the wrong one makes you invisible
- What the research shows earns a citation, and what measurably backfires
- Why AI referral traffic is systematically undercounted, and what server logs can tell you
- How to report on your own site honestly, including what you could not measure
Contents
- FoundationFoundations and mental model
- FoundationThe honest baseline
- Stream AAccess and security gates
- Stream BMachine-readable sites optimised for LLMs
- CapstoneApply both streams to your own site
- OptionalThe marketing and content-performance angle
- Before useCurrency, sources and what to re-verify
Written for someone who already has the basics working.
You should be comfortable editing HTML and CSS, using Git and a repository host, and deploying a static site. No backend or framework experience is assumed. If you have never published a site at all, start with the Putting a website online walkthrough and come back.
The two streams are self-contained enough to be taken separately. Someone in a communications or marketing role could take Stream B alone; someone security-curious could take Stream A alone. Taken together they make an argument that neither makes on its own.
- Comfort with HTML and CSS, and a text editor you like
- A repository host account and basic Git
- A site already deploying, or the willingness to set one up in the opening phase
- A browser with developer tools, which is all of them
- No paid accounts required to begin. Two phases reference paid tiers, always alongside a free path
Foundations and mental model
Neither stream makes sense until the picture is accurate. Before anything else, you need to know what actually happens when a request hits your site, and what happens when an AI system encounters it.
What the phase covers
The request and response cycle, and where code runs: in the browser, on the server, or at the edge. What "static site" really means, and what it can and cannot do as a result. What a crawler is and how it differs from a human visitor. And the difference between a generative model answering from what it absorbed during training and a retrieval system fetching your live page right now, which is a distinction most coverage of this subject blurs.
This phase sets up the single most important idea in the course: client-side versus server-side. That one distinction is the hinge for both streams. It is why a browser-checked passcode fails, and it is why some signals you publish are read and others are ignored.
Watch a page arrive
Open developer tools on any site and keep the network panel open while the page loads. Watch the requests go out and the responses come back. Read the response headers on the main document. Then look at the sources panel and find where the HTML ends and the JavaScript begins. Everything you can see there, the visitor can see too, which is the whole of Stream A in one observation.
The honest baseline
A short grounding in what is settled and what is contested, so you treat the rest of this course with the right amount of scepticism. This phase exists because the AI-and-the-web field is thick with vendor content marketing that inflates the importance of unproven techniques, and because the habit it teaches is worth more than any individual fact in the course.
What the phase covers
How to tell a genuine web standard, backed by a standards body and respected in practice, from a proposed convention published by one party and adopted variably. The worked example is robots.txt, a real standard codified as IETF RFC 9309 in 2022, held against llms.txt, a proposal from September 2024 with contested adoption. Then: how to read a vendor claim and find the primary source behind it, which is usually one hop away and frequently says something weaker than the claim.
What you should leave with is a reflex. Before adopting any technique in this field, ask two questions: who says, and is it adopted.
Trace one claim to its source
Take any confident claim from a vendor blog about AI search, and follow it back. Note what the primary source actually says, when it was published, and whether the claim survived the journey. Do this once and you will do it automatically afterwards.
Access and security gates
Built around a single sharp teaching point: a passcode checked in the browser protects nothing, and understanding exactly why teaches you how web security actually works. The stream is short, and it earns its length by being demonstrated rather than explained.
The broken gate
Build a frontend-only JavaScript passcode gate: the kind an AI coding assistant will happily generate if you ask it to make the page private. Then break it live with developer tools. Read the passcode straight out of the source. Fetch the "protected" content directly, gate never involved. Edit the check so it stops asking.
This is the security-theatre demonstration, and it is the spine of the stream. It is worth building the broken version yourself precisely so you can break it; the value is in the demonstration, not in the protection.
Why it failed
The client-side versus server-side distinction, made concrete by the thing you just did. Anything the browser can check, the visitor can read, because the browser downloaded it in order to check it. Real access control decides before the content is sent, not after.
Name the general principle here, because it transfers well beyond passcodes. Hidden prices, disabled buttons, client-side validation, feature flags: the same test applies to all of them. Ask where the decision is made. If the answer is "on the visitor's machine", it is not a control.
The real options for one shared passcode
Three genuine solutions for the common case: showing a site to people you choose, without running a subscriber system.
| Option | How it works | Trade-off |
|---|---|---|
| Server function with a signed cookie | A serverless function compares the passcode against a secret held in an environment variable, and on success returns a signed, HTTP-only, Secure cookie. Protected content is served only when a valid cookie arrives. | The most effort and the most control. Free at personal-site scale on typical function allowances. Per page or per section. This is the one worth learning, because it is where the security reasoning becomes concrete. |
| Edge access control with a one-time PIN | An access layer in front of the site intercepts every request at the network edge and emails a short-lived PIN to an allow-listed address. Nothing is served until the PIN is accepted. | No code, and strong, because the decision happens outside your site entirely. A free tier covers a small allow-list. Different mental model: it identifies people by inbox rather than sharing one secret, which is usually an improvement. |
| Host-level site password | One password for the whole site, set in the host's dashboard and enforced at the host's edge before anything is served. | The simplest of the three and genuine protection, but whole-site only and behind a paid plan on the hosts that offer it. |
Each phase of this stream includes the trade-offs that actually decide the choice: cost, setup complexity, per-page versus whole-site, and what each option protects against. It also covers one to know about and not reach for: built-in host identity products that have been deprecated, partially walked back, and left without active development. Alive enough to appear in old tutorials, dead enough that nothing will be fixed.
Pricing and tier names for all three options have changed within the past year, and at least one host moved to a different pricing model entirely. Dated figures with confidence labels are held in the access-gates resource written for this course; re-verify them against each vendor's current page before quoting them to anyone. Deliberately, no price appears on this page or in the interactive explainer.
Where secrets live
Why the repository must be private when it contains gate logic. Why secrets belong in environment variables rather than in committed code, and why deleting the line later does not help, because it is in the history. What an attacker can do with a public repository containing a signing secret: forge a valid token and walk in as a legitimate visitor.
This phase connects the access-control idea back to your actual repository and host settings, which is where it either holds or does not.
Machine-readable sites optimised for LLMs
The shift the course is named for. Search is no longer only about ranking links; increasingly it is about being the source an AI system reads, quotes and cites. This stream is deliberately balanced between the mechanics, meaning what you put in the files, and the strategy, meaning what actually earns a citation. It is also the stream where the honest-baseline habit does the most work, because this is where the hype is.
The vocabulary, and whether it is settled
SEO, GEO, AEO and the newer labels: what each means, where the terms came from, and the honest fact that the terminology is not settled. GEO is anchored to its academic origin rather than to a vendor blog, which is what lets the topic be taught as something with a research basis. AEO is Answer Engine Optimisation in mainstream use; the "Agent" expansion is an emerging minority reading and should be named as such.
The purpose of this phase is to stop you mistaking marketing jargon for established categories, which is easy to do when four vendors are each pushing their own acronym.
The layered model of machine readability
Machine readability taught as a stack, from most-standard to least, so any new technique can be placed on the spectrum rather than adopted uncritically.
| Layer | Status |
|---|---|
| robots.txt, including its AI-crawler directives | A genuine standard, IETF RFC 9309 (2022), respected in stated policy by the major search and most major AI crawlers. It controls permission, and nothing else. |
| Semantic HTML and XML sitemaps | Foundational and universally respected. The base layer of being readable by any machine at all. |
| Schema.org structured data as JSON-LD | Useful infrastructure with mixed evidence on citations. Confirmed helpful on some AI surfaces; a late-2025 study found no correlation with citation rates in several major assistants, and some engines missed data placed only in the JSON-LD. Treat it as amplification of an already-strong page, not as a citation switch. |
| llms.txt and similar proposed conventions | Taught explicitly as unproven. Proposed September 2024, no standards-body process, described by its own author as not an official standard, and largely unread by the systems it is pitched at on 2026 telemetry. Cheap to publish; not a standard, and not a replacement for robots.txt, which does a different job. |
The skill this phase builds is placing a technique on that spectrum before adopting it, rather than adopting all of them and hoping.
Controlling the AI crawlers
The actual user-agent names for the major AI crawlers, and the distinction that trips people up: training crawlers against retrieval and search crawlers. Blocking a training crawler is a content-rights choice and costs you nothing in AI-answer visibility. Blocking a retrieval or search crawler makes you invisible in that system's live answers, which is self-inflicted and surprisingly common.
Also in this phase: policy against reality. Your robots.txt states your policy; the server and edge logs show what actually happened. Some crawlers do not comply, and enforcing a block against one of those needs edge or firewall rules rather than a Disallow line.
Check the crawler names against the vendors
This is a live exercise, not a table to memorise. The crawler names change several times a year, and lists published on blogs go stale quickly. Take the list from this course, open each vendor's own bot documentation, and confirm or correct it. Note the date you checked. That habit is the deliverable, not the list.
Earning the citation
What the research actually shows moves the needle: specific statistics attributed to named sources, direct quotations, citations, fluent and authoritative prose, and clear headings framed as the questions people ask. Contrasted with what does not transfer from old search practice: keyword stuffing measurably backfires, scoring below an untouched baseline in the benchmark rather than merely failing to help.
Grounded in the academic GEO findings rather than assertion, and with the honest framing attached: those are research-benchmark results on the paper's own metrics, not promises about your site. One finding worth carrying: lower-ranked sites gained more from these tactics than sites already at the top.
Read the winning tactics back as a description and you get a page that says something specific, attributes its numbers, quotes its sources, reads well, and knows its subject. Which is to say: good, honest, well-sourced writing, rather than technical trickery.
Measuring it, and why it is hard
Why AI referral traffic is systematically undercounted in ordinary analytics: many AI surfaces pass no referrer, and many citations produce no click at all. The newer analytics channels and their gaps, including which assistants a given platform's published channel definition does and does not cover, and why that has to be checked rather than assumed. And why server-log analysis is the most reliable way to confirm AI crawler activity, since a tool that runs in the browser cannot see a crawler that never runs its code.
The phase finishes with vendor conversion claims and how to hold them: as vendor benchmarks, not peer-reviewed findings. The robust, teachable fact is not a multiplier; it is that a large share of the influence is unmeasured.
Apply both streams to your own site
Both streams, on a site you actually run. Four pieces of work, and one piece of writing that is the point of the whole thing.
- Add a real access gate to a chosen section, using whichever of the three options fits your case, and be able to say why you chose it
- Make the public part of the site properly machine-readable, working up the layered model rather than down it
- Verify AI crawler behaviour in your own logs, and record which crawlers you saw and when you looked
- Audit your own secrets: nothing sensitive in the repository or its history, and the repository private if it holds gate logic
- Write a short reflective piece on what you could and could not measure
That last item is assessed most heavily, and deliberately so. The capstone rewards honest reporting of uncertainty over inflated claims of success. Writing "I could not determine whether this made any difference, and here is why not" is a stronger result than an unfounded number, and it is the professional habit this course is really trying to build.
The marketing and content-performance angle
The two core streams stand alone, but Stream B adapts cleanly for a marketing or communications audience by adding an applied layer on top of the mechanics. The shape is shared foundations first, then the specialisation.
The shared foundations layer
What a large language model is, how it is trained, how it stores and recalls content, and the distinction between generative and agentic use. Plus whatever AI policy and compliance guardrails the delivering organisation works under, which varies enough that it cannot be written generically.
The marketing core layer
How AI systems and traditional search handle content differently. Rankings against citations. Crawlability and scannability. The performance trade-off between AI surfaces, which are strong for brand-perception and comparison queries, and traditional search, which is stronger for high-intent conversion. The practical question of how to become the source an AI reaches for. And a phase on where models form their impressions: which sources they weight, including community platforms, video and established references, and what that implies for how a brand surfaces.
This layer works best grounded in the organisation's own current performance data rather than generic examples, which is worth saying to anyone commissioning it.
What to re-verify before delivering this
Two categories of fact in this course go stale faster than the course does, and both are flagged wherever they appear.
- Pricing and plan names for every hosting and access-control option in Stream A. At least one vendor changed pricing model within the past year
- The AI crawler user-agent names in Stream B, which change several times a year
- Which assistants a given analytics platform's AI channel definition currently covers, which changed twice within a year
- Whether any major AI company has documented its production system reading llms.txt, which is the single fact that would change that phase's verdict
- Whether the "Agent" reading of AEO has moved from fringe toward mainstream, which could shift within months
The underlying material is held in two companion documents written for this course: a self-teaching resource on access gates and login options, and a report on the shift from SEO to GEO and how AI systems cite sources. Both carry confidence labels on their factual claims and dates on their figures, and both name the things their author could not verify. They draw on a consolidated research brief covering vendor documentation and pricing pages, the academic GEO paper, published crawler telemetry and standards documents.
Both are single self-contained files that run without a build step and work when opened directly. The security explainer is a closed simulation throughout: it contains no working passcodes, no signing code, and nothing that functions as a bypass against a live site. Its teaching value is in the demonstration of the principle. The GEO explainer keeps every status claim in line with the confidence labelling of the report behind it, hedging where the report hedges.
Last updated 14 August 2026