Technical Foundations

Track 1 · Lesson 4 of 6 · about 15 minutes

Technical foundations

Technical SEO has a reputation for being intimidating. Most of it is four files and one idea: make sure the pages you want found can be reached, understood and told apart from each other.

robots.txt controls crawling, not indexing

robots.txt tells cooperating bots which paths they may request. It does not remove anything from an index. A disallowed URL can still be listed in results — usually with no useful snippet, because the engine was never allowed to look inside.

This produces the single most common technical own-goal in SEO: blocking a page in robots.txt and adding a noindex tag, then wondering why it stays in the results. The engine cannot see the noindex, because you forbade it from fetching the page that carries it. If you want a page gone, allow the crawl and serve the noindex.

noindex is a directive; canonical is a hint

That distinction is worth memorising. A robots meta noindex will be obeyed once the page is crawled. A rel=canonical is a suggestion — a strong one, weighed against your internal links, redirects and sitemap entries, but Google can and does select a different canonical than the one you declared. Search Console will tell you which one it picked.

Sitemap priority values, for what it is worth, are ignored entirely.

XML sitemaps

A sitemap is a discovery aid, not a ranking device. List the canonical, indexable URLs you actually want found — nothing that redirects, nothing blocked, nothing noindexed. A sitemap full of URLs that contradict your other signals is worse than no sitemap, because it makes your intentions ambiguous.

Keep it current. A sitemap that has not updated since your last three publishing runs is a stale file quietly telling an engine that nothing has changed.

Redirects

A 301 says permanently moved; a 302 says temporarily. Both pass signals in practice, but 301 is the one that consolidates, and using 302 for a permanent move leaves the old URL alive in the index far longer than you want.

Redirect to the closest equivalent page. Sweeping every retired URL to the homepage is treated as a soft 404, and a chain of four hops loses more with each step — redirect straight to the destination.

HTTPS, duplication and the one true URL

Every page should have exactly one address. http and https, www and non-www, trailing slash and not, uppercase and lowercase, and every tracking parameter you have ever appended are all potentially separate URLs to a machine. Pick one form, redirect the rest, and make your internal links agree with the choice. Most duplicate-content problems are this, not plagiarism.

Speed, in the three measurements that count

LCP — how long until the main content appears. Usually a large image or a slow server.
CLS — how much the layout jumps while loading. Usually images without dimensions, or a webfont swapping in.
INP — how quickly the page responds to interaction, across the whole visit. It replaced First Input Delay as a Core Web Vital in March 2024, and it is harder to game because it does not stop measuring after the first tap.

Use field data where you have it. Lab scores from a fast laptop on office wifi will flatter you; real users on real phones will not.

Going deeper: Technical SEO guides · robots.txt explained · free technical tools.