(Lab 02)
The crawl
Scroll to crawl it ↓
01 — The map
This is your website, as a search engine sees it.
Not pages and pixels: a tangle of addresses, joined by whatever links happen to exist.
02 — The crawl
Crawlers follow your links.
Starting at the homepage, a crawler follows your links one click at a time, and gives up on pages buried too deep.
03 — The gaps
Some pages never get found.
Pages nothing links to, links to pages that are gone, and archives buried five clicks deep.
04 — The fix
So we bring it into order.
Every page linked from where it belongs, dead links redirected, nothing more than two clicks from home.
05 — Found
Now every page gets found.
A clean structure means search engines can crawl, understand and rank the whole site.
Pages found
0/24