Crawling

Crawling means tracking the links on a webpage to discover new pages, and then carrying on to find and track links on those new pages to even more new pages. A web crawler is like a computer program that goes through all the links on a page, which takes it to new pages, and it keeps doing this until there are no more new links or pages left to go to