Covert Crawler Descends on Web
The program comes from different internet addresses, simulates different browsers and throttles itself to human-like speeds.
Hoffman's program downloads everything that comes with a page -- images, JavaScript and components like ActiveX and Flash -- instead of just hitting the page itself like traditional spiders do. It also simulates a full web browser, keeping a cache and requesting only new material.
