Automated link-checker for system testing
automated-tests, system-testing, web-crawler
Solution
I use Xenu's Link Sleuth for this sort of thing. Quickly check for no deadlinks etc. on a/any site. Just point it at any URI and it'll spider all links on that site.
Desription from site:
Xenu's Link Sleuth (TM) checks Web sites for broken links. Link verification is done on "normal" links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and java applets. It displays a continously updated list of URLs which you can sort by different criteria. A report can be produced at any time.
It meets all you're requirements apart from being scriptable as it's a windows app that requires manually starting.
Problem
I often have to work with fragile legacy websites that break in unexpected ways when logic or configuration are updated. I don't have the time or knowledge of the system needed to create a Selenium script. Besides, I don't want to check a specific use case - I want to verify every link and page on the site. I would like to create an automated system test that will spider through a site and check for broken links and crashes. Ideally, there would be a tool that I could use to achieve this. It should have as many as possible of the following features, in descending order of priority: - Triggered via script - Does not require human interaction - Follows all links including anchor tags and links to CSS and js files - Produces a log of all found 404s, 500s etc. - Can be deployed locally to check sites on intranets - Supports cookie/form-based authentication - Free/Open source There are many partial solutions out there, like FitNesse, Firefox's LinkChecker and the W3C link checker, but none of them do everything I need. I would like to use this test with projects using a range of technologies and platforms, so the more portable the solution the better. I realise this is no substitute for proper system testing, but it would be very useful if I had a convenient and automatable way of verifying that no part of the site was obviously broken.