Broken links? No, thanks!
You may repeatedly have encountered these: Links that seem to point to a
valuable source of information, but once you clicked on them:
Bummer!
The page that you wanted to visit must have vanished into thin air!
This normally means that you have to start your search all over again (probably
with the site that didn't give you the desired information on your internal
blacklist) and look for another site that provides the desired information –
and pray that it exists there.
This is where BrokenLinksBot comes in. It is designed to dig up those annoying broken links and report back on its findings so that webmasters are able to fix any problems, thereby improving the overall experience for their visitors. The point is that if everything works smoothly with a given domain, visitors are more likely to return – and one prerequisite for a good experience is that all links present there are working as intended.
Unfortunately there are many possibilities for errors to be introduced. The
first and foremost of these are typos of any kind, resulting in broken links
where there should be a valid reference.
However, the ever-changing structure of the Web is another source for trouble:
Pages that existed before vanish and cannot be reached any more. Here you may
be able to control what's going on on your server and adjust any links
appropriately, but as soon as foreign domains are involved, you are just out of
luck. Your best bet would be to contact the admins of said domains to determine
where the pages in question have gone and adjust any links appropriately.
Unfortunately there could be some problems: You might not have the correct
contact info available and so cannot reach the site's operator, domain
ownership could have changed and the new admin is getting rid of the old
information, or incoming links from your server are locked out. Or even more
trivial: The entire domain at which a particular page has been available ceases
to exist.
Since manually looking for broken links and checking all of your documents or sifting through your logs is tedious at best and usually uses up a significant amount of your time, especially when larger sites are involved – that is, this time isn't available for productive work or requires that you task someone else with performing these checks – this is where BrokenLinksBot comes to your aid. It surveys your site at regular intervals and looks for anything that is broken or otherwise requires your attention, like unnecessary on-site redirects or documents that have moved to another location. So when BrokenLinksBot is your watchful eye, you can concentrate on other more productive tasks at hand instead of wasting your time on checking your site's integrity. Just let BrokenLinksBot keep you updated on the integrity of your links.
Current stage of development
BrokenLinksBot is still at an early stage of development, but the basic structure has already been established. Since it is currently alpha stage, you shouldn't see any of its activities in your logs. However, should it do show up, please drop me a note so that I can fix this problem (it means that BrokenLinksBot's internal restrictions are malfunctioning, which mandates immediate correction). This block is going to be lifted once it has matured to beta stage to check its behavior concerning foreign sites. If you feel that somehow BrokenLinksBot is starting a rampage across your sites or seems to otherwise misbehave, again drop me a note so that I can fix that. Since BrokenLinksBot abides by a strict ruleset to avoid swamping individual domains you should find neither multiple queries to the same portion of any of your sites during a particular run, nor should it be accessed by multiple threads simultaneously, and there should be a significant delay between individual requests.