The Internet is a big place, stuffed to the gills with just about every kind of information you could ever hope to find. Unfortunately, it sometimes seems that hoping to find what you're looking for is about as far as you get! See if this pattern rings any bells:
Enjoyable as this is (and it's one of the things that hooked me on the 'net!), it can be a little annoying if your initial search is important. Or for a paying client...
Search engines are very thorough these days, with most of the big ones indexing between thirty million and fifty million pages. What they have not yet mastered is the art of sorting the wheat from the chaff and, with around a thousand new sites coming online every day, there's an awful lot of chaff out there!
To help you find the grain of wheat that you're looking for here are a few tried and tested techniques that you may find useful.
The first of these is to pick your search terms with care. Searching on "Oasis" will not only return information about the desert, but also sites with musical references and various soft drinks facts too! Think hard about the words you are submitting. Do they have any other meanings? Can you be more precise?
Of course, even with the best will in the world, it is still possible to get it horribly wrong. I was once looking for information on a family member with the surname of Shergold. Unfortunately, "Craig Shergold" is a net-legend and so the search got completely swamped! It took me quite a while to filter the search down enough to get anything vaguely useful from it.
On some search engines it is possible to set whether a specific word must be included or, just as usefully, must not be included. In Alta Vista this is done by prefixing the mandatory words with a plus (+) and the prohibited words by a minus (-). So, in the Craig Shergold example above I would filter the search as follows "+Shergold, -Craig". Other engines use different syntax, so check with the usage instructions for the tool you are using.
Even with this form of filtering you will still probably get some unrelated hits. Try not to let yourself get side-tracked. Easier said than done, I know! One trick that I use is to have my bookmarks ordered by category, with the browser configured to add sites to a "New Stuff" folder. This allows me to pop into a site, quickly bookmark it, and then back out knowing that I can always visit it later. This is invaluable when preparing the Weekly Hotlist!
To make your searches even more precise, some engines support the use of Boolean algebra on search terms. A tutorial on this subject would go way beyond the bounds of this article, but a series of tutorials for the major search engines can be found here.
So far we have looked at narrowing searches to filter out unwanted hits. However, there are times when your subject is so nebulous that any hits at all would be welcome! This is when a meta-engine comes into its own. A search engine searches the Internet (or an aspect of it). A meta-engine passes your search terms through to a whole battery of search engines and then passes their results back to you.
For speed, one of the best meta-engines is Cyber411 which drives no fewer than fifteen search engines. However, its returns usually contain some duplicates and expired links, and its phrase searching tends to look for just the words rather than the words in the correct order. My choice would be MetaCrawler. This tool takes a little longer and "only" queries nine engines, but its phrase searching is spot on, and it also attempts to both remove duplicates and validate URL's (Uniform Resource Locators) before passing its results back to you.
It is important to use the appropriate search engine for the kind of search you want to do, as each has their own strengths and weaknesses. Here's a quick run down of what I tend to use each one for:
These are only a few of the many, many search tools that are now available on the World Wide Web. A comprehensive list would be far too long for this article, and would also be out of date as soon as it was published! However, there are people who maintain lists of the major search engines. Eureka is one site that holds a very extensive list, which might be just the thing for a special search.
There is one final strategy which is often overlooked by some and misused by others. It makes use of the internet's most powerful resource; people. Asking a question on Usenet draws on the knowledge of millions of people. Someone out there almost certainly has the information you're after. The trick is to ask the right question, in the right place, and in the right way to get the right result. My next article will try to show just how that can be done.
Frank Beaney, September 1996
for Baudwalk Webzine