Google is using external raters to test their experiments so that they can decide on whether a change is useful to their user’s experience. I wonder if this is the best approach.
The raters look for a number of basic things on which to evaluate a search experience and feed the qualified results back into their process to determine whether the search engine user’s experience has been OK. They also look for spam and check on page items to judge whether the search engine user is getting a quality service. Here are some details about spam and on page design:
The types of things being looked for are hidden keywords (cloaking), keyword stuffing, and sneaky redirects.
On Page Design
Overall, the on page check is focused on the ratio of main content, supplemental content and ads. The page needs to be easy to read, providing clear communication of information. Page evaluation is not about good professional looking pages but rather the user finding what they want without getting lost. Spelling and grammar are also of interest but not as essential. Main content should be clear and ads secondary and easily separated from main content.
Here’s a great video put out by Google http://youtu.be/J5RZOU6vK4Q
Despite some criticism around the Web Google is quite open (although sometimes very cryptic) about how they approach changes to their search algorithm. Check out some of the insider information here www.google.com/insidesearch/underthehood.html. Even more impressive is the outline of their testing process www.google.com/competition/howgooglesearchworks.html#section2.
While this approach by Google seems all worthwhile, I can’t help wondering whether they could develop and improve their search service much more quickly and effectively if they just get their users to evaluate their best experience. This could also be done through market testing except the Google approach is probably considered safer (from an engineer or scientist’s point of view).