Using Google’s Search Quality Evaluator Guidelines to assess your site

What are Google’s Search Quality Evaluator Guidelines?  

To make sure that Google’s search engine is returning the right answers, Google regularly uses human contractors to evaluate search results to make sure that the latest algorithm is returning the kind of results they want. Google’s Quality Evaluator Guidelines are the instructions that they give these evaluators.

Contents

 

How can Google’s Search Quality Evaluator Guidelines be used to help websites?  

The guidelines that Google gives Quality Evaluators give us insight into what Google is aiming for. Even if Quality Evaluators don’t visit the site, meeting these criteria will help organisations match what Google believes is a good quality website. Jennifer Slegg’s piece does a good job of detailing the 2019 update to the Search Quality Evaluator Guidelines.

We’ve created a checklist that distills Google’s Quality Evaluator Guidelines and aims to give us more information to succeed in search.

This post and accompanying checklist will allow you to:

1. Gain insight into what a website and organisation is doing well and could be doing better, in terms of facilitating a good user experience and the creation of quality content.
2. Provide a platform for a website to perform well in search, by aligning with Google’s perceived quality attributes.
3. Get internal buy in. The checklist is consistent, independent, quantifiable and based on Google’s guidelines. It could be filled out by internal teams or used as part of user testing.
4. Make sure content is optimised for launch. The checklist can be used for internal stakeholders to go through before publishing new content, website features or most things that could impact the UX of a website and content quality.

 

Methodology

 Creating the checklist

We took Google’s 167 page Search Quality Evaluator documentation, Distilled’s previous Panda survey and Google’s guide to building high quality websites and condensed it down to the 10 most important topics.

We then took the questions that Google is trying to answer in each topic and turned them into Pass/Fail items in our survey.

Perhaps not surprisingly, the principle of what Google believes constitutes a high quality website hasn’t changed too much throughout the years, although the qualifying criteria have certainly become more robust. 

Weighting of questions

We decided how important a question should be based on wording in Google’s documentation.

For example, Google rates pages “lowest quality” if there is intent to deceive users or search engines. 

Under section 7.6 Pages that Potentially Deceive Users, Google writes:

Google then goes on to describe misleading titles as a form of deception.

Section 7.6.1 Deceptive Page Purpose:

As a result of this wording in Google’s documentation, where they deem deception to be indicative of “lowest quality” pages, we have assigned strong relative importance to questions that pertain to deception. 

If your site fails the question below, it will fail that entire portion of the checklist, because of the importance that Google has placed on accurate titles.

Does the title in Google Search accurately describe the topic of the page?

How to use the checklist 

 Pagetype questions

When you are filling out the checklist, there are some questions which should be asked about multiple different types of pages. The page types include:

• Homepage
• Product pages
• Blog pages 

Applying topics to different page types will give a broader view of the overall UX and content quality and help pinpoint strengths and weaknesses.  

Of course, some page types may not be relevant to an organisation. For instance, publications don’t have product pages, so asking questions about a product page is not applicable.

Sitewide questions

Over the last year or so Google has really focussed on the profile of organisations and content authors in their search algorithms, with August’s 2018 medic update being of particular note.

The E-A-T (Expertise, Trustworthiness and Authoritativeness) of organisations and content creators is seen by Google to be indicative of quality and value. Importantly, Google places more weight on the external evidence of E-A-T, rather than an organisations’ or content creators’ self-endorsement. 

We’ve created specific questions that focus on identifying external reputation, with the idea that external reputation represents the potential E-A-T of a website.

We also created questions that focussed on more hygiene aspects of a website, specifically the adequacy of information on a business/content creators and how the design of a website affects the overall UX.

How you would use insight gained from the checklist

There could be many actions that could be taken from the checklist. Some insights we’ve provided for clients when running our checklist include:

Customer concerns/complaints on third party websites

External reputation information of a brand is important for both Google and users. 

So by identifying negative sentiment towards a brand and/or product on an influential industry related website, we convinced our client to not only review concerns on said website, but commission an off site review investigation for many other related websites.  

Content creators lacking perceived expertise

Google says that the E-A-T of content creators is indicative of content quality. Our checklist highlighted that content creators, despite being experienced within the vertical they work in, didn’t have an online presence. 

The outcome of this insight was to plan guest writing opportunities on industry publications to enhance employees perceived expertise.

Lack of information on content creators 

Users want to learn more about and validate the legitimacy of content creators. Our checklist highlighted that individual content creators didn’t have bio pages. 

The outcomes of this insight was to develop individual bio pages, with information on their experience and expertise.  

Product page insights

Some common insights include:

1. Not fully understanding the purpose and/or benefits of a product: often a common issue on B2B websites, this can be the result of jargon heavy content, long-winded descriptions that don’t make the benefits clear, or a lack of FAQ’s.  
2. Page design: content is sometimes difficult to digest. This can be the result of bunched up blocks of text, blurry or broken images, or key content placed near the bottom of the page.

Blog page insights

If executed properly, an organisation’s blog is key in driving top of the funnel traffic. This checklist highlights areas such as:

1. Inaccurate headlines:  users need to have an accurate  understanding and expectation of the benefits/knowledge they could gain by reading a blog post. We certainly don’t want to mislead readers in any form.
2. Fact validation: to enhance the authority of your content, you should link to reputable third party sites to validate facts when possible and appropriate.
3. Unique content: create more in depth and potentially data driven pieces of content to encourage bookmarking/sharing.

The 10 questionnaire topics and what they mean

 

Does [PAGE TYPE] have a beneficial purpose?

The goal of page quality rating is to evaluate how well a page achieves its purpose and objectives. The way a page is measured on “beneficial purpose” is dependant on the page type.

Under section 2.2, What is the Purpose of a Webpage? Google writes:

For example, the purpose of a product page is to inform the user about the product, such as the features and benefits of the product. 

If a product page fails to achieve this objective then it lacks a beneficial purpose and would be considered a low quality page and in theory not perform well in search results.

Is the main content on the [PAGE TYPE] created with substantial effort, time and talent/skill?  

Google needs to understand whether content provides enough informative, unique information, to be deemed high quality.

Under section 5.1, Very High Quality MC, Google writes:

Content that is vague and lacking in detail is unlikely to perform well, especially if other sites produce more comprehensive and useful pieces of content.

We’ve excluded questions that specifically tackle the E-A-T of the content creators in this section, with the aim to focus on evaluating the usefulness of existing content.

Not every business will be, or have industry experts producing content, but that shouldn’t detract from assessing the quality of content that is actually being produced. 

Google states that smaller businesses are likely to have a smaller web presence and a lack of external evidence of E-A-T is not an indication of low page quality. 

Under section 2.6.5, What to Do When You Find No Reputation Information, Google writes:

However, for “Your Money or Your Life” (YMYL) pages, such as medical, or financial information websites, the reputation of the website and/or the individual content creator is just as important as the potential usefulness of content.

Evaluating the E-A-T of a website and/or content creator will be addressed in the topics “Would you trust information from this website’s authors?” and “Would you trust information from this website?”.

Does the [PAGE TYPE] on this site have obvious errors?

This topic is about identifying errors on a website, with a focus on recognising unmaintained and/or defaced pages.

Under section 7.2.9, Unmaintained Websites, and Hacked, Defaced, or Spammed Pages, Google writes:

Although not explicitly mentioned in Google’s Search Quality Evaluator Guidelines, other signs of unmaintained websites are broken links and images as well as obvious missing blocks of main content.

If a website has a large number of glaring errors, it could compromise user experience and suggests the website may be insecure. Failing this topic indicates a website will likely struggle to perform well in search results.

Does the [PAGE TYPE] contain excessive adverts, or pop-ups?

Over the last couple of years, Google has been increasingly targeting websites that excessively contain adverts and/or block users’ access to content with pop-ups.

Under section 7.2.7, Obstructed or Inaccessible Main Content, Google writes:

Google also suggests ads don’t need to fully obstruct a page to be distracting to receive a “low” rating. Under section 6.4, Distracting Ads/SC, Google writes:

Whilst Google does state that ads can contribute to good user experience, they should be used  in moderation and not negatively impact the experience of consuming main content.

Sites that contain excessive adverts and obstructive pop-ups run the risk of penalisation, which will impact online visibility. 

This topic is particularly pertinent for mobile devices where screen space is limited.

The questions in our checklist for this topic also reflect Google’s firm stance on excessive advertisements. Any “Failed” question in our checklist will fail this entire topic.

Is [PAGE TYPE] deceptive?

Earlier, we mentioned how misleading page titles can be a means of deception, but Google cites other forms of deception. 

For example, under section 7.6.2, Deceptive Page Design, Google states the following are forms of deception:

• Pages that disguise Ads as MC. Actual MC may be minimal or created to encourage users to click on the Ads.
• Pages that disguise Ads as website navigation links.
• Pages where the MC is not usable or visible.
• Any page designed to trick users into clicking on links, which may be Ads or other links intended to serve the needs of the website rather than to the benefit of the user.

Pages that are misleading or attempt to cause harm in some way are deemed the “lowest” quality of page and will negatively impact the perception of the organisation and hurt online visibility.

Is there adequate information about the business and/or content creators?

Google wants websites to provide adequate information about a business and/or authors.

Under section 2.5.3, Finding About Us, Contact Information, and Customer Service Information, Google writes:

When it comes to individual content creators, Google doesn’t specifically mention that authors should have biography pages on a website. Google also state the E-A-T of authors should be primarily judged on external evidence; however, there is sound logic in providing an option to view biography pages on a website that demonstrate the E-A-T of authors.    

Under section 6.1, Lacking Expertise, Authoritativeness, or Trustworthiness (E-A-T), Google writes:   

• The creator of the MC does not have adequate expertise in the topic of the MC, e.g. a tax form instruction video made by someone with no clear expertise in tax preparation.
• The website is not an authoritative source for the topic of the page, e.g. tax information on a cooking website.”

Ultimately, users want to be able to contact and learn more about an organisation/content creator. A website that provides adequate and reliable information that demonstrates E-A-T will foster trust and credibility.

Would you trust information from this website?

Google is emphasising the importance of websites and organisations having credible external reputation. 

Google has stated they will trust external sources of reputation information over internal information.

Under section 2.6, Reputation of the Website or Creator of the Main Content, Google writes:

Google wants to evaluate the external reputation information of an organisation using credible third party sources like external news articles, blog posts, or even Wikipedia pages. 

Under section 2.6.2, Sources of Reputation Information, Google writes:

By identifying various sources of external reputation information, Google can better evaluate E-A-T. 

Content that is written by websites/authors who meet a high level of E-A-T is deemed to be of the “highest quality” and in theory should perform better in search results. This is particularly important for YMYL websites/topics, such as health and safety, finance and news and current events. 

Interestingly, for the September 2019 Search Quality Evaluator Guidelines update, Google adds “Shopping” as a type of YMYL. Under section 2.3, Your Money or Your Life (YMYL) Pages, Google defines “Shopping” as:

Under section 5.2, Very Positive Reputation, Google writes:

This suggests customer reviews for an organisation’s products/services can be used as a valid source of external reputation information for ecommerce and B2B websites. 

As part of our checklist, we’ve included questions to evaluate online reviews for a brand, which could also encompass specific review information on a brand’s products/services. 

Would you trust information from this website’s authors?

Like evaluating the E-A-T of a website/organisation, Google also wants to evaluate external reputation information for individual content creators.

Under section 9.2, Reputation and E-A-T: Website or the Creators of the Main Content?, Google writes: 

Sometimes a website/organisation will be creating content and other times it will be individual content creators. Either way, evaluating the E-A-T for both types of content creators is equally important.   

Does the design of the website make it difficult to use?

This topic focuses on the functionality of the website and in turn, user experience. 

Under section 7.2.3, Lowest Quality Main Content, Google writes: 

The design of a website plays a pivotal role in functionality and since Google rightfully views bad functionality with a grade of “lowest rating”, the design of a website deserves its own topic.

A website with a good design and functionality will contribute to high quality pages and in theory convert better and potentially perform better in search results.

Would you trust this website with personal details?

This topic is the culmination of all the previous questions. If the overarching results are positive then users could be more likely to hand over personal information, such as credit card details and email addresses, which is an objective for most businesses. 

How to use the checklist

Access the Search Quality Evaluator checklist for free here.

There will be an option to select “TRUE”, “FALSE”, or “OKAY” for each question in the checklist. 

Necessary

The importance (necessity) of each question will determine whether a section will “Pass”, or “Fail”. In the screenshot above, the question “Is it clear and easy to understand what the website offers?” is a necessity to pass and because the answer is “FALSE”, the section has “FAILED”.

You can view the necessity of each question by unhiding columns C-E but these additional columns can be confusing for people who don’t understand what the sheet is doing, so I would keep them hidden most of the time.

There are also two additional columns, “Ideal answer” and “Achieves ideal answer”

Ideal answer

“Ideal answer” determines what the “ideal” statement should be for each question. 

In some cases you want the statement to be “TRUE”. In other cases, you want the statement to be “FALSE”.

For instance, we want the statement to be “TRUE” for the question “Is it clear and easy to understand what the website offers?”.

However, the “ideal” answer for the question “Is the content too short, insubstantial or unhelpfully vague?” would be “FALSE”. “FALSE” states that the content is in fact not short, insubstantial, or unhelpfully vague.

Achieves ideal answer

“Achieves ideal answer” is determined by whether the “Grade” column matches the “Ideal answer” column. 

Using the above screenshot, we know the “Ideal answer” for the question “Is it clear and easy to understand what the website offers?” is “TRUE”“. 

However, because the answer given is “FALSE”, the ideal answer has not been achieved;  therefore, “FALSE” is the statement given for “Achieves ideal answer”.

This allows us to see all “Passed” and “Failed” sections of the checklist located in the“Passed and Failed section” tab.

We’ve also added a description and/or instructions and references column in the checklist questions tab to help explain questions, provide instructions and add context. Feel free to add to, or iterate the content found in these columns. 

You’ll also find references of “[ADD BRAND]” in the “Checklist questions” tab. You’ll need to update these references to reflect the organisation the checklist is being run against.   

The tab “Website content” concatenates example URLs and example titles to help make this checklist a little more efficient to use.

The easiest way to see how all of these fields work is just to fill them out with information from your brands first, and then notice how the questions have changed as you go through the checklist.

Who should use the checklist?

As we’ve mentioned, our Search Quality Evaluator Checklist can be used by a range of people.

1. SEO professionals: filling out this checklist is a great way to evaluate the UX and website quality of a client. It will allow you to pinpoint strengths and weaknesses and help you deliver priority recommendations.
2. Internal stakeholders: the checklist can be used by internal stakeholders to go through before publishing new content, website features or most things that could impact the UX and content quality. The checklist could also be filled out by internal stakeholders as a way of getting them on board with SEO priorities.
3. The public: if you want to get a true understanding of how users perceive your website and brand, then why not ask your potential customers? 

Members of the public NEED TO fill out all sitewide questions, but you should split page type questions to avoid a horribly long survey that they’ll give lazy answers to. For instance, get a respondent to only answer questions for the homepage, blog pages, or product pages.

Our other checklists

We like using checklists here at Distilled and if you do too, get your hands on our technical SEO audit checklist, Google Analytics audit checklist and our PPC audit checklist.