To test out Google's new Consumer Surveys service, we decided to create a quick survey to gain insight on data that we felt we knew pretty well, to put the validity of the response data to the test, as well as test the features of the service. Learn more about Google Consumer Surveys and how they might be used in our first post, "Google Consumer Surveys, a Game Changer?".
Since web browsers are something we always have to consider as we craft websites, we thought it would be interesting to see what consumers say is their favourite web browser. There is also a large body of data out there about what browsers people actually use, and we wondered if the usage data and what consumers say is their favourite browser would match up.
Set Up
We discussed throughout the office the one question we should ask, how we should ask it and what wording we would use before we set out to set up the survey. Wording is very important in surveys; you want to make sure that the question is not "leading", which may place a bias on one response over another, and you also want to make sure that the question you ask is the right question to get you the answer you are looking for.
With this in mind, we simply asked, "What is your favorite web browser?" (you'll note we used the US spelling of "favorite", since the demographic was in the US and not Canada or the UK). We included the major web browsers in use based on Clicky Web Analytics data, and included an "other" response as well, just in case we had missed something.
Once we had decided on what we were going to ask and how, we proceeded to actually creating the survey, which is a very quick process. In four steps, a survey is ready to be sent out and start providing you with insights.
1. Create Survey
Choose a title, brief description and tags for your survey. All of this information is only to help you keep your surveys organized, they are not shown to respondents.
2. Pick Your Audience
Here you have three options:
- An audience representing the US internet population. This option costs $0.10 per response and includes Men and Women aged 18+, in all US regions. This demographic includes consumers in urban, suburban, and rural setting with annual salaries from $0 to $150K+.
- An audience based on gender, age, or geography. This option costs $0.50 per response. Here you can select a combination of Men or Women within 3 distinct age groups; 18-24, 25-34, or 35+. You can also choose where in the US respondents are from such as; US (all of the US), US Midwest, US Northeast, US South, US West. Once your survey has been completed you may further segment your response data by audience income, age group, or geography (at the region or state level).
- A custom audience using a screening question. This option also costs $0.50 per response. You can choose to ask a multiple choice question (optionally, including an image) and use only those respondents who chose one of your flagged responses to continue with the survey.
We decided to survey the audience representing the US internet population, both to keep costs reasonable and to get an idea of a general sample's favourite browser.
3. Write Your Questions
You have the option of multiple question types:
- Multiple choice.
- Multiple choice with an image.
- Multiple selection. This option differs from multiple choice in that respondents can select multiple options as responses.
- Image choice. Respondents are offered their choice of two different images and may only select one. You provide your own images for respondents to choose from.
- Rating. A five star rating system where you enter the question text, and the high and low ends of the scale, such as 1 star = not satisfied and 5 stars = very satisfied.
- Rating with image. Similar to the above, but incorporates an image along with the rating scale, in addition to the question text.
- Rating with text. Similar to the Rating option, but incorporates a line of text above the scale, in addition to the question text.
One thing that wasn't immediately clear to us in this step is that the number of questions you choose to ask in your survey determines how many responses you will need to make the results statistically significant, and as a result, the minimum amount you will need to spend for those responses. For one question without a custom demographic you will need to spend a minimum of $200 per question ($0.10 per response, minimum of 1,000 responses).
For our survey we decided to ask only one multiple choice question, "What is your favorite web browser?".
4. Confirm Your Survey
At this stage you confirm the questions you have entered as well as your survey's title, description, and tags, and choose how many responses you receive per question. There are options to receive between 1,000 and 10,000 responses/question, with 1,500 responses/question being recommended. The reason for that recommendation is that this is the general amount of responses you would need to receive to get statistically relevant results with one or two filters applied to the response data.
Once you have chosen how many responses you'd like to receive, click the Buy Now button, pay Google, and your survey will be ready for mass consumption.
Responses
We received data for our survey very quickly, with hundreds of responses received within the first few hours of our purchase. Within 4 days, our 1,000 responses were received and the data began to be analyzed by Google's system.
In the reporting area of the management page, Google provides a Response Metrics section. Basic response metrics include how long respondents took to respond, the responses by respondent's local time, responses by the day of the week, how many responses were received, how many times the survey was shown, and the resultant response rate.
Results
Once we had received all of the responses we had paid for, the results of our survey emerged. Internet Explorer was chosen as the favourite web browser, with 30.1% of the respondents choosing this as their favourite. The second favourite was Google's Chrome browser, with 22.1% of the respondents choosing this as their favourite. Firefox was a close third with 22.0%. The results match very closely with Clicky's web browser market share data, which we didn't necessarily expect. We expected to some degree that market share data may have been skewed due to employees being stuck using a particular browser at work (Internet Explorer, in many cases), but preferring to use a different browser at home.
The next step was data mining. Advanced segmentation tools are provided, which allowed us to see, for example, if Males aged 18-24 responded differently than Females aged 18-24, or if different age groups, levels of income, or geographies played a role in how consumers responded. There is even an "Insights" tab which shows if there are demographic metrics which are strongly correlated. Our insights stated that 55-64 year-olds picked Internet Explorer twice as often as those aged 25-34, and 55-64 year-olds picked Internet Explorer twice as often as those aged 18-24.
With those insights, there is still a lot of room for interpretation as to why, and how this affects the question we asked. Do 55-64 year-olds not realize that there are other browser options? Do they prefer Internet Explorer because it is what they have always used? These insights open the opportunity to ask more questions. In order to ask these questions we could create a new survey with a custom audience using the screening question, "Is Internet Explorer your favorite web browser?", ask a question with multiple options as to why it is their favourite (simplicity, familiarity, design, came installed on system, didn't know there were others, etc.) and segment the results for those aged 55-64. In this case, we would likely need to purchase more than 1,000 responses to be able to get a relevant amount of respondents in that age range, however.
Statistical Significance and Bias
In the main response page we are also provided with a "Confidence" metric and a table showing the sampling bias of respondents.
The confidence metric shows you whether the results are statistically significant or not. In our case we received the message, "Winner statistically significant", which means that if we were to run the same survey again the results would show Internet Explorer as the favourite browser 95% of the time. If there was less than 95% confidence in the results we would see the message, "Too close to call".
These confidence messages are updated as you segment the results as well. For example, in our survey there is over 95% confidence that, if the survey were run again, women would choose Internet Explorer as their favourite browser, while men may not. This is helpful information to have available at a glance, especially when you are running a survey of which the results may be important to key business decisions.
The sampling bias table found below the results shows the percentages of respondents in different segments (such as sex, age group, and region) and compares the distribution to the US Census Current Population survey, to see how close to the US census data the survey's results are. In our case 66% or our respondents were male, whereas the current US census shows 48.9% of the US population as male. This gave our survey a male gender bias of about 17.1%. The table provides the bias values for each segmentation and at the end provides a RMSE score, the Root Mean Square Error. The lower this number, the less biased the survey is.
Conclusion
The only anomaly we saw in the survey results is that "Other", the response we included "just in case", received 19.0% of all responses, which is a large percentage. It could be that respondents chose this because their preferred browser was on their iPad, iPhone, or another tablet or mobile phone and they weren't sure how to respond. If this option was removed, the results may have been different. We could have also asked what their favourite desktop browser was on their home PC, which may have provided different results and eliminated the option of mobile/tablet browsers or what browsers they use at work.
At the end of the day, however, the results were about what we expected to find. For $200 were able to review the utility of a brand new service, write two blog posts, share this new service with you, and hopefully make your next small market research project a lot cheaper and faster to realize.
Check It Out For Free
One of the great features of the surveys is that you can choose to keep the data private, or share it with others. We made ours public so you could jump in, play with the tools, and analyze the results for yourself. Click this link to see how you can segment and filter the data (Google account not required).