Replies to This Discussion

Permalink Reply by Richard S. on February 15, 2012 at 4:27pm

I've done them, but I'm not an expert at it, and really, I'm not a fan either. 

If you know what you are trying to learn and have faith that respondents will be representative of your audience (those are two big ifs by btw), you can design a survey to get you some useful, though imperfect information.  If you are looking for general,  anything goes sort of feedback, a survey might be a challenging vehicle. 

Can you say more about what you are trying to learn from the survey? 

On the question of preferred output... you'll get answers all over the map. Some will ask for the usual suspects like PDF and HTML, but you'll get plenty of seemingly fringe requests too. If you're sample isn't representative one, this sort of question can lead you astray. So instead, try to get sense of what your capabilities are, then craft questions around the suitability or utility of what you can do rather than open ended questions about what people like. Also, it's important as you go into this that you have some sense that the data you collect might influence a decision. If it doesn't, there's no point in collecting it. In other words, if you're going to do HTML and PDF no matter what, a survey question about formats is probably a waste of time for all involved. 

On the question of layout (if you are talking about styling and presentation), you might be better served by A|B testing a few different alternatives.  

• ▶ Reply

Permalink Reply by Kai Weber on February 16, 2012 at 3:36am

Hi, Tiffany, it's a smart move to get to know your audience before you decide how to proceed. That said, you can get all kinds of answers from your customers, but depending on your questions, some answers will be more helpful than others.

I subscribe to Jakob Nielsen's position about usability: "Pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior." See http://www.useit.com/alertbox/20010805.html

So here are some questions that have helped me to understand who my customers are and how they use the documentation. I prefer to give users an odd number of choices, so I can slice up the results (instead of having to deal with freewheeling answers), and they have a chance to select the option in the middle if they want to.

1. What type of customer are you?
   1. Which version of <products> do you use most (please select only one)?
   2. For how long have you worked in the industry?
   3. For how long have you worked with <products like ours>?
   4. For how long have you worked with <our product>?
   5. Which of the following best describes your current job function?
2. How do you find information?
   1. When learning about a new feature or area in <product>, which of the following resources do you consult? <include available documentation formats, but also customer support and "asking a colleague"; if possible, ask to rank resources for frequency>
   2. When you have a problem with <our product>, which of the following resources do you consult? <same options as above>
   3. How often do you use the following sources of information? <same as above; for each, let users select Daily, Few times a week, Few times a month, Few times a year, Never>
   4. How often do you find the information that you need when using the following sources of information? <same as above; for each, let users select Frequently, Usually, Sometimes, Rarely, Never>

You can also consider to ask users to rate whether they Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree with statements, such as

• I read user manuals from end-to-end
• I look up specific information when I need answers
• When I have a problem or question about <product> functionality...
• • ... I contact a colleague
  • ... I contact customer support
  • ... I open a printed version of User Manual and search for information
  • ... I open the electronic (PDF) version of the User Manual and search for information
  • ... I open the online help and search for information
  • ... I consult the FAQ database.

Hope this helps.

• ▶ Reply

Permalink Reply by Tiffany Shoemaker on February 16, 2012 at 8:22am

Thanks for the replies! and Kai - you gave me a different approach to this survey and it was very helpful! Thank you so much!!!!!

• ▶ Reply

Permalink Reply by Kai Weber on February 16, 2012 at 9:41am

You're welcome, Tiffany. I just remembered that I blogged about this some time ago, maybe you'll find these two posts helpful as well:

• http://kaiweber.wordpress.com/2010/05/10/asking-your-users-part-1/
• http://kaiweber.wordpress.com/2010/05/17/asking-your-users-part-2/

• ▶ Reply

Permalink Reply by Mark Baker on February 16, 2012 at 10:40pm

Tiffany,

I would express the same kinds of reservations as Richard, and I would add this: by and large users don't actually care about the layout or format of the documentation. What they care about are:

• Can they find the information they want when they want it
• Do they trust the information (not a given!)
• Does the information solve their problem

A story to illustrate the point. At one client, we found that both internal users and customers were asking for PDF files even though the HTML version of the documentation was better in a number of ways. We investigated and found out that the reason they were asking for PDFs was so that they could search them. The HTML pages were fine, but the way the documentation was organized and delivered in the help system (it was an Eclipse based product, and the search engine in Eclipse help is not good) they could not search them effectively.

So, when we asked them what format they wanted, they said PDF, but what they were really saying was they wanted something they could search. Once we fixed the search issue, they were able to use the HTML effectively.

Rather than asking about format or layout, therefore, I would suggest asking what they want most from the docs. If one format or layout provides the features they want better than another, then you have your answer. If not, then the questions doesn't matter.

• ▶ Reply