questionnaire design issues

• purpose of questionnaires

  • research instrument consisting of a series of questions/statements for the purpose of gathering information from respondents
  • often designed for statistical analysis of the responses to explore hypothesis - not always the case
• questionnaires can be practical

  • large amounts of information can be collected from a large number of people in a short period of time - cheap
  • can be carried out by the researcher or by any number of people with limited affect to its validity and reliability
  • the results of the questionnaires can usually be quickly and easily quantified by either a researcher or through the use of a software package
• questionnaires can be practical

  • data can be analysed more 'scientifically' and objectively than other forms of research-debatable
  • when data has been quantified, it can be used to compare and contrast other research and may be used to measure change
  • quantitative data can be used to create new theories and test existing hypothesis
• practical uses of questionnaires

  • facts and knowledge
  • past behaviour
  • likely future behaviour
  • motives
  • opinions and attitudes
  • personality
• disadvantages of questionnaires

  • inadequate to understand some forms of information - i.e. changes of emotions, behaviour, feelings etc
  • asks only a limited amount of information without explanation
  • lacks validity
  • there is no way to tell how truthful or thoughtful a respondent is being
• disadvantages of questionnaires

  • standardised answers may frustrate users
  • respondents must be able to read the questions and respond to them
  • people may read differently into each question - i.e. what is 'good' to someone may be 'poor' to someone else = subjectivity
  • top down: meaning that when developing the questionnaire, the researcher is making their own decisions and assumptions as to what is important... therefore they may be missing something that is of importance
• stages in questionnaire design

  1. decode the information required
  2. define the target respondents
  3. choose the method(s) of reaching your target respondents
  4. decide on question content
  5. develop the question wording
  6. put questions into a meaningful order and formula
  7. check the length of the questionnaire
  8. pilot the questionnaire
• design stages

  after you pilot the questionnaire

  1. revising questionnaire
  2. administering questionnaire
  3. analysing data
  4. reporting study
• things to look out for

  • response bias - the tendency of a person to answer questions on a survey untruthfully or misleadingly. For example, they may feel pressure to give answers that are socially acceptable
  • social desirability tendency - the tendency for people to present a favourable image
  • acquiescence - tendency of a respondent to agree with a statement when in doubt:
• other things to look out for

  • good subject tendency - tendency of participants to act according to what they think the experimenter wants - wants to help but doesn't really know what the goals are
  • experimenter bias - when experimenter subtly communicates expectations about the outcome
• question types

  • open ended questions
  • fixed choice or close questions
  • leading questions - to be avoided at all costs
• open ended questions

  • cannot be answered with a simple 'yes' or 'no'
  • begin with phrases such as :
  • who? what? when? where? why? how?
• when to use open questions in a questionnaire

  • for complex or sensitive topics: allow participants to express themselves in a way that predefined response options might not cover. This can be especially important when dealing with emotions, personal experiences, or nuanced perspectives
• when to use open questions

  • when designing a questionnaire for a new topic, open questions in pilot testing can help identify potential issues, gather feedback, and refine the questionnaire before broader distribution
• when to use open questions

  • diverse perspectives: valuable for capturing diverse perspectives within a sample. participants can express themselves in their own terms, providing a more comprehensive view of the subject
• when to use open questions

  • contextual understanding: open questions are beneficial for obtaining a contextual understanding of a situation. participants can provide details that might be missed with close-ended questions
• when to use open questions

  • while open questions offer depth and flexibility, it's important to balance them with close-ended questions in a questionnaire
  • close-ended questions with predefined response options are useful for quantitative analysis and statistical comparisons. the choice between open and closed questions should align with the research objectives and the type of info you seek to gather from your participants
• advantages of open ended questions

  • they permit an unlimited number of possible answers
  • respondents can answer in detail and can qualify and clarify responses
  • unanticipated findings can be discovered
  • they permit adequate answers to complex issues
  • they permit creativity, self-expression, and richness of detail
  • they reveal a respondents logic, thinking process, and frame of reference
• disadvantages of open questions

  • different respondents give different degrees of detail in answers
  • responses may be irrelevant or buried in useless detail
  • comparisons and statistical analysis become difficult
  • coding responses is difficult - articulate and highly literate respondents have an advantage
• more disadvantages of open questions

  • questions may be too general for respondents who lose direction
  • a greater amount of respondent time, thought and effort is necessary
  • respondents can be intimidated by questions
  • answers take up a lot of space in the questionnaire
• close ended questions

  • can be answered with a simple 'yes' or 'no'
  • begin with phrases such as "how much do you agree or disagree"
• advantages of close ended questions

  • easier and quicker for respondents to answer
  • the answers of different respondents are easier to compare
  • answers are easier to code and statistically analyse
  • the response choices can clarify question meaning for respondents
  • respondents are more likely to answer about sensitive topics
• advantages to close ended questions

  • there are fewer irrelevant or confused answers to questions
  • less articulate or less literate respondents are not at a disadvantage
  • replication is easier
• disadvantages fo close ended questions

  • suggest ideas that the respondent would not otherwise have thought of
  • respondents with no opinion or no knowledge will answer anyway
  • respondents can be frustrated because their desired answer is not a choice
  • it is confusing if many response choices are offered
• disadvantages of close ended questions

  • misinterpretation of a question can go unnoticed
  • distinctions between respondent answers may be blurred
  • mistakes or marking the wrong response is possible
  • they force respondents to give simplistic responses to complex issues
  • they force people to make choices they would not make in the real world
• what type of question is this: "what is your evaluation of the efficacy of cognitive strategies in group work?

  over-complex vocabulary
• what type of question :how difficult do you expect the group work will be?

  leading questions
• what might be the varying perspectives on the necessity of a designated leader within a group dynamic?

  ambiguous questions
• should groups divide work equally and stick to deadlines?

  multiple-content questions
• there will be a practical session on improving group work. given its potential impact on behaviour, what do you think of this proposal?

  implicit questions