reliability and validity

• inter-observer reliability

  The extent to which multiple observers agree in their tallied recordings. High inter-observer reliability means that the observers recorded the same behaviours so ended up with consistent (reliable) findings.
• why is it important to assess inter-observer reliablity

  Using only one observer could result in subjective recording/analyses. This means that if another observer was to observe and record the same behaviour, they may end up with different (unreliable) results. By using multiple observers and assessing the consistency of their data, researchers can ensure the results are reliable.
• How do researchers assess inter-observer reliability?

  By comparing recordings from two observers
• What is the first step in assessing inter-observer reliability?

  • Two observers watch the same people
  • They tally their recordings independently
• What method is used to correlate the tallies from observers?

  Using a scattergram or correlation coefficient
• What does a correlation coefficient indicate in this context?

  It identifies the relationship between recordings
• What statistical action follows the correlation assessment?

  A statistical test to assess significance
• What correlation value indicates high inter-observer reliability?

  About 0.8
• What does a strong, significant correlation imply about the observers' recordings?

  It indicates high inter-observer reliability
• How can researchers improve inter-observer reliability?

  • Establish clear behavioural categories to avoid subjective interpretation when recording behaviours.

  • Train the observers with examples to ensure they understand the behavioural categories and how to tally them.

  • Film the participants so that behaviours could be reviewed as often as required so that none are missed.
• Test-retest reliability

  The extent to which the results of a study are consistent when the study is replicated.
• Why is it important to assess test-retest reliability?

  The results of a study may be affected by an extraneous variable. This means that if the study was conducted again, the researcher may gather different (unreliable) findings . By conducting the study again and assessing whether consistent results are gathered, the researcher can begin to assess whether it is likely their results were affected by an extraneous variable.
• How do researchers assess test-retest reliability?

  1. The researcher would carry out their study and then replicate it at a later time.

  2. The data from the original study and the replication study would then be correlated on a scattergram/using a correlation

  coefficient to identify whether there is a positive correlation between the two sets of data.

  3. The researcher would then use a statistical test to assess whether the correlation is significant.

  4. If there is a strong, significant correlation of about 0.8 between the two sets of data, there is said to be high test-retest

  reliability.
• How can researchers improve test-retest reliability?

  • Experiments: use control procedures such as standardisation and use the same experimenter(s).

  • Observations: Use clear behavioural categories and use the same observer(s).

  • Questionnaires: use closed questions.

  • Interviews: use structured interviews, use the same interviewer(s), train all interviewers.
• Validity refers to how well a piece of research measures what it says it measures.
• There are also different types of validity.

  • ecological validity
  • temporal validity
• Ecological validity:

  Whether the setting of the research is likely to result in participants behaving as they naturally would. This therefore involves examining whether the behaviour being investigated is being studied in a setting it would normally occur. If the participants are being observed in an artificial setting and end up producing unnatural behaviour, the findings would be criticised for lacking ecological validity.
• Temporal validity:

  Whether the findings of previous research still accurately reflect what researchers would find in today’s society.

  This therefore involves examining whether changes in society since the research was conducted could result in different findings if

  it was replicated. If the findings are found to no longer apply today, they would be criticised for lacking temporal validity.
• Face validity:

  A way of assessing validity that checks whether the behaviour being investigated appears, at first sight, to represent what is being

  measured. It therefore involves the researcher asking other researchers if their dependent variable/behavioural categories are a good

  measure of the general behaviour in focus.
• Concurrent validity:

  A way of assessing the validity of a new test (e.g. a depression questionnaire) by comparing participants’ results on it to the same

  participants’ results on a pre-existing test that has already been found to be valid measure of the same variable (e.g. depression).
• What is the first step in assessing concurrent validity of a new test?

   The researcher instructs participants to complete both the new test and an already validated test that measures the same construct.
• How is the data from the new and validated tests analyzed?

  The data are plotted on a scattergram and correlated using a correlation coefficient to identify a positive relationship
• What does the researcher do after calculating the correlation?

  They use a statistical test to assess whether the correlation is significant.
• What indicates high concurrent validity?

  A strong, significant correlation (around 0.8) between the new test and the validated test, showing the new test produces similar results.
• What does high concurrent validity imply about the new test?

  It suggests the new test is valid because it produces results consistent with the already validated test.