What to consider in user experiments

Basic knowledge you must know before doing any user experiments:

  1. What are IVs and DVs
  2. What are within- and between-subjects design and pros and cons
  3. What are the meanings of factors, levels, blocks, and trials, also learn the synonyms (e.g., IV = factors, levels = conditions, metrics = DV)
  4. Internal validity, external validity, construct validity
  5. Clear idea why you choose certain research methodologies (don't make up your own method)
  6. Clear idea why you choose to do the experiment in this way (why give users feedback versus not)
  7. Why we need to order conditions, and how many ways to order conditions: fully counterbalanced, partially counterbalanced, fully randomized;  What can be randomized and what cannot
  8. How to decide sample size using power analysis
  9. Balancing quantitative and qualitative data; what are the pros and cons of each data, and it is usually best to use mixed methods

When you design your experiment, keep in mind these common mistakes:

  1. Conduct pilot studies: ALWAYS conduct several pilot studies before the actual study
  2. Avoid making your own new methodologies without good reason; always learn what are the standard methodologies and tasks
  3. Be clear yourself why you choose a methodology over others, and what are the pros and cons
  4. Does not have clear hypothesis.  Carrying out a experiment without understanding what to expect will 99% gives you unfruitful results
  5. Make sure your participants are not too tired.  Manage your experimental time wisely.  1 hour is usually the maximum amount of time per participant.
  6. Make sure you log everything (logger, webcam, notes-taking).  The purpose of logging is so that we can replicate the whole experiment and see what happens. 
  7. Make sure you have enough participants.  Learn power analysis.
  8. Make sure your experiment pass the three validity test:
    1. Internal validity:  whether what you want to find out is well-isolated from other noise effects
      1. When you are comparing two conditions, did you make sure everything else is equal except what you are manipulating?
      2. Are you sure that the variables you are manipulating have really the desired effect that you want?  Run a usability test before an actual test can help.
      3. Did you correctly order the experimental conditions?  Order effects should be always minimized.
      4. Did you assign users to different groups in a randomized way?  If not, why?
      5. Did you take care of learning effects by applying appropriate training before the experiment or applying block design?
    2. External validity: is your result generalize across people and contexts
      1. Did you choose the participants that are representative of the world?
      2. Did you choose the experimental task that is representative of the world?
      3. Is your system or experimental tools representative of the world?
      4. At last, did I over claim my results in the paper?
      5. Did you discuss any design or real-world implications when the results are not significant?
    3. Construct validity: whether you are measuring things based on what you claim
      1. Measuring happiness but uses only interview or user preference  or measuring text-entry performance with only speed but not errors
      2. Measuring typing performance but ignore that people can type while they are walking or sitting or standing
      3. Try to use mixed methods - combining quantitative and qualitative
      4. Talking about habit formation but collect data using only five days of experiment

Checklist for apparatus:

  1. Recruit participants
    • Make sure they are either paid or are provided with some gift cards (it's important to provide enough motivation to conduct the experiment)
    • Make sure you do not recruit people that have some potential biases such as your close friend ( https://www.nngroup.com/articles/employees-user-test/)
  2. Informed consent
    • Before doing any experiment, it is ethical and it is the first thing in an experiment to let participants know about the experiment
  3. Inform participants before the experiment - for physiological experiments, please inform the participants prior to the experiment, what they should do, e.g., do not drink coffee, do not shower one hour before the experiment, etc.
  4. Demographic sheets
    • In the Participants section, you need to write who do you recruit, their age, their gender, any characteristics that may affect the study results (e.g., experience) and the criteria of inclusion and exclusion. 
  5. Your software or tools - make sure your software logs everything, key point is that you should be able to reconstruct the whole experiment and understand what exactly happens during the experiment.  For example, time log or any measurement tool should be in milliseconds, not seconds.
  6. A checklist of your procedure - make sure you follow the procedure.  This is to avoid human errors which are often the case when there are many tiny steps.
  7. Webcam, video camera, screen capture.  Very important to capture temporal behaviors of users.
  8. All questionnaires - use only standard questionnaires.  Please do not come up with your own questionnaire.  All standard questionnaires have been scientifically proven while your own is likely to be faulty.
  9. Make sure the setup is clean and quiet, and same across all conditions.  Don't overlook this.
     

---

Below contains some useful links where you can learn from.

Learn from Jakob Wobbrock - https://www.coursera.org/learn/designexperiments

Learn from Mackenzie - http://www.yorku.ca/mack/CourseNotes.pdf

Learn from Shengdon Zhao - http://www.slideshare.net/sszhao/the-5step-approach-to-controlled-experiment-design-for-human-computer-interaction

An excellent reference about research methods by Anna Cox and Paul Cairns - http://www.amazon.com/Research-Methods-Human-Computer-Interaction-Cairns/dp/0521690315/

For field experiment, check out this handbook - https://www.povertyactionlab.org/handbook-field-experiments