Survey linked to consented account (log) data

Description

Ask attitudes and motives in a survey and, with each respondent's permission, take their actual behaviour from their account data instead of asking them to recall it.

Recall questions about usage (how many tags, how much spent, how often) produce guesses and recall bias; facts a computer can record should be taken from the record. This design ends the survey by asking respondents to authorise researcher access to their account, then joins logged behaviour to their answers, so attitudes can be related to real behaviour. Nov et al. invited 1,373 Flickr users, obtained 237 valid responses and took tag counts (mean 370) from accounts. A variant samples each respondent's own content, stratified by a platform attribute, and shows it inside the survey for item-level questions: Kairam et al. selected 20 photos per person across five privacy settings and showed 10, alongside aggregate data from 638,930 active users. Inclusion rules can bias the sample, and informed consent for account access must be explicit and specific.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Define inclusion criteria from platform activity and check them for hidden biases (for example language or activity thresholds).
  2. Randomly sample eligible users and send invitations.
  3. Administer the attitude and motivation survey without asking for behaviours the logs can supply.
  4. Ask respondents to authorise access to specified account data, explaining exactly what will be extracted and why.
  5. Extract behavioural measures for consenting respondents and join them to survey answers under a data-protection plan.
  6. Variant: select each respondent's own items stratified by an attribute, show a random subset in the survey and ask item-specific questions.
  7. Compare consenters with non-consenters on available aggregate data to estimate consent bias.

Sources

  • Research Methods in HCI (Lazar et al., 2017)

Also known as

  • Survey plus authorised account data
  • Survey with data access permission
  • Survey-log linkage

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics