Description
Training or configuring entity recognition to detect UX-relevant entities, such as product features, competitors, devices, workflows and emotions, in transcripts and feedback.
Generic NER detects people, organisations and places, which is useful mainly for anonymisation. UX research needs domain entities: feature names, screens, competitor products, job roles, workarounds, devices and emotions. spaCy supports rule-based patterns (EntityRuler) and trainable statistical NER on annotated examples. A hybrid approach typically works best: rules for known vocabulary, trained models for variants and new mentions. Rigour comes from a clear annotation guideline, double annotation with agreement measurement, held-out evaluation with precision, recall and F1, and error analysis. Generic NER is also used to detect personal names and places for anonymisation, with high recall prioritised.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Unmigrated
- Goal
- Synthesise data
- Stage
- Definition, Post-launch
- Data
- Qualitative
- Effort
- Weeks
Steps
- Define an entity schema of 5-12 types with written definitions and examples.
- Build an EntityRuler with known terms (feature names from the product, competitor list).
- Annotate 300-1,000 sentences with two annotators, measuring agreement (e.g. F1 between annotators of 0.80 or above) and refining guidelines.
- Train or fine-tune the NER component on the annotated data, holding out 20% for evaluation.
- Evaluate per entity type and inspect errors, especially missed variants and false positives.
- Deploy with the rules first and model second, and re-evaluate when the product vocabulary changes.
Sources
- spaCy (en_core_web_trf transformer model)
- spacy-transformers
- textacy
- displacy visualisation
- Grishman, R. & Sundheim, B. (1996) Message Understanding Conference-6: A brief history. COLING
- Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. draft
- The Inquiry Series, C10 — spaCy — Building NLP Pipelines for Qualitative Research
Also known as
- Domain NER
- Custom entity extraction
Categories
- Analytics & Behaviour
- Computational & Machine Learning