Estimation from Zero-Truncated and Top-Coded Counts

Description

Fit the truncated or censored form of a count distribution when units with zero events are never recorded or counts above a cap are lumped together.

Many count datasets are shaped by how they were captured. Recording often starts only when an event happens, so units with zero events are missing: users appear in an error log only after their first error, customers in a support system only after their first ticket. Fitting an ordinary Poisson to such data overstates the rate. The zero-truncated (positive) Poisson conditions on at least one event; its maximum likelihood equation is solved iteratively, with quick starting values from Plackett's estimator. If the size of the full population matters, the number of unseen zeros can be estimated too. When high counts are recorded only as more than a cap, a censored likelihood applies if the number of capped units is known, a right-truncated one if they are dropped. Parallel results exist for the negative binomial and binomial. Truncation always costs precision.

Attributes

Loss type
Latent
Stage
Post-launch
Effort
Hours

Steps

  1. Establish how the data were captured: are zero counts unrecorded, are high counts capped, and is the number of capped units known?
  2. Choose the matching likelihood: zero-truncated, left-truncated at r1, right-truncated, censored at the cap, or doubly truncated.
  3. Check the count shape (an Ord plot still works under truncation) and choose Poisson, negative binomial or binomial.
  4. Get a starting value from a simple estimator, such as Plackett's for the positive Poisson.
  5. Solve the truncated or censored maximum likelihood equation iteratively and obtain its standard error.
  6. If the total population size matters, estimate the number of unseen zero units, and recommend capturing zeros and full counts in future logging.

Sources

  • Univariate Discrete Distributions (Johnson et al., 1992)

Also known as

  • Zero-truncated Poisson
  • Positive Poisson distribution
  • Conditional Poisson (Cohen)
  • Truncated negative binomial
  • Right-truncated Poisson
  • Censored (top-coded) Poisson
  • Estimation from zero-truncated, right-truncated and top-coded counts

Categories

  • Analytics & Behaviour
  • Statistics & Modelling