A Lead Score Can Rank the Queue and Still Misstate the Odds

A score may be a relative rank, a rule total, or an estimated probability. Those are different products. If sales reads 80 as an 80 percent chance of conversion, the model needs calibration evidence; a high AUC alone only says positive examples tend to rank above negative ones.

Scikit-learn describes a well-calibrated binary model as one where cases assigned near 0.8 become positive about 80 percent of the time. A reliability diagram compares predicted and observed rates by score band. It should be paired with ranking, volume, and commercial outcomes rather than treated as the only test.

A lead created yesterday cannot yet be labeled lost when the sales cycle takes ninety days. Build training and evaluation cohorts with complete outcome windows. Separate “not converted yet,” “not contacted,” and “confirmed lost” rather than collapsing all three into the negative class.

Historical labels also reflect earlier routing policy. If only high-scored leads received persistent follow-up, their better outcomes partly describe treatment, not just inherent quality. Capture contact attempts and service levels so the model is not rewarded for the attention its predecessor already allocated.

Use a time-based holdout that follows the training period. Monitor changes in source mix, missing fields, score distribution, conversion delay, observed outcome by band, and performance for regions, company sizes, products, and other legitimate operational segments.

A global metric can hide a collapsed segment. It can also hide scarcity: a score band with five examples is not stable evidence. Publish sample sizes and intervals, and delay fine-grained conclusions until the cohort matures.

Choose routing thresholds from available sales capacity, expected value, response-time commitments, and the cost of missing a suitable buyer. A fixed score of 70 has no universal operational meaning. Recalculate when capacity, offer, or market changes.

Keep reason codes, manual override, and an audit trail. Review false negatives and repeated overrides with sales, but do not automatically convert every opinion into a feature. The model should be a revisable prioritization aid, not an invisible veto on access to a human response.