Crafte Sessions analysis 2026 09 29 2026 10 29
sessions_analysis-2026-09-29-2026-10-29.csv · 2,246 rows · 15 columns · definition v3 saved 2026-10-29 14:08
Comparison v7 ordered · Random forest · built 2026-10-30 09:18 on segments v27
Answer
The model on the target segments trained on 30% of the snapshots at a 11.3% outcome rate instead of 4.0%, with no resampling. It did about as well as the model trained on everything (PR-AUC 0.20 vs 0.24). Across 200 resampled test sets it came out ahead 16% of the time, so the gap could be chance.Few test cases
Only 12 test snapshots have this outcome, so small differences between the approaches can be chance. The comparison below says when a difference holds up.Small top slice
The top 1.0% is only 4 snapshots, fewer than the 12 test cases, so it can catch 0 even with a good ranking. See 'How many each approach finds'.Small hold-out
20% held out for testing with under 200 cases leaves 12 to score on. Hold out 30% for a steadier answer.Label on every row
This outcome is set on every row of an entity from its first snapshot (90% of positive entities), so it's a label rather than an event with a 'before'. All snapshots are used, each labelled with the entity's eventual outcome.Outcome rate in training
4.0% → 11.3%
everyone → target segments
Of the outcome in the targets
82%
61 of 74 training cases
Training snapshots
542 vs 1,828
target segments vs everything
Test entities
403
Later customers (first seen after 2026-10-25)
How the approaches compare
Test: 418 snapshots, 12 with the outcome (2.9%)PR-AUC: target model 0.205 and everything model 0.242, against 0.029 for chance (7× and 8× better than picking at random).
- PR-AUC: target model 0.205 and everything model 0.242, against 0.029 for chance (7× and 8× better than picking at random).
- The rows are four ways of ranking the same 418 test snapshots (people the models never saw) from most to least likely to have the outcome. 12 of them did (2.9%).
- Chance: no information. Picking 4 at random finds about 0.1 with the outcome. Every other row has to beat this.
- Segments only: everyone gets their segment's rate from training, nothing else. Its PR-AUC of 0.157 against 0.029 for chance is what knowing the segment is worth on its own. Everyone in a segment scores the same, so its top 4 is a random pick from the best segment.
- Model on target segments: a model trained only inside the segments you ticked, where the outcome is 11.3% instead of 4.0%. It scores those segments' snapshots individually and ranks them above everyone else; snapshots outside the targets are only ranked by their segment's rate, below all of them.
- Model on everything: the same kind of model trained on all the training snapshots. It sees every buyer, but at the lower rate.
- PR-AUC scores the whole ranking (chance = 0.029, perfect = 1). Top 4 flagged is what you'd get acting only on the most likely few: caught, precision (how many of those 4 were right) and recall (how many of all 12 were found).
- Here: the target model scores 0.205 overall against 0.242 for the everything model, and catches 0 against 0 in the top 4. With only 12 cases in the test set, one or two either way change these, so check the headline for whether the gap holds up.
| Approach | Trained on | Test scores | Top 1.0% flagged (4 snapshots) | |||||
|---|---|---|---|---|---|---|---|---|
| Snapshots | With outcome | Rate | PR-AUC | ROC-AUC | Caught | Precision | Recall | |
| Chance (pick at random) | no training | 0.029 | 0.500 | ~0.1 of 12 | 2.9% | 1% | ||
| Segments only (no model) | rates only | 74 | 4.0% | 0.157 | 0.828 | 1 of 12 | 25% | 8% |
| Model on target segments | 542 | 61 | 11.3% | 0.205 | 0.900 | 0 of 12 | 0% | 0% |
| Model on everything | 1,828 | 74 | 4.0% | 0.242 | 0.933 | 0 of 12 | 0% | 0% |
PR-AUC is the fairer score for rare outcomes: chance scores the outcome rate (2.9%), 1 is perfect.
Precision is the share of flagged snapshots that really had the outcome; recall is the share of all outcome snapshots that were flagged.
How many each approach finds
Of the 12 test cases, how many are in each top sliceModel on everything finds 11 of the 12 cases in its top 20% (84 snapshots); picking at random would find about 2.4.
- Each cell is how many of the 12 test cases are in that approach's top slice. The ranking is what a model really gives you; the cut-off is your choice of how many people to act on.
- Model on everything finds half the cases in its top 5% (21 snapshots), where picking at random would find about 0.6.
- It finds 11 of 12 (92%) in its top 20%: act on 84 snapshots instead of all 418.
- Top 12 (as many as there are cases): if you flagged exactly as many as really had the outcome, how many would be right. A perfect ranking gets all 12.
- With 12 cases each one moves these counts a lot: read them as rough.
| Approach | Top 1% 4 | Top 2% 8 | Top 5% 21 | Top 10% 42 | Top 20% 84 | Top 30% 125 | Top 12 = cases |
|---|---|---|---|---|---|---|---|
| Chance | 0.1 | 0.2 | 0.6 | 1.2 | 2.4 | 3.6 | 0.3 |
| Segments only (no model) | 1 | 1 | 1 | 3 | 6 | 11 | 1 |
| Model on target segments | 0 | 1 | 6 | 8 | 10 | 10 | 2 |
| Model on everything | 0 | 1 | 6 | 9 | 11 | 11 | 3 |
Precision and recall
When each has found half the 12 cases, the target model is 35% precise and the everything model is 32% precise; picking at random is 2.9% precise at any point.
- Recall (across): of all the real cases, the share found so far as you go down the ranking.
- Precision (up): of the snapshots flagged so far, the share that really had the outcome.
- Each line walks down one approach's ranking. Flag more people and recall rises while precision usually falls; a better ranking stays higher for longer. Up and to the right is better.
- The dotted line is chance (2.9%): what precision you'd get flagging people at random.
- The area under each line is its PR-AUC, the single score in the table above.
Segments only (no model)Model on target segmentsModel on everything
Up and to the right is better. The dotted line is chance (2.9%).
What drives each model
Learned from the training snapshotsThe everything model's first cut is
minutes > 2: 26.4% have the outcome on that side against 1.2% on the other.- Importance: how much each input helps the trees separate outcome from no outcome. It shows how much, not which way.
- First splits: where the trees most often cut first, with the outcome rate on each side on the training snapshots. That gives the direction: e.g. more than 3 pages → 12% vs 1%.
- "segment … probability" and "fit to segments" are the segment inputs added to the models; everything else is your own inputs.
Model on target segments
Importance
minutes0.255
fit to segments0.178
pages0.128
tenure_days0.076
segment S3 probability0.070
segment confidence0.069
segment S2 probability0.067
rows0.027
landingPage=/0.026
landingPage=other0.022
First splits
| Split | Outcome rate if yes | if no |
|---|---|---|
minutes > 2vs ≤ 2 · used first in 46% of trees | 26.6% 188 snapshots | 3.1% 354 snapshots |
pages > 2vs ≤ 2 · used first in 34% of trees | 21.3% 192 snapshots | 5.7% 350 snapshots |
fit to segments > 1.90vs ≤ 1.90 · used first in 30% of trees | 43.8% 16 snapshots | 10.3% 526 snapshots |
segment S3 probability > 0.00vs ≤ 0.00 · used first in 24% of trees | 8.3% 395 snapshots | 19.1% 147 snapshots |
tenure_days > 11.06vs ≤ 11.06 · used first in 23% of trees | 26.5% 34 snapshots | 10.2% 508 snapshots |
segment confidence > 1.00vs ≤ 1.00 · used first in 18% of trees | 13.1% 442 snapshots | 3.0% 100 snapshots |
segment S2 probability > 1.00vs ≤ 1.00 · used first in 18% of trees | 23.8% 42 snapshots | 10.2% 500 snapshots |
source=Direct = 1vs = 0 · used first in 14% of trees | 21.1% 57 snapshots | 10.1% 485 snapshots |
Overall training rate: 11.3%.
Model on everything
Importance
minutes0.244
fit to segments0.172
pages0.137
tenure_days0.075
segment S2 probability0.057
segment S3 probability0.056
segment confidence0.044
rows0.032
landingPage=/0.029
segment S5 probability0.027
First splits
| Split | Outcome rate if yes | if no |
|---|---|---|
minutes > 2vs ≤ 2 · used first in 44% of trees | 26.4% 208 snapshots | 1.2% 1,620 snapshots |
fit to segments > -4.76vs ≤ -4.76 · used first in 44% of trees | 2.2% 1,623 snapshots | 18.5% 205 snapshots |
pages > 2vs ≤ 2 · used first in 34% of trees | 20.9% 220 snapshots | 1.7% 1,608 snapshots |
tenure_days > 11.06vs ≤ 11.06 · used first in 23% of trees | 27.0% 37 snapshots | 3.6% 1,791 snapshots |
rows > 2vs ≤ 2 · used first in 16% of trees | 19.4% 67 snapshots | 3.5% 1,761 snapshots |
segment S2 probability > 1.00vs ≤ 1.00 · used first in 16% of trees | 22.2% 54 snapshots | 3.5% 1,774 snapshots |
segment S1 probability > 0.26vs ≤ 0.26 · used first in 16% of trees | 0.4% 1,189 snapshots | 10.8% 639 snapshots |
segment S5 probability > 1.00vs ≤ 1.00 · used first in 14% of trees | 21.5% 65 snapshots | 3.4% 1,763 snapshots |
Overall training rate: 4.0%.
What the target-segment model uses
Measured on test snapshotspages0.075
minutes0.042
fit to segments0.040
rows0.004
segment S8 probability0.002
tenure_days0.002
How much the PR-AUC drops when each input is shuffled, measured on the test snapshots in the target segments.
What else describes the target segments
Columns left out of the segmentation, inside the targets vs everyoneEvery column feeds the segmentation, so there's nothing extra to describe with. Turn "Segments" off for attributes such as country or traffic source on the Columns tab to see them here.
Comparison versions
| Version | Built | Outcome | Model | Targets | Rate: all → targets | PR-AUC targets | PR-AUC everything |
|---|---|---|---|---|---|---|---|
| v7 | 2026-10-30 09:18 | ordered | Random forest | 3 | 4.0% → 11.3% | 0.205 | 0.242 |
| v6 | 2026-10-29 17:18 | ordered | Random forest | 3 | 4.0% → 10.9% | 0.349 | 0.377 |
| v5 | 2026-10-29 14:05 | ordered | Logistic regression | 3 | 3.9% → 12.6% | 0.162 | 0.210 |
| v4 | 2026-10-29 14:04 | ordered | Random forest | 3 | 4.3% → 13.8% | 0.118 | 0.300 |
| v3 | 2026-10-29 13:45 | ordered | Logistic regression | 1 | 4.3% → 10.0% | 0.597 | 0.674 |
| v2 | 2026-10-29 13:38 | ordered | Logistic regression | 1 | 4.3% → 10.0% | 0.597 | 0.674 |
| v1 | 2026-10-29 12:54 | ordered | Random forest | 3 | 4.3% → 14.1% | 0.491 | 0.734 |