Data & Elo methodology
Two retrospective studies, two different evidence questions
The Elo study asks whether map-win probabilities are calibrated on a chronological test period. The veto study asks whether greedy Elo regret is associated with observed series and picked-map outcomes. They use separate artifacts and neither establishes a causal effect.
Two evaluations, two questions
Probability calibration and veto outcome association are complementary, not interchangeable. A model-derived veto score cannot validate the probability model that produced it.
Probability calibration
Do forecast probabilities match observed map outcomes?
The chronological map replay reports Brier score, log loss, ECE, reliability, confidence bands, and a plain-Elo baseline. It controls candidate selection within each run.
Review calibration evidenceVeto outcome association
Does lower greedy Elo regret accompany better observed outcomes?
The match replay compares observational series and guaranteed-played BO3 pick outcomes. Bans and unchosen sequences remain counterfactual, so this is not evidence of coaching quality or causal win lift.
Review veto evidenceSource coverage
3,818 maps
Chronological test
835 maps / 319 series
Data through
Jun 21, 2026
Evaluated model
map-elo-v1.0.0
Deployed model: map-elo-v1.0.0. Evaluated artifact: map-elo-v1.0.0. The versions match.
This is a retrospective chronological replay, not a prospectively registered confirmatory study. The evaluated production configuration is the version deployed at analysis time. Repository history records its rating-scale change on Oct 4, 2025 at commit 99a0a87, after this test period began.
A second production comparator is reconstructed exactly from repository commit 2f134f1, dated Aug 26, 2025 and therefore frozen before the chronological test began. It used update scale 2000 but forecast scale 1000, K=74, production MOV scale 1, and a hard annual reset. Its later predictions are retrospectively out of sample, but this does not prove the original parameter selection was leakage-free or turn the already-viewed period into an untouched holdout.
Assessment: the pre-holdout production model is not proven better than plain Elo
On the 835-map chronological test, the pre-holdout frozen production model recorded Brier 0.2566 versus 0.2513 for plain overall-team Elo. The paired difference was +0.0054 with a 95% block-bootstrap interval of -0.0052 to +0.0160. The interval crosses zero, so this run does not establish an improvement.
The analysis-time production comparator was also inconclusive versus plain Elo (+0.0042, 95% CI -0.0060 to +0.0149). Among 14 experimental map-Elo candidates, validation selected Production map Elo with 8-game uncertainty prior; plain Elo itself had lower validation log loss (0.6815 versus 0.6899). The challenger's test change versus plain Elo was inconclusive (-0.0014, 95% CI -0.0101 to +0.0086). No production parameters were changed automatically.
Temporal split and leakage controls
Training warms rating state. Validation chooses one challenger by log loss from the experimental map-Elo grid; fixed references are not eligible. Within each run, the final test period begins on Sep 1, 2025 and is evaluated only after that choice is frozen.
| Partition | Observed dates (UTC) | Maps | Series |
|---|---|---|---|
| train | Feb 13, 2023 – Aug 25, 2024 | 1,799 | 712 |
| validation | Jan 11, 2025 – Aug 31, 2025 | 1,184 | 464 |
| test | Sep 12, 2025 – Jun 21, 2026 | 835 | 319 |
Prediction timing
- Teams are assigned to A/B lexically before the winner label is derived.
- All maps sharing a match timestamp are predicted before any result at that timestamp updates ratings.
- Roster appearances become visible only after those predictions, so the roster experiment is a last-observed proxy.
- Bootstrap samples are clustered by series, with a fixed seed and 500 draws.
data sha256: 1aa1306720b94850ca696befd7532b1408ff2e90734dd9def30b635c012acc14
source sha256: 57f19dc87cfcb41adfb7626c4cfe7d80bbb3742546b1642e65f16784db59b5c8
git: dfd4a7a0261a4505846d581a97adbc4d19a931e2 (dirty)
Final chronological test comparison
Lower Brier, log loss, and ECE are better; higher accuracy is better. Candidate selection used validation log loss only within the experimental map-Elo grid. Plain Elo, both production comparators, and the 0.5 reference were fixed comparisons rather than selection candidates.
| Model | Brier | Brier 95% CI | Log loss | Accuracy | ECE | Brier Δ vs plain |
|---|---|---|---|---|---|---|
| Plain overall-team Eloplain-overall-elo | 0.2513 | 0.2402 to 0.2603 | 0.6992 | 57.6% | 0.0578 | Reference |
| Pre-holdout frozen production map Eloproduction-map-elo-pre-holdout-frozen | 0.2566 | 0.2487 to 0.2645 | 0.7084 | 54.4% | 0.0512 | +0.005495% CI -0.0052 to +0.0160 |
| Production map Elo at analysis timeproduction-map-elo-fixed | 0.2554 | 0.2487 to 0.2628 | 0.7053 | 54.3% | 0.0503 | +0.004295% CI -0.0060 to +0.0149 |
| Uninformative 0.5 probabilitynaive-half | 0.2500 | 0.2500 to 0.2500 | 0.6931 | 49.3% | 0.0066 | -0.001395% CI -0.0122 to +0.0082 |
| Production map Elo with 8-game uncertainty priorcold-start-prior-8 | 0.2498 | 0.2478 to 0.2518 | 0.6928 | 54.6% | 0.0330 | -0.001495% CI -0.0101 to +0.0086 |
The 0.5 reference is close to this holdout's aggregate lexical A/B base rate (ECE 0.0066), but it carries no ranking information and is not perfectly calibrated. Accuracy and proper scoring rules must be read together.
Calibration and confidence
Reliability plot
Bubble area reflects maps per probability bin; tiny extreme bins are visually identifiable but should not drive conclusions.
Pre-holdout production accuracy by confidence
Accuracy does not rise consistently across the populated confidence bands. Each bar is labeled with its sample size; muted bars have fewer than 30 maps and should not drive conclusions.
| Band | Maps | Mean confidence | Accuracy | Accuracy 95% CI |
|---|---|---|---|---|
| 50%–60% | 548 | 53.6% | 54.7% | 50.6%–58.9% |
| 60%–70% | 226 | 64.0% | 52.7% | 46.2%–59.1% |
| 70%–80% | 50 | 73.3% | 56.0% | 42.3%–68.8% |
| 80%–90% | 11† | 83.8% | 63.6% | 35.4%–84.8% |
† Fewer than 30 maps; interval is correspondingly wide.
Exact reliability-bin values
| Model | Bin | Maps | Mean predicted | Observed | Observed 95% CI |
|---|---|---|---|---|---|
| Plain overall-team Elo | 0%–10% | 3 | 8.4% | 33.3% | 6.1%–79.2% |
| Plain overall-team Elo | 10%–20% | 18 | 15.3% | 55.6% | 33.7%–75.4% |
| Plain overall-team Elo | 20%–30% | 55 | 26.2% | 32.7% | 21.8%–45.9% |
| Plain overall-team Elo | 30%–40% | 131 | 35.6% | 43.5% | 35.3%–52.1% |
| Plain overall-team Elo | 40%–50% | 190 | 44.8% | 41.1% | 34.3%–48.2% |
| Plain overall-team Elo | 50%–60% | 254 | 55.3% | 54.3% | 48.2%–60.3% |
| Plain overall-team Elo | 60%–70% | 124 | 64.4% | 60.5% | 51.7%–68.6% |
| Plain overall-team Elo | 70%–80% | 54 | 74.2% | 53.7% | 40.6%–66.3% |
| Plain overall-team Elo | 80%–90% | 6 | 85.4% | 100.0% | 61.0%–100.0% |
| Pre-holdout frozen production map Elo | 10%–20% | 5 | 15.7% | 40.0% | 11.8%–76.9% |
| Pre-holdout frozen production map Elo | 20%–30% | 32 | 26.9% | 34.4% | 20.4%–51.7% |
| Pre-holdout frozen production map Elo | 30%–40% | 114 | 35.7% | 47.4% | 38.4%–56.5% |
| Pre-holdout frozen production map Elo | 40%–50% | 207 | 45.3% | 44.0% | 37.4%–50.8% |
| Pre-holdout frozen production map Elo | 50%–60% | 341 | 53.0% | 54.0% | 48.7%–59.2% |
| Pre-holdout frozen production map Elo | 60%–70% | 112 | 63.8% | 52.7% | 43.5%–61.7% |
| Pre-holdout frozen production map Elo | 70%–80% | 18 | 73.6% | 38.9% | 20.3%–61.4% |
| Pre-holdout frozen production map Elo | 80%–90% | 6 | 83.5% | 66.7% | 30.0%–90.3% |
| Production map Elo at analysis time | 10%–20% | 2 | 18.0% | 50.0% | 9.5%–90.5% |
| Production map Elo at analysis time | 20%–30% | 26 | 27.1% | 34.6% | 19.4%–53.8% |
| Production map Elo at analysis time | 30%–40% | 119 | 36.1% | 48.7% | 39.9%–57.6% |
| Production map Elo at analysis time | 40%–50% | 212 | 45.4% | 42.9% | 36.4%–49.7% |
| Production map Elo at analysis time | 50%–60% | 353 | 53.1% | 53.5% | 48.3%–58.7% |
| Production map Elo at analysis time | 60%–70% | 105 | 63.9% | 52.4% | 42.9%–61.7% |
| Production map Elo at analysis time | 70%–80% | 15 | 74.0% | 40.0% | 19.8%–64.3% |
| Production map Elo at analysis time | 80%–90% | 3 | 82.8% | 100.0% | 43.9%–100.0% |
| Production map Elo with 8-game uncertainty prior | 30%–40% | 12 | 37.1% | 41.7% | 19.3%–68.0% |
| Production map Elo with 8-game uncertainty prior | 40%–50% | 278 | 47.7% | 42.4% | 36.8%–48.3% |
| Production map Elo with 8-game uncertainty prior | 50%–60% | 540 | 51.0% | 53.1% | 48.9%–57.3% |
| Production map Elo with 8-game uncertainty prior | 60%–70% | 5 | 62.7% | 40.0% | 11.8%–76.9% |
Chronological test sample size by map
Map-level conclusions are less stable where the retrospective test period is sparse.
Cold-start context
Among the 14 experimental map-Elo variants, validation selected an eight-game cold-start prior. It was not the best model overall: plain Elo had lower validation log loss, and the challenger did not beat plain Elo decisively on the chronological test.
| Map | Holdout maps |
|---|---|
| Split | 125 |
| Haven | 124 |
| Breeze | 98 |
| Lotus | 97 |
| Pearl | 90 |
| Bind | 80 |
| Abyss | 72 |
| Corrode | 57 |
| Fracture | 45 |
| Ascent | 36 |
| Sunset | 11 |
VM-07 experiment comparison
Every experimental variant below is published on validation data. Within this run, only the lowest-log-loss experimental map-Elo candidate proceeds to the final holdout; fixed references are not eligible, and plain Elo scored better on validation than the selected challenger. The runner enforces that ordering, but it cannot prevent a person from rerunning and informally tuning after viewing test results; future decisions need a newly untouched period or rolling-origin confirmation.
| Family | Candidate | Configuration | Validation Brier | Validation log loss | Accuracy |
|---|---|---|---|---|---|
| season-carry | Production map Elo with 25% season carryseason-carry-25 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0.25; roster 0; prior 0 | 0.2522 | 0.6985 | 54.0% |
| season-carry | Production map Elo with 50% season carryseason-carry-50 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0.5; roster 0; prior 0 | 0.2529 | 0.7004 | 54.3% |
| season-carry | Production map Elo with 75% season carryseason-carry-75 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0.75; roster 0; prior 0 | 0.2554 | 0.7066 | 53.9% |
| season-carry | Production map Elo with 100% season carryseason-carry-100 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 1; roster 0; prior 0 | 0.2600 | 0.7193 | 53.8% |
| roster-regression | Last-observed roster proxy (25% strength)roster-regression-25 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0.25; prior 0 | 0.2526 | 0.6992 | 53.9% |
| roster-regression | Last-observed roster proxy (50% strength)roster-regression-50 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0.5; prior 0 | 0.2525 | 0.6990 | 53.7% |
| roster-regression | Last-observed roster proxy (75% strength)roster-regression-75 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0.75; prior 0 | 0.2525 | 0.6989 | 53.5% |
| cold-start-uncertainty | Production map Elo with 3-game uncertainty priorcold-start-prior-3 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0; prior 3 | 0.2483 | 0.6899 | 54.1% |
| cold-start-uncertainty | Production map Elo with 8-game uncertainty priorcold-start-prior-8 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0; prior 8 | 0.2484 | 0.6899 | 54.1% |
| cold-start-uncertainty | Production map Elo with 15-game uncertainty priorcold-start-prior-15 | K 74; update scale 1000; forecast scale 1000; MOV production/1; carry 0; roster 0; prior 15 | 0.2487 | 0.6906 | 54.1% |
| margin-of-victory | Map Elo without MOVmov-none | K 74; update scale 1000; forecast scale 1000; MOV none/0; carry 0; roster 0; prior 0 | 0.2487 | 0.6906 | 53.4% |
| margin-of-victory | Map Elo with log1p MOVmov-log1p | K 74; update scale 1000; forecast scale 1000; MOV log1p/0.75; carry 0; roster 0; prior 0 | 0.2510 | 0.6956 | 54.5% |
| margin-of-victory | Map Elo with square-root MOVmov-sqrt | K 74; update scale 1000; forecast scale 1000; MOV sqrt/0.75; carry 0; roster 0; prior 0 | 0.2492 | 0.6918 | 54.8% |
| margin-of-victory | Map Elo with capped-linear MOVmov-linear-capped | K 74; update scale 1000; forecast scale 1000; MOV linear-capped/0.25; carry 0; roster 0; prior 0 | 0.2485 | 0.6901 | 54.0% |
season carry
At least one season boundary exists before the holdout.
best on validation: season-carry-25
roster regression
Prior-map player appearances provide adequate last-observed roster coverage and observed changes.
best on validation: roster-regression-75
cold start uncertainty
Every row has observable prior map counts; shrinkage uses only counts strictly before prediction.
best on validation: cold-start-prior-8
margin of victory
Valid score margins are present before the holdout.
best on validation: mov-linear-capped
The evaluated update is intentionally aggressive
At equal 1000 ratings, the fixed production formula moves each team by approximately 86.3 points for 13–11 and 114.8 points for 13–0. The holdout evidence above does not validate those magnitudes as superior.
expected = 1 / (1 + 10^((loser − winner) / 1000))
margin = ln(5.95 × √(round difference + 1))
change = 74 × margin × (1 − expected)
Separate observational study
Veto outcome association, not calibration or causal lift
This replay asks whether lower regret from a deterministic greedy Elo pick/ban rule is associated with observed outcomes. It does not observe what would have happened on banned maps or under an unchosen sequence, and it cannot establish that following the rule causes wins.
Holdout conclusion: no incremental evidence for veto regret
The lower-regret team won 52.2% of 291 non-tied holdout matches (152 wins). The event-cluster 95% interval was 47.3% to 57.3%, which includes 50%.
Adding relative regret to the calibrated selected-map forecast changed holdout Brier by +0.0025 (95% interval -0.0059 to +0.0118; positive is worse). The interval crosses zero, so this run provides no incremental predictive evidence.
- Holdout matches
- 310
- Regret ties excluded
- 19
- 291 non-tied matches included
- Holdout cold starts
- 0
- matches with any 1000 fallback
- Source quality
- 1,438 / 57
- accepted / rejected matches
Five holdout forecast tracks
Lower Brier, log loss, and ECE are better. Accuracy is descriptive for non-neutral tracks; the neutral track's accuracy is intentionally suppressed.
| Track | Matches | Brier | Log loss | Accuracy | ECE |
|---|---|---|---|---|---|
| Neutral p=0.5 | 310 | 0.2500 | 0.6931 | Suppressed† | 0.0065 |
| Pre-veto pool Elo | 310 | 0.2516 | 0.6965 | 52.3% | 0.0255 |
| Selected-map series Elo | 310 | 0.2543 | 0.7023 | 51.3% | 0.0511 |
| Calibrated selected-map | 310 | 0.2556 | 0.7051 | 51.6% | 0.0570 |
| Calibrated selected-map + regret | 310 | 0.2582 | 0.7109 | 51.6% | 0.0654 |
† At p=0.5 the shared threshold chooses lexical team A, so neutral accuracy is team-A outcome prevalence rather than simulated coin-flip accuracy; use neutral Brier/log loss instead. Neutral Brier and log loss remain meaningful.
Mean-map plug-in baseline: average the seven pre-veto map probabilities, then plug that mean into the BO3/BO5 formula. It is not the full expected value of a pre-veto policy or an average over legal veto sequences.
Series win rate by regret-gap band
Observed lower-regret-team win rates with a 50% reference. Hollow bars have fewer than 30 matches; the pattern is not monotonic and is not causal.
Exact series association values
Intervals resample the 11 observed events as clusters. Band upper bounds are exclusive.
| Regret gap | Matches | Wins | Win rate | Event-cluster 95% interval |
|---|---|---|---|---|
| 0–<25 Elo | 24 | 15 | 62.5% | 42.0% to 83.0% |
| 25–<50 Elo | 13 | 8 | 61.5% | 37.5% to 91.7% |
| 50–<100 Elo | 33 | 19 | 57.6% | 34.7% to 71.9% |
| 100+ Elo | 221 | 110 | 49.8% | 41.2% to 58.3% |
Exploratory BO3 picked-map outcomes
Of 656 holdout pick decisions, 584 guaranteed-played BO3 picks were included. 72 BO5 picks were excluded and 0 outcomes were unresolved. These rates are exploratory, visibly non-monotonic, and do not validate the greedy rule.
| Pick regret | Included picks | Map wins | Observed win rate | Event-cluster 95% interval |
|---|---|---|---|---|
| 0–<25 Elo | 272 | 151 | 55.5% | 48.5% to 61.7% |
| 25–<50 Elo | 19 | 15 | 78.9% | 61.5% to 91.7% |
| 50–<100 Elo | 61 | 31 | 50.8% | 34.4% to 63.2% |
| 100+ Elo | 232 | 112 | 48.3% | 40.2% to 56.0% |
Interpretation boundary. Stored rating-history model version: unknown-unversioned. The database replay uses recorded match completion time as its cutoff proxy because start time is not stored, while explicitly excluding ratings produced by the target match. Source-less history is accepted only for exact 1000-point January 1 UTC hard resets: this snapshot contains 2,112 such resets out of 2,112 source-less rows and 0 outside the signature. Generation and replay fail closed for every outside row because target-series exclusion cannot otherwise be proven. Greedy regret is observational and omits side choice, preparation, roster plans, and private strategy. Ban values and unchosen sequences remain counterfactual; their outcomes are not observed. Event-cluster intervals cover sampled events, not model-selection, counterfactual, timestamp, or scrape uncertainty.
Open pick/ban model alignmentData quality and provenance
- Accepted
- 3,818
- Rejected
- 0
- Series ID coverage
- 100.0%
- Roster proxy coverage
- 100.0%
Extraction universe
maps where processed = true; required winner and loser team joins; completed_at and score validity are checked during normalization.
- All map rows
- 3,818
- Processed
- 3,818
- Unprocessed
- 0
- Missing completion time
- 0
- Processed, missing time
- 0
- Required-team join loss
- 0
These counts describe the database snapshot, not events that VLR.gg never exposed or that the scraper never discovered.
Limitations that remain
- This is an observational temporal backtest, not evidence of causal model improvement.
- The reported chronological test period is retrospective and can be rerun; it is not a prospectively untouched holdout, and future event mixes and patches may differ.
- The pre-holdout production comparator is externally frozen by repository history before the chronological test start, making its later rows retrospectively out of sample; this does not prove its original parameter selection was leakage-free or restore an untouched holdout after these results have been viewed.
- Roster change is a last-observed appearance proxy, not a confirmed pre-match roster announcement.
- Data-quality rejection counts start from maps marked processed that survive required team joins; they do not measure events never scraped, source-universe completeness, or all upstream ETL failures.
- Confidence intervals quantify resampling uncertainty for the observed series blocks and do not cover source bias or model-selection uncertainty.
- Validation-only selection and one test evaluation are enforced within each run; there is no persistent access ledger that can detect reruns after holdout results have been viewed.
- No experimental result automatically changes the production model; adoption requires an explicit review and calibration decision.
VLR.gg is a third-party source; tracked-event scope, delayed corrections, and scrape failures can affect coverage. The public freshness panel reports successful ingestion separately from full-pipeline health.
Decision boundary
This evidence argues against silently shipping a new formula. The runner did not change production parameters or adopt its challenger. A future adoption decision should pre-register the candidate and wait for a new untouched period or rolling-origin confirmation.