Understanding Chess Improvement

Analysis of the factors that influence chess improvement by combining online behavior from Lichess, official over-the-board rating history from FIDE, and a small verified survey sample.

The project investigates two complementary questions:

• The dark side of online play: can highly repetitive fast-paced online play be associated with weaker long-term improvement in official FIDE rating?
• The virtuous side of online practice: which online behaviors are associated with stable improvement, better move quality, resilience, and deliberate practice?

The work was developed as a Business Intelligence and data analysis project for the 2024-25 academic year.

Presentation

View the full project presentation (PDF)

Project Overview

Chess produces a rare mix of structured behavioral data and long-term skill measurements. Online platforms such as Lichess expose large-scale activity, game metadata, openings, clocks, engine evaluations, and puzzle activity. FIDE provides official over-the-board rating histories, which are a stronger proxy for classical chess improvement.

This project connects those two worlds:

1. Extract active Lichess users from public PGN dumps.
2. Filter users by account age and game activity.
3. Integrate verified users from a Google Forms survey.
4. Match Lichess users to FIDE identities using real names and FIDE public data.
5. Scrape FIDE rating history for matched users.
6. Download Lichess activity and analyzed games from 2023 onward.
7. Build analytical datasets for compulsivity, rating growth, move quality, openings, time management, puzzle solving, and resilience.
8. Visualize results through Power BI and Tableau dashboards.

Data Sources

• Lichess public game dumps: used to extract active usernames from recent PGN files.
• Lichess API: used to collect user profiles, rating histories, activity, and analyzed game details.
• FIDE XML player database: used for matching Lichess real names to FIDE IDs.
• FIDE profile pages: scraped to retrieve rating history and player metadata.
• Google Forms survey: used to add verified Lichess usernames and real-name information where available.

Main Analyses

1. Compulsivity Index and FIDE Rating Growth

The first analysis studies whether a highly fast-paced online playing style correlates with FIDE rating change.

The project defines a Compulsivity Index (CI) between 0 and 1:

CI = 0.5 * frequency + 0.5 * fast_quality

Where:

• frequency measures annual intensity of UltraBullet, Bullet, and Blitz games normalized against the sample maximum.
• fast_quality measures the weighted share of faster time controls:

fast_quality = (1.00 * UltraBullet + 0.60 * Bullet + 0.30 * Blitz) / total_games

FIDE rating changes are computed monthly and annually. Missing months are filled with the last known rating and marked as inactive, so rating plateaus caused by inactivity can be separated from active periods with no rating gain.

The project also corrects the FIDE March 2024 rating reform for ratings below 2000. When needed, post-reform ratings are mapped back to the pre-reform scale:

elo_pre = (elo_post - 800) / 0.60

2. Virtuous Online Patterns

The second analysis focuses on online behaviors associated with stable improvement on Lichess.

Metrics include:

• Rating growth by game type and rating band.
• Puzzle volume.
• Average centipawn loss (ACPL).
• Accuracy.
• Inaccuracies, mistakes, and blunders.
• Opening variety through ECO codes.
• Deviation from theoretical opening lines.
• Average time spent in opening, middlegame, and endgame.
• Rating difference against opponents.
• Wins and draws from disadvantaged positions.
• Resign rate.

These features are aggregated globally and monthly, then compared across improvement quartiles.

Repository Structure

config/
  secrets/                      Local secrets such as Lichess API tokens

data/
  raw/                          Original input files
  interim/                      Intermediate JSONL, GZ, ZIP, and AWS outputs
  processed/                    Final analysis tables and processed assets

dashboards/
  source/                       Power BI and Tableau source files
  exports/                      Dashboard images and PDF exports
  extensions/                   Dashboard extensions

docs/
  presentation/                 Project presentation
  survey/                       Survey text and related material

src/
  data_collection/              Lichess and FIDE collection scripts
  matching/                     Lichess-FIDE matching scripts
  analysis/                     Compulsivity, virtuous-pattern, and Elo analyses
  aws/                          AWS Glue jobs and helper scripts
  utilities/                    Utility scripts for splitting, merging, and reading files

Setup

Create a virtual environment and install the local dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For AWS helper scripts, install the AWS-specific requirements:

pip install -r requirements-aws.txt

AWS Glue jobs also require the awsglue Python modules provided by the AWS Glue runtime. They are not part of the standard local requirements.txt.

Expected Local Secrets

Lichess API tokens are expected in:

config/secrets/TOKEN.txt

Use one token per line. This file must not be committed to GitHub.

Pipeline

The main local pipeline is organized as follows:

1. src/data_collection/lichess/00_a_lichess_get_users.py
   Extract unique usernames from Lichess PGN dumps.

2. src/data_collection/lichess/00_b_lichess_get_users_filtered.py
   Filter users by account age and rated game count.

3. src/data_collection/lichess/00_c_lichess_survey_integration.py
   Integrate additional users from the survey.

4. src/data_collection/lichess/01_a_lichess_users.py
   Collect Lichess profile data, rating histories, and performance data.

5. src/matching/02_lichess_fide_match.py
   Match Lichess users to FIDE records.

6. src/data_collection/fide/03_fide_scraping_users.py
   Scrape FIDE rating history for matched users.

7. src/matching/04_lichess_get_matched.py
   Create the final matched-user list for downstream Lichess game collection.

8. src/data_collection/lichess/01_b_lichess_activity_jsonl.py
   Collect activity data for matched users.

9. src/data_collection/lichess/01_b_lichess_games_jsonl.py
   Collect analyzed game details for matched users.

10. src/analysis/compulsivity/ and src/analysis/virtuous_patterns/
    Build final analytical datasets.

Dashboards

The project includes Power BI and Tableau dashboards:

• dashboards/source/powerbi/
• dashboards/source/tableau/

Exported dashboard views are available under:

dashboards/exports/

Data Notes

Some files in data/interim/ are large and are better handled with Git LFS, external storage, or regenerated from the pipeline rather than committed directly to GitHub.

The project also contains survey-derived and identity-matching data. Before publishing the repository, sensitive or personally identifiable information should be anonymized or excluded.

Current Caveat

The analysis scripts that compute game-level statistics expect:

data/interim/lichess_games/lichess_games_matched.jsonl

The current repository snapshot contains:

data/interim/lichess_games/lichess_games_matched.zip

If the JSONL is not present, extract it from the ZIP or regenerate it with the game collection script before running those analyses.