This repository contains a complete data science project for predicting the outcome of Kickstarter crowdfunding campaigns. The target variable is binary: a campaign is classified as either successful (1) or failed (0).
The project combines dataset engineering, web scraping, exploratory analysis, statistical inference, machine learning modeling, hyperparameter tuning, threshold analysis, and final model assessment.
The work was developed as an academic project for the Bicocca Academy / AIDA Master, academic year 2024–2025.
View the full project presentation (PDF)
The goal is to build a predictive model able to estimate whether a Kickstarter campaign will succeed or fail using only ex-ante or launch-time information. The project has two main objectives:
- Identify which campaign characteristics are most associated with crowdfunding success or failure.
- Compare multiple classification models under a common evaluation framework and select the most reliable model for risk-oriented prediction.
The problem is treated as a supervised binary classification task. In the final assessment, the positive class is set to 0, corresponding to failed campaigns. This choice reflects a risk-oriented setting: missing a failed campaign is considered more critical than incorrectly flagging a successful campaign as risky.
Kickstarter is a reward-based crowdfunding platform. A creator defines a campaign, an economic goal, a duration, and a set of rewards for backers. Kickstarter operates with an all-or-nothing funding mechanism: if the target is reached, the creator receives the funds; otherwise, backers are not charged.
The project therefore models campaign outcome as a function of financial ambition, category, country, campaign duration, creator experience, reward structure, and communication/visibility indicators.
The base dataset comes from WebRobots.io Kickstarter datasets, which provide structured CSV/JSON data obtained through web scraping. Additional variables are engineered or collected through custom scraping scripts.
The final modeling dataset contains 49,945 campaigns, 16 predictors, the binary target, and the campaign ID.
The dataset pipeline starts from raw Kickstarter CSV files distributed inside a ZIP archive. The code merges the raw files, extracts structured columns from nested JSON fields, removes duplicate campaign IDs, converts timestamps, filters the dataset to the two target states of interest, scrapes missing project-level information, converts monetary variables to USD, and finally joins all feature groups into a single modeling table.
The final dataset is generated through the following conceptual stages:
- Raw Kickstarter ZIP ingestion.
- Deduplication by campaign
id. - Extraction of core campaign variables, URL fields, and marketing/media fields.
- Conversion of Unix timestamps into calendar dates.
- Engineering of campaign duration and launch month/year variables.
- Filtering to dichotomous outcomes:
successfulandfailed. - Conversion of binary fields such as
staff_pick,prelaunch_activated,video_present, andphoto_present. - Scraping of reward information from Kickstarter reward pages.
- Scraping of creator experience through the number of previously created projects.
- Conversion of reward amounts into USD.
- Inner join across campaign variables, reward variables, and creator variables.
- Export of the final modeling dataset.
The final dataset includes the following predictors.
| Variable | Type | Description |
|---|---|---|
state |
Binary target | Campaign outcome. 1 = successful, 0 = failed. |
goal_usd |
Numeric | Campaign funding goal converted into USD. |
country |
Categorical | Country of the project. |
created_at_year |
Numeric | Year of project creation or publication. |
created_at_month |
Numeric | Month of project creation; later also used to derive seasonality. |
season |
Categorical | Engineered seasonal feature: Winter, Spring, Summer, Autumn. |
slug_category |
Categorical | Kickstarter macro-category. |
slug_sub_category |
Categorical | Kickstarter sub-category. |
campaign_duration |
Numeric | Campaign duration in days, computed from launch and deadline dates. |
prelaunch_activated |
Binary | Whether the project used a pre-launch page. |
staff_pick |
Binary | Whether the project was selected as a Kickstarter staff pick. |
video_present |
Binary | Whether the main campaign page includes an introductory video. |
photo_present |
Binary | Whether a campaign image is available. This variable is removed from some model specifications. |
rewards_count |
Numeric | Number of reward tiers. |
min_amount_usd |
Numeric | Minimum reward amount converted into USD. |
max_amount_usd |
Numeric | Maximum reward amount converted into USD. |
var_amount_usd |
Numeric | Variance of reward amounts, converted into USD squared. |
created_projects |
Numeric | Number of projects previously created by the same creator. |
The raw category structure contains 15 macro-categories and 147 sub-categories. Since high-cardinality categorical variables can introduce sparse dummy encodings and unstable estimates, some models apply category lumping, especially on slug_sub_category.
The repository is organized as follows. Python and R scripts are numbered according to the main pipeline and modeling sequence.
.
├── README.md
├── requirements.txt
├── dataset/
│ ├── 00.dataset_webrobots/ # Original WebRobots archive
│ ├── 01.dataset_clean/ # Clean thematic datasets
│ ├── 02.dataset_dicotomico/ # Binary target and scraping outputs
│ ├── 03.dataset_cutoff50k/ # Modeling inputs limited to 50k rows
│ └── Kickstarter_2024-12_JOIN_50k.csv
├── docs/
│ ├── Kickstarter_predictive_analysis_cover.png
│ └── Kickstarter_predictive_analysis_presentation.pdf
├── outputs/
│ ├── descriptive statistics graphs/
│ └── reports/
└── src/
├── datase_scraping/
│ ├── kickstarter_2022-2024_dataset/ # Dataset correction utilities
│ ├── 00.kickstarter_dataset_merge_cleaning.py
│ ├── 01.kickstarter_dataset_dicotomico.py
│ ├── 02.kickstarter_scraping_playwright_rewards_PROXY.py
│ ├── 02.kickstarter_scraping_selenium_rewards.py
│ ├── 03.kickstarter_scraping_creator.py
│ ├── 04.kickstarter_scraping_playwright_photo-video_PROXY.py
│ ├── 05.kickstarter_rewards_usd_convertion.py
│ └── 06.dataset_join.py
└── R/
├── DEV/
│ └── 06_Kickstarter_NEURAL_NETWORK.Rmd
├── 00-Kickstarter_ descrittiva_inferenza.R
├── 01_Kickstarter_LOGISTICA_lasso_stepwise.Rmd
├── 02_Kickstarter_TREE.Rmd
├── 03_Kickstarter_RANDOM_FOREST.Rmd
├── 04_Kickstarter_ADA_BOOST.Rmd
├── 05_Kickstarter_KNN.Rmd
└── 07_Kickstarter_ASSESSMENT.R
The numbered dataset directories represent successive processing stages. The final joined table used by the modeling workflows is dataset/Kickstarter_2024-12_JOIN_50k.csv. Generated charts and rendered model reports are stored under outputs/.
The repository should not include raw credentials, proxy usernames, API keys, browser profiles, or private local paths. Scraping scripts should be adapted to read credentials from environment variables or from a local .env file excluded through .gitignore.
This script reads the raw Kickstarter ZIP archive, concatenates all CSV files, removes invalid or duplicate IDs, converts timestamp columns, and exports three thematic datasets:
- campaign variables,
- project URLs,
- marketing/media variables.
It also extracts JSON fields such as category slugs and URL fields. It computes goal_usd, campaign duration, video_present, and photo_present.
This script filters the dataset to the two campaign states used in the classification task: successful and failed. It creates the dichotomous version of the variables, URL, and marketing datasets.
For the variables file, it also converts logical/string fields such as prelaunch_activated and staff_pick into robust 0/1 integer encodings.
This is a Selenium-based reward scraper. It reads the campaign reward URLs and extracts the available reward tiers from the Kickstarter /rewards page. It computes:
rewards_count,min_amount,max_amount,median_amount,q1_amount,q3_amount,mean_amount,var_amount,currency,- serialized reward amount lists.
It includes browser handling, cookie acceptance, reward page hydration, Cloudflare/Turnstile detection, and fallback navigation logic.
This is an asynchronous Playwright version of the reward scraper. It processes campaign reward pages in parallel, applies a rolling rate limit, rotates user agents, and optionally rotates proxies. It was designed to scale the reward extraction step over large row ranges while reducing the probability of failed requests.
The script should be sanitized before publication. Proxy credentials must not be committed to GitHub.
This script enriches the campaign dataset with creator-level information. It visits the creator page associated with each project and extracts the number of previously created projects. The resulting variable, created_projects, acts as a proxy for creator experience and platform reputation.
The script uses asynchronous Playwright execution, concurrent tabs, random user agents, checkpoint saving, and fallback extraction from page state when available.
This script counts media elements in the campaign story section. It extracts:
photo_num, the number of images in the story section,video_num, the number of videos or embedded video references.
It includes robust scrolling, lazy-image handling, story-section detection, media counting, proxy rotation, zero-output logging, and request-rate control.
This script joins reward variables with the campaign exchange-rate variable static_usd_rate. It creates USD-normalized reward features. Linear reward amounts are multiplied by the exchange rate, while the variance is multiplied by the squared exchange rate.
The key output is a reward dataset with fields such as min_amount_usd, max_amount_usd, and var_amount_usd.
This script joins the campaign variable dataset, reward USD dataset, and creator dataset into the final modeling dataset. It performs an inner join on id, preserving only campaigns present in all three input tables.
It also enforces a controlled column order and applies robust data type casting for numeric, categorical, integer, and date-like fields.
This R script performs descriptive and inferential analysis. It computes category and sub-category frequencies, success/failure distributions, contingency tables, chi-square tests, seasonality analysis, country-level summaries, distribution checks, outlier inspection, and preliminary decision-tree variable importance.
It also generates descriptive plots used in the presentation, including category distributions, success/failure proportions, and outlier diagnostics.
This R Markdown file implements the logistic regression workflow. It includes:
- preprocessing,
- categorical dummy encoding,
- train/test split,
- outlier handling on the training set,
- rare category lumping,
- multicollinearity checks,
- baseline logistic regression,
- LASSO logistic regression,
- coefficient interpretation,
- odds ratio interpretation,
- ROC/AUC evaluation,
- confusion matrices,
- ASE calculation.
The LASSO step is used for variable selection and model regularization.
This R Markdown file implements decision tree models using rpart. It compares multiple tree specifications, including shallow trees, complete trees, and pruned trees. The tuning focuses on parameters such as cp, minsplit, minbucket, and maxdepth.
The tree workflow emphasizes interpretability, visual decision rules, variable importance, pruning, and the trade-off between underfitting and overfitting.
This R Markdown file implements a Random Forest classifier. It applies preprocessing, missing-value handling, sub-category lumping, train/test split, and hyperparameter tuning over parameters such as:
ntree,mtry,nodesize,maxnodes.
The model is evaluated through accuracy, ASE/Brier score, ROC/AUC, confusion matrices, and variable importance based on Mean Decrease Gini.
This R Markdown file implements AdaBoost using weak decision-tree learners. The workflow tunes the number of boosting iterations and tree depth, then evaluates the model using train/test accuracy, ASE, ROC/AUC, and confusion matrices.
AdaBoost is particularly relevant for the project because it sequentially focuses on observations that previous weak learners classified incorrectly.
This R Markdown file implements K-Nearest Neighbors. It includes one-hot encoding of categorical variables, transformation and standardization of numeric predictors, and comparison of different distance metrics:
- Euclidean,
- Manhattan,
- Cosine.
The tuning loop compares several values of k and computes accuracy and ASE on both train and test sets. The best observed configuration uses cosine distance with k = 13.
This development R Markdown file implements a neural network model using neuralnet. The workflow includes:
- stratified train/test split,
- train-only outlier handling,
- winsorization based on training-set thresholds,
- dummy encoding,
- standardization,
- neural network architecture tuning,
- probability prediction,
- ROC/AUC evaluation,
- Brier/ASE calculation,
- precision-recall analysis.
The neural network provides a nonlinear benchmark, although it requires heavier preprocessing and longer computation compared with simpler models.
This script compares the main candidate models under a common assessment protocol. It trains Decision Tree, Random Forest, and AdaBoost models using a consistent target definition and computes:
- ROC and AUC,
- default-threshold metrics,
- Youden threshold metrics,
- F1-optimized threshold metrics,
- max-sensitivity threshold with minimum specificity constraint,
- F-beta threshold with higher weight on sensitivity,
- accuracy,
- ASE,
- sensitivity,
- specificity,
- positive predictive value,
- negative predictive value.
The assessment sets the positive class to 0, corresponding to failed campaigns.
The preprocessing is model-specific but follows common principles.
Categorical predictors are converted to factors or dummy variables depending on the model. Tree-based models can directly handle factor variables in R, while Logistic Regression, KNN, and Neural Networks require dummy encoding or numeric matrices.
Rare sub-categories are collapsed into an Other class to reduce sparsity and improve generalization. This is particularly important because the original data contains 147 sub-categories.
Economic outliers are handled carefully. In several workflows, outlier treatment is applied only on the training set to avoid information leakage from the test set. In the neural network workflow, winsorization is explicitly based on training-set thresholds and then applied consistently to the test set.
Distance-based models require standardization. KNN applies centering and scaling so that variables measured in different units do not dominate the distance computation.
The project uses a 70/30 holdout split. The training set is used for preprocessing, outlier thresholds, encoding design, hyperparameter tuning, and model fitting. The test set is kept for final assessment.
This structure is intended to avoid leakage and to obtain a more realistic estimate of model generalization.
The logistic regression model estimates the probability of campaign success or failure through a logit link. It is used both as an interpretable baseline and as a framework for coefficient and odds-ratio interpretation.
The workflow includes a baseline logistic regression and a LASSO-regularized logistic regression. LASSO introduces an L1 penalty, which shrinks some coefficients to zero and therefore performs variable selection.
Strengths:
- interpretable coefficients,
- odds-ratio interpretation,
- useful baseline,
- low computational cost.
Limitations:
- assumes a linear relationship in the log-odds,
- less flexible with complex nonlinear interactions,
- depends on correct encoding and careful treatment of multicollinearity.
The decision tree provides an interpretable rule-based classifier. It recursively partitions campaigns according to variables that reduce node impurity.
The project compares:
- a limited-depth tree,
- a more complete tree,
- a post-pruned optimized tree.
The pruning process is controlled by the complexity parameter cp. A small cp allows a larger tree and may overfit; a large cp produces a simpler tree and may underfit.
Strengths:
- high interpretability,
- visual decision rules,
- no need for feature scaling,
- useful for exploratory understanding.
Limitations:
- unstable if used as a single model,
- prone to overfitting without pruning,
- weaker predictive robustness than ensemble methods.
Random Forest is an ensemble of decision trees trained on bootstrap samples and random subsets of predictors. Each tree votes for a class, and the final prediction is based on aggregated votes.
The project uses Random Forest to reduce the variance of individual decision trees. Tuning considers parameters such as number of trees, number of predictors tried at each split, terminal node size, and maximum number of nodes.
Variable importance is measured with Mean Decrease Gini. In the project outputs, variables such as goal_usd, rewards_count, var_amount_usd, slug_category, campaign_duration, max_amount_usd, and created_projects emerge as important predictors.
Strengths:
- robust generalization,
- good balance between predictive power and stability,
- reduced overfitting compared with a single tree,
- useful variable importance ranking.
Limitations:
- less interpretable than a single tree,
- computationally heavier than logistic regression,
- probability thresholds still require careful calibration for risk-oriented use.
AdaBoost combines sequential weak learners, usually shallow decision trees. Each new learner gives more weight to observations misclassified by previous learners.
In this project, AdaBoost is evaluated as a robust ensemble model able to balance sensitivity and specificity. It is particularly relevant in the final assessment because it performs well when the objective is to detect failed campaigns while preserving acceptable specificity.
Strengths:
- strong classification performance,
- useful for difficult observations,
- good risk-oriented behavior after threshold tuning.
Limitations:
- sensitive to noisy labels and outliers,
- less transparent than a single tree,
- requires tuning of boosting iterations and learner complexity.
KNN classifies a campaign according to the majority class among its nearest neighbors in feature space. Since the method is distance-based, preprocessing is critical.
The project compares Euclidean, Manhattan, and Cosine distances. The best configuration reported in the project workflow is cosine distance with k = 13.
Strengths:
- conceptually simple,
- non-parametric,
- can capture nonlinear local patterns.
Limitations:
- sensitive to feature scaling,
- computationally expensive on large datasets,
- less interpretable at global model level,
- affected by high-dimensional dummy-encoded spaces.
The neural network is used as a nonlinear benchmark. It requires the most extensive preprocessing: numeric conversion, dummy encoding, standardization, train-only outlier handling, and architecture tuning.
Strengths:
- flexible nonlinear modeling,
- can capture complex interactions,
- useful as a benchmark against classical models.
Limitations:
- longer training time,
- lower interpretability,
- more sensitive to preprocessing choices,
- less convenient for direct business explanation.
The project evaluates the models using both threshold-independent and threshold-dependent metrics.
AUC-ROC measures how well the model separates the two classes across all possible classification thresholds.
The main threshold-dependent metrics are:
- Accuracy: share of correct predictions.
- ASE / Brier score: average squared error between predicted probability and actual class.
- Sensitivity / Recall: share of failed campaigns correctly detected when failure is the positive class.
- Specificity: share of successful campaigns correctly identified as non-failures.
- Precision / Positive Predictive Value: reliability of failure predictions.
- Negative Predictive Value: reliability of success predictions.
- F1-score: harmonic mean between precision and recall.
- Cohen's Kappa: improvement over random agreement.
The final assessment does not rely only on the default 0.5 threshold. Several threshold policies are compared:
- Default threshold:
0.5. - Youden Index: maximizes
Sensitivity + Specificity - 1. - F1-maximizing threshold: maximizes the harmonic balance between precision and recall.
- F-beta threshold: gives more weight to sensitivity.
- Max Sensitivity with minimum Specificity constraint: maximizes recall while imposing
Specificity >= 0.7.
The last policy is important for a risk-oriented use case. If the objective is to detect potential campaign failures, false negatives are more costly than false positives. However, specificity must not collapse, otherwise the model would classify too many successful campaigns as failures.
The final comparison indicates that ensemble methods provide the best compromise between predictive accuracy and robustness.
Decision Trees are highly interpretable but tend to overfit if they are not pruned and may become unstable as standalone predictors.
Logistic Regression is useful for interpretation, odds ratios, and statistical explanation, but it is less flexible than ensemble models.
KNN can perform reasonably after strong preprocessing and distance tuning, but it is more sensitive to feature scaling and less efficient on large datasets.
Neural Networks provide a nonlinear benchmark but require more preprocessing and longer computation, with lower interpretability.
Random Forest and AdaBoost are the most convincing models in terms of robustness. Under the risk-oriented threshold policy, AdaBoost is slightly favored because it preserves high specificity while maintaining good sensitivity for failed campaign detection.
The most relevant predictors are linked to financial ambition, project category, reward design, creator experience, and visibility signals.
The following variables repeatedly appear as relevant across models or analyses:
goal_usd, because highly ambitious funding targets are associated with different success probabilities.rewards_count, because the structure of reward tiers captures campaign design effort.var_amount_usd, because diversified reward pricing can indicate a more articulated crowdfunding strategy.slug_categoryandslug_sub_category, because success dynamics differ strongly by project type.campaign_duration, because the campaign time window affects exposure and conversion dynamics.staff_pick, because platform-level visibility is a strong signal of perceived quality.video_present, because campaign communication quality matters.created_projects, because creator experience acts as a proxy for reputation and familiarity with the platform.
Create a Python virtual environment:
python -m venv .venvActivate it on Windows:
.venv\Scripts\activateOr on Linux and macOS:
source .venv/bin/activateThen install the Python dependencies:
pip install -r requirements.txtFor Playwright-based scraping, install browser binaries:
playwright install chromiumRun the data preparation scripts from the repository root, adapting the input/output constants in each script when necessary:
python src/datase_scraping/00.kickstarter_dataset_merge_cleaning.py
python src/datase_scraping/01.kickstarter_dataset_dicotomico.py
python src/datase_scraping/05.kickstarter_rewards_usd_convertion.py
python src/datase_scraping/06.dataset_join.pyThe scraping scripts are optional and should be executed only if the reward, creator, or media variables need to be rebuilt:
python src/datase_scraping/02.kickstarter_scraping_playwright_rewards_PROXY.py
python src/datase_scraping/03.kickstarter_scraping_creator.py
python src/datase_scraping/04.kickstarter_scraping_playwright_photo-video_PROXY.pyOpen the R Markdown files in RStudio or render them from R. The recommended order is:
src/R/00-Kickstarter_ descrittiva_inferenza.R
src/R/01_Kickstarter_LOGISTICA_lasso_stepwise.Rmd
src/R/02_Kickstarter_TREE.Rmd
src/R/03_Kickstarter_RANDOM_FOREST.Rmd
src/R/04_Kickstarter_ADA_BOOST.Rmd
src/R/05_Kickstarter_KNN.Rmd
src/R/DEV/06_Kickstarter_NEURAL_NETWORK.Rmd
src/R/07_Kickstarter_ASSESSMENT.R
The final model comparison is performed in 07_Kickstarter_ASSESSMENT.R.
The Python scripts require at least:
pandas
numpy
playwright
selenium
Depending on the selected scraper, additional browser drivers or Playwright binaries may be required.
The R workflows use the following packages:
caret
dplyr
forcats
ggplot2
tidyr
rpart
rpart.plot
randomForest
adabag
pROC
glmnet
car
funModeling
FNN
class
proxy
neuralnet
kernlab
PRROC
reshape2
pdp
nortest
scalesRendering the .Rmd files also requires rmarkdown and knitr. PDF output requires a working LaTeX distribution, such as TinyTeX; this is a system-level requirement rather than a Python dependency.
The scripts use fixed seeds in several model workflows to improve reproducibility. However, exact results can still vary depending on package versions, scraping completeness, operating system, and whether all external enrichment variables are available.
The most important reproducibility principles in this project are:
- keep the train/test split fixed,
- apply outlier treatment using training-set statistics only,
- fit encoders on training data and apply them to the test data,
- avoid leakage from test data during preprocessing,
- keep the positive class definition consistent during assessment,
- report both default-threshold and optimized-threshold metrics.
This repository is intended for academic and educational purposes. The original dataset is derived from publicly available Kickstarter data distributed through WebRobots.io, while additional variables are collected from publicly accessible campaign pages.
Before publishing the repository, verify the license and redistribution conditions of the raw dataset. If the raw dataset cannot be redistributed, include only code, documentation, and possibly small synthetic or sample data files.
This repository is intended for academic use. Add a license file before public release. If raw datasets are not redistributable, publish only the code and documentation.