Assessing Wage Inequality with Machine Learning: Approaches for Measuring the Adjusted Gender Pay Gap

Abstract

This paper investigates the methodological performance of Ordinary Least Squares (OLS) regression and Random Forest machine learning algorithms in measuring adjusted gender pay gaps. The research is motivated by the European Union’s Pay Transparency Directive (2023/970), which mandates that employers report adjusted gender pay gaps. While Oaxaca-Blinder Decomposition and the underlying OLS regression have served as the industry standard for gap estimation, this paper examines whether machine learning approaches can better capture complex, nonlinear compensation relationships. Using synthetic datasets with controlled discrimination parameters, the study compares both methods across two sample sizes and multiple discrimination scenarios. Key findings demonstrate that both methods successfully distinguish between occupational segregation and direct wage discrimination at large sample sizes. However, at smaller sample sizes, Random Forest exhibits substantial instability whereas OLS remains slightly more stable. A methodological adjustment, training Random Forest on the larger population before applying predictions to subsets substantially improves small-sample performance. The paper concludes that OLS regression remains preferable for formal regulatory compliance due to its interpretability and stability, while Random Forest can serve as a complementary validation tool for large-scale analysis.