For the impatient, head directly to Getting Started
Model Matrix is a framework/tool for solving large scale feature engineering problem: building model features for machine learning.
It’s build on top Spark DataFrames and can read input data, and write ‘featurized’ from/to HDFS (CSV, Parquet) and Hive.
With Model Matrix CLI you can control all lifecycle of Model Matrix:
Doing machine learning is fun and cool. Feature engineering (process of using domain knowledge of the data to create features that make machine learning algorithms work better) is tedious and boring. However good feature selection is bedrock to good models, none of machine learning techniques can produce predictive model if input features are bad.
Take for example this data set:
Producing a feature vector for every visitor (cookie) row and every piece of information about a visitor as an p-size vector, where p is the number of predictor variables multiplied by cardinality of each variable (number of states in US, number of unique websites, etc …). It is impractical both from the data processing standpoint and because the resulting vector would only have about 1 in 100,000 non-zero elements.
Model Matrix uses feature transformations (top, index, binning) to reduce dimensionality to arrive at between one and two thousand predictor variables, with data sparsity of about 1 in 10. It removes irrelevant and low frequency predictor values from the model, and transforms continuous variable into bins of the same size.
|visitor_id||Nike||OtherAd||NY||OtherState||price ∈ [0.01, 0.20)||price ∈ [0.20, 0.90)||…|
You can rad more about motivation and modeling approach in Machine Learning at Scale paper.