Model Matrix

Feature Engineering tool

Machine Learning at Scale: Feature Engineering

News : v0.0.2 released

For the impatient, head directly to Getting Started

Model Matrix is a framework/tool for solving large scale feature engineering problem: building model features for machine learning.

It’s build on top Spark DataFrames and can read input data, and write ‘featurized’ from/to HDFS (CSV, Parquet) and Hive.

Model Matrix Philosophy

Feature Matrix

Feature Transformations

Model Matrix Catalog

Model Matrix CLI

With Model Matrix CLI you can control all lifecycle of Model Matrix:

Specify model matrix database

Work with model matrix definitions

Work with model matrix instances

Feature extraction

Why Model Matrix?

Doing machine learning is fun and cool. Feature engineering (process of using domain knowledge of the data to create features that make machine learning algorithms work better) is tedious and boring. However good feature selection is bedrock to good models, none of machine learning techniques can produce predictive model if input features are bad.

Take for example this data set:

visitor_id ad_campaign ad_id ad_ctr pub_site state city price timestamp
bob Nike_Sport 1 0.01 NY New York 0.17 1431032702135
bill Burgers_Co 2 0.005 CA Los Angeles 0.42 1431032705167
mary Macys 3 0.015 CA Los Angeles 0.19 1431032708384

Producing a feature vector for every visitor (cookie) row and every piece of information about a visitor as an p-size vector, where p is the number of predictor variables multiplied by cardinality of each variable (number of states in US, number of unique websites, etc …). It is impractical both from the data processing standpoint and because the resulting vector would only have about 1 in 100,000 non-zero elements.

visitor_id Nike_Sport Burgers_Co Macys NY CA
bob 1.0     1.0  
bill   1.0     1.0
mary     1.0   1.0

Model Matrix uses feature transformations (top, index, binning) to reduce dimensionality to arrive at between one and two thousand predictor variables, with data sparsity of about 1 in 10. It removes irrelevant and low frequency predictor values from the model, and transforms continuous variable into bins of the same size.

visitor_id Nike OtherAd NY OtherState price ∈ [0.01, 0.20) price ∈ [0.20, 0.90)
bob 1.0   1.0   1.0  
bill   1.0   1.0   1.0
mary   1.0   1.0   1.0

You can rad more about motivation and modeling approach in Machine Learning at Scale paper.