Actable AI vs DataRobot

While Actable AI’s focus is on extracting actionable insights from data, we in no way care less about the quality of our AI models. In this blog post, we show a performance benchmark of our predictive models with DataRobot Autopilot, one of the most popular DSML vendors in the market.

In this benchmark, we select randomly 10 popular data sets from Kaggle and other sources. The selected data sets vary in both number of rows and columns. Columns can be either numbers, categories or text. We then ran regression models for half of the data sets and classification for the other half. We use almost all the available features except features that leak predicted targets. In Actable AI, for each of the model, we set the options to optimize for quality, training time limit of 2 hours with a 10-fold cross validation. In DataRobot, we use the same set of features and ran with Autopilot mode.

For regression, we use R2, RMSE and MAE for benchmarking. The results for each of the data set are listed below:

Task 1: Predict weekly sales, data set : Walmart sales

Task 2: Predict customer lifetime value, data set: IBM Watson Marketing Customer Value Data

Task 3: Predict customer propensity score, data set: Customer propensity to purchase

Task 4: Predict mobile price, data set: Mobile prices.

Task 5: Predict movie ratings, data set: Netflix shows and movies

For classification, we use AUC for binary classification and Accuracy/Balanced Accuracy for multiclass classification. The results for 5 data sets are reported below:

Task 6: Predict if an individual income is more than $50K, data set: UCI Income.

Task 7: Predict credit card defaults, data set: Default of Credit Card Clients

Task 8: Predict customer churn, data set: IBM Telco Churn

Task 9: Drug classification, data set: Drug200

Task 10: Predict marketing responses, data set: Bank marketing responses.

The results show Actable AI’s models perform better than DataRobot Autopilot (statistically significant) in most cases except with IBM Telco Churn data set. We suspect that DataRobot Autopilot has a smart way to transform geospatial columns into useful features in this case.With Actable AI, it is possible to increase training time limit to achieve better results. DataRobot also has a Comprehensive mode.

We will report another benchmark for our Actable AI models without training time limit and DataRobot Comprehensive mode in another blog. This benchmark is designed to give a rough comparison between our AutoML and DataRobot Autopilot mode. It is by no meant an exhaustive comparison and the results might be different depending on the nature of the input data.

Actable AI
Actable AI Technologies LTD
UK Company Number: 12669336
Malta Company Number: C99699