R interface for MLeap
mleap is a sparklyr extension that provides an interface to MLeap, which allows us to take Spark pipelines to production.
Getting started
mleap can be installed from CRAN via
install.packages("mleap")
or, for the latest development version from GitHub, using
::install_github("rstudio/mleap") devtools
Once mleap has been installed, we can install the external dependencies using
library(mleap)
install_maven()
# Alternatively, if you already have Maven installed, you can
# set options(maven.home = "path/to/maven")
install_mleap()
We can now export Spark ML pipelines from sparklyr.
library(sparklyr)
<- spark_connect(master = "local", version = "2.4.0")
sc <- sdf_copy_to(sc, mtcars, overwrite = TRUE)
mtcars_tbl
# Create a pipeline and fit it
<- ml_pipeline(sc) %>%
pipeline ft_binarizer("hp", "big_hp", threshold = 100) %>%
ft_vector_assembler(c("big_hp", "wt", "qsec"), "features") %>%
ml_gbt_regressor(label_col = "mpg")
<- ml_fit(pipeline, mtcars_tbl)
pipeline_model
# Export model
<- file.path(tempdir(), "mtcars_model.zip")
model_path ml_write_bundle(pipeline_model, sample_input = mtcars_tbl, path = model_path)
# Disconnect from Spark
spark_disconnect(sc)
## NULL
At this point, we can share mtcars_model.zip
with our deployment/implementation engineers, and they would be able to embed the model in another application. See the MLeap docs for details.
We also provide R functions for testing that the saved models behave as expected. Here we load the previously saved model:
<- mleap_load_bundle(model_path)
model model
## MLeap Transformer
## <7e2f61ed-154b-4c9e-9926-85fa326d69ef>
## Name: pipeline_c1754f374a53
## Format: json
## MLeap Version: 0.14.0
We can retrieve the schema associated with the model:
mleap_model_schema(model)
## # A tibble: 6 x 5
## name type nullable dimension io
## <chr> <chr> <lgl> <chr> <chr>
## 1 qsec double TRUE <NA> input
## 2 hp double FALSE <NA> input
## 3 wt double TRUE <NA> input
## 4 big_hp double FALSE <NA> output
## 5 features double TRUE (3) output
## 6 prediction double FALSE <NA> output
Then, we create a new data frame to be scored, and make predictions using our model:
<- tibble::tribble(
newdata ~qsec, ~hp, ~wt,
16.2, 101, 2.68,
18.1, 99, 3.08
)
# Transform the data frame
<- mleap_transform(model, newdata)
transformed_df ::glimpse(transformed_df) dplyr
## Observations: 2
## Variables: 6
## $ qsec <dbl> 16.2, 18.1
## $ hp <dbl> 101, 99
## $ wt <dbl> 2.68, 3.08
## $ big_hp <dbl> 1, 0
## $ features <list> [[[1, 2.68, 16.2], [3]], [[0, 3.08, 18.1], [3]]]
## $ prediction <dbl> 21.00084, 20.56445