library(sparklyr)
sc <- spark_connect(master = "local")
# compute element-wise products of 2 arrays from each row of `left` and `right`
# and store the resuling array in `res`
copy_to(
sc,
tibble::tibble(
left = list(1:5, 21:25),
right = list(6:10, 16:20),
res = c(0, 0)
)
) %>%
hof_zip_with(~ .x * .y)
#> # Source: spark<?> [?? x 3]
#> left right res
#> <list> <list> <list>
#> 1 <dbl [5]> <dbl [5]> <dbl [5]>
#> 2 <dbl [5]> <dbl [5]> <dbl [5]>Combines 2 Array Columns
R/dplyr_hof.R
hof_zip_with
Description
Applies an element-wise function to combine elements from 2 array columns (this is essentially a dplyr wrapper for the zip_with(array<T>, array<U>, function<T, U, R>): array<R> built-in function in Spark SQL)
Usage
hof_zip_with(x, func, dest_col = NULL, left = NULL, right = NULL, ...) Arguments
| Arguments | Description |
|---|---|
| x | The Spark data frame to process |
| func | Element-wise combining function to be applied |
| dest_col | Column to store the query result (default: the last column of the Spark data frame) |
| left | Any expression evaluating to an array (default: the first column of the Spark data frame) |
| right | Any expression evaluating to an array (default: the second column of the Spark data frame) |
| … | Additional params to dplyr::mutate |