Copy an R Data Frame to Spark
R/dplyr_spark.R
copy_to.spark_connection
Description
Copy an R data.frame to Spark, and return a reference to the generated Spark DataFrame as a tbl_spark. The returned object will act as a dplyr-compatible interface to the underlying Spark table.
Usage
## S3 method for class 'spark_connection'
copy_to(
dest,
df,
name = spark_table_name(substitute(df)),
overwrite = FALSE,
memory = TRUE,
repartition = 0L,
...
) Arguments
| Arguments | Description |
|---|---|
| dest | A spark_connection. |
| df | An R data.frame. |
| name | The name to assign to the copied table in Spark. |
| overwrite | Boolean; overwrite a pre-existing table with the name nameif one already exists? |
| memory | Boolean; should the table be cached into memory? |
| repartition | The number of partitions to use when distributing the table across the Spark cluster. The default (0) can be used to avoid partitioning. |
| … | Optional arguments; currently unused. |
Value
A tbl_spark, representing a dplyr-compatible interface to a Spark DataFrame.