Read the Schema of a Spark DataFrame
R/sdf_wrapper.R
sdf_schema
Description
Read the schema of a Spark DataFrame.
Usage
sdf_schema(x, expand_nested_cols = FALSE, expand_struct_cols = FALSE) Arguments
| Arguments | Description |
|---|---|
| x | A spark_connection, ml_pipeline, or a tbl_spark. |
| expand_nested_cols | Whether to expand columns containing nested array of structs (which are usually created by tidyr::nest on a Spark data frame) |
| expand_struct_cols | Whether to expand columns containing structs |
Details
The type column returned gives the string representation of the underlying Spark type for that column; for example, a vector of numeric values would be returned with the type "DoubleType". Please see the Spark Scala API Documentation
for information on what types are available and exposed by Spark.
Value
An R list, with each list element describing the name and type of a column.