Ytukyg

I am new to Scala. Excuse my lack of knowledge.
This is my dataset:

val bfDS = sessions.select("bf")

sessions.select("bf").printSchema





 |-- bf: array (nullable = true)

 |    |-- element: struct (containsNull = true)

 |    |    |-- s: struct (nullable = true)

 |    |    |    |-- a: string (nullable = true)

 |    |    |    |-- b: string (nullable = true)

 |    |    |    |-- c: string (nullable = true)

 |    |    |-- a: struct (nullable = true)

 |    |    |    |-- a: integer (nullable = true)

 |    |    |    |-- b: long (nullable = true)

 |    |    |    |-- c: integer (nullable = true)

 |    |    |    |-- d: array (nullable = true)

 |    |    |    |    |-- element: struct (containsNull = true)

 |    |    |    |    |    |-- a: string (nullable = true)

 |    |    |    |    |    |-- b: integer (nullable = true)

 |    |    |    |    |    |-- c: long (nullable = true)

 |    |    |-- tr: struct (nullable = true)

 |    |    |    |-- a: integer (nullable = true)

 |    |    |    |-- b: long (nullable = true)

 |    |    |    |-- c: integer (nullable = true)

 |    |    |    |-- d: array (nullable = true)

 |    |    |    |    |-- element: struct (containsNull = true)

 |    |    |    |    |    |-- e: string (nullable = true)

 |    |    |    |    |    |-- f: integer (nullable = true)

 |    |    |    |    |    |-- g: long (nullable = true)

 |    |    |-- cs: struct (nullable = true)

 |    |    |    |-- a: integer (nullable = true)

 |    |    |    |-- b: long (nullable = true)

 |    |    |    |-- c: integer (nullable = true)

 |    |    |    |-- d: array (nullable = true)

 |    |    |    |    |-- element: struct (containsNull = true)

 |    |    |    |    |    |-- e: string (nullable = true)

 |    |    |    |    |    |-- f: integer (nullable = true)

 |    |    |    |    |    |-- g: long (nullable = true)

1) I don't think I understand Scala datasets very well. A dataset is composed of rows, but when I print the schema, it shows an array. How does the dataset map to the array? Is each row is an element in the array?

2) I want to convert my dataset into a case class.

case class Features( s: Iterable[CustomType], a: Iterable[CustomType], tr: Iterable[CustomType], cs: Iterable[CustomType])

How do I convert my dataset and how do I use encoders?

Many thanks.

Welcome to StackOverflow. Sadly this question is too board for SO, take a look at "how to ask" to improve this and future questions.

However I will try to answer a few of your questions.

First, Spark Rows can encode a variety of values, including Arrays & Structures.

Second, your dataframe's rows are composed of only one column of type Array[...].

Third, if you want to create a Dataset from your df, your case class must match your schema, in such case it should be something like:

case class Features(array: Array[Elements])

case class Elements(s: CustomType, a: CustomType, tr: CustomType, cs: CustomType)

Finally, an Encoder is used to transform your case classes and their values to the Spark internal representation. You shouldn't bother too much about them yet - you just need to import spark.implicits._ and all the encoders you need will be there automatically.

val spark = SparkSession.builder.getOrCreate()

import spark.implicits._

val ds: Dataset[Features] = df.as[Features]

Also, you should take a look to this as a reference.