CSV

Allows you to read or write a delimited file (often called Comma Separated File, CSV)

Source Parameters

CSV Source supports all the available spark read options for CSV.

The below list contains the additional parameters to read a CSV file:

Parameter		Description	Required
Dataset Name		Name of the Dataset (read more about Datasets)	True
Location		Location of the file(s) to be loaded Eg: dbfs:/data/test.csv	True
Schema		Schema to applied on the loaded data. Can be defined/edited as json or inferred using Infer Schema button	True

Target Parameters

CSV Target supports all the available spark write options for CSV.

The below list contains the additional parameters to write a CSV file:

Parameter		Description	Required
Dataset Name		Name of the Dataset (read more about Datasets)	True
Location		Location of the file(s) to be loaded Eg: dbfs:/data/output.csv	True

Loading a CSV file

Step 1 - Create Source Component

Python

def load_csv(spark: SparkSession) -> DataFrame:
    return spark.read\
        .schema(
          StructType([
            StructField("order_id", IntegerType(), True),
            StructField("customer_id", IntegerType(), True),
            StructField("order_status", StringType(), True),
            StructField("order_category", StringType(), True),
            StructField("order_date", DateType(), True),
            StructField("amount", DoubleType(), True)
        ])
        )\
        .option("header", True)\
        .option("quote", "\"")\
        .option("sep", ",")\
        .csv("dbfs:/Prophecy/anshuman@simpledatalabs.com/OrdersDatasetInput.csv")

Scala

object load_csv {

  def apply(spark: SparkSession): DataFrame =
    spark.read
      .format("csv")
      .option("header", true)
      .option("quote",  "\"")
      .option("sep",    ",")
      .schema(
        StructType(
          Array(
            StructField("order_id",       IntegerType, true),
            StructField("customer_id",    IntegerType, true),
            StructField("order_status",   StringType,  true),
            StructField("order_category", StringType,  true),
            StructField("order_date",     DateType,    true),
            StructField("amount",         DoubleType,  true)
          )
        )
      )
      .load("dbfs:/Prophecy/anshuman@simpledatalabs.com/OrdersDatasetInput.csv")

}

---

Writing a CSV file

Step 1 - Create Target Component

Python

def write_as_csv(spark: SparkSession, in0: DataFrame):
    in0.write\
        .option("header", True)\
        .option("sep", ",")\
        .mode("error")\
        .option("separator", ",")\
        .option("header", True)\
        .csv("dbfs:/Prophecy/anshuman@simpledatalabs.com/output.csv")

Scala

object write_as_csv {
  def apply(spark: SparkSession, in: DataFrame): Unit =
    in.write
      .format("csv")
      .option("header", true)
      .option("sep",    ",")
      .mode("error")
      .save("dbfs:/Prophecy/anshuman@simpledatalabs.com/output.csv")
}