I am working on Azure Databrick. I run the python script on the notebook and gets the data from the SQL. I tried to split the date time column into date and time columns. Here is the syntax of python:
pushdown_query = "(SELECT * FROM STAGE.OutagesAndInterruptions) int_alias" df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties) df['INTERRUPTION_DATE']=df['INTERRUPTION_TIME'].dt.date df['INTERRUPTION_TIME'] looks like:
+-------------------+ | INTERRUPTION_TIME| +-------------------+ |1997-05-12 09:57:00| |1998-03-08 13:00:00| |1998-02-26 13:00:00| |1998-02-26 13:00:00| |1998-03-03 10:04:00| |1998-05-20 09:27:00| |1998-11-21 08:51:00| |1998-11-27 08:44:00| |1998-10-19 01:19:00| |1998-10-19 01:44:00| |2000-03-13 07:00:00| |2000-03-19 07:30:00| |2000-08-04 12:55:00| |2002-09-30 18:11:00| |2002-09-30 18:11:00| |2002-05-06 09:22:00| |2002-01-16 13:15:00| |2003-01-08 15:46:00| |2003-02-04 10:25:00| |2003-02-04 10:25:00| +-------------------+ When I ran the code, it throws an error message:
TypeError: 'DataFrame' object does not support item assignment --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <command-2244924718685919> in <module> ----> 1 df['INTERRUPTION_DATE']=df['INTERRUPTION_TIME'].dt.date TypeError: 'DataFrame' object does not support item assignment Could we create new columns in the data frame on Data frame? How can we create new columns on the data frame in Azure data bricks?
31 Answer
This should work
from pyspark.sql.types import DateType df2 = df.withColumn('INTERRUPTION_DATE', ,df['INTERRUPTION_TIME'].cast(DateType())) Edit After Comment:
from pyspark.sql.functions import date_format df.select(date_format('INTERRUPTION_TIME', 'M/d/yyyy').alias('INTERRUPTION_DATE'), date_format('INTERRUPTION_TIME', 'h:m:s a').alias('TIME')) 
