Ask Question

The Spark Python API (PySpark) exposes the apache-spark programming model to Python.

33,695 questions 1

0votes0answers7views

Unable to launch pyspark with kafka when trying to import the kafka external package

I am trying to use pyspark with the following setup. pyspark version 3.2.1 spark version 3.2.1 hadoop version 3.2 jdk 11 So far, this setup works well. I am interested in working with Kafka, which ... user avatar Zeruno

  • 1,164
-2votes0answers12views

I want to use when on pyspark dataframe but i have multiple columns df.withcolumn [closed]

I have a column "Esc_char" if it is NULL. I want to use df.withColumm on 6-7 column and with when condition i can change only one column Any suggestions user avatar Sonali Bisht

  • 1
-1votes1answer15views

Convert Unix timing PySpark 13 digits

I've been trying to change the UNIX date (13 digits the one on the first column on the pic) to a readable date: from pyspark.sql import functions as F #TRY TO CHANGE THE DATA FORMAT sd = df.... user avatar Federico Gottardi

  • 19
1vote1answer19views

PySpark structured streaming with window get earliest & latest record based upon a timestamp value

I have a structured streaming process that reads from a deltalake. The data contains values that are increasing with time. Within each window, I would like to get the (difference between the) earliest ... user avatar Kermit754

  • 261
-1votes1answer22views

PySpark - Handle csv file with a 3 character quote

I just loaded a csv file to work on a new script and I've noticed that the quotes in some of the columns are """sometext""", tried setting .option('quote', '""&... user avatar VigyX

  • 1
1vote1answer21views

Create columns with upper and lower borders obtained from list

I have a DataFrame and a list of borders: test = spark.createDataFrame( [ (1,), (2,), (234,), (0,), (6,), (7,), (35,), (46,), ... user avatar Bohdan Pylypenko

  • 641
0votes1answer18views

How to zip-package a python module for pyspark executor?

I am parsing Protobuf binary-messages from Kafka topic, using pyspark-streaming. My UDF is: def parse_protobuf_from_bytes(pb_bytes): data = contacts_pb2.Contacts() data.ParseFromString(... user avatar aiman

  • 1,025
0votes1answer47views

Adding a new column to a dataframe with a value which is based on the values from next rows

I have a dataframe as shown below, +-----+----------+---------+-------+-------------------+ |jobid|fieldmname|new_value|coltype| createat| +-----+----------+---------+-------+----------------... user avatar Mohana B C

  • 3,994
1vote1answer9views

DataBricks: Ingesting CSV data to a Delta Live Table in Python triggers "invalid charactres in table name" error - how to set column mapping mode?

First off, can I just say that I am learning DataBricks at the time of writing this post, so I'd like simpler, cruder solutions as well as more sophisticated ones. I am reading a CSV file like this: ... user avatar Asfand Qazi

  • 6,197
1vote2answers34views

Unique element count in array column

I have this dataset with a column of array type. From this column, we need to create another column which will have list of unique elements and its counts. Example [a,b,e,b] results should be [[b,a,e],... user avatar Piyush Baranwal

  • 19
0votes0answers27views

Pyspark read file only if it exists

I have some parquet files in my hdfs directory /dir1/dir2/. The name of the files contain some timestamps but those are pretty random. For example, one file path is: /dir1/dir2/2022-06-16-03-12-36-086.... user avatar aishik roy chaudhury

  • 129
0votes1answer22views

pyspark performance and processing time

I have 5 steps which produced df_a,df_b,df_c,df_e and df_f. Each step generates a dataframe (df_a for instance), and persist as parquet files. The file is used in the sequential step (df_b for example)... user avatar NewPy

  • 561
-2votes0answers20views

ThreadPoolExecutor executor script with questions

Hi the code below is what I have my question on. I am trying to fully understand/learn the code. 1.) What is (line 17) "row for row in " doing? 2.) Why are we writing "future_to_url[... user avatar user1529514

  • 71
0votes0answers18views

Spark - Large Dataset ML -- How to set properties

Im hoping to get some basic information. I am using lightgbm on spark : and my data is large : about 40 ... user avatar B_Miner

  • 1,716
0votes0answers20views

Null value appeared in non-nullable field

I am trying to write the following pyspark dataframe to csv by using df_final.write.csv("v1_results/run1",header=True, emptyValue='',nullValue='') Please help me locate on why this issue ... user avatar Sree Aurovindh

  • 665

153050per page

1 2 3 4 5 … 2247