pyspark.pipelines.temporary_view#

pyspark.pipelines.temporary_view(query_function=None, *, name=None, comment=None, spark_conf=None)[source]#

(Return a) decorator to define a view in the pipeline and mark a function as the view’s query function.

@view can be used with or without parameters. If called without parameters, Python will implicitly pass the decorated query function as the query_function param. If called with parameters, @view will return a decorator that is applied on the decorated query function.

Parameters

• query_function – The view’s query function. This parameter should not be explicitly passed by users. This is passed implicitly by Python if the decorator is called without parameters.

• name – The name of the dataset. If unspecified, the query function’s name will be used.

• comment – Description of the dataset.

• spark_conf – A dict whose keys are the conf names and values are the conf values. These confs will be set when the query for the dataset is executed and they can override confs set for the pipeline or on the cluster.