pyspark.sql.datasource.DataSourceReader.partitions#

DataSourceReader.partitions()[source]#

Returns an iterator of partitions for this data source.

Partitions are used to split data reading operations into parallel tasks. If this method returns N partitions, the query planner will create N tasks. Each task will execute DataSourceReader.read() in parallel, using the respective partition value to read the data.

This method is called once during query planning. By default, it returns a single partition with the value InputPartition(None). Subclasses can override this method to return multiple partitions.

It’s recommended to override this method for better performance when reading large datasets.

Returns:

sequence of InputPartitions: A sequence of partitions for this data source. Each partition value must be an instance of InputPartition or a subclass of it.

Notes

All partition values must be picklable objects.

Examples

Returns a list of integers:

>>> def partitions(self):
...     return [InputPartition(1), InputPartition(2), InputPartition(3)]

Returns a list of string:

>>> def partitions(self):
...     return [InputPartition("a"), InputPartition("b"), InputPartition("c")]

Returns a list of ranges:

>>> class RangeInputPartition(InputPartition):
...    def __init__(self, start, end):
...        self.start = start
...        self.end = end

>>> def partitions(self):
...     return [RangeInputPartition(1, 3), RangeInputPartition(5, 10)]