Explain the concept of lineage in Spark.

Lineage in Spark refers to the logical execution plan that captures the dependencies between RDDs (Resilient Distributed Datasets) or DataFrames. It represents the sequence of transformations applied to create a particular RDD or data frame from its source data. Lineage information enables Spark to recover lost data by recomputing the lost partitions based on the available lineage information, making the data processing pipeline fault-tolerant.