1. What is the difference between a DataFrame and an RDD in PySpark?

• RDD (Resilient Distributed Dataset):
  • Low-level API for distributed data processing.
  • Immutable and distributed collections of objects.
  • Offers fine-grained control but requires more manual optimization.
  • Does not support optimizations like Catalyst and Tungsten.
• DataFrame:
  • High-level abstraction built on top of RDDs.
  • Represents structured data as rows and columns (like a table in a database).
  • Provides built-in optimizations (Catalyst Optimizer).
  • Easier to work with because of SQL-like syntax.

Example in your project:
While working on the Retail Sales Analytics Platform project, I used DataFrames instead of RDDs for data cleaning and transformation in Azure Databricks. DataFrames allowed me to leverage Spark’s Catalyst Optimizer for efficient query execution, significantly improving performance.

2. Why is it recommended to use DataFrames over RDDs in most scenarios?

• DataFrames provide:
  • Optimization: Catalyst Optimizer and Tungsten Execution Engine optimize query plans.
  • Ease of use: SQL-like syntax simplifies complex transformations.
  • Integration: Works seamlessly with APIs like Spark SQL and MLlib.
  • Performance: Automatically manages serialization and data shuffling.

Example:
During the Modern Analytics and Insights – ADB Migration project, I used DataFrames for writing queries to transform data between Azure SQL and Data Lake. This reduced memory usage and improved pipeline performance.

3. What is lazy evaluation in PySpark? How does it impact pipeline performance?

• Lazy Evaluation:
  • Transformations in PySpark (e.g., filter, map) are not executed immediately.
  • Spark builds a logical execution plan (DAG) and executes it only when an action (e.g., show, collect) is triggered.
• Impact:
  • Optimizes execution by combining transformations and reducing unnecessary computation.

Example:
In the ADB Migration project, I built a series of transformations (filters, joins, aggregations) on DataFrames, but these were executed only when I triggered a write operation to Azure Data Lake. This approach minimized unnecessary intermediate computations.

4. Why is partitioning important in PySpark? How would you partition a large dataset for better performance?

• Importance of Partitioning:
  • Allows Spark to process data in parallel by dividing it into smaller chunks.
  • Reduces data shuffling and improves query performance.

How to Partition:

• Partition data by columns frequently used in filtering (e.g., date or region).
• Save partitioned data in formats like Parquet or Delta for easy retrieval.

Example:
For the Retail Sales Analytics Platform, I partitioned sales data by year and month before storing it in Azure Data Lake. This improved query performance by ensuring that only relevant partitions were read during analysis.

5. Can you explain how “repartition” and “coalesce” differ and when to use each?

• Repartition:
  • Increases or decreases the number of partitions.
  • Creates a new DataFrame and performs a full data shuffle.
  • Use when increasing partitions or redistributing data evenly.
• Coalesce:
  • Decreases the number of partitions without a full shuffle.
  • Use when reducing partitions (e.g., after a large filter operation).

Example:
In the Employee Payroll System, after filtering out inactive employees, I used coalesce to reduce partitions from 10 to 3, optimizing the write process to Azure SQL. For initial data ingestion, I used repartition to evenly distribute data for better parallel processing.

6. What types of joins are supported in PySpark, and how do you optimize join operations on large datasets?

• Join Types:
  • Inner, Left, Right, Outer, Semi, Anti, Cross joins.
• Optimization Techniques:
  • Use broadcast joins for smaller datasets.
  • Ensure both datasets are partitioned on the join key.
  • Use Delta Lake for efficient updates and merges.

Example:
While processing retail sales data, I joined customer and transaction data using a broadcast join, as the customer dataset was small. This avoided unnecessary shuffling and improved join performance.

7. How do you handle large aggregations in PySpark while preventing out-of-memory errors?

• Use window functions instead of groupBy for large datasets.
• Persist intermediate results in memory or on disk using .persist() or .cache().
• Optimize aggregations by using partitioning.

Example:
To calculate total sales per region in the Retail Sales Analytics Platform, I used a window function instead of a groupBy. This avoided excessive shuffling and improved memory efficiency.

8. What file formats have you worked with (e.g., Parquet, CSV, Delta Lake)? Why is Parquet preferred for big data processing?

• Worked with:
  • Parquet: Columnar format; optimized for read-heavy operations.
  • CSV: Simple format; used for interoperability.
  • Delta Lake: ACID-compliant format for real-time data.
• Why Parquet:
  • Supports compression and faster queries due to columnar storage.
  • Schema evolution and predicate pushdown enhance performance.

Example:
In the ADB Migration project, I converted on-premises data to Parquet before storing it in Azure Data Lake, reducing storage costs and speeding up transformations.

9. How do you handle scenarios where a late-arriving record affects previously loaded data?

• Use Delta Lake‘s upsert functionality (MERGE INTO) to handle late-arriving data.
• Implement watermarking in streaming jobs to manage late data.

Example:
While optimizing real-time analytics in the Retail Sales Analytics Platform, I used Delta Lake’s MERGE statement to update sales data when late-arriving transactions were ingested.

10. Explain the Medallion architecture (Bronze, Silver, Gold) you implemented. How did PySpark fit into each layer?

• Bronze Layer:
  • Raw data ingestion from various sources (e.g., Azure Blob Storage).
  • PySpark used for schema enforcement.
• Silver Layer:
  • Data cleaning, deduplication, and transformations.
  • PySpark used for applying business logic and joins.
• Gold Layer:
  • Aggregated, ready-for-analysis datasets.
  • PySpark used for aggregations and storing results in Delta Lake.

Example:
In the Retail Sales Analytics Platform, PySpark transformations (filtering, joins) were applied in the Silver layer to clean and validate raw sales data. Aggregations were performed in the Gold layer to generate region-wise sales reports.

11. What transformations would typically occur in the Silver and Gold layers?

• Silver Layer:
  • Deduplication, cleaning, and data validation.
  • Enrichment using joins with lookup tables.
• Gold Layer:
  • Aggregations, calculations, and summarization.
  • Formatting data for specific reporting use cases.

Example:
For the Modern Analytics project, the Silver layer involved removing duplicates from customer data. In the Gold layer, I aggregated customer transaction data to generate monthly revenue reports.

12. How would you configure PySpark for processing a streaming dataset? What libraries or tools in Spark enable real-time data processing?

• Configuration:
  • Use Structured Streaming for real-time data processing.
  • Configure checkpointing for fault tolerance.
• Tools:
  • Delta Lake for ACID compliance.
  • Kafka or Event Hub as a source.

Example:
In the Retail Sales Analytics Platform, I processed real-time transaction data using Spark Structured Streaming. Kafka served as the source, and Delta Lake was used for storing and querying the data.

13. In the migration process, how did you validate data accuracy between on-premise and Azure environments using PySpark?

• Used dataframe counts to verify row-level consistency.
• Applied checksum/hash techniques to compare data.
• Wrote custom PySpark scripts to compare datasets for schema and data differences.

Example:
During the ADB Migration, I compared the source SQL data with Azure Data Lake data by calculating record counts and hash values for critical columns.

14. How do you handle corrupted records or schema mismatches in PySpark pipelines?

• Use badRecordsPath in PySpark to log corrupted records.
• Validate schemas before processing data using PySpark’s inferSchema or predefined schemas.
• Apply exception handling using try…except.

Example:
In the Employee Payroll System, I handled corrupted records by writing invalid rows to a separate log file using badRecordsPath. Schema mismatches were addressed by enforcing a strict schema during ingestion.