Never call spark.stop() in Databricks workflows. Explicitly call sys.exit().