Spark-submit

Latest version: v1.4.0

Safety actively analyzes 626436 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 2

1.4.0

Features:
- Report if framework not detected by `spark_submit.system_info`

Bug Fixes
- Fix quotes in spark-sumbit binary path in `spark_submit.system_info`

Misc/Internal
- Better way to construct the spark-submit command

1.3.0

Features:
- Addition of `spark_submit.SparkJob.get_id` method to obtain the Spark submission ID
- Use all environment variables from `os.environ` if `use_env_vars=True` in `spark_submit.SparkJob.submit`
- Additional environment variables can be used in `spark_submit.SparkJob.submit with `extra_env_vars` argument

Misc/Internal
- `spark_submit.SparkJob.submit` now uses keyword arguments only
- Argument `await_result` is changed to `poll_time` in `spark_submit.SparkJob.submit`
- No default environment variables to be used with `spark_submit.SparkJob.submit`
- Code standards improvements and general refactoring

1.2.1

Misc/Internal
- Refactor to comply with `pylint`, `isort` and `mypy` checks
- Minor logic improvements

1.2.0

Features
- Added optional `timeout` argument for `spark_submit.SparkJob.submit`

Improvements
- Improve support for Yarn and Kubernetes clusters on polling and killing submitted tasks

Misc/Internal
- Use dedicated file for default arguments and Spark states
- Use consistent path separators where possible

1.1.0

Features
- Addition of `spark_submit.SparkJob.get_submit_cmd` method
- Introduction of `spark_submit.SparkJob.concluded` attribute to keep track of submitted job

Improvements
- Improve `spark_submit.system_info` functionality
- More generic status polling
- Minor refactoring

1.0.1

Bug Fixes
- Fix return code bug in `spark_submit.SparkJob.get_code`

Improvements
- Change argument order in `spark_submit.SparkJob.submit`
- Minor improvements

Page 1 of 2

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.