mlm_insights.mlm_spark.execution_engine package¶

Submodules¶

mlm_insights.mlm_spark.execution_engine.spark_engine module¶

class mlm_insights.mlm_spark.execution_engine.spark_engine.SparkEngine¶

Bases: ExecutionEngine

Class for Spark execution engine functionality.

parse_data_frame_result(profile_dataframe: DataFrame) → Profile¶

Method to parse the profile dataframe based on different Execution engines As Spark converts to a bytearray whereas Dask provides a byte string. We need to handle this distinction between different Execution engines

Parameters¶

profile_dataframe: DataFrame: Pandas profile dataframe

Returns¶

Profile: Contains data summary and includes profile header, information about the features, metrics, SFCs.