mlm_insights.core.data_sources.filters package

Submodules

mlm_insights.core.data_sources.filters.contains_filter module

class mlm_insights.core.data_sources.filters.contains_filter.ContainsFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for contains filter-type to search/filter out the list of file location which contains the filter value in file location .

filter_files(files_location: List[str], filter_value: str, object_storage: Any) → List[str]

Parameters

files_location :

List of files present on the file path in the oci object system

filter_value :

A filter/search string if file name contains the string

kwargs:

Key-value pair for dynamic arguments

Returns

List[str]:

List of files present on the file path in the oci object system

mlm_insights.core.data_sources.filters.date_range_filter module

class mlm_insights.core.data_sources.filters.date_range_filter.DateRangeFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for date range filter-type to search/filter out the list of file location which were last modified in between the provided date range . Two arguments are taken as input in this filter : start and end, start being older date than end.

filter_files(files_location: List[str], filter_value: Any, object_storage: OCIFileSystem) → List[str]

Parameters

files_location:

List of files to filter from

filter_value:

dictionary containing the arguments for filters

object_storage:

OCIFileSystem object

Returns

List[str]:

List of files present on the file path in the oci object system

Raises

DataSourceException: DataSourceException

Exception if the file list is empty

mlm_insights.core.data_sources.filters.file_extension_filter module

class mlm_insights.core.data_sources.filters.file_extension_filter.FileExtensionFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for file extension filter-type to search/filter out the list of file location based on the file extension provided by user.

filter_files(files_location: List[str], filter_value: str, object_storage: Any) → List[str]

Parameters

files_location :

List of files present on the file path in the oci object system

filter_value :

string argument for file extension filter

object_storage:

OCIFileSystem object

Returns

List[str]:

List of files present on the file path in the oci object system

mlm_insights.core.data_sources.filters.last_n_days_filter module

class mlm_insights.core.data_sources.filters.last_n_days_filter.LastNDaysFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for last_n_days filter-type to search/filter out the list of file location modified in last n days , where n is the filter value provided by user .

filter_files(files_location: List[str], filter_value: str, object_storage: OCIFileSystem) → List[str]

Parameters

files_location:

List of files to filter from

filter_value:

string argument for last n days filter

object_storage:

OCIFileSystem object

Returns

List[str]:

List of files present on the file path in the oci object system based on last n days filter type

mlm_insights.core.data_sources.filters.oci_object_storage_data_source_filter module

class mlm_insights.core.data_sources.filters.oci_object_storage_data_source_filter.OCIObjectStorageDataSourceFilter(name: str = '', **kwargs: Any)

Bases: ABC

Abstract Base Class for OCI Object Storage DataSource Filters . The child class must implement filter_files methods to write the filtering logic .

abstract filter_files(files_location: List[str], filter_value: Any, object_storage: Any) → List[str]

Read and Return the List of files based on the filter_value and implemented filtering logic

Parameters

files_location :

List of files present on the file path in the oci object system

filter_value :

A string to filter/search the files

kwargs:

Key-value pair for dynamic arguments

Returns

List[str]:

List of files present on the file path in the oci object system

get_name() → str

Returns the name of filter which was set during initialization.

Returns

filter_name: str

Name of the filter

mlm_insights.core.data_sources.filters.partition_based_date_range_filter module

class mlm_insights.core.data_sources.filters.partition_based_date_range_filter.PartitionBasedDateRangeFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for date range filter-type to search/filter out the list of file location partitioned on basis of date range. Two arguments are taken as input in this filter : start, end and date_format, start being older date than end and date format is the regex required to search the date in object location.

filter_files(files_location: List[str], filter_value: Any, object_storage: OCIFileSystem) → List[str]

Parameters

files_location:

List of files to filter from

filter_value:

dictionary containing the arguments for filters

object_storage:

OCIFileSystem object

Returns

List[str]:

List of files present on the file path in the oci object system

Raises

DataSourceException: DataSourceException

Exception if the file list is empty

mlm_insights.core.data_sources.filters.prefix_filter module

class mlm_insights.core.data_sources.filters.prefix_filter.PrefixFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for prefix filter-type to search/filter out the list of file location which start with the filter value in file location .

filter_files(files_location: List[str], filter_value: str, object_storage: Any) → List[str]

Parameters

files_location :

List of files present on the file path in the oci object system

filter_value :

A filter/search string if file name prefixes the string

Returns

List[str]:

List of files present on the file path in the oci object system matching the filter

mlm_insights.core.data_sources.filters.suffix_filter module

class mlm_insights.core.data_sources.filters.suffix_filter.SuffixFilter(**kwargs: Any)

Bases: OCIObjectStorageDataSourceFilter

This class implements the OCIObjectStorageDataSourceFilter class for suffix filter-type to search/filter out the list of file location which end with the filter value in file location .

filter_files(files_location: List[str], filter_value: str, object_storage: Any) → List[str]

Parameters

files_location :

List of files present on the file path in the oci object system

filter_value :

A filter/search string if file name suffixes the string

Returns

List[str]:

List of files present on the file path in the oci object system matching the filter

Module contents