lib.sedna.datasources

Subpackages

• lib.sedna.datasources.kafka
  • lib.sedna.datasources.kafka.consumer
  • lib.sedna.datasources.kafka.kafka_manager
  • lib.sedna.datasources.kafka.producer
• lib.sedna.datasources.obs
  • lib.sedna.datasources.obs.connector

Package Contents

Classes

BaseDataSource	An abstract class representing a BaseDataSource.
TxtDataParse	txt file which contain image list parser
CSVDataParse	csv file which contain Structured Data parser

class lib.sedna.datasources.BaseDataSource(data_type='train', func=None)

An abstract class representing a BaseDataSource.

All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite parse`, supporting get train/eval/infer data by a function. Subclasses could also optionally overwrite __len__, which is expected to return the size of the dataset.overwrite x for the feature-embedding, y for the target label.

Parameters

• data_type (str) – define the datasource is train/eval/test

• func (function) – function use to parse an iter object batch by batch

num_examples(self) → int

__len__(self)

abstract parse(self, *args, **kwargs)

property is_test_data(self)

save(self, output='')

class lib.sedna.datasources.TxtDataParse(data_type, func=None)

Bases: BaseDataSource, abc.ABC

txt file which contain image list parser

parse(self, *args, **kwargs)

class lib.sedna.datasources.CSVDataParse(data_type, func=None)

Bases: BaseDataSource, abc.ABC

csv file which contain Structured Data parser

static parse_json(lines: dict, **kwargs) → pandas.DataFrame

parse(self, *args, **kwargs)