lib.sedna.datasources

Subpackages

• lib.sedna.datasources.kafka
  • lib.sedna.datasources.kafka.consumer
  • lib.sedna.datasources.kafka.kafka_manager
  • lib.sedna.datasources.kafka.producer
• lib.sedna.datasources.obs
  • lib.sedna.datasources.obs.connector

Package Contents

Classes

BaseDataSource	An abstract class representing a BaseDataSource.
TxtDataParse	txt file which contain image list parser
CSVDataParse	csv file which contain Structured Data parser
JSONDataParse	json file which contain Structured Data parser

class lib.sedna.datasources.BaseDataSource(data_type='train', func=None)

An abstract class representing a BaseDataSource.

All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite parse`, supporting get train/eval/infer data by a function. Subclasses could also optionally overwrite __len__, which is expected to return the size of the dataset.overwrite x for the feature-embedding, y for the target label.

Parameters:

• data_type (str) – define the datasource is train/eval/test

• func (function) – function use to parse an iter object batch by batch

property is_test_data

num_examples() → int

__len__()

abstract parse(*args, **kwargs)

save(output='')

class lib.sedna.datasources.TxtDataParse(data_type, func=None)

Bases: BaseDataSource, abc.ABC

txt file which contain image list parser

parse(*args, **kwargs)

class lib.sedna.datasources.CSVDataParse(data_type, func=None)

Bases: BaseDataSource, abc.ABC

csv file which contain Structured Data parser

static parse_json(lines: dict, **kwargs) → pandas.DataFrame

parse(*args, **kwargs)

class lib.sedna.datasources.JSONDataParse(data_type, func=None)

Bases: BaseDataSource, abc.ABC

json file which contain Structured Data parser

parse(*args, **kwargs)

load_anno_from_ids(id_)