Store
client for the store that contains all data (datasets, training configuration files, models, results)
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Optional[str]
|
path to store's main directory |
create(verbose=False)
classmethod
create store at cls.path
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
verbose |
bool
|
output |
False
|
get_path()
classmethod
Returns:
| Type | Description |
|---|---|
Optional[str]
|
cls.path: path to store's main directory |
get_training_results()
classmethod
get results for all trainings
Returns:
| Name | Type | Description |
|---|---|---|
training_results_list |
List[TrainingResults]
|
TODO: instead of list return dict that maps training_name to TrainingResults? |
get_training_results_single(training_name, update_trainings=True, verbose=False)
classmethod
get results for single training
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
training_name |
str
|
e.g. 'exp0' |
required |
update_trainings |
bool
|
whether to update cls.training_id2name & cls.training_name2id |
True
|
verbose |
bool
|
output |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
training_results |
Tuple[bool, TrainingResults]
|
for training with training_name |
mlflow(action)
classmethod
start or stop the mlflow server at http://127.0.0.1:5000 or check its status
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
action |
str
|
"start", "status", "stop" |
required |
parse_training_result_single(results=None, metric='f1', level='entity', label='micro', phase='test', average=True)
staticmethod
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
results |
Optional[TrainingResults]
|
TrainingResults gotten from get_training_results_single() |
None
|
metric |
str
|
"f1", "precision", "recall" |
'f1'
|
level |
str
|
"entity" or "token" |
'entity'
|
label |
str
|
"micro", "macro", "PER", .. |
'micro'
|
phase |
str
|
"val" or "test" |
'test'
|
average |
bool
|
if True, return average result of all runs. if False, return result of best run. |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
result |
Optional[str]
|
e.g. "0.9011 +- 0.0023" (average = True) or "0.9045" (average = False) |
set_path(path)
classmethod
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path |
str
|
path to store's main directory |
required |
Returns:
| Type | Description |
|---|---|
Optional[str]
|
cls.path: path to store's main directory |
show_trainings(as_df=True)
classmethod
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
as_df |
bool
|
if True, return pandas DataFrame. if False, return dict |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
trainings |
Union[pd.DataFrame, Dict[str, str]]
|
overview of trainings that have been run |
tensorboard(action)
classmethod
start or stop the tensorboard server at http://127.0.0.1:6006 or check its status
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
action |
str
|
"start", "status", "stop" |
required |