meerqat.data.labelstudio module

Used to manipulate the output of Label Studio, see also ANNOTATION.md

• assign takes annotations out of the TODO list in a tasks.json file (input to LS)

• save images similar to kilt2vqa download, not used for the final dataset

• merge merges several LS outputs, also compute inter-annotator agreement and saves disagreements

• agree merges the output of merge along with the corrected disagreements

Usage: labelstudio.py save images <path>… labelstudio.py merge <output> <path>… labelstudio.py assign <output> <todo> <start> <end> [–overlap=<n> –zip <config>…] labelstudio.py agree <dataset> <agreements>

Options:

--overlap=<n>

Number of questions to leave in todo [default: 0].

--zip

Whether to zip the output directory

meerqat.data.labelstudio.load_completions(completions_path)

Handles individual completions path and JSON exports

meerqat.data.labelstudio.save_images(completions_paths)

meerqat.data.labelstudio.merge(output_path, completions_paths)

meerqat.data.labelstudio.annotator_agreement(dataset)

Parameters:

dataset (dict[str, list]) – output of merge

Returns:

disagreements – questions of dataset where the annotators disagree on whether discarding the question or not

Return type:

dict

meerqat.data.labelstudio.retrieve_vqa(completion)

meerqat.data.labelstudio.assign_annotations(todo, start, end, overlap=0)

meerqat.data.labelstudio.assign(output_path, todo_path, start, end, overlap=0, zip=False, configs=[])

meerqat.data.labelstudio.agree(dataset_path, agreements_path)

meerqat.data.labelstudio.main()