undr.check

Implementation of datasets conformity tests (file structure, hashes, event coordinates…).

Overview

Classes

CheckFile	Uses a switch to process files and wraps messages into undr.check.Error.
CheckLocalDirectoryRecursive	Dispatches check actions to validate the invariants of local files.
Error	Reports invariant violations while checking data files.

Functions

format_index_recursive	Validates the given index and formats its content.
handle_aps	Checks the invariants of an APS file.
handle_directory	Checks that system files are listed in the index, and vice-versa.
handle_dvs	Checks the invariants of a DVS file.
handle_imu	Checks the invariants of an IMU file.
handle_other	Checks the invariants of an “other” file.
structure_recursive	Rexucrively checks that the given path exists and that it has an UNDR structure (-index.json).

Module Contents

class undr.check.CheckFile(file: undr.path.File, switch: undr.formats.Switch)

Bases: undr.json_index_tasks.ProcessFile

Uses a switch to process files and wraps messages into undr.check.Error.

Parameters:

• file (path.File) – The file to process.

• switch (formats.Switch) – Switch that maps file types to actions.

run(session: requests.Session, manager: undr.task.Manager)

class undr.check.CheckLocalDirectoryRecursive(path_root: pathlib.Path, path_id: pathlib.PurePosixPath, priority: int)

Bases: undr.task.Task

Dispatches check actions to validate the invariants of local files.

This can be used to make sure that a dataset has been properly downloaded or to check files before uploading a dataset to a server.

Parameters:

• path_root (pathlib.Path) – The root path used to generate local file paths.

• path_id (pathlib.PurePosixPath) – The path ID of the directory that will be scanned recursively.

• priority (int) – Priority of this task and all recursively created tasks (tasks with lower priorities are scheduled first).

run(session: requests.Session, manager: undr.task.Manager)

class undr.check.Error

Reports invariant violations while checking data files.

message: str

Description of the problem.

path_id: pathlib.PurePosixPath

Identifier of the problematic resource.

undr.check.format_index_recursive(path: pathlib.Path, handle_path: Callable[[pathlib.Path], None])

Validates the given index and formats its content.

Parameters:

• path (pathlib.Path) – Path of the index’s parent directory.

• handle_path (Callable[[pathlib.Path], None]) – Called if the index was reformatted.

undr.check.handle_aps(file: undr.formats.ApsFile, send_message: undr.formats.SendMessage)

Checks the invariants of an APS file.

Parameters:

• file (formats.ApsFile) – The file to check.

• send_message (formats.SendMessage) – Callback channel for errors.

undr.check.handle_directory(directory: undr.path_directory.Directory, send_message: undr.formats.SendMessage)

Checks that system files are listed in the index, and vice-versa.

Parameters:

• directory (path_directory.Directory) – The directory to check.

• send_message (formats.SendMessage) – Callback channel for errors.

undr.check.handle_dvs(file: undr.formats.DvsFile, send_message: undr.formats.SendMessage)

Checks the invariants of a DVS file.

Parameters:

• file (formats.DvsFile) – The file to check.

• send_message (formats.SendMessage) – Callback channel for errors.

undr.check.handle_imu(file: undr.formats.ImuFile, send_message: undr.formats.SendMessage)

Checks the invariants of an IMU file.

Parameters:

• file (formats.ImuFile) – The file to check.

• send_message (formats.SendMessage) – Callback channel for errors.

undr.check.handle_other(file: undr.path.File, send_message: undr.formats.SendMessage)

Checks the invariants of an “other” file.

This function simply consumes the file bytes. This guarantees that the file can be read and that the index contains the right hash and number of bytes.

Parameters:

• file (formats.ImuFile) – The file to check.

• send_message (formats.SendMessage) – Callback channel for errors.

undr.check.structure_recursive(path: pathlib.Path)

Rexucrively checks that the given path exists and that it has an UNDR structure (-index.json).

Parameters:

path (pathlib.Path) – The local file path to check.

Raises:

RuntimeError – if the path is not a directory or does not contain a -index.json file.