Manager#
- class simplesimdb.Manager(directory='./data', filetype='nc', executable='./execute.sh')[source]#
Bases:
objectLightweight Simulation Database Manager
Create, access and display simulation data of a given code as pairs
(inputfile.json : outputfile [, restarted_01, restarted_02, ...]), where all input is given by python dictionaries (stored as json files) and the output files (of arbitrary type) are generated by an executable that takes the json file as input. This executable is provided by the user.Note
an executable may only take one sinlge input file and may only generate one single output file (except for RESTART, see below)
Note
the executable can be a bash script. For example if the actual program does not take json input files you could write a converter and let the bash script chain the two programs. There are endless possibilities.
Restart Addon (ignore if not used)#
Sometimes simulation outputs cannot be created in a single run (due to file size, time limitations, etc.) but rather a simulation is partitioned (in time) into a sequential number of separate smaller runs. Correspondingly each run sequentially generates and stores only a partition of the entire output file. Each run restarts the simulation with the result of the previous run. The Manager solves this problem via the simulation number n in its member functions. For n>0 create will pass the result of the previous simulation as a third input to the executable
Naming Scheme#
By default the naming of the inputs and outputs is based on the sha1 of the python dictionary as a sorted string. The user can override this behaviour by giving a human readable name in the create function or register one manually. This name is then mapped to the sha1 and stored in the file “simplesimdb.json” in the data directory. All subsequent references to the input file then map to the correct name.
- __init__(directory='./data', filetype='nc', executable='./execute.sh')[source]#
Init the Manager class
Restart Addon (ignore if not used)#
If you intend to use the restart option (by passing a simulation number
n > 0to create), executable is called with:subprocess.run( [ executable, directory/hashid.json, directory/hashid0xN.filetype, directory/hashid0x(N-1).filetype, ], ...)
that is it must take a third argument (the previous simulation)
- type directory:
- param directory:
the path of the folder this class manages. Folder is created if it does not exist yet. The class will recognize existing files that were generated by a previous session.
- type filetype:
- param filetype:
file extension of the output files
- type executable:
- param executable:
The executable that generates the data. Can be a string or list of strings. Since executable is passed to subprocess.run, it can be a command with arguments. If your code is an executable script or single command, this can be passed just as a string (e.g. “./execute.sh” or “cp”). If your code needs to be run with an interpreter or needs additional arguments (e.g. “python script.py”, “ls -l”), then you must pass the executable as a list of strings (e.g. [“python”, “script.py”], [“ls”, “-l”]). Same as it would be given to subprocess.run. executable is called using subprocess.run([executable, directory/hashid.json, directory/hashid.filetype],…) with a json file as input (do not change the input else the file is not recognized any more) and one output file (that executable needs to generate) (if your code does not take json as input you can for example parse json in bash using jq)
- count(js)[source]#
(RESTART ADDON) Count number of output files for given input
Count the output files associated with the given input parameters that currently exist in directory (i.e. count n as long as self.exists( js, n) returns True).
- create(js, n=0, name='', error='raise', stdout='ignore', *, check=True, capture_output=True, **kwargs)[source]#
Run a simulation if outfile does not exist yet
Create (write) the in.json file to disc Use subprocess.run( [executable, in.json, out]) If the executable returns a non-zero exit code the inputfile (if n!= 0) and outputfile are removed
Attention
in order to generate a unique identifier js needs to be normalized in the sense that the datatype must match the required datatype documented (e.g. don’t write 10 in a field requiring float but 10.0, otherwise it is interpreted as an integer and thus produces a different hash)
- Parameters:
js (
dict[str,Any]) – the complete input file as a python dictionary. All keys must be strings such that js can be converted to JSON.n (
int) – (RESTART ADDON) the number of the simulation beginning with 0. If n>0, we will use the previous simulation as a third argument subprocess.run( [executable, in.json, out_n, out_(n-1)]) This can be used to restart simulationsname (
str) – [OPTIONAL] human readable name/id that is used in the naming of all files associated with js. For example the input is called ‘<name>.json’. If empty, the simplesimdb.json file will be searched for a name, afterwards the default sha based naming scheme will be used, for example ‘<sha1>.json’. Once a name is given it can only be changed by deleting and resimulating the output. Exceptions will be raised on name clashes. Names will be stored and mapped to their sha1 in the file “simplesimdb.json” See also: registererror (
str) –“raise”: raise a subprocess.CalledProcessError error if the executable returns a non-zero exit code
”display”: print(stderr ) then return
”ignore”: return
stdout (
str) –“ignore”: throw away std output
”display”:
print( process.stdout)
check (
bool) – passed to subprocess.run, if True a non-zero exit code raises a subprocess.CalledProcessError, if False, no exception is raised and you can check the return code with process.returncodecapture_output (
bool) – passed to subprocess.run, if True, stdout and stderr are captured and can be accessed with process.stdout and process.stderr,kwargs – additional arguments passed to subprocess.run in the create method
- Returns:
filename of new entry if it did not exist before existing filename else
- Return type:
- delete(js, n=0)[source]#
Delete an entry if it exists
- Parameters:
js (
dict[str,Any]) – the complete input file as a python dictionary. All keys must be strings such that js can be converted to JSON.n (
int) –(RESTART ADDON) the number of the simulation to select beginning with 0.
In case n>0, only the outputfile outfile(js,n) will be removed.
In case n==0, both the outfile as well as the jsonfile(js) and any eventual registered names will be removed
- delete_all()[source]#
Delete all file pairs id’d by the files method
and the directory itself (if empty)
Attention
if you want to continue to use the object afterwards remember to reset the directory:
m.directory = '...'
- property directory: PathLike | str#
Data directory
If the directory does not exist it will be created
- property executable: list[str]#
The executable that generates the data.
The create method calls subprocess.run([executable, directory/hashid.json, directory/hashid.filetype],…) that is it must take 2 arguments - a json file as input (do not change the input else the file is not recognized any more) and one output file (that executable needs to generate) (if your code does not take json as input you can for example parse json in bash using jq)
Restart Addon (ignore if not used)#
If you intend to use the restart option (by passing a simulation number n>0 to create), the executable is called with subprocess.run([*executable, directory/hashid.json, directory/hashid0xN.filetype, directory/hashid0x(N-1).filetype],…) that is it must take a third argument (the previous simulation)
- exists(js, n=0)[source]#
Check for existence of data
- Parameters:
- Returns:
True if output data corresponding to js exists, False else
- Return type:
- files()[source]#
Return a list of dictionaries (sorted by id and number) with ids and files existing in directory.
The purpose here is to give the user an iterable object to search or tabularize the content of outputfiles
- get_registry()[source]#
Get a dictionary containing the mapping from sha to names
Read the file “simplesimdb.json”
- hashinput(js)[source]#
Hash the input dictionary
Warning
in order to generate a unique identifier,
jsneeds to be normalized in the sense that the datatype must match the required datatype documented (e.g. 10 in a field requiring float is interpreted as an integer and thus produces a different hash)
- jsonfile(js)[source]#
File path to json file from the input
Does not check if the file actually exists
- outfile(js, n=0)[source]#
File path to output file from the input
Do not check if the file actually exists
- recreate(js, n=0, name='', error='raise', stdout='ignore')[source]#
Force a re-simulation:
deletefollowed bycreate- Return type:
- register(js, name)[source]#
Register a human readable name for the given input dictionary
The registry is stored in the file “simplesimdb.json” If the given dictionary already has a name associated to it the name is already in use or the input file exists under a different name an Exception will be raised
- select(js, n=0)[source]#
Select an output file based on its input parameters
Raise a ValueError exception if the file does not exist else it just returns self.outfile( js, n)
Warning
in order to generate a unique identifier js needs to be normalized in the sense that the datatype must match the required datatype documented (e.g. 10 in a field requiring float is interpreted as an integer and thus produces a different hash)
- set_registry(registry)[source]#
Set the registry with a dictionary containing mapping from sha to names
Warning
Use with care as this operation can corrupt your naming scheme!