11.08., 9:00 - 11:00: Due to updates GitLab will be unavailable for some minutes between 09:00 and 11:00.

Commit 974b435a authored by Daniel Lehmberg's avatar Daniel Lehmberg

Squashed 'Tools/SUQController/' changes from f75af9422..2bcc6f3ee

2bcc6f3ee Merge branch 'omnet_add_on_final' into 'master'
ca96ef284 Merge branch 'master' into omnet_add_on_final
34d78229e minor changes
42f00a252 minor changes (pyDOE not global dependency)
980041184 minor changes
2ef30133c corrected vadere server name
5666a686b add vadere server id
db95e7a4e moved file to repo Forschung
1781dbeb3 social dist calibration
99a919b35 refactored
83d944bd7 moved util function into rover class CoupledDictVariation
66bac6af3 read data for succesfull sim runs
3ff758a1a refactored
45d5b59d0 use vadere default seed instead of seed "42"
17dd13cfe get enviromenment_manager info
933181b19 create subset of simulations
108747469 set seeds with omnet ini file
b1c5bee19 only copy folder structure and xml files
ef848d982 use nlevels
08df62298 added some explanations
c3b4d6bc3 Merge branch 'onnet_add_on3' into omnet_add_on_final
639822bc8 Auto stash before merge of "omnet_add_on_final" and "origin/omnet_add_on_final"
c044e1bc3 Apply suggestion to suqc/parameter/sampling.py
733769944 further refactoring
51123ef2d applied some suggestions
6eecb3e8c do not copy cache results
6d70e73a4 Apply suggestion to suqc/environment.py
7629d24b6 Merge branch 'omnet_add_on_final0' into omnet_add_on_final
c2468db5c update
dd8afe40c Update README.md
b6c68b8f2 Update README.md
c852a6d74 Update README.md
1fae57c3d Update README.md
d4d1944c4 Update README.md
4605c5942 Apply suggestion to requirements.txt
ce488ff72 remove output after each simulation run in rover
c4d3c688f implemented AbstractScenarioCreation
8c7f67c1d abstract 2
f14585306 VaderQoI s parent class is Qoi
a7b990113 use scenario file defined in config
a6b02378f moved classed into sampling.py
41776cf94 set runscript to class attribute
4b54ba0fb give qoi to external python script
5039ddf2d added GenerealOutputFile
6a1c75540 use general QoI
34d88a1fc removed waiting
1176810db format code
664225e14 allow multiple runs
101905a4b wrong value
fceb0dbbc final setup of parameter studies
d46f84f2c removed figures
95d0bede4 removed seed
83fd95dca added shadowing study
94300edbd fixed bugs and refactored
b488aae21 extended sampling methods
85086f7ab before introducing new sampling feature
c68f3abb7 repition test
0516249ab some parameterstudies to compare with LTE
872dc90ce use different numbr of repititions
98b8d22e8 refactored for rover sim
015c13f10 adapted postprocessing for Python 3.8
f6da2f347 fixed bug
af30aa71b prepared for FP
55cd34485 added postprocessing
12012a827 added postprocessing
6e9565929 added utility for vector parameters
419bd6d74 added dependent parameters
ff94bcd59 read qoi
606324287 changed scenario
5951e9187 no parralel runs
e066af5e9 produce error in deep_look_up
8f9822a69 extended framework with abstract consolewrapper
97c6bc77d moved methods in abstract class
5f50ddecc sorted scripts
f588d5ed1 also parse inifiles
4b34227b4 create rover structure in output folder
577133bac minor: add TODO
d5114d6a7 Merge branch 'delete_me' into test
c4d7b7cbf added classes and tried to access data
216dc10b6 Merge branch 'CM' into 'master'
9fa54dc33 added forgotten if query for reading data
824d6f9cb adapt read from existing output
791e3e13b apply black and isort
9ff248e3d Merge branch 'master' into omnet_add_on (apply black isort)
ea9e16349 apply black and isort
64d7b4afc Merge remote-tracking branch 'origin/omnet_add_on' into omnet_add_on
1623c7f3e created sampling with units
c4f62c728 add ConfigParser for OMNeT++ ini files.
b9b446001 Merge remote-tracking branch 'origin/master'
f62bf74b3 scenario runs for FolderExistScenarios

git-subtree-dir: Tools/SUQController
git-subtree-split: 2bcc6f3eeb520df2c455a112016b86fda95cc607
parent 7a9937ed
......@@ -45,3 +45,27 @@ not installed successfully.
### Introduction
See [SRC_PATH]/tutorial
#### Using SUQC and Vadere
Here a few hints for your .scenario file for Vadere:
1. ScenarioChecker
Before running your scenario automatically on suqc, activate the ``ScenarioChecker`` (Project > Activate ScenarioChecker) and run it in the ``VadereGui``.
The ScenarioChecker will point out potential problems with your scenario file.
2. Default parameters
Make sure to set ``realTimeSimTimeRatio`` to 0.0. (Values > 0.0 slow down the simulation for the visualisation)
Another point that may cost a lot of computation time is the ``optimizationType``, consider using ``DISCRETE`` (discrete optimization) instead of ``NELDER_MEAD``. Please note, that ``varyStepDirection`` should always be activated with discrete optimization.
Remove ``attributesCar`` from the .scenario file if you are not using any vehicles to avoid confusion of attributes.
3. Visual check
Visually check the results of your simulation, maybe check upper and lower parameter bounds.
4. Clean topography
Remove elements in your topography that are not used. Sometimes through the interaction with the mouse, tiny obstacles or targets are created unintentionally.
Check the elements in your topography, you can take a look at the ``ElementTree`` in the Topography creator tab. Remove all elements that are unused, especially focusing on targets.
5. Data processors
Remove all data processors and output files that you don't use. In particular, remove the overlap processors, they are intended for testing purposes.
6. Reproducibility
Make sure that your runs are reproducible - work with a ``fixedSeed`` by activating ``useFixedSeed`` or save all the ``simulationSeed``s that have been used.
(Another way is to provide a ``fixedSeed`` for each runs with suqc, in this case make sure that ``useFixedSeed`` is true.)
setuptools>=39.1.0
fabric==2.4.0
pandas>=0.24.0,<1.0.0
pandas>=1.0.3
numpy>=1.17.0
scikit-learn>=0.21.0
......@@ -2,15 +2,15 @@
import os
from setuptools import setup, find_packages
from setuptools import find_packages, setup
from suqc.configuration import SuqcConfig
from suqc import __version__
from suqc.configuration import SuqcConfig
# To generate a new requirements.txt file run in console (install vis 'pip3 install pipreqs'):
# pipreqs --use-local --force /home/daniel/REPOS/suq-controller
with open('requirements.txt', "r") as f:
with open("requirements.txt", "r") as f:
requirements = f.read().splitlines()
# Writes a file that gives information about the version such that "suqc.__version__" provides the current version,
......@@ -27,7 +27,7 @@ setup(
url="www.vadere.org",
packages=find_packages(),
install_requires=requirements,
data_files=[('suqc', ["suqc/PACKAGE.txt"])]
data_files=[("suqc", ["suqc/PACKAGE.txt"])],
)
os.remove(SuqcConfig.path_package_indicator_file())
......
......@@ -5,4 +5,4 @@ from suqc.parameter.postchanges import PostScenarioChangesBase
from suqc.qoi import *
from suqc.request import *
__version__ = "2.0"
__version__ = "2.1"
#!/usr/bin/env python3
import os
import json
import os
import os.path as p
import pathlib
# configuration of the suq-controller
DEFAULT_SUQC_CONFIG = {"default_vadere_src_path": "TODO", # TODO Feature: #25
"server": {
"host": "",
"user": "",
"port": -1}}
DEFAULT_SUQC_CONFIG = {
"default_vadere_src_path": "TODO", # TODO Feature: #25
"server": {"host": "", "user": "", "port": -1},
}
def check_setup(_paths_class):
if not os.path.exists(_paths_class.path_cfg_folder()) and _paths_class.is_package_paths():
if (
not os.path.exists(_paths_class.path_cfg_folder())
and _paths_class.is_package_paths()
):
print(f"INFO: Setting up configuration folder {_paths_class.path_cfg_folder()}")
# the next two checks will fail automatically too, because the folder is empty
os.mkdir(_paths_class.path_cfg_folder())
if not os.path.exists(_paths_class.path_suq_config_file()):
print(f"INFO: Setting up default configuration file located at {_paths_class.path_suq_config_file()}")
print(
f"INFO: Setting up default configuration file located at "
f"{_paths_class.path_suq_config_file()}"
)
_paths_class.store_config(DEFAULT_SUQC_CONFIG)
if not os.path.exists(_paths_class.path_container_folder()):
print(f"INFO: Setting up the default container path (which will store output of simulation runs). "
f"Location {_paths_class.path_container_folder()}")
print(
f"INFO: Setting up the default container path "
f"(which will store output of simulation runs). "
f"Location {_paths_class.path_container_folder()}"
)
os.mkdir(_paths_class.path_container_folder())
return _paths_class
......
This diff is collapsed.
This diff is collapsed.
import os
import unittest
from suqc.opp.config_parser import OppConfigFileBase, OppConfigType
class OppConfigFileBaseTest(unittest.TestCase):
NEW_FILE = "omnetpp_2.ini"
NEW_FILE_COMP = "omnetpp_compare.ini"
NEW_FILE_DEL = "omnetpp_del.ini"
@staticmethod
def get_object(
config, cfg_type=OppConfigType.EDIT_LOCAL, is_parent=False, path="omnetpp.ini"
):
return OppConfigFileBase.from_path(
os.path.join(os.path.split(__file__)[0], path), config, cfg_type, is_parent
)
@staticmethod
def save_object(obj: OppConfigFileBase, path):
with open(os.path.join(os.path.split(__file__)[0], path), "w") as f:
obj.writer(f)
@staticmethod
def get_lines(path):
with open(os.path.join(os.path.split(__file__)[0], path), "r") as f:
return f.readlines()
def tearDown(self) -> None:
f = os.path.join(os.path.split(__file__)[0], self.NEW_FILE)
if os.path.exists(f):
os.remove(f)
def test_set_default_exits(self):
opp = self.get_object("HighTrafficSettings", OppConfigType.READ_ONLY)
# set default on exiting must work
self.assertEqual(opp.setdefault("opt_3", '"val_3"'), '"val_3"')
# set new will ignore default on existing
self.assertEqual(opp.setdefault("opt_3", '"new"'), '"val_3"')
def test_set_default_not_exits(self):
opp = self.get_object("HighTrafficSettings", OppConfigType.READ_ONLY)
# must raise error on OppConfigType.READ_ONLY
self.assertRaises(NotImplementedError, opp.setdefault, "new_key", "42")
opp = self.get_object("HighTrafficSettings", OppConfigType.EDIT_LOCAL)
# must raise error on OppConfigType.EDIT_LOCAL
self.assertRaises(NotImplementedError, opp.setdefault, "new_key", "42")
opp = self.get_object("HighTrafficSettings", OppConfigType.EXT_DEL_LOCAL)
# must work on OppConfigType.EXT_DEL_LOCAL
ret = opp.setdefault("new_key", "42")
self.assertEqual(ret, "42")
self.assertEqual(opp["new_key"], "42")
# new key must be local
self.assertTrue(opp.is_local("new_key"))
def test_read_only(self):
opp = self.get_object("HighTrafficSettings", OppConfigType.READ_ONLY)
# reading must work
self.assertEqual(opp["opt_3"], '"val_3"')
self.assertEqual(opp["general_option"], '"VAL1"')
# setting new values must not work for (local and parent options)
self.assertRaises(NotImplementedError, opp.__setitem__, "opt_3", '"new_val"')
self.assertRaises(
NotImplementedError, opp.__setitem__, "general_option", '"new_val"'
)
def test_edit_local(self):
opp = self.get_object("HighTrafficSettings", OppConfigType.EDIT_LOCAL)
# reading must work
self.assertEqual(opp["opt_3"], '"val_3"')
self.assertEqual(opp["general_option"], '"VAL1"')
# setting new values must work for local options only
opp["opt_3"] = '"new_val"'
self.assertEqual(opp["opt_3"], '"new_val"')
# general_option belongs to parent config 'General'
self.assertRaises(
NotImplementedError, opp.__setitem__, "general_option", '"new_val"'
)
def test_hierarchy(self):
""" Ensure correct lookup order for extended configurations"""
opp = self.get_object("SlottedAloha2b", OppConfigType.EDIT_LOCAL)
self.assertListEqual(
opp.section_hierarchy,
[
"Config SlottedAloha2b",
"Config SlottedAloha2",
"Config SlottedAlohaBase",
"Config HighTrafficSettings",
"General",
],
)
opp = self.get_object("SlottedAloha1", OppConfigType.EDIT_LOCAL)
self.assertListEqual(
opp.section_hierarchy,
[
"Config SlottedAloha1",
"Config SlottedAlohaBase",
"Config LowTrafficSettings",
"General",
],
)
opp = self.get_object("General", OppConfigType.EDIT_LOCAL)
self.assertListEqual(opp.section_hierarchy, ["General"])
opp = self.get_object("SlottedAlohaBase", OppConfigType.EDIT_LOCAL)
self.assertListEqual(
opp.section_hierarchy, ["Config SlottedAlohaBase", "General"]
)
def test_override(self):
opp = self.get_object("SlottedAloha2", OppConfigType.EXT_DEL_LOCAL)
opp["opt_5"] = '"val_55"'
self.save_object(opp, self.NEW_FILE)
opp2 = opp = self.get_object(
"SlottedAloha2", OppConfigType.EXT_DEL_LOCAL, path=self.NEW_FILE
)
self.assertEqual(opp2["opt_5"], '"val_55"')
lines_new = [
line for line in self.get_lines(self.NEW_FILE) if not line.startswith("\n")
]
lines_comp = [
line
for line in self.get_lines(self.NEW_FILE_COMP)
if not line.startswith("\n")
]
self.assertListEqual(lines_new, lines_comp)
def test_safe_to_file(self):
sa_2 = self.get_object("SlottedAloha2", OppConfigType.EXT_DEL_LOCAL)
hts = self.get_object("HighTrafficSettings", OppConfigType.EXT_DEL_LOCAL)
self.assertEqual(sa_2["opt_HT"], '"overwritten_val_HT"')
self.assertEqual(
sa_2["general_option"],
'"general_option option overwritten by SlottedAloha2"',
)
self.assertEqual(hts["opt_HT"], '"val_HT"')
self.assertEqual(hts["general_option"], '"VAL1"')
def test_delete_key(self):
opp = self.get_object("SlottedAloha2", OppConfigType.EXT_DEL_LOCAL)
self.assertEqual(
opp["general_option"],
'"general_option option overwritten by SlottedAloha2"',
)
del opp["general_option"]
# after deletion value from General section must be accessible.
self.assertEqual(opp["general_option"], '"VAL1"')
self.save_object(opp, self.NEW_FILE)
lines_new = [
line for line in self.get_lines(self.NEW_FILE) if not line.startswith("\n")
]
lines_del = [
line
for line in self.get_lines(self.NEW_FILE_DEL)
if not line.startswith("\n")
]
self.assertListEqual(lines_new, lines_del)
[General]
general_option = "VAL1"
[Config SlottedAlohaBase]
opt_1 = "val_1"
[Config LowTrafficSettings]
opt_2 = "val_2"
[Config HighTrafficSettings]
opt_3 = "val_3"
opt_HT = "val_HT"
[Config SlottedAloha1]
extends = SlottedAlohaBase, LowTrafficSettings
opt_4 = "val_4"
[Config SlottedAloha2]
extends = SlottedAlohaBase, HighTrafficSettings
opt_5 = "val_5"
opt_HT = "overwritten_val_HT"
general_option = "general_option option overwritten by SlottedAloha2"
[Config SlottedAloha2a]
extends = SlottedAloha2
[Config SlottedAloha2b]
extends = SlottedAloha2
opt_6 = "val_6"
[General]
general_option = "VAL1"
[Config SlottedAlohaBase]
opt_1 = "val_1"
[Config LowTrafficSettings]
opt_2 = "val_2"
[Config HighTrafficSettings]
opt_3 = "val_3"
opt_HT = "val_HT"
[Config SlottedAloha1]
extends = SlottedAlohaBase, LowTrafficSettings
opt_4 = "val_4"
[Config SlottedAloha2]
extends = SlottedAlohaBase, HighTrafficSettings
opt_5 = "val_55"
opt_HT = "overwritten_val_HT"
general_option = "general_option option overwritten by SlottedAloha2"
[Config SlottedAloha2a]
extends = SlottedAloha2
[Config SlottedAloha2b]
extends = SlottedAloha2
opt_6 = "val_6"
[General]
general_option = "VAL1"
[Config SlottedAlohaBase]
opt_1 = "val_1"
[Config LowTrafficSettings]
opt_2 = "val_2"
[Config HighTrafficSettings]
opt_3 = "val_3"
opt_HT = "val_HT"
[Config SlottedAloha1]
extends = SlottedAlohaBase, LowTrafficSettings
opt_4 = "val_4"
[Config SlottedAloha2]
extends = SlottedAlohaBase, HighTrafficSettings
opt_5 = "val_5"
opt_HT = "overwritten_val_HT"
[Config SlottedAloha2a]
extends = SlottedAloha2
[Config SlottedAloha2b]
extends = SlottedAloha2
opt_6 = "val_6"
......@@ -8,7 +8,6 @@ from suqc.utils.dict_utils import change_dict
class PostScenarioChangesBase(object):
def __init__(self, apply_default=False):
self._apply_scenario_changes = {}
......@@ -22,28 +21,38 @@ class PostScenarioChangesBase(object):
self.add_scenario_change(AlwaysEnableMetaData())
self.add_scenario_change(ChangeDescription())
def add_scenario_change(self, scenario_change: 'PostScenarioChange'):
def add_scenario_change(self, scenario_change: "PostScenarioChange"):
# ABCScenarioChange in '' to support forward reference,
# see https://www.python.org/dev/peps/pep-0484/#forward-references
if scenario_change.name in self._apply_scenario_changes.keys():
raise KeyError(f"Scenario change with {scenario_change.name} is already present.")
raise KeyError(
f"Scenario change with {scenario_change.name} is already present."
)
self._apply_scenario_changes[scenario_change.name] = scenario_change
def _collect_changes(self, scenario, parameter_id, run_id, parameter_variation):
changes = {}
for chn in self._apply_scenario_changes.values():
changes.update(chn.get_changes_dict(scenario=scenario,
changes.update(
chn.get_changes_dict(
scenario=scenario,
parameter_id=parameter_id,
run_id=run_id,
parameter_variation=parameter_variation))
parameter_variation=parameter_variation,
)
)
return changes
def change_scenario(self, scenario, parameter_id, run_id, parameter_variation):
return change_dict(scenario, changes=self._collect_changes(scenario, parameter_id, run_id, parameter_variation))
return change_dict(
scenario,
changes=self._collect_changes(
scenario, parameter_id, run_id, parameter_variation
),
)
class PostScenarioChange(metaclass=abc.ABCMeta):
def __init__(self, name):
self.name = name
......@@ -61,12 +70,15 @@ class AlwaysEnableMetaData(PostScenarioChange):
class ChangeRealTimeSimTimeRatio(PostScenarioChange):
def __init__(self):
super(ChangeRealTimeSimTimeRatio, self).__init__(name="real_time_sim_time_ratio")
super(ChangeRealTimeSimTimeRatio, self).__init__(
name="real_time_sim_time_ratio"
)
def get_changes_dict(self, scenario, parameter_id, run_id, parameter_variation):
return {"realTimeSimTimeRatio": 0.0} # Speeds up the non-visual computations in case a ratio was set!
return {
"realTimeSimTimeRatio": 0.0
} # Speeds up the non-visual computations in case a ratio was set!
class ChangeRandomNumber(PostScenarioChange):
......@@ -75,7 +87,9 @@ class ChangeRandomNumber(PostScenarioChange):
KEY_SIM_SEED = "simulationSeed"
def __init__(self, fixed=False, randint=False, par_and_run_id=False):
assert fixed + randint + par_and_run_id == 1, "Exactly one parameter has to be set to true"
assert (
fixed + randint + par_and_run_id == 1
), "Exactly one parameter has to be set to true"
self._isfixed = fixed
self._fixed_randnr = None
......@@ -91,17 +105,21 @@ class ChangeRandomNumber(PostScenarioChange):
def get_changes_dict(self, scenario, parameter_id, run_id, parameter_variation):
if self._isfixed:
assert self._fixed_randnr is not None, "Fixed random number has to be set with method set_fixed_random_nr"
assert (
self._fixed_randnr is not None
), "Fixed random number has to be set with method set_fixed_random_nr"
rnr = self._fixed_randnr
elif self._israndint:
# 4294967295 = max unsigned 32 bit integer
rnr = np.random.randint(0, 4294967295)
else: # --> self._isparid
rnr = (parameter_id * 1E6 + run_id) # the 1E6 is required to not have
rnr = parameter_id * 1e6 + run_id # the 1E6 is required to not have
return {ChangeRandomNumber.KEY_FIXED: True,
return {
ChangeRandomNumber.KEY_FIXED: True,
ChangeRandomNumber.KEY_SEED: rnr,
ChangeRandomNumber.KEY_SIM_SEED: rnr}
ChangeRandomNumber.KEY_SIM_SEED: rnr,
}
class ChangeScenarioName(PostScenarioChange):
......@@ -125,6 +143,11 @@ class ChangeDescription(PostScenarioChange):
super(ChangeDescription, self).__init__(name="description")
def get_changes_dict(self, scenario, parameter_id, run_id, parameter_variation):
changes_in_description = " ".join(["applied parameter variation=", str(parameter_variation)])
return {ChangeDescription.KEY_DESCRIPTION: "--".join([f"par_id={parameter_id} and run_id={run_id}",
changes_in_description])}
changes_in_description = " ".join(
["applied parameter variation=", str(parameter_variation)]
)
return {
ChangeDescription.KEY_DESCRIPTION: "--".join(
[f"par_id={parameter_id} and run_id={run_id}", changes_in_description]
)
}
......@@ -2,19 +2,22 @@
import os
import re
import time
from typing import *
import pandas as pd
from suqc.environment import EnvironmentManager
from suqc.environment import VadereEnvironmentManager
from suqc.utils.dict_utils import deep_dict_lookup
class FileDataInfo(object):
# Implemented in Vadere merge request !38, this is only a fallback mode and requires manual updating if there are
# changes in Vadere. See also vadere issue #199 and #201.
map_outputtype2index = {"IdOutputFile": 1,
# Implemented in Vadere merge request !38, this is only a fallback mode and requires
# manual updating if there are changes in Vadere. See also Vadere issue #199 and #201.
# all output types are from Vadere BUT "GeneralOutputFile"
map_outputtype2index = {
"IdOutputFile": 1,
"LogEventOutputFile": 1,
"NoDataKeyOutputFile": 0,
"PedestrianIdOutputFile": 1,
......@@ -22,13 +25,20 @@ class FileDataInfo(object):
"TimestepPedestrianIdOutputFile": 2,
"TimestepPedestrianIdOverlapOutputFile": 3,
"TimestepPositionOutputFile": 3,
"TimestepRowOutputFile": 2}
"TimestepRowOutputFile": 2,
"GeneralOutputFile": 1,
}
printFallbackMsg = False
def __init__(self, process_file, processors):
def __init__(
self, process_file, processors=None, outputkey=None,
):
self.filename = process_file["filename"]
if outputkey is None:
self.output_key = process_file["type"].split(".")[-1]
else:
self.output_key = outputkey
self.processors = processors # not really needed yet, but maybe in future.
try:
......@@ -36,72 +46,39 @@ class FileDataInfo(object):
except KeyError:
if not self.printFallbackMsg:
self.printFallbackMsg = True
print(f"WARNING: file type {self.output_key} was not found in list, this may require an update. Setting "
f"number of index columns to 1.")
print(
f"WARNING: file type {self.output_key} was not found in list, this may require an update. Setting "
f"number of index columns to 1."
)
self.nr_row_indices = 1 # use simply first column as index
class QuantityOfInterest(object):
def __init__(self, basis_scenario: dict, requested_files: Union[List[str], str]):
def __init__(self, requested_files: Union[List[str], str]):
assert isinstance(requested_files, (list, str))
if isinstance(requested_files, str):
requested_files = [requested_files]
user_set_writers, _ = deep_dict_lookup(basis_scenario, "processWriters")
process_files = user_set_writers["files"]
processsors = user_set_writers["processors"]
self.req_qois = self._requested_qoi(requested_files, process_files, processsors)
def _requested_qoi(self, requested_files, process_files, processsors):
self.req_qois = self._requested_qoi(requested_files)
def _requested_qoi(self, requested_files):
req_qois = list()
for pf in process_files:
for rf in requested_files:
pf = dict()
pf["filename"] = rf
pf["type"] = rf
# TODO: This has to exactly match, maybe make more robust to allow without the file-ending
filename = pf["filename"] # TODO: see issue #33
if filename in requested_files:
sel_procs = self._select_corresp_processors(pf, processsors)
req_qois.append(FileDataInfo(process_file=pf, processors=sel_procs))
requested_files.remove(filename) # -> processed, list should be empty when leaving function
if requested_files: # has to be empty
raise ValueError(f"The requested files {requested_files} are not set in the Vadere scenario: \n "
f"{process_files}")
req_qois.append(
FileDataInfo(process_file=pf, outputkey="GeneralOutputFile")
)
return req_qois
def _select_corresp_processors(self, process_file, processors):
proc_ids = process_file["processors"]
selected_procs = list()
# TODO: see issue #33
for pid in proc_ids:
found = False
for p in processors:
if pid == p["id"]:
selected_procs.append(p)
if not found:
found = True
else:
raise ValueError("The Vadere scenario is not correctly set up! There are two processors with "
f"the id={pid}.")
if not found:
raise ValueError(f"The Vadere scenario is not correctly set up! Processor id {pid} could not be found "
"in 'processors'.")
return selected_procs
def _read_csv(self, req_qoi: FileDataInfo, filepath):
# make sure that Vadere writes
with open(filepath) as f:
first_line = f.readline()
......@@ -140,12 +117,92 @@ class QuantityOfInterest(object):
for k in self.req_qois:
filepath = os.path.join(output_path, k.filename)
df_data = self._read_csv(k, filepath)
read_data[k.filename] = self._add_parid2idx(df_data, par_id, run_id) # filename is identifier for QoI
read_data[k.filename] = self._add_parid2idx(
df_data, par_id, run_id
) # filename is identifier for QoI
return read_data
class VadereQuantityOfInterest(QuantityOfInterest):
def __init__(self, basis_scenario: dict, requested_files: Union[List[str], str]):
assert isinstance(requested_files, (list, str))
if isinstance(requested_files, str):
requested_files = [requested_files]
user_set_writers, _ = deep_dict_lookup(basis_scenario, "processWriters")
self.process_files = user_set_writers["files"]
self.processsors = user_set_writers["processors"]
super().__init__(requested_files)
def get_process_files(self):
return self.process_files
def get_processors(self):
return self.processsors
def _requested_qoi(self, requested_files):
req_qois = list()
process_files = self.get_process_files()
processsors = self.get_processors()
for pf in process_files:
# TODO: This has to exactly match, maybe make more robust to allow without the file-ending
filename = pf["filename"] # TODO: see issue #33