ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2019-03-21T10:47:08+01:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/245Toolkit2: Allow users to share binaries, move them accross computers, etc.2019-03-21T10:47:08+01:00Ghost UserToolkit2: Allow users to share binaries, move them accross computers, etc.The new ExaHyPE Parser C++ infrastructure currently does a `suprocess` to call the Toolkit in order to convert old-fashioned specification files. To do so, the call to `Toolkit/toolkit.sh` is hardcoded, including the path.
This raises p...The new ExaHyPE Parser C++ infrastructure currently does a `suprocess` to call the Toolkit in order to convert old-fashioned specification files. To do so, the call to `Toolkit/toolkit.sh` is hardcoded, including the path.
This raises problems in certain use cases:
* The user shares his Executable to another user on the system/cluster but does not make the access permissions correctly for the Toolkit
* The user copies only the Executable to another machine but not the overall Code (especially not the Toolkit code)
* The user renames the path to his installation (probably when cleaning up his home directory) but wants to keep an ExaHyPE build working (obviously this is not possible)
We have no solution for all these use cases, but we should include checks in the C++ code to deal with them:
* Check whether `Path/to/Toolkit/toolkit.sh` exists, is readable and executablehttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/242Toolkit2: Fix old merge2019-03-21T10:46:28+01:00Ghost UserToolkit2: Fix old mergehttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/d92031f56b4bf6b6ba92f1d774cc9271ec6d13c5#note_215226
@Sven: Check whether this is done or not.
Two changes I did and you unfortunately undid with your merge (should be easy to fix...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/d92031f56b4bf6b6ba92f1d774cc9271ec6d13c5#note_215226
@Sven: Check whether this is done or not.
Two changes I did and you unfortunately undid with your merge (should be easy to fix though):
* Make sure you are using `ValueError` instead of `JSONDecodeError(ValueError)`.
Otherwise, the code requires python3 version>=3.5.0.
This is a typical error message you get:
```
controller.py:215(getSpec):ERROR Could not read specification file '../Euler_ADERDG.exahype': 'module' object has no attribute 'JSONDecodeError'
```
* Make sure `mexa.py` is python3.
I regex-replaced
`print (".+"\s*(%\s*\(.+\))?)$` with `print(\1)`https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/274Toolkit: add flux limiter choice to FV solvers2019-03-20T09:31:43+01:00Ghost UserToolkit: add flux limiter choice to FV solversThis is basically low priority because we have a working solution on the [minmod branch](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/minmod): A replacement of the `minmod` flux limiter in the 2nd order FV scheme with something more...This is basically low priority because we have a working solution on the [minmod branch](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/minmod): A replacement of the `minmod` flux limiter in the 2nd order FV scheme with something more sophisticated.
However, users need to be able to choose which flux limiter they want to have with a FV solver. Therefore, the task of this issue is the integration into the ExaHyPE options language, i.e. implement what is discussed in the comments of https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/bc9063c5d8dc9ccb77eae346b00ff8f676808353:
> Thanks for the comment, @domcha. I think the way to go is to implement the *flux limiter* as a choice in the FV solver section in the toolkit, and to call a `solver` method right in the `kernels::finitevolumes::musclhancock::c::solutionUpdate` in place of the former `minmod` function. Choices would be then: `minmod`, `koren`, `user-defined`, where the latter creates a function stub in the generated user FV solver. Toolkit code generation as usual.
>
> I will implement this in a seperate feature branch and merge it then into various branches (such as `master`, as well as Lukes current working branch, i.e. the `minmod` branch).
>
> One principal thing is that this function is scalar, while it probably should be vectorized. That is easily possible with `inline` functions, but not any more with `user-defined` ones. But I don't think there is an urgent need for `user-defined` anyway. Maybe we can just offer a choice. Adding another one should then be an easy task.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/152Dynamic plotter registration2019-03-07T19:39:00+01:00Ghost UserDynamic plotter registrationUsers shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plot...Users shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plotter registration function (semi pseudocode)
```c++
Writer* kernels::getNamedWriter(std::string name, Solver& solver) {
if(name == "ConservedWriter") return new GRMHD::ConservedWriter(*static_cast<exahype::solvers::LimitingADERDGSolver*>(solver));
if(name == "usw") return new GRMHD::SomeOtherWriter(*static_cast<exahype::solvers::ADERDGSolver*>(solver));
if(name == ...
else
failure: Dont now this plotter type.
}
```
We then can replace the generated current section
```c++
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,0,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,1,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,2,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,3,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,4,parser,new GRMHD::IntegralsWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
```
with a non-generated section (pseudocode)
```c+++
for(const int& solvernum : Parser->getSolvers()) {
int plotternum=0;
for( const string& plottername : Parser->getPlotterNamesForSolver(solvernum)) {
exahype::plotters::RegisteredPlotters.push_back(new exahype::plotters::Plotter(solvernum,plotternum++,parser,getNamedWriter(plottername, exahype::solvers::RegisteredSolvers[solvernum]));
}
}
```
This is something I can do on my own. If the Parser gives the neccessary data...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/252Toolkit validation shows non-deterministic behaviour2018-09-21T11:30:46+02:00Ghost UserToolkit validation shows non-deterministic behaviourI discovered that the toolkit validation shows non-deterministic behaviour.
I use a specification file where I remove the 'cores' option which is required
in the 'shared_memory' section.
The validation does not complain in some runs but...I discovered that the toolkit validation shows non-deterministic behaviour.
I use a specification file where I remove the 'cores' option which is required
in the 'shared_memory' section.
The validation does not complain in some runs but fails correctly in other
runs.
1. It works the first time (it shouldn't as the 'shared_memory/core' is missing.
```
17:14 $ ../../../Toolkit/toolkit.sh -v -format=any ../Euler_ADERDG.exahype
controller.py:77(__init__):INFO
______ __ __ ____ ______
/ ____/ ______ _/ / / /_ __/ __ \/ ____/ *************************
/ __/ | |/_/ __ `/ /_/ / / / / /_/ / __/ The ExaHyPE Toolkit v2
/ /____> </ /_/ / __ / /_/ / ____/ /___ www.exahype.eu
/_____/_/|_|\__,_/_/ /_/\__, /_/ /_____/ Commit: 77efa02
/____/ *************************
This project has received funding from the European Union's Horizon 2020
research and innovation programme under grant agreement No 671698 (ExaHyPE).
Copyright (c) 2018, All rights reserved
ExaHyPE is based on the PDE framework Peano (www.peano-framework.org).
Released under the BSD 3 Open Source License.
controller.py:85(__init__):INFO Read input file /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/ApplicationExamples/Euler/Euler_ADERDG.exahype.
omni_reader.py:46(__init__):INFO 4 file formats registered: json,hjson,exahype-v1,mexa
omni_reader.py:156(read):INFO No specific file format requested, trying all available file formats in order.
omni_reader.py:124(read_omni):INFO Trying to read file format json
omni_reader.py:136(read_omni):INFO The input file is certainly not written in the format json as a ParserError occured: Expecting value: line 1 column 1 (char 0)
omni_reader.py:124(read_omni):INFO Trying to read file format hjson
omni_reader.py:130(read_omni):INFO Cannot check file format hjson because neccessary library is not installed. The missing library is: No module named 'hjson'
omni_reader.py:131(read_omni):INFO Will silently ignore this problem
omni_reader.py:124(read_omni):INFO Trying to read file format exahype-v1
specfile1_reader.py:521(read_lines):INFO Converting legacy specification file (95 lines) to INI file...
specfile1_reader.py:523(read_lines):INFO OK
specfile1_reader.py:532(read_lines):INFO Mapping INI structure to JSON input object...
specfile1_reader.py:534(read_lines):INFO OK
omni_reader.py:126(read_omni):INFO Success reading file format exahype-v1
directories.py:37(check):INFO Peano kernel path: /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/Peano ... ok
directories.py:37(check):INFO Peano kernel path: /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/Peano holds tarch ... ok
[...]
/Euler_ADERDG/Makefile'
controller.py:152(run):INFO please change into directory /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ApplicationExamples/Euler/Euler_ADERDG and type make
ensure that you set all environment variables before:
export COMPILER=GNU Select GNU compiler
export COMPILER=Intel Select Intel compiler (default)
[...]
export USE_IPO=on Compile using IPO for the solvers and kernels (only with Intel compilers)
If USE_IPO is not specified, it falls back to "off" (compilation without IPO).
If you run CSH, please replace "export ARG=VALUE" with "setenv ARG VALUE".
```
2. It fails the second time which is the correct behaviour:
```
✔ ~/dev/codes/c/upstream/ExaHyPE-Engine/ApplicationExamples/Euler/Euler_ADERDG [master|✚ 16…1135⚑ 58]
17:18 $ ../../../Toolkit/toolkit.sh -v -format=any ../Euler_ADERDG.exahype
rm: cannot remove ‘*.o’: No such file or directory
rm: cannot remove ‘cipofiles.mk’: No such file or directory
rm: cannot remove ‘ffiles.mk’: No such file or directory
rm: cannot remove ‘cfiles.mk’: No such file or directory
rm: cannot remove ‘kernels’: No such file or directory
controller.py:77(__init__):INFO
______ __ __ ____ ______
/ ____/ ______ _/ / / /_ __/ __ \/ ____/ *************************
/ __/ | |/_/ __ `/ /_/ / / / / /_/ / __/ The ExaHyPE Toolkit v2
/ /____> </ /_/ / __ / /_/ / ____/ /___ www.exahype.eu
/_____/_/|_|\__,_/_/ /_/\__, /_/ /_____/ Commit: 77efa02
/____/ *************************
This project has received funding from the European Union's Horizon 2020
research and innovation programme under grant agreement No 671698 (ExaHyPE).
Copyright (c) 2018, All rights reserved
ExaHyPE is based on the PDE framework Peano (www.peano-framework.org).
Released under the BSD 3 Open Source License.
controller.py:85(__init__):INFO Read input file /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/ApplicationExamples/Euler/Euler_ADERDG.exahype.
omni_reader.py:46(__init__):INFO 4 file formats registered: json,hjson,exahype-v1,mexa
omni_reader.py:156(read):INFO No specific file format requested, trying all available file formats in order.
omni_reader.py:124(read_omni):INFO Trying to read file format json
omni_reader.py:136(read_omni):INFO The input file is certainly not written in the format json as a ParserError occured: Expecting value: line 1 column 1 (char 0)
omni_reader.py:124(read_omni):INFO Trying to read file format hjson
omni_reader.py:130(read_omni):INFO Cannot check file format hjson because neccessary library is not installed. The missing library is: No module named 'hjson'
omni_reader.py:131(read_omni):INFO Will silently ignore this problem
omni_reader.py:124(read_omni):INFO Trying to read file format exahype-v1
specfile1_reader.py:521(read_lines):INFO Converting legacy specification file (95 lines) to INI file...
specfile1_reader.py:523(read_lines):INFO OK
specfile1_reader.py:532(read_lines):INFO Mapping INI structure to JSON input object...
specfile1_reader.py:534(read_lines):INFO OK
omni_reader.py:126(read_omni):INFO Success reading file format exahype-v1
controller.py:218(validateAndSetDefaults):ERROR Specification file does not hold a valid ExaHyPE specification, it did not pass the schema validation step. The error message is: 'cores' is a required property
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/153Constants in specfiles2018-09-10T19:42:06+02:00Ghost UserConstants in specfilesI have too much constants that a specfile syntax
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size...I have too much constants that a specfile syntax
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = initial_data:tovstar,boundary_x_lower:reflective,boundary_y_lower:reflective,boundary_z_upper:outgoing,tovstar-mass:1.234,tovstar-rl-ratio:2.345
```
would make sense. Tobias thinks in such a case a user would do something like
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = configuration-file:foobar.txt
```
and outsource the configuration in a language the user likes. However, this breaks with the idea of a single specfile for a single run. Therefore, I see two options
## Add a DATA section after the specfile
This is what many script languages do, for instance perl ([the DATA syntax in perl](https://stackoverflow.com/questions/13463509/the-data-syntax-in-perl)). The idea is just that the parsers ignore what goes after the end of the specfile (ie. the line containing `end exahype-project`). Users could dump there any content in their favourite language. I would vote for this as it is super easy to implement, allows file concatenation and flexibility.
## Allow user constant section
We could also just allow users to do something like
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = parameters:appended
limiter-kernel const = generic::musclhancock
limiter-language const = C
dmp-observables = 2
dmp-relaxation-parameter = 1e-2
dmp-difference-scaling = 1e-3
steps-till-cured = 0
simulation-parameters
foo = bar
baz = bar
blo = bar
blu = bar
etc.
end simulation-parameters
plot vtk::Cartesian::vertices::limited::binary ConservedWriter
variables const = 19
time = 0.0
repeat = 0.00166667
output = ./vtk-output/conserved
end plot
...
```
This would go well with the specfile syntax. In order to implement, we need
* Such a section with any key-value pairs added to the grammar, so the toolkit does not complain
* Support in the `Parser.cpp` (which is not too hard to add)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/157Plotter variables are not constants2018-09-10T19:41:33+02:00Ghost UserPlotter variables are not constantsActually the ExaHyPE runtime treats the number of writtenUnknowns per plotter as a runtime variable, ie.
```
plot hdf5::flash ConservedWriter
variables = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/c...Actually the ExaHyPE runtime treats the number of writtenUnknowns per plotter as a runtime variable, ie.
```
plot hdf5::flash ConservedWriter
variables = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/conserved
end plot
```
one can change `variables = x` without recompiling. However, the toolkit wants this to be a constant:
```
plot hdf5::flash ConservedWriter
variables const = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/conserved
end plot
```
otherwise it says `ERROR: eu.exahype.parser.ParserException: [71,17] expecting: 'const'`
This should not happen, ie. the toolkit grammar should accept without `const`.
As always, there is assumably no case when somebody wants to do this **except benchmarking plotter file formats which is exactly what I'm doing now** and the typical generated code by the toolkit is not aware of a non-constexpr number of variables, but all the `ExaHyPE/exahype/plotters/` code actually treats the number as runtime constant and I see no reason why to artificially introduce something constexpr here.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/239Toolkit2: Be consistent with interactivity2018-09-10T19:38:15+02:00Ghost UserToolkit2: Be consistent with interactivityThe toolkit.sh is blocking for user input if dependency problems occur. Will be fatal when called from Engine. Simple solution: Take --interactive etc flags into account.
Can fix that.The toolkit.sh is blocking for user input if dependency problems occur. Will be fatal when called from Engine. Simple solution: Take --interactive etc flags into account.
Can fix that.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/237Toolkit extensibility threatened by java method size limit2018-09-10T19:37:48+02:00Ghost UserToolkit extensibility threatened by java method size limitProblem description
-------------------
When adding a few more tokens to the `exahype.grammar` file, the ExaHyPE Toolkit does not compile
anymore. The compilation stops with the following error:
```
javac -cp ../lib/commons-cli.jar -sou...Problem description
-------------------
When adding a few more tokens to the `exahype.grammar` file, the ExaHyPE Toolkit does not compile
anymore. The compilation stops with the following error:
```
javac -cp ../lib/commons-cli.jar -sourcepath . eu/exahype/GenerateSolverRegistration.java
eu/exahype/parser/Parser.java:104: error: code too large
public Start parse() throws ParserException, LexerException, IOException
^
1 error
```
Function `parse()` in file `eu/exahype/parser/Parser.java` contains a large `switch`-`case` statement
considering nearly 3000 cases and has more than 17000 lines of code.
`Parser.java` is unfortunately autogenerated by SableCC.
Further reading:
* https://dzone.com/articles/method-size-limit-java
* https://stackoverflow.com/a/17422590
The Typical *Let's rewrite everything!* Proposal
------------------------------------------------
From my point of view, above issue is just another reason why we should abandon SableCC grammars and switch to a proper data serialisation
format like JSON or YAML (internally). Validation can be performed via JSON schemata. It would be still possible
to keep our current specification file format. We would just introduce a precompilation step.
Additionally, we could consider to rewrite the toolkit in python. A lot is realised via template files anyway and
the code generator for the optimised kernels is written in python, too.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/247Toolkit2: Make subprocess invocation of ExaHyPE main code optional2018-09-06T15:37:41+02:00Ghost UserToolkit2: Make subprocess invocation of ExaHyPE main code optionalSome people might not like the idea that the ExaHyPE code calls the Toolkit in order to (pre)parse specification files.
We should offer a specfile option within the specfiles to allow to generate code which does not calls a subprocess b...Some people might not like the idea that the ExaHyPE code calls the Toolkit in order to (pre)parse specification files.
We should offer a specfile option within the specfiles to allow to generate code which does not calls a subprocess but instead expects the input to be JSON.
This could be seen as advance hotfix when people report problems in complex MPI environments.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/107An idea to solve both the toolkit overwriting and file location problem2018-06-15T15:13:43+02:00Ghost UserAn idea to solve both the toolkit overwriting and file location problemThe Java toolkit creates C++ files in the application directory ("output Directory"). This creates (amongst others) two problems:
1. In large applications, users cannot freely structure their code in subdirectories
2. Users cannot u...The Java toolkit creates C++ files in the application directory ("output Directory"). This creates (amongst others) two problems:
1. In large applications, users cannot freely structure their code in subdirectories
2. Users cannot understand which files are overwritten and which not
To address problem (1), I recently introduced an heuristic approach, [FileSearch.java](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/6e7e9552fb3d65b4c5feb5c70e36038be06499ad/Toolkit/src/eu/exahype/FileSearch.java). This allows to put a file *with a similar name* in any subdirectory in the output directory. Thus, for instance, we can have an application to be structured in folders like
```
SRMHD
├── C2P-MHD.f90
├── C2P-MHD.h
├── C2PRoutines.f90 -> ../SRHD/C2PRoutines.f90
├── ExaHyPE-MHDSolver-p2
├── extended.log
├── InitialDataAdapter.cpp
├── InitialDataAdapter.h
├── InitialData.f90
├── KernelCalls.cpp
├── Makefile
├── make-p2.log
├── MHDSolver.cpp
├── MHDSolver_generated.cpp
├── MHDSolver.h
├── Parameters.f90
├── parameters.mod
├── PDE.f90
├── PDE.h
├── run_withEnv.sh
└── Writers
├── ConservedWriter.cpp
├── ConservedWriter.h
├── ErrorWriter.cpp
├── ErrorWriter.h
├── ExactPrimitivesWriter.cpp
├── ExactPrimitivesWriter.h
├── IntegralsWriter.cpp
├── IntegralsWriter.h
├── PrimitivesWriter.cpp
├── PrimitivesWriter.h
├── RelativeErrorWriter.cpp
├── RelativeErrorWriter.h
└── TimeSeriesReductions.h -> ../../EulerFlow/TimeSeriesReductions.h
```
ie. note the `Writers` subdirectory.
Actually, this ticket shall propose an idea which solves the problems (1) and (2) altogether.
## Defining a header
My idea proposes to define a magic line at any place (ie. header or footer) of the generated `.h` or `.cpp` files, containing something like:
```
/** ExaHyPE.jar: autogenerated at Fr 3. Feb 19:08:51 CET 2017 **/
/** ExaHyPE.jar: File identifier: Plotter[i].HeaderCode **/ <- this is the identifier line
/** ExaHyPE.jar: Rebuild options: [x] Rebuild on every toolkit call [ ] Manually disable rebuild [ ] Rebuild on structural change **/ <- This can be changed by user
```
The identifier line solves the problem to find an appropriate file. It allows users to rename them as he wants. The toolkit has to inspect all possible files with a `grep` like search, but this is not too bad. We don't really expect too big applications...
The options line allows the user both to understand the default setting and to enforce his own rules. This is very helpful when working at ExaHyPE internals and to stop this long lasting conflict with the toolkit overwriting stuff. It also allows users to request an overwrite of a file. The specific syntax is open to discussion, ie. one could also choose something better machine readable here.