ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2019-09-21T15:09:46+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/296GradQ format2019-09-21T15:09:46+02:00Jean-Matthieu GallardGradQ formatThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*...The format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/289Luh padding and alternative layout2019-07-23T10:59:21+02:00Jean-Matthieu GallardLuh padding and alternative layoutCurrently the optimized kernels do not change luh. This was done for historical reasons (the non temp arrays had a set size independently of the kernels, now changed) and to ensure that the plotter, limiter and AMR would work no matter t...Currently the optimized kernels do not change luh. This was done for historical reasons (the non temp arrays had a set size independently of the kernels, now changed) and to ensure that the plotter, limiter and AMR would work no matter the kernels.
However being able to work with a padded luh or even further an SoS-layout luh (x,n,y,z coordinates instead of the current n,x,y,z) could provide a speedup. Currently the linear vectorized split_ck implementation does the back and forth transpositions to SoS layout and the cost of these is non negligible (even if more than compensated by the vectorization speedup).
The Limiter projections+tests and AMR can now be done using optimized kernels too so the compatibility issue here could easily be solved by generating the right kernels.
Regarding the plotters a preprocessing step to translate back luh locally could be added in case luh is non standard.
Is there a good way to do this?
Open questions: would this impact other domain of the code
@domcha @leo.rannabauer
Occurrences of luh:
-------------------
Search Term: ``getSolution()``
**exahype plotters**
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Carpet/ADERDG2Carpet.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/CSV/ADERDG2LegendreCSV.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/CSV/Patch2CSV.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FlashHDF5/ADERDG2FlashHDF5.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/ADERDG2CartesianPeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/ADERDG2LegendrePeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/FiniteVolumes2PeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Tecplot/ExaHyPE2Tecplot.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LegendreDivergenceVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LegendreVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LobattoVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/FiniteVolumes2VTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/LimitingADERDG2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/LimitingADERDGSubcells2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/Patch2VTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/ADERDG2ProbeAscii.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/ADERDG2UserDefined.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FiniteVolumes2ProbeAscii.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FiniteVolumes2UserDefined.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/LimitingADERDG2UserDefined.cpp
**exhaype/solvers**
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_FusedTimeStepJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_PredictionJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/FiniteVolumesSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver_LocalRecomputationJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cppJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/288Separate Variables from Parameters2019-07-23T10:40:15+02:00Jean-Matthieu GallardSeparate Variables from ParametersProblem: Currently parameters are stored right after the variables in luh, Q and its related arrays (lQi, lQhi, lQhbnd). This make it difficult to provide vectorized PDE options and make some operations where the parameter have to be ski...Problem: Currently parameters are stored right after the variables in luh, Q and its related arrays (lQi, lQhi, lQhbnd). This make it difficult to provide vectorized PDE options and make some operations where the parameter have to be skipped less intuitive and slower with SIMD
Solution: Introduce a new array like luh to store the parameters. This array is to be passed to the user functions using parameters (nullptr if no parameters) like luh or Q is. The linear space time predictor with split_ck already do this locally.
Issue: Will change the signatures of some user functions and formulation if they are indeed using parameters.
@ga96nuvhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/275User-friendly API, use AbstractSolver as Adapter2019-07-23T10:40:16+02:00Ghost UserUser-friendly API, use AbstractSolver as AdapterIn order to make the code more easy to use, we could reestablish the MySolver_Variables
data access classes.
We could implement the user hooks that take a raw pointer in the AbstractSolver,
and would then call the user solver's implemen...In order to make the code more easy to use, we could reestablish the MySolver_Variables
data access classes.
We could implement the user hooks that take a raw pointer in the AbstractSolver,
and would then call the user solver's implementation which takes the
data access object as argument.
**Example:**
```
void MyAbstractSolver::adjustPointSolution(double* const Q,...) {
Variables vars(Q);
static_cast<MyUserSolver*>(this)->adjustPointSolution(vars,...);
}
```
We could then have additional data access classes for the full solution array, e.g. ``Solution`` and ``ReadOnlySolution``.
Potentially, we could then introduce dimension-aware iterators with appropriate typecast to
a ``Variables`` or ``ReadOnlyVariables`` object.
This would allow writing code like the following.
**Example:**
```
void MyAbstractSolver::refinementCriterion(ReadOnlySolution solution,...) {
double rhoMax = 0;
for ( ReadOnlyVariables& Q : solution ) {
rhoMax = std::max( Q.rho(), rhoMax );
}
}
```
Remarks
-------
``(ReadOnly)Variables`` and ``Variables`` must be split or declared in such a way that
it is clear that there is not necessarily a contiguous array storing
and variables and parameters. We should get rid of the index operator and
have separate functions ``double`` parameter(int i), ``double& variable(int i)`` instead.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/271Specification File Defaults in C++ Parser2019-05-08T13:41:02+02:00Ghost UserSpecification File Defaults in C++ ParserWe have a C++ JSON file parser.
The specification file schema is a JSON file itself.
Hence, we could read default values also directly from the schema.
**Edit:**
We could generate most of the C/C++ parser directly from the JSON schema....We have a C++ JSON file parser.
The specification file schema is a JSON file itself.
Hence, we could read default values also directly from the schema.
**Edit:**
We could generate most of the C/C++ parser directly from the JSON schema.
**Edit 2:**
There is now also a C/C++ header-only library for
JSON file validation.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/241Improve Plotter infrastructure2018-09-04T20:38:44+02:00Ghost UserImprove Plotter infrastructureThis is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendr...This is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendre basis plotter cannot do patchwise mapping of quantities while VTK Cartesian basis plotter can).
The obvious answer to this problem is *abstraction*. I already started once when introducing a `Slicer` interface class to allow various kind of slicing operations without having to specify them in the individual plotters. They just pass the parameters given by the specfile.
In a similar way, we need more abstraction for the quantities mapping, for labeling of the written quantities (named instead of `Q0...QN`), interpolation on different subcell grids, Support of the *isTroubled* flag or in general bitmasks of patch statusses, etc. The following picture is a sketchy overview:
![plotters](/uploads/876457ebab375ee2e1a15bd32a43c1f3/plotters.png)
To be honest I made this ticket here only becaue I wanted to throw away that sketch paper. It is at least one year old.
Figure as PDF: [Sbizhub1318082520330.pdf](/uploads/d4279ca1621927df8155582c66726dd0/Sbizhub1318082520330.pdf)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/226Use sensible Fortran flags2018-04-24T11:58:40+02:00Ghost UserUse sensible Fortran flagsAt the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently te...At the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently testing:
`-xCORE-AVX512 -fma -align array64byte`
and has mentioned that `-qopt-prefetch=3` worked well with the prototype code.
To be updated...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/219Cancel all predictor background jobs when a predictor rerun is necessary2018-03-26T11:15:21+02:00Ghost UserCancel all predictor background jobs when a predictor rerun is necessaryhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/218Horizontal detection of insufficiently refined mesh for LimitingADERDGSolver2018-03-07T11:58:59+01:00Ghost UserHorizontal detection of insufficiently refined mesh for LimitingADERDGSolverCurrently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
r...Currently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
requests which force the time stepping to stop.
Restricting to the next coarser parent implies non-global master-worker communication
in MPI builds. This is not good.
To get rid of this master-worker communication during the time-stepping,
I propose to extend the limiter status range by a few more than one OK statuses.
If such an OK status is then propagating into a virtual child cell (Descendant),
we know that the mesh is not sufficiently refined and we halt the time stepping.
Some philosophy:
From the updates of the flags, we should further be able to predict in which direction a shock
propagates. We can then select more carefully which cell to refine next.
I should further rethink my whole limiter-based mesh refinement. Maybe it is more advantageous,
to do some bottom-up flagging for refinement. Instead of the current top-down approach
where I use halo-refinement around the limited regions.
With MPI switched on, I wonder however how well or badly this will interplay with the
load balancing during the initial mesh refinement.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/214Faster mesh refinement2018-03-02T11:23:39+01:00Ghost UserFaster mesh refinementProblem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If...Problem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If a new cells is introduced to the tree it might not be yet be identified
as part of a regular subgrid and the operations performed on the cell
are thus parallelised.
Solution: If imposing initial conditions or evaluating a refinement criterion is too
expensive we could let the user choose to perform it as background task.
This might lead to more mesh setup iteration but potentially to a better
exploitation of the available cores. It should also benefit the hiding of MPI communication
during the mesh setup.
Problem 2 (Overall concurrency)
-------------------------------
ExaHyPE's regular shared-memory parallelism during the mesh setup is currently further limited as
multiple cells might write to the same vertex in order to set refinement events.
Solution: We should be able to solve this by inverting the control. The
vertex checks in touchVertexLastTime if any cell has set a refinement event,
and refines if that is the case. This would increase the concurrency of the
enterCell operations.
Problem 3 (Memory)
-------------------------------
ExaHyPE's initial mesh setup is performed at the beginning by a single rank.
Gradually more and more ranks are added. In order to prevent that
any of the ranks runs out of memory during the initial mesh setup,
it might make sense to only temporarily allocate memory, impose initial
conditions, evaluate the refinement criterion, and then free the memory again (better:
recylce it).
After the initial mesh setup, we would then allocate memory on all ranks
and impose initial conditions.
Problem 4 (Load Balancing)
-------------------------------
The load-balancing does currently only count the number of cells.
It does not take the different cell types in ExaHyPE's grid into account.
The helper cell types Descendant and Ancestor have way less work to do than
the compute cells of type Cell.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TU...As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Ghost UserToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always...> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/197Let tarch VTK plotters pass file creation errors to ExaHyPE plotters2018-03-02T12:46:18+01:00Ghost UserLet tarch VTK plotters pass file creation errors to ExaHyPE plottersThis is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetH...This is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetHDF5 writer) while other silently ignore this issue (Tobias tarch-based VTK writers). Clearly, we want the user to be able to control whether problems at output shall be severe or not.
I can easily implement this in the ExaHyPE codebase, but I don't control peano, so I have to make sure Peano passes such errors into exahype's domain.
=> TODO @ Sven.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/175Symbolic flux calculations reduce speed significantly2019-03-21T10:56:35+01:00Ghost UserSymbolic flux calculations reduce speed significantlyI just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classi...I just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classic array indexing (optimised-nonsymbolic).
The files are suffixed by a ".likwid.csv".
I further attached measured Peano adapter times
The files are suffixed by a ".csv".
Setup
--------
* Compressible Euler equations (Euler_Flow)
* pure ADER-DG scheme (no limiter)
* polynomial orders p=3,5,7,9;
regular 27^3 grid (3D)
* TBB threads=1,12,24.
* Intel icpc17 (USE_IPO=on).
* nonfused (3 algorithmic phases) vs. fused (a single pipelined algorithmic phase) ADER-DG implementation
* no predictor reruns did occur for the fused implementation
Preliminary Results
----------------------------
* Optimised kernels are faster than the generic ones (I kind of expected this 😉)
* Raw array access (optimised-nonsymbolic) is significantly faster than using the "symbolic variables"(optimised).
* Fused scheme pays off (as long as number of reruns is low; very interesting for linear PDEs (no reruns here))
Files
-----
[Euler_ADERDG-no-output-generic.csv](/uploads/f671c93a33e70c9549853f1f51518c9d/Euler_ADERDG-no-output-generic.csv)
[Euler_ADERDG-no-output-generic.likwid.csv](/uploads/0d52ad214f0b1d6cfae4d658a9997cb5/Euler_ADERDG-no-output-generic.likwid.csv)
[Euler_ADERDG-no-output-optimised.csv](/uploads/422288cdc222b1b250b3aad9e2ad73a1/Euler_ADERDG-no-output-optimised.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.csv](/uploads/0f84240941230d4bc9f06c76b154396b/Euler_ADERDG-no-output-optimised-nonsymbolic.csv)
[Euler_ADERDG-no-output-optimised.likwid.csv](/uploads/135bd71c668ca0c4c4ee5b7988ea2f17/Euler_ADERDG-no-output-optimised.likwid.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv](/uploads/7606eefe559499a1f6de62b5f33905d0/Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/129Access to neighbours for which kernels?2018-06-15T15:13:07+02:00Ghost UserAccess to neighbours for which kernels?Finite Volumes Solver:
- source; ex: bathymetry in SWEFinite Volumes Solver:
- source; ex: bathymetry in SWE