ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2019-09-21T15:09:46+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/296GradQ format2019-09-21T15:09:46+02:00Jean-Matthieu GallardGradQ formatThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/277FV Substeps / Particle Simulations2019-04-23T14:16:30+02:00Dominic Etienne CharrierFV Substeps / Particle SimulationsAdd possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver class in order to plug in
other projectors.
Rollbacks are difficult to implement with anarchic time stepping.
I have no clue how to do this yet.Add possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver class in order to plug in
other projectors.
Rollbacks are difficult to implement with anarchic time stepping.
I have no clue how to do this yet.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/241Improve Plotter infrastructure2018-09-04T20:38:44+02:00Sven KöppelImprove Plotter infrastructureThis is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendre basis plotter cannot do patchwise mapping of quantities while VTK Cartesian basis plotter can).
The obvious answer to this problem is *abstraction*. I already started once when introducing a `Slicer` interface class to allow various kind of slicing operations without having to specify them in the individual plotters. They just pass the parameters given by the specfile.
In a similar way, we need more abstraction for the quantities mapping, for labeling of the written quantities (named instead of `Q0...QN`), interpolation on different subcell grids, Support of the *isTroubled* flag or in general bitmasks of patch statusses, etc. The following picture is a sketchy overview:
![plotters](/uploads/876457ebab375ee2e1a15bd32a43c1f3/plotters.png)
To be honest I made this ticket here only becaue I wanted to throw away that sketch paper. It is at least one year old.
Figure as PDF: [Sbizhub1318082520330.pdf](/uploads/d4279ca1621927df8155582c66726dd0/Sbizhub1318082520330.pdf)This is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendre basis plotter cannot do patchwise mapping of quantities while VTK Cartesian basis plotter can).
The obvious answer to this problem is *abstraction*. I already started once when introducing a `Slicer` interface class to allow various kind of slicing operations without having to specify them in the individual plotters. They just pass the parameters given by the specfile.
In a similar way, we need more abstraction for the quantities mapping, for labeling of the written quantities (named instead of `Q0...QN`), interpolation on different subcell grids, Support of the *isTroubled* flag or in general bitmasks of patch statusses, etc. The following picture is a sketchy overview:
![plotters](/uploads/876457ebab375ee2e1a15bd32a43c1f3/plotters.png)
To be honest I made this ticket here only becaue I wanted to throw away that sketch paper. It is at least one year old.
Figure as PDF: [Sbizhub1318082520330.pdf](/uploads/d4279ca1621927df8155582c66726dd0/Sbizhub1318082520330.pdf)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/228stableTimeStepSize kernels should return maximum eigenvalue for (classical) l...2018-06-14T16:10:38+02:00Dominic Etienne CharrierstableTimeStepSize kernels should return maximum eigenvalue for (classical) local time steppingIn general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step size by a factor k (k=3 in Peano) with decreasing mesh level.
The kernels should thus return the maximum eigenvalue, too, or only the maximum eigenvalue.
The minimum local time step size would then be computed according to:
```
dt_min = CFL * max_{over all cells} lambda / min_{over all cells} cellSize
```
which is different to what we are currently doing for global time stepping:
```
dt_min = CFL * min_{over all cells} ( lambda / cellSize )
```
The ADERDGSolver superclass would then decide which minimisation to
perform.In general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step size by a factor k (k=3 in Peano) with decreasing mesh level.
The kernels should thus return the maximum eigenvalue, too, or only the maximum eigenvalue.
The minimum local time step size would then be computed according to:
```
dt_min = CFL * max_{over all cells} lambda / min_{over all cells} cellSize
```
which is different to what we are currently doing for global time stepping:
```
dt_min = CFL * min_{over all cells} ( lambda / cellSize )
```
The ADERDGSolver superclass would then decide which minimisation to
perform.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Sven KöppelToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/193Inconsitencies with the solver constructors (ParserView, cmd line passing)2017-10-18T09:07:53+02:00Sven KöppelInconsitencies with the solver constructors (ParserView, cmd line passing)Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and LimitingADERDG solvers. While the Command line options passing (`std::vector<std::string>& cmdlineargs`) is mandatory in all solvers, they only get the `exahype::Parser::ParserView& constants` when there *are* constants in the specfile. This creates a lot of trouble for the users, for instance when they decide to introduce constants on an existing application because they need to introduce the new signature then on their own.
Even worse: Currently, the `ParserView&` works for FV solvers, but not for ADERDG solvers (at some point, a `ParserView` instead of a `ParserView&` is needed) and therefore also not for coupled LimitingADERDG solvers.
I know that Tobias wants to keep the first code the user see's in his solver as small as possible. So what about always passing the `cmdlineargs` and a `constants` reference and either storing everything in some class
```
struct SpecfileOptions { // or so
std::vector<std::string>& cmdlineargs;
exahype::Parser::ParserView& constants;
// could store even more, for instance general access to the parser,
// static build information, paths, etc.
};
```
and just pass a `SpecfileOptions& options` to the constructors and the `init` function.
As an alternative or additionally, we can make the `init` function virtual and put an empty implementation into the `Abstract*Solver.{cpp,h}`. Then the user could introduce it into his solver only when he needs it.
We are talking about startup code which is only called once, so this should really be as comfortable as possible, performance does not matter at all.Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and LimitingADERDG solvers. While the Command line options passing (`std::vector<std::string>& cmdlineargs`) is mandatory in all solvers, they only get the `exahype::Parser::ParserView& constants` when there *are* constants in the specfile. This creates a lot of trouble for the users, for instance when they decide to introduce constants on an existing application because they need to introduce the new signature then on their own.
Even worse: Currently, the `ParserView&` works for FV solvers, but not for ADERDG solvers (at some point, a `ParserView` instead of a `ParserView&` is needed) and therefore also not for coupled LimitingADERDG solvers.
I know that Tobias wants to keep the first code the user see's in his solver as small as possible. So what about always passing the `cmdlineargs` and a `constants` reference and either storing everything in some class
```
struct SpecfileOptions { // or so
std::vector<std::string>& cmdlineargs;
exahype::Parser::ParserView& constants;
// could store even more, for instance general access to the parser,
// static build information, paths, etc.
};
```
and just pass a `SpecfileOptions& options` to the constructors and the `init` function.
As an alternative or additionally, we can make the `init` function virtual and put an empty implementation into the `Abstract*Solver.{cpp,h}`. Then the user could introduce it into his solver only when he needs it.
We are talking about startup code which is only called once, so this should really be as comfortable as possible, performance does not matter at all.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/177Batching required for LTS and ExaHyPE's limiter-based refinement2017-09-04T14:16:41+02:00Dominic CharrierBatching required for LTS and ExaHyPE's limiter-based refinementLimiting ADER-DG in ExaHyPE
---------------------------
In ExaHyPE, we do not consider adaptive refinement for the Finite Volumes (FV)
solver.
We thus only use FV on the finest level of the adaptive mesh
if we employ a limiting ADER-DG solver.
If a limiting ADER-DG solver detects a shock, i.e. a cell where we need to
limit non-physical oscillations, we refine it down to
the finest adaptive mesh level.
And we further refine its neighbours down to the finest mesh level.
Local Time Stepping
-------------------
TBC
Local Time Stepping plus limiting ADER-DG in ExaHyPE
-----------------------------------------------------
It is possible that a cell is marked as troubled during the
local time stepping. This will require potentially
more refinement around the cell, or to refine the cell itself.
The newly refined cells might not have done any time stepping
at all during the current LTS batch since
their parents stem from a coarser level.
They lag behind in time.
The question is what to do in those scenarios.
Batching
--------
One idea is to consider the number of local time steps to run as a batch.
We remember the solution before a batch starts.
In case (limiter-based) refinement is triggered, we memorise the cells to refine,
perform a rollback to the memorised solution.
Afterwards, we refine the memorised cells and run the batch again.
It can happen multiple times that we have to rerun a batch.
On the other hand, we ensure that our grid is always tracking a shock correctly.
Further techniques
------------------
On the expense of computational cost, we could refine the mesh generously (according to the restricted limiter status)
This would reduce the number of batch reruns.Limiting ADER-DG in ExaHyPE
---------------------------
In ExaHyPE, we do not consider adaptive refinement for the Finite Volumes (FV)
solver.
We thus only use FV on the finest level of the adaptive mesh
if we employ a limiting ADER-DG solver.
If a limiting ADER-DG solver detects a shock, i.e. a cell where we need to
limit non-physical oscillations, we refine it down to
the finest adaptive mesh level.
And we further refine its neighbours down to the finest mesh level.
Local Time Stepping
-------------------
TBC
Local Time Stepping plus limiting ADER-DG in ExaHyPE
-----------------------------------------------------
It is possible that a cell is marked as troubled during the
local time stepping. This will require potentially
more refinement around the cell, or to refine the cell itself.
The newly refined cells might not have done any time stepping
at all during the current LTS batch since
their parents stem from a coarser level.
They lag behind in time.
The question is what to do in those scenarios.
Batching
--------
One idea is to consider the number of local time steps to run as a batch.
We remember the solution before a batch starts.
In case (limiter-based) refinement is triggered, we memorise the cells to refine,
perform a rollback to the memorised solution.
Afterwards, we refine the memorised cells and run the batch again.
It can happen multiple times that we have to rerun a batch.
On the other hand, we ensure that our grid is always tracking a shock correctly.
Further techniques
------------------
On the expense of computational cost, we could refine the mesh generously (according to the restricted limiter status)
This would reduce the number of batch reruns.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/175Symbolic flux calculations reduce speed significantly2019-03-21T10:56:35+01:00Dominic CharrierSymbolic flux calculations reduce speed significantlyI just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classic array indexing (optimised-nonsymbolic).
The files are suffixed by a ".likwid.csv".
I further attached measured Peano adapter times
The files are suffixed by a ".csv".
Setup
--------
* Compressible Euler equations (Euler_Flow)
* pure ADER-DG scheme (no limiter)
* polynomial orders p=3,5,7,9;
regular 27^3 grid (3D)
* TBB threads=1,12,24.
* Intel icpc17 (USE_IPO=on).
* nonfused (3 algorithmic phases) vs. fused (a single pipelined algorithmic phase) ADER-DG implementation
* no predictor reruns did occur for the fused implementation
Preliminary Results
----------------------------
* Optimised kernels are faster than the generic ones (I kind of expected this 😉)
* Raw array access (optimised-nonsymbolic) is significantly faster than using the "symbolic variables"(optimised).
* Fused scheme pays off (as long as number of reruns is low; very interesting for linear PDEs (no reruns here))
Files
-----
[Euler_ADERDG-no-output-generic.csv](/uploads/f671c93a33e70c9549853f1f51518c9d/Euler_ADERDG-no-output-generic.csv)
[Euler_ADERDG-no-output-generic.likwid.csv](/uploads/0d52ad214f0b1d6cfae4d658a9997cb5/Euler_ADERDG-no-output-generic.likwid.csv)
[Euler_ADERDG-no-output-optimised.csv](/uploads/422288cdc222b1b250b3aad9e2ad73a1/Euler_ADERDG-no-output-optimised.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.csv](/uploads/0f84240941230d4bc9f06c76b154396b/Euler_ADERDG-no-output-optimised-nonsymbolic.csv)
[Euler_ADERDG-no-output-optimised.likwid.csv](/uploads/135bd71c668ca0c4c4ee5b7988ea2f17/Euler_ADERDG-no-output-optimised.likwid.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv](/uploads/7606eefe559499a1f6de62b5f33905d0/Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv)I just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classic array indexing (optimised-nonsymbolic).
The files are suffixed by a ".likwid.csv".
I further attached measured Peano adapter times
The files are suffixed by a ".csv".
Setup
--------
* Compressible Euler equations (Euler_Flow)
* pure ADER-DG scheme (no limiter)
* polynomial orders p=3,5,7,9;
regular 27^3 grid (3D)
* TBB threads=1,12,24.
* Intel icpc17 (USE_IPO=on).
* nonfused (3 algorithmic phases) vs. fused (a single pipelined algorithmic phase) ADER-DG implementation
* no predictor reruns did occur for the fused implementation
Preliminary Results
----------------------------
* Optimised kernels are faster than the generic ones (I kind of expected this 😉)
* Raw array access (optimised-nonsymbolic) is significantly faster than using the "symbolic variables"(optimised).
* Fused scheme pays off (as long as number of reruns is low; very interesting for linear PDEs (no reruns here))
Files
-----
[Euler_ADERDG-no-output-generic.csv](/uploads/f671c93a33e70c9549853f1f51518c9d/Euler_ADERDG-no-output-generic.csv)
[Euler_ADERDG-no-output-generic.likwid.csv](/uploads/0d52ad214f0b1d6cfae4d658a9997cb5/Euler_ADERDG-no-output-generic.likwid.csv)
[Euler_ADERDG-no-output-optimised.csv](/uploads/422288cdc222b1b250b3aad9e2ad73a1/Euler_ADERDG-no-output-optimised.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.csv](/uploads/0f84240941230d4bc9f06c76b154396b/Euler_ADERDG-no-output-optimised-nonsymbolic.csv)
[Euler_ADERDG-no-output-optimised.likwid.csv](/uploads/135bd71c668ca0c4c4ee5b7988ea2f17/Euler_ADERDG-no-output-optimised.likwid.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv](/uploads/7606eefe559499a1f6de62b5f33905d0/Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/129Access to neighbours for which kernels?2018-06-15T15:13:07+02:00Dominic CharrierAccess to neighbours for which kernels?Finite Volumes Solver:
- source; ex: bathymetry in SWEFinite Volumes Solver:
- source; ex: bathymetry in SWE