ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2019-09-20T15:43:37+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/254Finite Volumes Kernel Issues2019-09-20T15:43:37+02:00Ghost UserFinite Volumes Kernel IssuesA list of known issues with the FV kernels.
This list is open for extension. It may be split into separate issues
later on.
* MUSCL-Hancock implementation is second-order accurate but allows unlimited slopes
in the first ghost layer at ...A list of known issues with the FV kernels.
This list is open for extension. It may be split into separate issues
later on.
* MUSCL-Hancock implementation is second-order accurate but allows unlimited slopes
in the first ghost layer at corners and edges of the patch. This caused certain applications to crash.
* Stack allocation of multiple temporary arrays leads to Seg Faults for applications where these arrays
become huge (e.g. CCZ4). (This issue might be resolved after issue #255 has been resolved.)
* No (JM) optimisation has been performed yet. For most applications, the FV updates
are bandwith-bound indicating potential for speedups by vectorisation.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/57Finite volumes solver / Limiter2019-09-20T15:46:56+02:00Ghost UserFinite volumes solver / Limiter# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to g...# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to get rid of the "update" field if we directly
add a weighted (by dt and quad weight) update to the "solution" field
from the volume integral and surface integral evaluation.
# Status of the implementation of the limiting ADER-DG scheme (unordered notes)
* The limiter workflow works now for uniform meshes with Intel's TBBs.
[sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/b71d24c7b82f805c604f31c1b322fcae/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi)
* Found and fixed some more bugs. Now everything looks fine. The limiter has a local characteristic and only
requires a rollback/reallocation of memory in every fifth time step (49 out of 255) in the experiment
shown [sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/1e065c7c5c2a52aefb551ef0a03f0ac5/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi), where we ran an experiment with P=3 order ADER-DG approx, \delta_0=1e-2, \epsilon=1e-3.
As long as the limiter domain does not change, we do not need to reallocate new memory and do a
recomputation of certain cells in our novel implementation.
* Find another simulation using a 9-th order ADER-DG approximation and parameters \delta_0=10^-4 , \epsilon=10^-3 here:
[sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi](/uploads/fb466322f7df5339790aac4afca9ce6c/sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi)
~~We observee a strange x velocity in this benchmark. Can't be seen in the video. The both profile of the Shock-Tube differs signifantly
in vicinity of the contact discontinuity.~~This is actually the momentum density. Everything is fine.
* Explosion problem with P=5 and delta_0=1e-4 and epsilon=1e-3:
[explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi](/uploads/4ace50cdcbfcbc1706432d0cc180fcea/explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi). Here the limiter domain must be adjusted after most of the time steps
* I performed some further optimisations of the DMP evaluation. I found that we can easily perform a
"loop" fusion of the min and max computation and the DMP.
* Implemented the physical admissibility detection (PAD) now as well and use it to
detect non-physical oscillations in the initial conditions. Here, I found that we can directly
pass it the solution min and max we computed as part of the DMP calculation. No need
to loop over the whole degrees of freedom and the Lobatto nodes again.
* ~~There is still an issue with the discrete maximum principle which detects too many
cells to be troubled.~~ Was resolved by tuning the DMP parameters.
* AMR and Limiting need to be combined. This is in principle "a simple" wiring of FV patches
along the ADER-DG tree. We further need spatial averaging and interpolation operators.
Then, we can follow the AMR timestepping implementation of the ADER-DG scheme.
# Writing a Godunov type first order method
Due to the issues I encountered while [rewriting](#issueswithrewritingthefinitevolumessolver) the
pseudo-TVD donor cell type finite volumes solver, Tobias and me decided
to write a simple first order Godunov type finite volumes solver that only exchanges volume averages/fluxes
between direct (face) neighbours.
Replacement of this simple method by a more complex one will be tackled in later stages of the project -- if necessary.
# Issues with higher order finite volumes solvers
* Higher-order methods have a reconstruction stencil which is larger than the
Riemann solver stencil (simple star).
* Reconstruction and time evolution of boundary extrapolated values at one face of a patch can only be performed
after all volume averages from the direct neighbour and corner neighbour patches are available.
I have identified the following phases of the FVM solver now:
* Gather: Just get the neighbour values (arithmetic intensity = 0). This will move into the Merging mapping.
* Spatial reconstruction: Compute spatial slopes in the cells or something similar. This will move into the SolutionUpdate mapping.
* Temporal evolution of boundary-extrapolated values: This will move into the SolutionUpdate mapping.
* Riemann solve: This will move into the SolutionUpdate mapping.
* Solution update; This will move into SolutionUpdate mapping.
* Extrapolation of volume averages: This will send layers of the volume averages to the neighbours.
We send layers instead of a single layer in order to only have one data exchange instead of two.
With the above strategy, we can merge FVM solvers into the current framework. The difference to the ADER-DG solver
is that the computational intensity of the neighbour merging operation is zero.
We will further need to consider neighbour data exchange over edges (3-d) and corners in our
merging scheme. Neighbour data exchange over edges in 3-d will require additional falgs.
Exchange over corners does not require additional flags.
# Issues with rewriting the finite volumes solver
I tried to decompose our pseudo-TVD donor cell type finite volumes
solver into a Riemann solve and a solution update part,
and ran into the following issues.
## Open issues with the donor cell type pseudo-TVD finite volumes solver implementation:
* The current solver uses updated solution values in the solution update.
We have to distinguish between old values and new values or
have to introduce an update.
* We need to consider two layers of the neighbour to compute all extrapolated boundary
values (wLx,wRx,wLy,wRy). Currently only one layer ist considered. This means
that the boundary extrapolated values and thus the face fluxes at the outermost faces
are computed wrongly.
* To tackle the above problem, we either need to exchange a single cell of
the diagonal neighbours or we need to rely on two data exchanges.
Tobias told me there exists however a trick to circumvent this:
Reconstructing the diagonal neighbours' contributions with values from
the direct neighbours.
## Limitations of the donor cell type pseudo-TVD finite volumes solver:
* The solver is not TVD in multiple dimensions.
* We do neither perform corner correction nor (dimensional) operator splitting. We should
thus observe loss of mass conservation, large dispersion errors, and a reduced CFL stability limit.
The reduced CFL stability limit was taken into account in our implementation.
This is also a problem of the multi-dim. Godunov method in the unsplit form.
## Follow up: Low-order Euler-DG patch with direct coupling to ADER-DG method?
## Dissemination
For the following benchmarks I always used copy boundary conditions which are equal to
outflow/inflow boundary conditions as long as everything flows out of the domain.
* Euler (Compressible Hydrodynamics)
* Explosion Problem with 27^2 grid, P=3, and N=2*3+1 FV subgrid:
![lim-aderdg_explosion](/uploads/6459f9b4a775a482cdfec24e129c56ae/lim-aderdg_explosion.png)
[lim-aderdg_explosion.avi](/uploads/6a38fa00a6bf41fe3aefbee380a03396/lim-aderdg_explosion.avi)
* Sod Shock Tube with 27^2 grid, P=3 and N=2*3+1 FV subgrid:
![lim-aderdg_euler_sod-shock-tube](/uploads/eb2a5e57ca8f37665c2dffe48dbf1e22/lim-aderdg_euler_sod-shock-tube.png)
[lim-aderdg_euler_sod-shock-tube.avi](/uploads/61dfedf7db4ead996d7fd6b9796ba5c0/lim-aderdg_euler_sod-shock-tube.avi)
![hires_antialias](/uploads/26843a0f50e52ded919cfc12b777360b/draft2_hires_antialias.png)
* SRMHD (Special Relativistic Magnetohydrodynamics; basically: Special Relativistic Euler+Maxwell)
* Blast Wave setup as in 10.1016/j.cpc.2014.03.018 with 27^2 grid, P=5, and N=2*5+1 FV subgrid:
![lim-aderg_mhd__blast-wave](/uploads/feb026167c7c2d882c132f3b775c37ac/lim-aderg_mhd__blast-wave.png)
[lim-aderdg_mhd_blast-wave.avi] (/uploads/4d715c47a03bd3850d3c0398e39da58c/lim-aderdg_mhd_blast-wave.avi)
* Rotor setup as in http://adsabs.harvard.edu/abs/2004MSAIS...4...36D with 9^2 grid, P=9, and N=2*9+1 FV subgrid:
![lim-aderdg_mhd_rotor_9x9_P9](/uploads/b35e7dc2dc137068e7416a1952d02a1a/lim-aderdg_mhd_rotor_9x9_P9.png)
[lim-aderdg_mhd_rotor_9x9_P9.avi](/uploads/c5ee8ff29f18a7201be66a1a26b664dc/lim-aderdg_mhd_rotor_9x9_P9.avi)
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/297FiniteVolumeSolver error about time step2020-11-20T17:30:31+01:00Duo Lid.li@lmu.deFiniteVolumeSolver error about time stepWhen running FV, there is this error. I use CFL=0.7. How to fix the warning or how does it matter to the code.
```cpp
if ( !tarch::la::equals(cellDescription.getTimeStepSize(), 0.0) && tarch::la::smaller(admissibleTimeStepSize,cellDesc...When running FV, there is this error. I use CFL=0.7. How to fix the warning or how does it matter to the code.
```cpp
if ( !tarch::la::equals(cellDescription.getTimeStepSize(), 0.0) && tarch::la::smaller(admissibleTimeStepSize,cellDescription.getTimeStepSize()) ) {
logWarning("updateSolution(...)","Finite volumes solver time step size harmed CFL condition. dt="<<
cellDescription.getTimeStepSize()<<", dt_adm=" << admissibleTimeStepSize << ". cell=" <<cellDescription.toString());
}
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/90Fix convergence submission <-> reporting interface2018-06-15T15:13:44+02:00Ghost UserFix convergence submission <-> reporting interfaceIt's still broken:
```
...
INFO:ConvergenceTest(EulerShuVortex):Starting p=9, maxmeshsize=1.83333333333 with runTemplatedSpecfile.sh
INFO:ConvergenceTest(EulerShuVortex):Starting p=9, maxmeshsize=0.611111111111 with runTemplatedSpecfile...It's still broken:
```
...
INFO:ConvergenceTest(EulerShuVortex):Starting p=9, maxmeshsize=1.83333333333 with runTemplatedSpecfile.sh
INFO:ConvergenceTest(EulerShuVortex):Starting p=9, maxmeshsize=0.611111111111 with runTemplatedSpecfile.sh
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p7, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p7-meshsize0.203703703704/run-iboga31.log
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p7, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p7-meshsize0.611111111111/run-iboga31.log
INFO:ConvergenceTest(EulerShuVortex):34 processes have been started with PIDS:
INFO:ConvergenceTest(EulerShuVortex):143265 143266 143270 143275 143283 143295 143308 143325 143349 143373 143401 143425 143458 143487 143507 143532 143568 143596 143625 143661 143692 143729 143760 143795 143832 143866 143905 143941 143989 144028 144069 144109 144155 144194
Traceback (most recent call last):
File "./ShuVortexTest.py", line 71, in <module>
ConvergenceFrontend(test, description=__doc__)
File "../libconvergence/convergence_frontend.py", line 53, in __init__
self.actions.call(args.action, self)
File "../libconvergence/convergence_helpers.py", line 278, in call
return self.actions[action_key](selfRef, *args, **kwargs)
File "../libconvergence/convergence_frontend.py", line 109, in startConvergenceTestAndWaitAndReporter
exitcode = self.startConvergenceTestAndWait()
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p8, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p8-meshsize5.5/run-iboga31.log
File "../libconvergence/convergence_helpers.py", line 273, in _impl
method(self, *method_args, **method_kwargs)
File "../libconvergence/convergence_frontend.py", line 103, in startConvergenceTestAndWait
exitcode = self.waitForProcesses(processes)
NameError: global name 'processes' is not defined
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p8, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p8-meshsize1.83333333333/run-iboga31.log
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p8, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p8-meshsize0.611111111111/run-iboga31.log
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p9, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p9-meshsize5.5/run-iboga31.log
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p9, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p9-meshsize1.83333333333/run-iboga31.log
runTemplatedSpecfile.sh: Calling ExaHyPE-Euler-p9, redirecting output to /mnt/beegfs/koeppel/exahype-iboga/ExaHyPE-Engine/Miscellaneous/ConvergenceAnalysis/ShuVortex/simulations/p9-meshsize0.611111111111/run-iboga31.log
koeppel@iboga31 ShuVortex$
` ``https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/138Fix Fortran module detection Makefile again2018-06-15T15:13:43+02:00Ghost UserFix Fortran module detection Makefile againNote for @svenk.
It's extremely nerving to restart builds all the time, especially when using out-of-tree builds.Note for @svenk.
It's extremely nerving to restart builds all the time, especially when using out-of-tree builds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/196Fix musclhancock scheme for certain patch sizes2018-02-22T09:45:39+01:00Ghost UserFix musclhancock scheme for certain patch sizesFor me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applicatio...For me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applications (GRMHD,CCZ4).
I will do this, ticket just for book keeping.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/222Fix the M/h computation in the CCZ4/Writers/TimingStatisticsWriter.cpph2019-04-15T12:41:12+02:00Ghost UserFix the M/h computation in the CCZ4/Writers/TimingStatisticsWriter.cpphFrom a mail at Luke:
>
I just noticed there is something wrong in the M/h determination:
>
As explained in my last mail to Tobias and you, I just divide these
two numbers. However, the time in the first column of stdout measures
the tim...From a mail at Luke:
>
I just noticed there is something wrong in the M/h determination:
>
As explained in my last mail to Tobias and you, I just divide these
two numbers. However, the time in the first column of stdout measures
the time since program start.
>
In ExaHyPE, the grid setup sometimes takes a considerable amount --
like 10 minutes. If you measure the M/h straight after the first
timesteps after these 10 minutes, you get of course totally wrong
numbers. However, if you measure after 1000 minutes runtime, the 10
minutes grid setup do not change the result so much.
>
It is not hard to substract the time the grid setup needs in order to
improve the correctness of the number. You can do this either by hand
(just look up when the first time step started) or we can do this in
code (CCZ4/Writers/TimingStatisticsWriter.cpph).https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/276Fused Time Stepping for LimitingADERDGSolver crashes with MPI if recomputatio...2019-03-29T17:07:03+01:00Ghost UserFused Time Stepping for LimitingADERDGSolver crashes with MPI if recomputation is triggeredNonfused variants seem to work fine.
EDIT: A run with generic kernels and noarch did not crash.Nonfused variants seem to work fine.
EDIT: A run with generic kernels and noarch did not crash.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/135FV + MPI crashes2018-06-15T15:13:43+02:00Ghost UserFV + MPI crashesApplication: `GRMHD_FV` with current grid as in repository, start with
$ `mpirun -np 4 ./ExaHyPE-GRMHD ../GRMHD_FV.exahype`
and you get
```
assertion in file /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/Time...Application: `GRMHD_FV` with current grid as in repository, start with
$ `mpirun -np 4 ./ExaHyPE-GRMHD ../GRMHD_FV.exahype`
and you get
```
assertion in file /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/TimeStepSizeComputation.cpp, line 229 failed: solver->getNextMaxCellSize()>0
parameter solver->getNextMaxCellSize(): -1.00000000000000000000e+00
parameter _maxCellSizes[solverNumber]: -1.79769313486231570815e+308
parameter _minCellSizes[solverNumber]: 5.3047 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 3 (199 message(s))
5.30473 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 2 (78 message(s))
5.30475 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 0 (42 message(s))
1.79769313486231570815e+308
ExaHyPE-GRMHD: /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/TimeStepSizeComputation.cpp:229: void exahype::mappings::TimeStepSizeComputation::endIteration(exahype::State&): Assertion `false' failed.
```
* Voller Log: [mpi.log](/uploads/e598f84752224c16f03ebee9b0d646ba/mpi.log)
* Gepackter Coredump: [core.gz](/uploads/bd610012c8e810aaac9fa6f9ccd9214d/core.gz)
* Version information: [exa.version](/uploads/e30e199b8ef94ee4d5839ecb3ab08843/exa.version) (MPI, kein TBB, GCC)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/122FV kernels NCP / gradQ2018-06-15T15:13:43+02:00Ghost UserFV kernels NCP / gradQNote also that the Godunov kernels still lack support for NCP and gradQ and all that.
It's something I could try work on.Note also that the Godunov kernels still lack support for NCP and gradQ and all that.
It's something I could try work on.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/277FV Substeps / Particle Simulations2019-04-23T14:16:30+02:00Ghost UserFV Substeps / Particle SimulationsAdd possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver...Add possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver class in order to plug in
other projectors.
Rollbacks are difficult to implement with anarchic time stepping.
I have no clue how to do this yet.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/113Generic and Optimised solvers use abstract base class handling the kernel calls2018-06-15T15:13:43+02:00Ghost UserGeneric and Optimised solvers use abstract base class handling the kernel callshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/174Generic kernels: Select fluxes,ncp similar as for optimised kernels?2017-10-04T16:06:01+02:00Ghost UserGeneric kernels: Select fluxes,ncp similar as for optimised kernels?This issue is open for discussion.
Issue
-----
Working with new ExaHyPE users has revealed that they often are not familar
with the object-oriented programming concept of inheritance.
Especially, they do not know how to overriding vir...This issue is open for discussion.
Issue
-----
Working with new ExaHyPE users has revealed that they often are not familar
with the object-oriented programming concept of inheritance.
Especially, they do not know how to overriding virtual functions
of the AbstractMySolver class.
They are further not familiar with the keywords "virtual" and "override".
This issue makes it difficult for them to select the
right PDE kernels (flux,ncp,...) for their application.
Even worse: The code might even compile and run but it will not perform
the expected calculations.
Such an error is very hard to detect in practice.
Especially for a new ExaHyPE user.
Toolkit-based solution (open for discussion)
--------------------------------------------
JM has moved the selection of the
PDE kernels to the toolkit by requiring the
user to specify the kernels in the following way:
```
kernels const = optimised::fluxes::nonlinear // flux only
```
or
```
kernels const = optimised::fluxes::ncp::nonlinear // flux and ncp
```
or
```
kernels const = optimised::fluxes::ncp::source::nonlinear // flux and ncp, source
```
In my opinion, this is the better approach.
The "const" modifier of "kernels" indicates that the user has to
rerun the toolkit everytime he selects different PDE-kernels
The toolkit will then update the AbstractSolver Header file.
The compiler will deal with any inconsistencies between
the files:
* The compiler will tell you if you have not implemented a
kernel you have specified - not an assertion
as it is the case right now.
(Users often do not even know about the Asserts Mode.)
* The compiler will tell you if you have implemented
a method which is not called. You should then
comment out the implementation or remove it.
What is your opinion?
---------------------
Please comment below.Jean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/262Get CarpetASCII 1d cut working on different machines such as hazelhen2019-03-06T18:27:47+01:00Ghost UserGet CarpetASCII 1d cut working on different machines such as hazelhenLuke reported that it did not work for him.
But it does no more depend on HDF5 so we should get this working...
@lbovardLuke reported that it did not work for him.
But it does no more depend on HDF5 so we should get this working...
@lbovardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/181Get rid of TemporaryVariables2018-03-20T16:13:53+01:00Ghost UserGet rid of TemporaryVariables- We still have to debate if this makes sense.- We still have to debate if this makes sense.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/188Global Reduction of Time Stepping Data2017-10-11T11:50:37+02:00Ghost UserGlobal Reduction of Time Stepping DataI currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker...I currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker to master
all the way up to the global master rank.
I could use a simple MPI_Reduce and a simple MPI_Gather to
perform the above steps.~~
Postponed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/296GradQ format2019-09-21T15:09:46+02:00Jean-Matthieu GallardGradQ formatThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*...The format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/167Grid cells are not erased properly2017-08-16T20:03:29+02:00Ghost UserGrid cells are not erased properly... only the cell description is deleted. Empty remains in the grid.
Furthermore, we do not erase all cells and cell descriptions at the end of the simulation.
We rely on the OS's process manager to free the application memory.... only the cell description is deleted. Empty remains in the grid.
Furthermore, we do not erase all cells and cell descriptions at the end of the simulation.
We rely on the OS's process manager to free the application memory.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/118Grid extends are wrong in 3D2018-06-15T15:13:43+02:00Ghost UserGrid extends are wrong in 3DLook at this screenshot, the Z range specified in the specfile is not correctly interpreted.
![exahype-grid-size](/uploads/fcc13d63a1b6e1fa14c54b72d27500e1/exahype-grid-size.png)Look at this screenshot, the Z range specified in the specfile is not correctly interpreted.
![exahype-grid-size](/uploads/fcc13d63a1b6e1fa14c54b72d27500e1/exahype-grid-size.png)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/121Grid Setup takes forever for aspect-ratio<0.4 domains.2018-06-15T15:13:43+02:00Ghost UserGrid Setup takes forever for aspect-ratio<0.4 domains.This was confirmed for the Finite Volumes, ADER-DG and Limiting ADER-DG solver
**Update**: width=1.0,0.4 works but not width = 1.0,0.3 and smaller aspect ratios width_y/width_x.
This might be a Peano bug since the if-body in the sn...This was confirmed for the Finite Volumes, ADER-DG and Limiting ADER-DG solver
**Update**: width=1.0,0.4 works but not width = 1.0,0.3 and smaller aspect ratios width_y/width_x.
This might be a Peano bug since the if-body in the snippet below
is not entered anymore after the first grid setup iteration.
```
void exahype::mappings::MeshRefinement::refineVertexIfNecessary(
exahype::Vertex& fineGridVertex,
const tarch::la::Vector<DIMENSIONS, double>& fineGridH,
bool isCalledByCreationalEvent
) const {
for (const auto& p : exahype::solvers::RegisteredSolvers) {
if (
fineGridVertex.getRefinementControl() == Vertex::Records::Unrefined
&&
tarch::la::allGreater(fineGridH,p->getMaximumMeshSize())
) {
logInfo("refineVertexIfNecessary(...)",fineGridH);
#ifdef Parallel
if (isCalledByCreationalEvent) {
fineGridVertex.enforceRefine();
}
else {
fineGridVertex.refine();
}
#else
fineGridVertex.refine();
#endif
}
}
}
```
Output:
```
0.00654268 info peano::utils::UserInterface::writeHeader() Application based upon the PDE framework Peano - 3rd Generation
0.0065589 info peano::utils::UserInterface::writeHeader() revision: 2509
0.00657392 info peano::utils::UserInterface::writeHeader() build: dim=2
0.00658727 info peano::utils::UserInterface::writeHeader() optimisations: d-loop persistent-attributes packed opt-static-subtrees recursion-unrolling persistent-regular-subtrees
0.00660133 info peano::utils::UserInterface::writeHeader() (C) 2005 - 2016 www.peano-framework.org
0.00661516 info peano::utils::UserInterface::writeHeader() processes: 1, threads: 1
0.00667906 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [1,1]
0.00671291 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [1,1]
0.00695658 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00709009 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00744748 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00749516 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00753284 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00765061 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00768328 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00771189 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00837231 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00840974 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00844097 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00893974 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00898051 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00901055 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0099957 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106082 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106461 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106778 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0154214 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.015456 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0154729 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0201931 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0202355 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0202694 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0283382 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0284777 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:208,numberOfBoundaryVertices:154,numberOfOuterVertices:352,numberOfInnerCells:144,numberOfOuterCells:306,numberOfInnerLeafVertices:104,numberOfBoundaryLeafVertices:64,numberOfOuterLeafVertices:176,numberOfInnerLeafCells:135,numberOfOuterLeafCells:266,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0285568 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.041122 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.04128 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:208,numberOfBoundaryVertices:154,numberOfOuterVertices:1106,numberOfInnerCells:144,numberOfOuterCells:729,numberOfInnerLeafVertices:104,numberOfBoundaryLeafVertices:64,numberOfOuterLeafVertices:534,numberOfInnerLeafCells:135,numberOfOuterLeafCells:642,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:0,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.041399 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0546217 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0550416 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1550,numberOfInnerCells:225,numberOfOuterCells:972,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:744,numberOfInnerLeafCells:135,numberOfOuterLeafCells:858,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:0,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0551374 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0675743 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0676742 info exahype::runners::Runner::createGrid() grid setup iteration #4, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:358,numberOfBoundaryVertices:94,numberOfOuterVertices:1026,numberOfInnerCells:225,numberOfOuterCells:702,numberOfInnerLeafVertices:179,numberOfBoundaryLeafVertices:34,numberOfOuterLeafVertices:498,numberOfInnerLeafCells:135,numberOfOuterLeafCells:618,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0677187 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0781031 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0782623 info exahype::runners::Runner::createGrid() grid setup iteration #5, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1106,numberOfInnerCells:225,numberOfOuterCells:729,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:534,numberOfInnerLeafCells:135,numberOfOuterLeafCells:642,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:0,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0783317 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0897303 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0898073 info exahype::runners::Runner::createGrid() grid setup iteration #6, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:358,numberOfBoundaryVertices:94,numberOfOuterVertices:1550,numberOfInnerCells:225,numberOfOuterCells:972,numberOfInnerLeafVertices:179,numberOfBoundaryLeafVertices:34,numberOfOuterLeafVertices:744,numberOfInnerLeafCells:135,numberOfOuterLeafCells:858,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:0,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0898409 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.098608 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0987325 info exahype::runners::Runner::createGrid() grid setup iteration #7, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1026,numberOfInnerCells:225,numberOfOuterCells:702,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:498,numberOfInnerLeafCells:135,numberOfOuterLeafCells:618,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
```