ExaHyPE issueshttps://gitlab.lrz.de/groups/exahype/-/issues2018-02-09T22:50:28+01:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/206CCZ4: Run BH until T=10002018-02-09T22:50:28+01:00Ghost UserCCZ4: Run BH until T=1000We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical t...We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical time in hours". M means "Mass" and roughly indicates for us that we refer to the simulation time.*https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/207"error: Neighbours cannot communicate." (Minimal working example Application)2018-02-12T15:25:32+01:00Ghost User"error: Neighbours cannot communicate." (Minimal working example Application)In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria....In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria.
In the submitted example, the code crashes after the first timestep with **error: Neighbours cannot communicate**.
```
12.1608 info memoryUsage =333 MB
12.5276 info grid setup iteration #40, idle-nodes=1, vertical solver communication=0
12.5276 info memoryUsage =333 MB
12.8925 info grid setup iteration #41, idle-nodes=1, vertical solver communication=0
12.8925 info memoryUsage =333 MB
13.2603 info grid setup iteration #42, idle-nodes=1, vertical solver communication=0
13.2603 info memoryUsage =333 MB
13.2604 info finished grid setup after 42 iterations
13.2604 info finalise mesh refinement and compute first time step size
13.4246 info initialised all data and computed first time step size
13.7191 info plotted initial solution (if specified) and computed first predictor
13.7191 info step 0 t_min =0
13.7192 info dt_min =0.0125926
13.7192 info memoryUsage =333 MB
13.7192 info plot
13.78 error Neighbours cannot communicate.
cell1=(solverNumber:0,neighbourMergePerformed:[1,0,1,0],isInside:[1,1,1,1],parentIndex:920,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.428,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:42627,solutionAverages:42630,solutionCompressed:-1,previousSolution:42626,previousSolutionAverages:42629,previousSolutionCompressed:-1,update:42628,updateAverages:42631,updateCompressed:-1,extrapolatedPredictor:42632,extrapolatedPredictorAverages:42634,extrapolatedPredictorCompressed:-1,fluctuation:42633,fluctuationAverages:42635,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[1,0,1,0],helperStatus:2,facewiseLimiterStatus:[2,0,1,0],limiterStatus:2,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36444784,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1287913656)
.cell2=(solverNumber:0,neighbourMergePerformed:[0,0,1,0],isInside:[1,1,1,1],parentIndex:933,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.5926,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:8218,solutionAverages:8221,solutionCompressed:-1,previousSolution:8217,previousSolutionAverages:8220,previousSolutionCompressed:-1,update:8219,updateAverages:8222,updateCompressed:-1,extrapolatedPredictor:8223,extrapolatedPredictorAverages:8225,extrapolatedPredictorCompressed:-1,fluctuation:8224,fluctuationAverages:42960,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[0,0,1,0],helperStatus:2,facewiseLimiterStatus:[0,0,0,0],limiterStatus:0,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36491056,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1285749088) (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp,line:1570)
terminate called without an active exception
Abgebrochen
```
The specfile describes a PDE with 1 scalar field (doing nothing), the setup runs on a notebook in <10 seconds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/208Random initial conditions for LimitingADERDGSolver pose a problem2019-04-15T12:39:44+02:00Ghost UserRandom initial conditions for LimitingADERDGSolver pose a problemCurrently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy...Currently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy certain conditions (PAD, DMP).
During the imposition of initial conditions, we then allocate a new FV patch and
impose the initial conditions on its FV solution degrees of freedom.
Random initial conditions
-------------------------
We have to ensure that FV and ADER-DG solution are consistent.
This is currently only ensured if we have deterministic initial conditions.
In case of random initial conditions, the imposition of initial conditions
on the ADER-DG solution and on the FV solution might lead to two different
outcomes.
A cell might encounter rough initial conditions while imposing the initial conditions
on the ADER-DG solution but it then might encounter smooth initial conditions
for the FV solution.
But more severely, both differ from each other.
A solution?
-----------
In case a cell allocates an FV patch and has a limiter status such that it will
compute with Finite Volumes in the first iteration, we will then treat this FV solution as
the correct one and will project it back on the ADER-DG solution.
This way ADER-DG and FV solution are consistent.
On the other hand, if the limiter status of the cell is such that the cell
will compute with ADER-DG in the first iteration, we will treat the ADER-DG solution as the correct
one, we will project the ADER-DG solution onto the FV solution. We will not impose initial
conditions on the FV solution.
FV and ADER-DG solution are consistent in this case as well.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Ghost UserToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always...> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TU...As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/212Number of Plotters is not detected2018-09-10T19:43:27+02:00Ghost UserNumber of Plotters is not detectedWhen you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even a...When you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even are hardly understandable in DEBUG mode:
```
0.0310583 15:50:15 [nils]rank:0 debug exahype::parser::Parser::getIdentifierForPlotter() found token notoken (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:999)
assertion in file /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp, line 1001 failed: token.compare(_noTokenFound) != 0
parameter token: notoken
parameter solverNumber: 0
parameter plotterNumber: 3
ExaHyPE-GRMHD: /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp:1001: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> exahype::parser::Parser::getIdentifierForPlotter(int, int) const: Assertion `false' failed.
Abgebrochen
```
but completely nonunderstandable in Release mode:
```
0.00535548 15:32:20 [nils]rank:0 error exahype::parser::Parser::getFirstSnapshotTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'time' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1046)
0.00538875 15:32:20 [nils]rank:0 error exahype::parser::Parser::getRepeatTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'repeat' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1067)
```
note that in this example, `n=4` and `m=3`, so especially plotter 3 looked allright where the error message tried to complain actually about the nonexisting plotter 4.
This is very bad. We need some better enforcement of this rule.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/213Named variables in VTK plots2018-09-10T19:38:52+02:00Ghost UserNamed variables in VTK plotsI want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for p...I want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for peano.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/214Faster mesh refinement2018-03-02T11:23:39+01:00Ghost UserFaster mesh refinementProblem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If...Problem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If a new cells is introduced to the tree it might not be yet be identified
as part of a regular subgrid and the operations performed on the cell
are thus parallelised.
Solution: If imposing initial conditions or evaluating a refinement criterion is too
expensive we could let the user choose to perform it as background task.
This might lead to more mesh setup iteration but potentially to a better
exploitation of the available cores. It should also benefit the hiding of MPI communication
during the mesh setup.
Problem 2 (Overall concurrency)
-------------------------------
ExaHyPE's regular shared-memory parallelism during the mesh setup is currently further limited as
multiple cells might write to the same vertex in order to set refinement events.
Solution: We should be able to solve this by inverting the control. The
vertex checks in touchVertexLastTime if any cell has set a refinement event,
and refines if that is the case. This would increase the concurrency of the
enterCell operations.
Problem 3 (Memory)
-------------------------------
ExaHyPE's initial mesh setup is performed at the beginning by a single rank.
Gradually more and more ranks are added. In order to prevent that
any of the ranks runs out of memory during the initial mesh setup,
it might make sense to only temporarily allocate memory, impose initial
conditions, evaluate the refinement criterion, and then free the memory again (better:
recylce it).
After the initial mesh setup, we would then allocate memory on all ranks
and impose initial conditions.
Problem 4 (Load Balancing)
-------------------------------
The load-balancing does currently only count the number of cells.
It does not take the different cell types in ExaHyPE's grid into account.
The helper cell types Descendant and Ancestor have way less work to do than
the compute cells of type Cell.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/218Horizontal detection of insufficiently refined mesh for LimitingADERDGSolver2018-03-07T11:58:59+01:00Ghost UserHorizontal detection of insufficiently refined mesh for LimitingADERDGSolverCurrently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
r...Currently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
requests which force the time stepping to stop.
Restricting to the next coarser parent implies non-global master-worker communication
in MPI builds. This is not good.
To get rid of this master-worker communication during the time-stepping,
I propose to extend the limiter status range by a few more than one OK statuses.
If such an OK status is then propagating into a virtual child cell (Descendant),
we know that the mesh is not sufficiently refined and we halt the time stepping.
Some philosophy:
From the updates of the flags, we should further be able to predict in which direction a shock
propagates. We can then select more carefully which cell to refine next.
I should further rethink my whole limiter-based mesh refinement. Maybe it is more advantageous,
to do some bottom-up flagging for refinement. Instead of the current top-down approach
where I use halo-refinement around the limited regions.
With MPI switched on, I wonder however how well or badly this will interplay with the
load balancing during the initial mesh refinement.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/219Cancel all predictor background jobs when a predictor rerun is necessary2018-03-26T11:15:21+02:00Ghost UserCancel all predictor background jobs when a predictor rerun is necessaryhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/226Use sensible Fortran flags2018-04-24T11:58:40+02:00Ghost UserUse sensible Fortran flagsAt the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently te...At the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently testing:
`-xCORE-AVX512 -fma -align array64byte`
and has mentioned that `-qopt-prefetch=3` worked well with the prototype code.
To be updated...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/228stableTimeStepSize kernels should return maximum eigenvalue for (classical) l...2018-06-14T16:10:38+02:00Ghost UserstableTimeStepSize kernels should return maximum eigenvalue for (classical) local time steppingIn general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step ...In general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step size by a factor k (k=3 in Peano) with decreasing mesh level.
The kernels should thus return the maximum eigenvalue, too, or only the maximum eigenvalue.
The minimum local time step size would then be computed according to:
```
dt_min = CFL * max_{over all cells} lambda / min_{over all cells} cellSize
```
which is different to what we are currently doing for global time stepping:
```
dt_min = CFL * min_{over all cells} ( lambda / cellSize )
```
The ADERDGSolver superclass would then decide which minimisation to
perform.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/145SegFault occurs if I turn conservative flux off2018-06-15T15:13:07+02:00Ghost UserSegFault occurs if I turn conservative flux offProject: DIM_LimitingADERDGProject: DIM_LimitingADERDGhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/129Access to neighbours for which kernels?2018-06-15T15:13:07+02:00Ghost UserAccess to neighbours for which kernels?Finite Volumes Solver:
- source; ex: bathymetry in SWEFinite Volumes Solver:
- source; ex: bathymetry in SWEhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/110Compression2018-06-15T15:13:07+02:00Ghost UserCompression## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedom## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedomhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/229Add level as parameter to adjustSolution function2018-06-19T11:10:57+02:00Ghost UserAdd level as parameter to adjustSolution functionThe seismic applications need this for topology interpolation.The seismic applications need this for topology interpolation.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/230In-Code documentation of "adjustSolution" function2018-06-21T14:16:01+02:00Ghost UserIn-Code documentation of "adjustSolution" functionIs it possible to add a comment for adjustSolution() in the header.
/**
* @see FiniteVolumesSolver
*/
void adjustSolution(const double* const x,const double t,const double dt, double* Q) override;
Also is there a...Is it possible to add a comment for adjustSolution() in the header.
/**
* @see FiniteVolumesSolver
*/
void adjustSolution(const double* const x,const double t,const double dt, double* Q) override;
Also is there any advise on how to get the corner points of the cell. I guess there was a adjustPointSolution() in the past but not anymore.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/232EulerFV headers2018-07-25T12:41:20+02:00Ghost UserEulerFV headersHi,
in EulerFV there seems to be a mismatch/inconsistency between the MyEulerSolver.cpp and MyEulerSolver.h
on the repository this is the function used:
void EulerFV::MyEulerSolver::eigenvalues(const double* const Q, const int normalNo...Hi,
in EulerFV there seems to be a mismatch/inconsistency between the MyEulerSolver.cpp and MyEulerSolver.h
on the repository this is the function used:
void EulerFV::MyEulerSolver::eigenvalues(const double* const Q, const int normalNonZeroIndex, double* lambda)
but there is no such signature in the header generated by the toolkit. I assume const int normalNonZeroIndex, const int d and const int dIndex refer to the same variable.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/241Improve Plotter infrastructure2018-09-04T20:38:44+02:00Ghost UserImprove Plotter infrastructureThis is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendr...This is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendre basis plotter cannot do patchwise mapping of quantities while VTK Cartesian basis plotter can).
The obvious answer to this problem is *abstraction*. I already started once when introducing a `Slicer` interface class to allow various kind of slicing operations without having to specify them in the individual plotters. They just pass the parameters given by the specfile.
In a similar way, we need more abstraction for the quantities mapping, for labeling of the written quantities (named instead of `Q0...QN`), interpolation on different subcell grids, Support of the *isTroubled* flag or in general bitmasks of patch statusses, etc. The following picture is a sketchy overview:
![plotters](/uploads/876457ebab375ee2e1a15bd32a43c1f3/plotters.png)
To be honest I made this ticket here only becaue I wanted to throw away that sketch paper. It is at least one year old.
Figure as PDF: [Sbizhub1318082520330.pdf](/uploads/d4279ca1621927df8155582c66726dd0/Sbizhub1318082520330.pdf)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/243Toolkit2: Create .gitignore2018-09-05T14:23:33+02:00Ghost UserToolkit2: Create .gitignoreWe could think about such an option to let the toolkit create a `.gitignore` in the output directory. That could hold something like
```
Abstract*
KernelCalls*
README_generated.md
```
However, these files are not solver-name-dependent ...We could think about such an option to let the toolkit create a `.gitignore` in the output directory. That could hold something like
```
Abstract*
KernelCalls*
README_generated.md
```
However, these files are not solver-name-dependent and easy to track in the global `.gitignore`, too.
Would a local `.gitignore` have any features beyond?