ExaHyPE issueshttps://gitlab.lrz.de/groups/exahype/-/issues2018-04-24T11:58:40+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/226Use sensible Fortran flags2018-04-24T11:58:40+02:00Ghost UserUse sensible Fortran flagsAt the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently te...At the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently testing:
`-xCORE-AVX512 -fma -align array64byte`
and has mentioned that `-qopt-prefetch=3` worked well with the prototype code.
To be updated...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/219Cancel all predictor background jobs when a predictor rerun is necessary2018-03-26T11:15:21+02:00Ghost UserCancel all predictor background jobs when a predictor rerun is necessaryhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/218Horizontal detection of insufficiently refined mesh for LimitingADERDGSolver2018-03-07T11:58:59+01:00Ghost UserHorizontal detection of insufficiently refined mesh for LimitingADERDGSolverCurrently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
r...Currently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
requests which force the time stepping to stop.
Restricting to the next coarser parent implies non-global master-worker communication
in MPI builds. This is not good.
To get rid of this master-worker communication during the time-stepping,
I propose to extend the limiter status range by a few more than one OK statuses.
If such an OK status is then propagating into a virtual child cell (Descendant),
we know that the mesh is not sufficiently refined and we halt the time stepping.
Some philosophy:
From the updates of the flags, we should further be able to predict in which direction a shock
propagates. We can then select more carefully which cell to refine next.
I should further rethink my whole limiter-based mesh refinement. Maybe it is more advantageous,
to do some bottom-up flagging for refinement. Instead of the current top-down approach
where I use halo-refinement around the limited regions.
With MPI switched on, I wonder however how well or badly this will interplay with the
load balancing during the initial mesh refinement.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/214Faster mesh refinement2018-03-02T11:23:39+01:00Ghost UserFaster mesh refinementProblem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If...Problem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If a new cells is introduced to the tree it might not be yet be identified
as part of a regular subgrid and the operations performed on the cell
are thus parallelised.
Solution: If imposing initial conditions or evaluating a refinement criterion is too
expensive we could let the user choose to perform it as background task.
This might lead to more mesh setup iteration but potentially to a better
exploitation of the available cores. It should also benefit the hiding of MPI communication
during the mesh setup.
Problem 2 (Overall concurrency)
-------------------------------
ExaHyPE's regular shared-memory parallelism during the mesh setup is currently further limited as
multiple cells might write to the same vertex in order to set refinement events.
Solution: We should be able to solve this by inverting the control. The
vertex checks in touchVertexLastTime if any cell has set a refinement event,
and refines if that is the case. This would increase the concurrency of the
enterCell operations.
Problem 3 (Memory)
-------------------------------
ExaHyPE's initial mesh setup is performed at the beginning by a single rank.
Gradually more and more ranks are added. In order to prevent that
any of the ranks runs out of memory during the initial mesh setup,
it might make sense to only temporarily allocate memory, impose initial
conditions, evaluate the refinement criterion, and then free the memory again (better:
recylce it).
After the initial mesh setup, we would then allocate memory on all ranks
and impose initial conditions.
Problem 4 (Load Balancing)
-------------------------------
The load-balancing does currently only count the number of cells.
It does not take the different cell types in ExaHyPE's grid into account.
The helper cell types Descendant and Ancestor have way less work to do than
the compute cells of type Cell.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/213Named variables in VTK plots2018-09-10T19:38:52+02:00Ghost UserNamed variables in VTK plotsI want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for p...I want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for peano.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/212Number of Plotters is not detected2018-09-10T19:43:27+02:00Ghost UserNumber of Plotters is not detectedWhen you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even a...When you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even are hardly understandable in DEBUG mode:
```
0.0310583 15:50:15 [nils]rank:0 debug exahype::parser::Parser::getIdentifierForPlotter() found token notoken (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:999)
assertion in file /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp, line 1001 failed: token.compare(_noTokenFound) != 0
parameter token: notoken
parameter solverNumber: 0
parameter plotterNumber: 3
ExaHyPE-GRMHD: /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp:1001: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> exahype::parser::Parser::getIdentifierForPlotter(int, int) const: Assertion `false' failed.
Abgebrochen
```
but completely nonunderstandable in Release mode:
```
0.00535548 15:32:20 [nils]rank:0 error exahype::parser::Parser::getFirstSnapshotTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'time' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1046)
0.00538875 15:32:20 [nils]rank:0 error exahype::parser::Parser::getRepeatTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'repeat' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1067)
```
note that in this example, `n=4` and `m=3`, so especially plotter 3 looked allright where the error message tried to complain actually about the nonexisting plotter 4.
This is very bad. We need some better enforcement of this rule.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TU...As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Ghost UserToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always...> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/208Random initial conditions for LimitingADERDGSolver pose a problem2019-04-15T12:39:44+02:00Ghost UserRandom initial conditions for LimitingADERDGSolver pose a problemCurrently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy...Currently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy certain conditions (PAD, DMP).
During the imposition of initial conditions, we then allocate a new FV patch and
impose the initial conditions on its FV solution degrees of freedom.
Random initial conditions
-------------------------
We have to ensure that FV and ADER-DG solution are consistent.
This is currently only ensured if we have deterministic initial conditions.
In case of random initial conditions, the imposition of initial conditions
on the ADER-DG solution and on the FV solution might lead to two different
outcomes.
A cell might encounter rough initial conditions while imposing the initial conditions
on the ADER-DG solution but it then might encounter smooth initial conditions
for the FV solution.
But more severely, both differ from each other.
A solution?
-----------
In case a cell allocates an FV patch and has a limiter status such that it will
compute with Finite Volumes in the first iteration, we will then treat this FV solution as
the correct one and will project it back on the ADER-DG solution.
This way ADER-DG and FV solution are consistent.
On the other hand, if the limiter status of the cell is such that the cell
will compute with ADER-DG in the first iteration, we will treat the ADER-DG solution as the correct
one, we will project the ADER-DG solution onto the FV solution. We will not impose initial
conditions on the FV solution.
FV and ADER-DG solution are consistent in this case as well.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/207"error: Neighbours cannot communicate." (Minimal working example Application)2018-02-12T15:25:32+01:00Ghost User"error: Neighbours cannot communicate." (Minimal working example Application)In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria....In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria.
In the submitted example, the code crashes after the first timestep with **error: Neighbours cannot communicate**.
```
12.1608 info memoryUsage =333 MB
12.5276 info grid setup iteration #40, idle-nodes=1, vertical solver communication=0
12.5276 info memoryUsage =333 MB
12.8925 info grid setup iteration #41, idle-nodes=1, vertical solver communication=0
12.8925 info memoryUsage =333 MB
13.2603 info grid setup iteration #42, idle-nodes=1, vertical solver communication=0
13.2603 info memoryUsage =333 MB
13.2604 info finished grid setup after 42 iterations
13.2604 info finalise mesh refinement and compute first time step size
13.4246 info initialised all data and computed first time step size
13.7191 info plotted initial solution (if specified) and computed first predictor
13.7191 info step 0 t_min =0
13.7192 info dt_min =0.0125926
13.7192 info memoryUsage =333 MB
13.7192 info plot
13.78 error Neighbours cannot communicate.
cell1=(solverNumber:0,neighbourMergePerformed:[1,0,1,0],isInside:[1,1,1,1],parentIndex:920,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.428,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:42627,solutionAverages:42630,solutionCompressed:-1,previousSolution:42626,previousSolutionAverages:42629,previousSolutionCompressed:-1,update:42628,updateAverages:42631,updateCompressed:-1,extrapolatedPredictor:42632,extrapolatedPredictorAverages:42634,extrapolatedPredictorCompressed:-1,fluctuation:42633,fluctuationAverages:42635,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[1,0,1,0],helperStatus:2,facewiseLimiterStatus:[2,0,1,0],limiterStatus:2,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36444784,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1287913656)
.cell2=(solverNumber:0,neighbourMergePerformed:[0,0,1,0],isInside:[1,1,1,1],parentIndex:933,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.5926,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:8218,solutionAverages:8221,solutionCompressed:-1,previousSolution:8217,previousSolutionAverages:8220,previousSolutionCompressed:-1,update:8219,updateAverages:8222,updateCompressed:-1,extrapolatedPredictor:8223,extrapolatedPredictorAverages:8225,extrapolatedPredictorCompressed:-1,fluctuation:8224,fluctuationAverages:42960,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[0,0,1,0],helperStatus:2,facewiseLimiterStatus:[0,0,0,0],limiterStatus:0,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36491056,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1285749088) (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp,line:1570)
terminate called without an active exception
Abgebrochen
```
The specfile describes a PDE with 1 scalar field (doing nothing), the setup runs on a notebook in <10 seconds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/206CCZ4: Run BH until T=10002018-02-09T22:50:28+01:00Ghost UserCCZ4: Run BH until T=1000We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical t...We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical time in hours". M means "Mass" and roughly indicates for us that we refer to the simulation time.*https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/202Segfault in some VTK plotters if variables = 02019-03-21T10:48:34+01:00Ghost UserSegfault in some VTK plotters if variables = 0https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/199Extend batching to global time stepping2017-11-21T14:41:53+01:00Ghost UserExtend batching to global time steppingI am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll ba...I am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll back to the state at the beginning of the batch.
This WiP issue is for tracking my progress and issues I ran into.
Reduction of time step size
---------------------------
While we will not update the currently used CFL time step size in every single iteration
in the planned batched global time stepping scheme,
we will search the minimum CFL time step size over all batch iterations.
This allows us to check if the used time step size did violate the CFL constraint
during the execution of the batch.
The consequence might then be to use the newly determined minimum batch time step size as
time step size for a rerun of the whole batch.
Basic modifications to the fused time stepping algorithm
--------------------------------------------------------
- We will store the previous solution only at the beginning of a batch.
The same happens with the previous time stamp and time step size.
- TBC...
Rollbacks:
----------
TBC...
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/197Let tarch VTK plotters pass file creation errors to ExaHyPE plotters2018-03-02T12:46:18+01:00Ghost UserLet tarch VTK plotters pass file creation errors to ExaHyPE plottersThis is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetH...This is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetHDF5 writer) while other silently ignore this issue (Tobias tarch-based VTK writers). Clearly, we want the user to be able to control whether problems at output shall be severe or not.
I can easily implement this in the ExaHyPE codebase, but I don't control peano, so I have to make sure Peano passes such errors into exahype's domain.
=> TODO @ Sven.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/193Inconsitencies with the solver constructors (ParserView, cmd line passing)2017-10-18T09:07:53+02:00Ghost UserInconsitencies with the solver constructors (ParserView, cmd line passing)Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and Limit...Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and LimitingADERDG solvers. While the Command line options passing (`std::vector<std::string>& cmdlineargs`) is mandatory in all solvers, they only get the `exahype::Parser::ParserView& constants` when there *are* constants in the specfile. This creates a lot of trouble for the users, for instance when they decide to introduce constants on an existing application because they need to introduce the new signature then on their own.
Even worse: Currently, the `ParserView&` works for FV solvers, but not for ADERDG solvers (at some point, a `ParserView` instead of a `ParserView&` is needed) and therefore also not for coupled LimitingADERDG solvers.
I know that Tobias wants to keep the first code the user see's in his solver as small as possible. So what about always passing the `cmdlineargs` and a `constants` reference and either storing everything in some class
```
struct SpecfileOptions { // or so
std::vector<std::string>& cmdlineargs;
exahype::Parser::ParserView& constants;
// could store even more, for instance general access to the parser,
// static build information, paths, etc.
};
```
and just pass a `SpecfileOptions& options` to the constructors and the `init` function.
As an alternative or additionally, we can make the `init` function virtual and put an empty implementation into the `Abstract*Solver.{cpp,h}`. Then the user could introduce it into his solver only when he needs it.
We are talking about startup code which is only called once, so this should really be as comfortable as possible, performance does not matter at all.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/192[Peano] Number of persistent subgrids is changing from iteration to iteration2017-10-16T11:09:26+02:00Ghost User[Peano] Number of persistent subgrids is changing from iteration to iterationWe observed that if a rank has distributed all its local work to workers,
the number of cells of this rank which are stored in regular subgrids
might change every second iteration.We observed that if a rank has distributed all its local work to workers,
the number of cells of this rank which are stored in regular subgrids
might change every second iteration.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/191Pass std::vectors to patchwise functions2017-10-13T20:55:05+02:00Ghost UserPass std::vectors to patchwise functions- It's safer - size is available e.g.
- Signatures are clearer. User kernels still get the raw pointers.
- We can just do vector.data() to pass a pointer to the existing kernels.- It's safer - size is available e.g.
- Signatures are clearer. User kernels still get the raw pointers.
- We can just do vector.data() to pass a pointer to the existing kernels.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/188Global Reduction of Time Stepping Data2017-10-11T11:50:37+02:00Ghost UserGlobal Reduction of Time Stepping DataI currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker...I currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker to master
all the way up to the global master rank.
I could use a simple MPI_Reduce and a simple MPI_Gather to
perform the above steps.~~
Postponed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/180Compression - take ghostLayers and padding into account2017-09-26T10:19:58+02:00Ghost UserCompression - take ghostLayers and padding into account- FiniteVolumesSolver - Compression should take ghostLayers+padding (alignment) into account
- ADERDGSolver - Compression should take padding (alignment) into account- FiniteVolumesSolver - Compression should take ghostLayers+padding (alignment) into account
- ADERDGSolver - Compression should take padding (alignment) into accounthttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/178Limiting ADER-DG: Solution min and max computation is very expensive2017-09-04T16:02:23+02:00Ghost UserLimiting ADER-DG: Solution min and max computation is very expensiveWe found that the min/max computation (in ExaHyPE) can currently be by a factor 10 more
expensive than the space-time predictor computation.
Test case was Euler equations in 3D with p=5 polynomials and a Sod shock tube scenario.We found that the min/max computation (in ExaHyPE) can currently be by a factor 10 more
expensive than the space-time predictor computation.
Test case was Euler equations in 3D with p=5 polynomials and a Sod shock tube scenario.