ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2017-07-12T17:49:05+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/163Have a maximum-timestep-size in the spec file2017-07-12T17:49:05+02:00Ghost UserHave a maximum-timestep-size in the spec fileThis value can then be used to prescribe a time step size as long.
As the solver does not require a smaller one at least.This value can then be used to prescribe a time step size as long.
As the solver does not require a smaller one at least.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/164Extrapolate space-time polynomials and wrap boundary condition imposition in ...2017-07-12T20:24:40+02:00Ghost UserExtrapolate space-time polynomials and wrap boundary condition imposition in time-integralThis will further be a first step for local time steppingThis will further be a first step for local time steppinghttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/165Minimise limiter status based halo refinement2017-08-15T16:59:19+02:00Ghost UserMinimise limiter status based halo refinementIf the a-posteriori troubled cell indicators indicate a troubled cell, we refine
this cell down to the finest mesh level specified by the user.
I further have to ensure that I can place helper cell layers around the troubled cell
also o...If the a-posteriori troubled cell indicators indicate a troubled cell, we refine
this cell down to the finest mesh level specified by the user.
I further have to ensure that I can place helper cell layers around the troubled cell
also on the finest mesh level. Thus, I need some halo refinement.
Currently, I refine all neighbours of a cell with limiter status 1 and 2 in order to ensure
that the diagonal neighbours are refined as well.
In principle, I could however refine the cells with have a limiter status of 1 and
the cells which have two neighbours with a limiter status 1.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/169Do not impose initial conditions in every mesh refinement iteration2017-11-21T13:42:50+01:00Ghost UserDo not impose initial conditions in every mesh refinement iterationIt turned out that this is quite costly.
It might be responsible for time outs encountered during the
initial grid setup.
Min and max search:
--------------------
- This should be done once in FinaliseMeshRefinement
since we have a lar...It turned out that this is quite costly.
It might be responsible for time outs encountered during the
initial grid setup.
Min and max search:
--------------------
- This should be done once in FinaliseMeshRefinement
since we have a larger concurrency level here.
- We further should move around some of the behaviour of LocalRecomputation
into FinaliseMeshRefinement.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/172High time outs required during mesh refinement iterations2017-08-25T11:24:13+02:00Ghost UserHigh time outs required during mesh refinement iterationsI could get a p=9 243^3 Euler scenario run (64x14 ranks) but only if
I set the timeout to a large value of 360 sec.
Issue presumed to be related to imposing initial conditions
on-the-fly during the grid setup iterations.
This would exp...I could get a p=9 243^3 Euler scenario run (64x14 ranks) but only if
I set the timeout to a large value of 360 sec.
Issue presumed to be related to imposing initial conditions
on-the-fly during the grid setup iterations.
This would explain the worse behaviour in 3d as well as
for the limiting ADER-DG solver.
For the latter, also the min and max is computed on-the-fly.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/173LimitingADERDGSolver with limiter layers = 1 crashes in certain scenarios2017-08-29T18:48:26+02:00Ghost UserLimitingADERDGSolver with limiter layers = 1 crashes in certain scenarios* ~~I might need to stop merging the limiter status on-the-fly if
there is only 1 limiter layer.~~
* Does also affect helper-layers =2 runs.
* The crash is prevented by choosing a stricter dmp-relaxation-parameter.
It seems to be a ...* ~~I might need to stop merging the limiter status on-the-fly if
there is only 1 limiter layer.~~
* Does also affect helper-layers =2 runs.
* The crash is prevented by choosing a stricter dmp-relaxation-parameter.
It seems to be a numerical issue with the MUSCL-Hancock scheme.
* I will check if the issue does not appear if I use the Godunov scheme again.
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/175Symbolic flux calculations reduce speed significantly2019-03-21T10:56:35+01:00Ghost UserSymbolic flux calculations reduce speed significantlyI just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classi...I just completed some Likwid performance measurements for the generic and optimised kernels.
For the optimised kernels, I tested a variant using "symbolic variables" in the
flux and eigenvalue computation and another variant using
classic array indexing (optimised-nonsymbolic).
The files are suffixed by a ".likwid.csv".
I further attached measured Peano adapter times
The files are suffixed by a ".csv".
Setup
--------
* Compressible Euler equations (Euler_Flow)
* pure ADER-DG scheme (no limiter)
* polynomial orders p=3,5,7,9;
regular 27^3 grid (3D)
* TBB threads=1,12,24.
* Intel icpc17 (USE_IPO=on).
* nonfused (3 algorithmic phases) vs. fused (a single pipelined algorithmic phase) ADER-DG implementation
* no predictor reruns did occur for the fused implementation
Preliminary Results
----------------------------
* Optimised kernels are faster than the generic ones (I kind of expected this 😉)
* Raw array access (optimised-nonsymbolic) is significantly faster than using the "symbolic variables"(optimised).
* Fused scheme pays off (as long as number of reruns is low; very interesting for linear PDEs (no reruns here))
Files
-----
[Euler_ADERDG-no-output-generic.csv](/uploads/f671c93a33e70c9549853f1f51518c9d/Euler_ADERDG-no-output-generic.csv)
[Euler_ADERDG-no-output-generic.likwid.csv](/uploads/0d52ad214f0b1d6cfae4d658a9997cb5/Euler_ADERDG-no-output-generic.likwid.csv)
[Euler_ADERDG-no-output-optimised.csv](/uploads/422288cdc222b1b250b3aad9e2ad73a1/Euler_ADERDG-no-output-optimised.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.csv](/uploads/0f84240941230d4bc9f06c76b154396b/Euler_ADERDG-no-output-optimised-nonsymbolic.csv)
[Euler_ADERDG-no-output-optimised.likwid.csv](/uploads/135bd71c668ca0c4c4ee5b7988ea2f17/Euler_ADERDG-no-output-optimised.likwid.csv)
[Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv](/uploads/7606eefe559499a1f6de62b5f33905d0/Euler_ADERDG-no-output-optimised-nonsymbolic.likwid.csv)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/177Batching required for LTS and ExaHyPE's limiter-based refinement2017-09-04T14:16:41+02:00Ghost UserBatching required for LTS and ExaHyPE's limiter-based refinementLimiting ADER-DG in ExaHyPE
---------------------------
In ExaHyPE, we do not consider adaptive refinement for the Finite Volumes (FV)
solver.
We thus only use FV on the finest level of the adaptive mesh
if we employ a limiting ADER-DG ...Limiting ADER-DG in ExaHyPE
---------------------------
In ExaHyPE, we do not consider adaptive refinement for the Finite Volumes (FV)
solver.
We thus only use FV on the finest level of the adaptive mesh
if we employ a limiting ADER-DG solver.
If a limiting ADER-DG solver detects a shock, i.e. a cell where we need to
limit non-physical oscillations, we refine it down to
the finest adaptive mesh level.
And we further refine its neighbours down to the finest mesh level.
Local Time Stepping
-------------------
TBC
Local Time Stepping plus limiting ADER-DG in ExaHyPE
-----------------------------------------------------
It is possible that a cell is marked as troubled during the
local time stepping. This will require potentially
more refinement around the cell, or to refine the cell itself.
The newly refined cells might not have done any time stepping
at all during the current LTS batch since
their parents stem from a coarser level.
They lag behind in time.
The question is what to do in those scenarios.
Batching
--------
One idea is to consider the number of local time steps to run as a batch.
We remember the solution before a batch starts.
In case (limiter-based) refinement is triggered, we memorise the cells to refine,
perform a rollback to the memorised solution.
Afterwards, we refine the memorised cells and run the batch again.
It can happen multiple times that we have to rerun a batch.
On the other hand, we ensure that our grid is always tracking a shock correctly.
Further techniques
------------------
On the expense of computational cost, we could refine the mesh generously (according to the restricted limiter status)
This would reduce the number of batch reruns.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/178Limiting ADER-DG: Solution min and max computation is very expensive2017-09-04T16:02:23+02:00Ghost UserLimiting ADER-DG: Solution min and max computation is very expensiveWe found that the min/max computation (in ExaHyPE) can currently be by a factor 10 more
expensive than the space-time predictor computation.
Test case was Euler equations in 3D with p=5 polynomials and a Sod shock tube scenario.We found that the min/max computation (in ExaHyPE) can currently be by a factor 10 more
expensive than the space-time predictor computation.
Test case was Euler equations in 3D with p=5 polynomials and a Sod shock tube scenario.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/180Compression - take ghostLayers and padding into account2017-09-26T10:19:58+02:00Ghost UserCompression - take ghostLayers and padding into account- FiniteVolumesSolver - Compression should take ghostLayers+padding (alignment) into account
- ADERDGSolver - Compression should take padding (alignment) into account- FiniteVolumesSolver - Compression should take ghostLayers+padding (alignment) into account
- ADERDGSolver - Compression should take padding (alignment) into accounthttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/188Global Reduction of Time Stepping Data2017-10-11T11:50:37+02:00Ghost UserGlobal Reduction of Time Stepping DataI currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker...I currently do the following to reduce and broadcast global
values, like e.g. the minimum time step size:
* I broadcast time step data from master to worker
all the way down to the "lowest" worker.
* I reduce time step data from worker to master
all the way up to the global master rank.
I could use a simple MPI_Reduce and a simple MPI_Gather to
perform the above steps.~~
Postponed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/191Pass std::vectors to patchwise functions2017-10-13T20:55:05+02:00Ghost UserPass std::vectors to patchwise functions- It's safer - size is available e.g.
- Signatures are clearer. User kernels still get the raw pointers.
- We can just do vector.data() to pass a pointer to the existing kernels.- It's safer - size is available e.g.
- Signatures are clearer. User kernels still get the raw pointers.
- We can just do vector.data() to pass a pointer to the existing kernels.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/192[Peano] Number of persistent subgrids is changing from iteration to iteration2017-10-16T11:09:26+02:00Ghost User[Peano] Number of persistent subgrids is changing from iteration to iterationWe observed that if a rank has distributed all its local work to workers,
the number of cells of this rank which are stored in regular subgrids
might change every second iteration.We observed that if a rank has distributed all its local work to workers,
the number of cells of this rank which are stored in regular subgrids
might change every second iteration.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/193Inconsitencies with the solver constructors (ParserView, cmd line passing)2017-10-18T09:07:53+02:00Ghost UserInconsitencies with the solver constructors (ParserView, cmd line passing)Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and Limit...Note: This is a _minor_ long term issue.
I noticed (once again) that we have inconsistencies in the handling of the user solver constructor, the abstract solver constructor and the user `init` function, all this for FV, ADERDG and LimitingADERDG solvers. While the Command line options passing (`std::vector<std::string>& cmdlineargs`) is mandatory in all solvers, they only get the `exahype::Parser::ParserView& constants` when there *are* constants in the specfile. This creates a lot of trouble for the users, for instance when they decide to introduce constants on an existing application because they need to introduce the new signature then on their own.
Even worse: Currently, the `ParserView&` works for FV solvers, but not for ADERDG solvers (at some point, a `ParserView` instead of a `ParserView&` is needed) and therefore also not for coupled LimitingADERDG solvers.
I know that Tobias wants to keep the first code the user see's in his solver as small as possible. So what about always passing the `cmdlineargs` and a `constants` reference and either storing everything in some class
```
struct SpecfileOptions { // or so
std::vector<std::string>& cmdlineargs;
exahype::Parser::ParserView& constants;
// could store even more, for instance general access to the parser,
// static build information, paths, etc.
};
```
and just pass a `SpecfileOptions& options` to the constructors and the `init` function.
As an alternative or additionally, we can make the `init` function virtual and put an empty implementation into the `Abstract*Solver.{cpp,h}`. Then the user could introduce it into his solver only when he needs it.
We are talking about startup code which is only called once, so this should really be as comfortable as possible, performance does not matter at all.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/197Let tarch VTK plotters pass file creation errors to ExaHyPE plotters2018-03-02T12:46:18+01:00Ghost UserLet tarch VTK plotters pass file creation errors to ExaHyPE plottersThis is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetH...This is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetHDF5 writer) while other silently ignore this issue (Tobias tarch-based VTK writers). Clearly, we want the user to be able to control whether problems at output shall be severe or not.
I can easily implement this in the ExaHyPE codebase, but I don't control peano, so I have to make sure Peano passes such errors into exahype's domain.
=> TODO @ Sven.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/199Extend batching to global time stepping2017-11-21T14:41:53+01:00Ghost UserExtend batching to global time steppingI am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll ba...I am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll back to the state at the beginning of the batch.
This WiP issue is for tracking my progress and issues I ran into.
Reduction of time step size
---------------------------
While we will not update the currently used CFL time step size in every single iteration
in the planned batched global time stepping scheme,
we will search the minimum CFL time step size over all batch iterations.
This allows us to check if the used time step size did violate the CFL constraint
during the execution of the batch.
The consequence might then be to use the newly determined minimum batch time step size as
time step size for a rerun of the whole batch.
Basic modifications to the fused time stepping algorithm
--------------------------------------------------------
- We will store the previous solution only at the beginning of a batch.
The same happens with the previous time stamp and time step size.
- TBC...
Rollbacks:
----------
TBC...
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/202Segfault in some VTK plotters if variables = 02019-03-21T10:48:34+01:00Ghost UserSegfault in some VTK plotters if variables = 0https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/206CCZ4: Run BH until T=10002018-02-09T22:50:28+01:00Sven KöppelCCZ4: Run BH until T=1000We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical t...We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical time in hours". M means "Mass" and roughly indicates for us that we refer to the simulation time.*https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/207"error: Neighbours cannot communicate." (Minimal working example Application)2018-02-12T15:25:32+01:00Sven Köppel"error: Neighbours cannot communicate." (Minimal working example Application)In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria....In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria.
In the submitted example, the code crashes after the first timestep with **error: Neighbours cannot communicate**.
```
12.1608 info memoryUsage =333 MB
12.5276 info grid setup iteration #40, idle-nodes=1, vertical solver communication=0
12.5276 info memoryUsage =333 MB
12.8925 info grid setup iteration #41, idle-nodes=1, vertical solver communication=0
12.8925 info memoryUsage =333 MB
13.2603 info grid setup iteration #42, idle-nodes=1, vertical solver communication=0
13.2603 info memoryUsage =333 MB
13.2604 info finished grid setup after 42 iterations
13.2604 info finalise mesh refinement and compute first time step size
13.4246 info initialised all data and computed first time step size
13.7191 info plotted initial solution (if specified) and computed first predictor
13.7191 info step 0 t_min =0
13.7192 info dt_min =0.0125926
13.7192 info memoryUsage =333 MB
13.7192 info plot
13.78 error Neighbours cannot communicate.
cell1=(solverNumber:0,neighbourMergePerformed:[1,0,1,0],isInside:[1,1,1,1],parentIndex:920,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.428,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:42627,solutionAverages:42630,solutionCompressed:-1,previousSolution:42626,previousSolutionAverages:42629,previousSolutionCompressed:-1,update:42628,updateAverages:42631,updateCompressed:-1,extrapolatedPredictor:42632,extrapolatedPredictorAverages:42634,extrapolatedPredictorCompressed:-1,fluctuation:42633,fluctuationAverages:42635,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[1,0,1,0],helperStatus:2,facewiseLimiterStatus:[2,0,1,0],limiterStatus:2,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36444784,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1287913656)
.cell2=(solverNumber:0,neighbourMergePerformed:[0,0,1,0],isInside:[1,1,1,1],parentIndex:933,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.5926,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:8218,solutionAverages:8221,solutionCompressed:-1,previousSolution:8217,previousSolutionAverages:8220,previousSolutionCompressed:-1,update:8219,updateAverages:8222,updateCompressed:-1,extrapolatedPredictor:8223,extrapolatedPredictorAverages:8225,extrapolatedPredictorCompressed:-1,fluctuation:8224,fluctuationAverages:42960,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[0,0,1,0],helperStatus:2,facewiseLimiterStatus:[0,0,0,0],limiterStatus:0,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36491056,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1285749088) (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp,line:1570)
terminate called without an active exception
Abgebrochen
```
The specfile describes a PDE with 1 scalar field (doing nothing), the setup runs on a notebook in <10 seconds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/208Random initial conditions for LimitingADERDGSolver pose a problem2019-04-15T12:39:44+02:00Dominic Etienne CharrierRandom initial conditions for LimitingADERDGSolver pose a problemCurrently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy...Currently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy certain conditions (PAD, DMP).
During the imposition of initial conditions, we then allocate a new FV patch and
impose the initial conditions on its FV solution degrees of freedom.
Random initial conditions
-------------------------
We have to ensure that FV and ADER-DG solution are consistent.
This is currently only ensured if we have deterministic initial conditions.
In case of random initial conditions, the imposition of initial conditions
on the ADER-DG solution and on the FV solution might lead to two different
outcomes.
A cell might encounter rough initial conditions while imposing the initial conditions
on the ADER-DG solution but it then might encounter smooth initial conditions
for the FV solution.
But more severely, both differ from each other.
A solution?
-----------
In case a cell allocates an FV patch and has a limiter status such that it will
compute with Finite Volumes in the first iteration, we will then treat this FV solution as
the correct one and will project it back on the ADER-DG solution.
This way ADER-DG and FV solution are consistent.
On the other hand, if the limiter status of the cell is such that the cell
will compute with ADER-DG in the first iteration, we will treat the ADER-DG solution as the correct
one, we will project the ADER-DG solution onto the FV solution. We will not impose initial
conditions on the FV solution.
FV and ADER-DG solution are consistent in this case as well.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Sven KöppelToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always...> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TU...As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/212Number of Plotters is not detected2018-09-10T19:43:27+02:00Sven KöppelNumber of Plotters is not detectedWhen you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even a...When you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even are hardly understandable in DEBUG mode:
```
0.0310583 15:50:15 [nils]rank:0 debug exahype::parser::Parser::getIdentifierForPlotter() found token notoken (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:999)
assertion in file /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp, line 1001 failed: token.compare(_noTokenFound) != 0
parameter token: notoken
parameter solverNumber: 0
parameter plotterNumber: 3
ExaHyPE-GRMHD: /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp:1001: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> exahype::parser::Parser::getIdentifierForPlotter(int, int) const: Assertion `false' failed.
Abgebrochen
```
but completely nonunderstandable in Release mode:
```
0.00535548 15:32:20 [nils]rank:0 error exahype::parser::Parser::getFirstSnapshotTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'time' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1046)
0.00538875 15:32:20 [nils]rank:0 error exahype::parser::Parser::getRepeatTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'repeat' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1067)
```
note that in this example, `n=4` and `m=3`, so especially plotter 3 looked allright where the error message tried to complain actually about the nonexisting plotter 4.
This is very bad. We need some better enforcement of this rule.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/213Named variables in VTK plots2018-09-10T19:38:52+02:00Sven KöppelNamed variables in VTK plotsI want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for p...I want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for peano.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/214Faster mesh refinement2018-03-02T11:23:39+01:00Dominic Etienne CharrierFaster mesh refinementProblem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If...Problem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If a new cells is introduced to the tree it might not be yet be identified
as part of a regular subgrid and the operations performed on the cell
are thus parallelised.
Solution: If imposing initial conditions or evaluating a refinement criterion is too
expensive we could let the user choose to perform it as background task.
This might lead to more mesh setup iteration but potentially to a better
exploitation of the available cores. It should also benefit the hiding of MPI communication
during the mesh setup.
Problem 2 (Overall concurrency)
-------------------------------
ExaHyPE's regular shared-memory parallelism during the mesh setup is currently further limited as
multiple cells might write to the same vertex in order to set refinement events.
Solution: We should be able to solve this by inverting the control. The
vertex checks in touchVertexLastTime if any cell has set a refinement event,
and refines if that is the case. This would increase the concurrency of the
enterCell operations.
Problem 3 (Memory)
-------------------------------
ExaHyPE's initial mesh setup is performed at the beginning by a single rank.
Gradually more and more ranks are added. In order to prevent that
any of the ranks runs out of memory during the initial mesh setup,
it might make sense to only temporarily allocate memory, impose initial
conditions, evaluate the refinement criterion, and then free the memory again (better:
recylce it).
After the initial mesh setup, we would then allocate memory on all ranks
and impose initial conditions.
Problem 4 (Load Balancing)
-------------------------------
The load-balancing does currently only count the number of cells.
It does not take the different cell types in ExaHyPE's grid into account.
The helper cell types Descendant and Ancestor have way less work to do than
the compute cells of type Cell.Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/218Horizontal detection of insufficiently refined mesh for LimitingADERDGSolver2018-03-07T11:58:59+01:00Dominic Etienne CharrierHorizontal detection of insufficiently refined mesh for LimitingADERDGSolverCurrently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
r...Currently, we restrict the limiter status up to the next coarser parent
in every iteration of the time stepping. We then evaluate on the coarser grids
if the limiter status is such that we need to refine. This then triggers refinement
requests which force the time stepping to stop.
Restricting to the next coarser parent implies non-global master-worker communication
in MPI builds. This is not good.
To get rid of this master-worker communication during the time-stepping,
I propose to extend the limiter status range by a few more than one OK statuses.
If such an OK status is then propagating into a virtual child cell (Descendant),
we know that the mesh is not sufficiently refined and we halt the time stepping.
Some philosophy:
From the updates of the flags, we should further be able to predict in which direction a shock
propagates. We can then select more carefully which cell to refine next.
I should further rethink my whole limiter-based mesh refinement. Maybe it is more advantageous,
to do some bottom-up flagging for refinement. Instead of the current top-down approach
where I use halo-refinement around the limited regions.
With MPI switched on, I wonder however how well or badly this will interplay with the
load balancing during the initial mesh refinement.Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/219Cancel all predictor background jobs when a predictor rerun is necessary2018-03-26T11:15:21+02:00Dominic Etienne CharrierCancel all predictor background jobs when a predictor rerun is necessaryhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/226Use sensible Fortran flags2018-04-24T11:58:40+02:00Ben HazelwoodUse sensible Fortran flagsAt the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently te...At the moment the fortran code is compiled with minimal flags, and therefore the compiler has its hands tied. However, the code is written in a way which should be easily vectorisable by the compiler.
Ekatherine from RSC is currently testing:
`-xCORE-AVX512 -fma -align array64byte`
and has mentioned that `-qopt-prefetch=3` worked well with the prototype code.
To be updated...Ben HazelwoodBen Hazelwoodhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/228stableTimeStepSize kernels should return maximum eigenvalue for (classical) l...2018-06-14T16:10:38+02:00Dominic Etienne CharrierstableTimeStepSize kernels should return maximum eigenvalue for (classical) local time steppingIn general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step ...In general, we cannot associate the smallest time step size with the finest mesh
level as there might be larger eigenvalues present on coarser mesh levels.
This is an issue for local time stepping where you scale the smallest time
step size by a factor k (k=3 in Peano) with decreasing mesh level.
The kernels should thus return the maximum eigenvalue, too, or only the maximum eigenvalue.
The minimum local time step size would then be computed according to:
```
dt_min = CFL * max_{over all cells} lambda / min_{over all cells} cellSize
```
which is different to what we are currently doing for global time stepping:
```
dt_min = CFL * min_{over all cells} ( lambda / cellSize )
```
The ADERDGSolver superclass would then decide which minimisation to
perform.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/145SegFault occurs if I turn conservative flux off2018-06-15T15:13:07+02:00Ghost UserSegFault occurs if I turn conservative flux offProject: DIM_LimitingADERDGProject: DIM_LimitingADERDGhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/129Access to neighbours for which kernels?2018-06-15T15:13:07+02:00Ghost UserAccess to neighbours for which kernels?Finite Volumes Solver:
- source; ex: bathymetry in SWEFinite Volumes Solver:
- source; ex: bathymetry in SWEhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/110Compression2018-06-15T15:13:07+02:00Ghost UserCompression## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedom## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedomhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/229Add level as parameter to adjustSolution function2018-06-19T11:10:57+02:00Ghost UserAdd level as parameter to adjustSolution functionThe seismic applications need this for topology interpolation.The seismic applications need this for topology interpolation.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/230In-Code documentation of "adjustSolution" function2018-06-21T14:16:01+02:00Ghost UserIn-Code documentation of "adjustSolution" functionIs it possible to add a comment for adjustSolution() in the header.
/**
* @see FiniteVolumesSolver
*/
void adjustSolution(const double* const x,const double t,const double dt, double* Q) override;
Also is there a...Is it possible to add a comment for adjustSolution() in the header.
/**
* @see FiniteVolumesSolver
*/
void adjustSolution(const double* const x,const double t,const double dt, double* Q) override;
Also is there any advise on how to get the corner points of the cell. I guess there was a adjustPointSolution() in the past but not anymore.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/232EulerFV headers2018-07-25T12:41:20+02:00Ghost UserEulerFV headersHi,
in EulerFV there seems to be a mismatch/inconsistency between the MyEulerSolver.cpp and MyEulerSolver.h
on the repository this is the function used:
void EulerFV::MyEulerSolver::eigenvalues(const double* const Q, const int normalNo...Hi,
in EulerFV there seems to be a mismatch/inconsistency between the MyEulerSolver.cpp and MyEulerSolver.h
on the repository this is the function used:
void EulerFV::MyEulerSolver::eigenvalues(const double* const Q, const int normalNonZeroIndex, double* lambda)
but there is no such signature in the header generated by the toolkit. I assume const int normalNonZeroIndex, const int d and const int dIndex refer to the same variable.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/241Improve Plotter infrastructure2018-09-04T20:38:44+02:00Sven KöppelImprove Plotter infrastructureThis is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendr...This is a long term goal:
The plotters were growing features in a weird way with tons of code duplication and the fact that now some plotters have features which others don't and there is no obvious reason why (for instance: VTK Legendre basis plotter cannot do patchwise mapping of quantities while VTK Cartesian basis plotter can).
The obvious answer to this problem is *abstraction*. I already started once when introducing a `Slicer` interface class to allow various kind of slicing operations without having to specify them in the individual plotters. They just pass the parameters given by the specfile.
In a similar way, we need more abstraction for the quantities mapping, for labeling of the written quantities (named instead of `Q0...QN`), interpolation on different subcell grids, Support of the *isTroubled* flag or in general bitmasks of patch statusses, etc. The following picture is a sketchy overview:
![plotters](/uploads/876457ebab375ee2e1a15bd32a43c1f3/plotters.png)
To be honest I made this ticket here only becaue I wanted to throw away that sketch paper. It is at least one year old.
Figure as PDF: [Sbizhub1318082520330.pdf](/uploads/d4279ca1621927df8155582c66726dd0/Sbizhub1318082520330.pdf)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/243Toolkit2: Create .gitignore2018-09-05T14:23:33+02:00Sven KöppelToolkit2: Create .gitignoreWe could think about such an option to let the toolkit create a `.gitignore` in the output directory. That could hold something like
```
Abstract*
KernelCalls*
README_generated.md
```
However, these files are not solver-name-dependent ...We could think about such an option to let the toolkit create a `.gitignore` in the output directory. That could hold something like
```
Abstract*
KernelCalls*
README_generated.md
```
However, these files are not solver-name-dependent and easy to track in the global `.gitignore`, too.
Would a local `.gitignore` have any features beyond?https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/244Toolkit2: Should have --pretend and --clean option2019-04-15T12:40:31+02:00Sven KöppelToolkit2: Should have --pretend and --clean optionEvery invocation of the toolkit currently overwrites certain files. However, as a user, I miss two options:
* Clean the generated files, especially the glue files. For a *novice user*, it is not obvious what to clean when wanting certai...Every invocation of the toolkit currently overwrites certain files. However, as a user, I miss two options:
* Clean the generated files, especially the glue files. For a *novice user*, it is not obvious what to clean when wanting certain files to be regenerated. A `--clean` file should exeute `rm Abstract* KernelCalls*`
* Explain what the toolkit would do. Some tools such as [rename(1)](http://man7.org/linux/man-pages/man1/rename.1.html) or [rsync(1)](https://linux.die.net/man/1/rsync) have an option called `--pretend`, `--dry-run` or `--no-act` which *explain* what the tool would do. We should offer such an option to make it more transparent what especially a second run of the toolkit does.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/246Toolkit2: If it is configure, it should test the make system2018-09-01T15:54:55+02:00Sven KöppelToolkit2: If it is configure, it should test the make systemCodes using the popular [Autotools/Autoconf ecosystem](https://www.gnu.org/software/autoconf/manual/autoconf.html) expect users to compile code by something like
```
wget http://example.com/code.tar.gz && unpack || git clone ...
./confi...Codes using the popular [Autotools/Autoconf ecosystem](https://www.gnu.org/software/autoconf/manual/autoconf.html) expect users to compile code by something like
```
wget http://example.com/code.tar.gz && unpack || git clone ...
./configure --with-MPI --with-TBB --without-lapack
make
```
Configure does several things, but typically it
* Checks whether the neccessary *compilers and libraries* are installed by doing test compilations and executions
* Generates a Makefile
We probably all dislike configure for doing these lengthy tests which always consume time.
In similarity to *configure*, some ExaHyPE users (notably Dominic) interpret the ExaHyPE toolkit as a *configure step*:
```
wget http://exahype.eu/obtain/exahype.tar.gz ...
./toolkit.sh PathToSpecfile.exahype
make
```
We know this analogy at least does not hold because some build decisions are made at making time, for instance the choice for MPI, TBB and Assertions. Our toolkit never executes any compiler, it only generates code. There is a clean distinction.
However, frequently compilation fails and users must find out why. What a real *configure* step would provide is a check for compiler capabilities. With tough problems, users have to debug these capabilities anyway, for instance by [compiling and running my compiler challanges](https://bitbucket.org/svek/compiler-challenges).
I suggest we add an option `--test` to the Toolkit which brings in some *configure* flavour. When enabled, after the invocation of the regular Toolkit it could compile and run some tests and test for
* A valid C++11 and Fortran (long lines + REAL=8 bytes) compiler (test programs)
* Presence of a modern and working TBB (test program)
* Presence of a working MPI (test program)
Of course, running these tests probably requires to have some flags such as `EXAHYPE_CC` already set.
As an alternative, I suggest we have another Makefile target for our make system: `make test`, which does not compile the code but instead compiles (and runs) these tests.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/249Toolkit2: Proper handling of parameters to plotters2019-04-15T12:36:17+02:00Sven KöppelToolkit2: Proper handling of parameters to plottersJM reports:
```
vers/0/plotters/0/parameters
0.0855118 19:11:35 [mpp2r02c02s08]rank:0 info exahype::plotters::Plotter::Plotter(...) write snapshot to file ./reference_solution every 0.005 time units with f...JM reports:
```
vers/0/plotters/0/parameters
0.0855118 19:11:35 [mpp2r02c02s08]rank:0 info exahype::plotters::Plotter::Plotter(...) write snapshot to file ./reference_solution every 0.005 time units with first snapshot at 0. plotter type is probe::ascii. Plotter configuration=(solver no=0,plotter identifier (type)=probe::ascii,written unknowns=9,time=0,repeat=0.005,file name=./reference_solution,plotter parameters={"select":{"x":0.5,"y":0.5,"z":0.8}},device configured=0)
0.085593 19:11:35 [mpp2r02c02s08]rank:0 error exahype::plotters::ADERDG2ProbeAscii::init() Probe location is invalid. Require x,y,z values. Have {"select":{"x":0.5,"y":0.5,"z":0.8}} (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./ExaHyPE/exahype/plotters/ADERDG2ProbeAscii.cpp,line:56)
0.085651 19:11:35 [mpp2r02c02s08]rank:0 error exahype::parser::Parser::getIntFromPath() Missing entry /solvers/0/plotters/0/parameters/x ([json.exception.out_of_range.403] key 'x' not found) (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./ExaHyPE/exahype/parser/Parser.cpp,line:346)
0.0856928 19:11:35 [mpp2r02c02s08]rank:0 error exahype::parser::Parser::getIntFromPath() Missing entry /solvers/0/plotters/0/parameters/y ([json.exception.out_of_range.403] key 'y' not found) (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./ExaHyPE/exahype/parser/Parser.cpp,line:346)
0.0857256 19:11:35 [mpp2r02c02s08]rank:0 error exahype::parser::Parser::getIntFromPath() Missing entry /solvers/0/plotters/0/parameters/z ([json.exception.out_of_range.403] key 'z' not found) (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./ExaHyPE/exahype/parser/Parser.cpp,line:346)
0.0879261 error was not able to open input file exahype.log-filter (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./Peano/tarch/logging/LogFilterFileReader.cpp,line:88)
0.0879611 warning filter file exahype.log-filter was invalid. Switch on all log statements (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./Peano/tarch/logging/LogFilterFileReader.cpp,line:105)
0.0879824 error do not run code as parser reported errors (file:/home/hpc/pr63so/di57wuf/ExaHyPE-Engine/toolkit2-linear_flux_ncp_mm.exahype-build/./ExaHyPE/exahype/runners/Runner.cpp,line:691)
0.0879934 info quit with error code 1
status=COMPLETED
```
This does not seem to be a fault by the C++ Parser but maybe instead by the `classic-exahype -> JSON` converter in the toolkit.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/258Minfy JSON (filter out comments and son) before deserialisation2018-11-13T18:10:26+01:00Dominic Etienne CharrierMinfy JSON (filter out comments and son) before deserialisationThe JSON parser allows specifying a callback for this purpose:
```
/** [...]
@param[in] i input to read from
@param[in] cb a parser callback function of type @ref parser_callback_t
which is used to control the deseriali...The JSON parser allows specifying a callback for this purpose:
```
/** [...]
@param[in] i input to read from
@param[in] cb a parser callback function of type @ref parser_callback_t
which is used to control the deserialization by filtering unwanted values
(optional)
*/
static basic_json parse(detail::input_adapter&& i,
const parser_callback_t cb = nullptr,
const bool allow_exceptions = true)
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/260Some nonlinear behavior with MPI2019-01-18T10:27:26+01:00m.tavelliSome nonlinear behavior with MPII got some strange behavior when I try to run GPR on the new skylake i9 here in Trento. The application is the GPR one used with the GPR_2D.exahype spec file. The test is a travelling smooth shock and it runs with non static limiter and ...I got some strange behavior when I try to run GPR on the new skylake i9 here in Trento. The application is the GPR one used with the GPR_2D.exahype spec file. The test is a travelling smooth shock and it runs with non static limiter and AMR. I report some results we got:
2D 4 TBB 1MPI 1 AMR NoOpt ok in 177,54s
2D 4 TBB 1MPI 1 AMR Opt ok in 170,85s
2D 8 TBB 1MPI 1 AMR Opt ok in 145,24
2D 2 TBB 1MPI 1 AMR Opt ok in 292,18
2D 2 TBB 4MPI 1 AMR Opt do not work (issue in musclhancock)
2D 1 and 2 TBB 4MPI 1 AMR Opt using Godunov, do not work, it frozen in initialization and doesn't do anything
2D 1 TBB 4MPI 0 AMR NoOpt using Godunov, it explode
2D 1 TBB 1MPI 0 AMR NoOpt ok in 37,42s (every spawn optimization disabled)
2D 1 TBB 1MPI 1 AMR NoOpt ok in 219.433s (every spawn optimization disabled)
In summary it seems to work all the times we use 1 MPI, more than one seems to be in trouble.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/261Delete branch sven/ccz4-testcase from Engine repository again2019-02-27T18:12:37+01:00Sven KöppelDelete branch sven/ccz4-testcase from Engine repository againhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/sven/ccz4-testcase
This is only for Jenkins testing. Should never be merged in master.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/sven/ccz4-testcase
This is only for Jenkins testing. Should never be merged in master.Sven KöppelSven Köppelhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/262Get CarpetASCII 1d cut working on different machines such as hazelhen2019-03-06T18:27:47+01:00Sven KöppelGet CarpetASCII 1d cut working on different machines such as hazelhenLuke reported that it did not work for him.
But it does no more depend on HDF5 so we should get this working...
@lbovardLuke reported that it did not work for him.
But it does no more depend on HDF5 so we should get this working...
@lbovardSven KöppelSven Köppelhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/263Optimised kernels won't compile if Architecture is noarch2019-03-05T22:06:05+01:00Sven KöppelOptimised kernels won't compile if Architecture is noarchThe toolkit should stop processing if `Architecture=="noarch" but optimisation="optimised"` is selected and raise an error, instead of building nonworking code which won't compile. Otherwise, users will be confronted with C compiler erro...The toolkit should stop processing if `Architecture=="noarch" but optimisation="optimised"` is selected and raise an error, instead of building nonworking code which won't compile. Otherwise, users will be confronted with C compiler errors like
```
/zhome/academic/HLRS/xfp/xfpskoep/ExaHyPE/Engine/./AstroApplications/CCZ4GRMHD/AbstractCCZ4GRMHDSolver_ADERDG.h(257): error: identifier "ALIGNMENT" is undefined
inline double* allocateArray(const int size) {return ((double* ) _mm_malloc(sizeof(double)*size, ALIGNMENT));}
^
/zhome/academic/HLRS/xfp/xfpskoep/ExaHyPE/Engine/./AstroApplications/CCZ4GRMHD/AbstractCCZ4GRMHDSolver_ADERDG.cpp(89): error: identifier "ALIGNMENT" is undefined
double tempUpdate[CCZ4GRMHD::CCZ4GRMHDSolver_ADERDG_kernels::aderdg::getUpdateSize()] __attribute__((aligned(ALIGNMENT))); // TODO no heap variant implemented
^
```
which are hard to understand.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/266Write plotters which really plot limited DOF where the limiter is active2019-03-21T10:41:52+01:00Sven KöppelWrite plotters which really plot limited DOF where the limiter is activeFor VTK, this is straightforward.
For other formats, this is a nice to have.For VTK, this is straightforward.
For other formats, this is a nice to have.Sven KöppelSven Köppelhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/267Domain sizes of TOV star2019-03-07T17:26:15+01:00Sven KöppelDomain sizes of TOV starInterior: `[-3,+3]`, `maximum-mesh-size = 0.23`, DG order 3
Plotting repeat: `integrals=0.1, volumetric=0.5`Interior: `[-3,+3]`, `maximum-mesh-size = 0.23`, DG order 3
Plotting repeat: `integrals=0.1, volumetric=0.5`Sven KöppelSven Köppelhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/268Lucianos tasks as of meeting Thu 2019-03-07, 7pm2019-03-07T17:27:57+01:00Sven KöppelLucianos tasks as of meeting Thu 2019-03-07, 7pm![WhatsApp_Image_2019-03-07_at_17.23.40](/uploads/965440c582658dcf300fa06544a0e637/WhatsApp_Image_2019-03-07_at_17.23.40.jpeg)![WhatsApp_Image_2019-03-07_at_17.23.40](/uploads/965440c582658dcf300fa06544a0e637/WhatsApp_Image_2019-03-07_at_17.23.40.jpeg)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/269Carpet Plotter Warnings in 2D2019-03-21T10:41:29+01:00Sven KöppelCarpet Plotter Warnings in 2D> In 2D, I get the following warnings all the time. Can you take a look at it:
```
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp:202:8: warning: unused variable ‘isNotInAny...> In 2D, I get the following warnings all the time. Can you take a look at it:
```
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp:202:8: warning: unused variable ‘isNotInAnyGhostLayer’ [-Wunused-variable]
> bool isNotInAnyGhostLayer = tarch::la::allSmaller(icell,numberOfCellsPerAxis+ghostLayerWidth) && tarch::la::allGreater(icell,ghostLayerWidth-1);
> ^~~~~~~~~~~~~~~~~~~~
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp:205:8: warning: unused variable ‘isNotInCornerGhostLayer’ [-Wunused-variable]
> bool isNotInCornerGhostLayer = !(tarch::la::allSmaller(icell, ghostLayerWidth) || tarch::la::allGreater(icell, numberOfCellsPerAxis+ghostLayerWidth));
> ^~~~~~~~~~~~~~~~~~~~~~~
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp:207:8: warning: unused variable ‘justIgnoreTheCornerProblem’ [-Wunused-variable]
> bool justIgnoreTheCornerProblem = true;
> ^~~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.h:177:0,
> from /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/parser/Parser.h:33,
> from /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Plotter.h:21,
> from /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.h:19,
> from /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp:16:
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph: In member function ‘void exahype::plotters::FiniteVolume2Carpet::interpolateCartesianSlicedVertexPatch(const tarch::la::Vector<2, double>&, const tarch::la::Vector<2, double>&, double*, double*, double, const exahype::plotters::CartesianSlicer&)’:
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
> /home/dominic/dev/codes/c/upstream/ExaHyPE-Engine/./Peano/peano/../tarch/la/Vector.cpph:31:10: warning: array subscript is above array bounds [-Warray-bounds]
> _values[2] = value2;
> ~~~~~~~^
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/270Plotting: Obtain number of elements hold by an MPI rank before plotting grid ...2019-03-19T10:00:24+01:00Francesco FambriPlotting: Obtain number of elements hold by an MPI rank before plotting grid sweepI was looking for any strategy able to give me, for every MPI rank, the total number of cells (if possible, distinguishing between limited and non-limited cells).
A possibility was to use the plotpatches of ADERDG, counting unlimited cel...I was looking for any strategy able to give me, for every MPI rank, the total number of cells (if possible, distinguishing between limited and non-limited cells).
A possibility was to use the plotpatches of ADERDG, counting unlimited cells, and of FV, counting limited cells, but the problem is that...
I need this info for the startPlotting of the userdefined Writer, because for my plotter I need to allocate a very big array that, a-priori, should be able to contain all the info that has to be plotted for a given MPI rank.
besthttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/271Specification File Defaults in C++ Parser2019-05-08T13:41:02+02:00Dominic Etienne CharrierSpecification File Defaults in C++ ParserWe have a C++ JSON file parser.
The specification file schema is a JSON file itself.
Hence, we could read default values also directly from the schema.
**Edit:**
We could generate most of the C/C++ parser directly from the JSON schema....We have a C++ JSON file parser.
The specification file schema is a JSON file itself.
Hence, we could read default values also directly from the schema.
**Edit:**
We could generate most of the C/C++ parser directly from the JSON schema.
**Edit 2:**
There is now also a C/C++ header-only library for
JSON file validation.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/272Find out exact properties of the Cactus schemes2019-03-18T14:04:53+01:00Sven KöppelFind out exact properties of the Cactus schemes* WhiskyTHC: Find details; probably in papers; maybe find comparisons with other codes
* FIL/IllinoisGRMHD: Run a 2nd order FV Cowling star, in order to compare different codes and see how they compare* WhiskyTHC: Find details; probably in papers; maybe find comparisons with other codes
* FIL/IllinoisGRMHD: Run a 2nd order FV Cowling star, in order to compare different codes and see how they compareSven KöppelSven Köppelhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/273Replace libc assertions by Peano assertions2019-03-18T17:49:44+01:00Sven KöppelReplace libc assertions by Peano assertions[LibC assertions](http://www.cplusplus.com/reference/cassert/assert/) are always checked, unless `NDEBUG` is set (we never do that). Instead, we have [Peano assertions](https://gitlab.lrz.de/gi26det/Peano/blob/master/src/tarch/Assertions...[LibC assertions](http://www.cplusplus.com/reference/cassert/assert/) are always checked, unless `NDEBUG` is set (we never do that). Instead, we have [Peano assertions](https://gitlab.lrz.de/gi26det/Peano/blob/master/src/tarch/Assertions.h). `tarch/Assertions.h` also defines the `assert` macro (so one cannot exactly know which one is called, if one is not picky about the include file ordering). **For safety, always use Peanos `assert1` in a one-parametric assertion**.
An instance for a *wrong* use is the current master `minmod` function: https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/master/ExaHyPE/kernels/finitevolumes/musclhancock/c/3d/musclhancock.cpph#L28
It is easy to search other occurances of *wrong* assertion uses.
The overall idea is that in Release mode, such assertions are not checked, especially not on hot paths such as the scalar `minmod` function in the given example.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/275User-friendly API, use AbstractSolver as Adapter2019-07-23T10:40:16+02:00Dominic Etienne CharrierUser-friendly API, use AbstractSolver as AdapterIn order to make the code more easy to use, we could reestablish the MySolver_Variables
data access classes.
We could implement the user hooks that take a raw pointer in the AbstractSolver,
and would then call the user solver's implemen...In order to make the code more easy to use, we could reestablish the MySolver_Variables
data access classes.
We could implement the user hooks that take a raw pointer in the AbstractSolver,
and would then call the user solver's implementation which takes the
data access object as argument.
**Example:**
```
void MyAbstractSolver::adjustPointSolution(double* const Q,...) {
Variables vars(Q);
static_cast<MyUserSolver*>(this)->adjustPointSolution(vars,...);
}
```
We could then have additional data access classes for the full solution array, e.g. ``Solution`` and ``ReadOnlySolution``.
Potentially, we could then introduce dimension-aware iterators with appropriate typecast to
a ``Variables`` or ``ReadOnlyVariables`` object.
This would allow writing code like the following.
**Example:**
```
void MyAbstractSolver::refinementCriterion(ReadOnlySolution solution,...) {
double rhoMax = 0;
for ( ReadOnlyVariables& Q : solution ) {
rhoMax = std::max( Q.rho(), rhoMax );
}
}
```
Remarks
-------
``(ReadOnly)Variables`` and ``Variables`` must be split or declared in such a way that
it is clear that there is not necessarily a contiguous array storing
and variables and parameters. We should get rid of the index operator and
have separate functions ``double`` parameter(int i), ``double& variable(int i)`` instead.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/276Fused Time Stepping for LimitingADERDGSolver crashes with MPI if recomputatio...2019-03-29T17:07:03+01:00Dominic Etienne CharrierFused Time Stepping for LimitingADERDGSolver crashes with MPI if recomputation is triggeredNonfused variants seem to work fine.
EDIT: A run with generic kernels and noarch did not crash.Nonfused variants seem to work fine.
EDIT: A run with generic kernels and noarch did not crash.Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/277FV Substeps / Particle Simulations2019-04-23T14:16:30+02:00Dominic Etienne CharrierFV Substeps / Particle SimulationsAdd possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver...Add possibility to choose FV patch time step size freely.
This should be done when we implement anarchic time stepping
for the limiting ADER-DG solver.
Furthermore, there should be the possibility to override
the generated Limiter solver class in order to plug in
other projectors.
Rollbacks are difficult to implement with anarchic time stepping.
I have no clue how to do this yet.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/279Plot NaNs2019-04-24T23:40:18+02:00Dominic Etienne CharrierPlot NaNsVTK is able to visualise NaNs.
Currently, the plotters omit NaNs. This is not necessary.VTK is able to visualise NaNs.
Currently, the plotters omit NaNs. This is not necessary.Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/280Pass coordinates to nonConservativeProduct2019-05-02T17:55:34+02:00Dominic Etienne CharrierPass coordinates to nonConservativeProduct~~Currently, it is the only PDE function where we do not pass the coordinates in.~~
We actually do not pass the coordinates to the flux and the eigenvalues kernels, too.
18 kernel files need to be changed.
Workaround
----------
Specif...~~Currently, it is the only PDE function where we do not pass the coordinates in.~~
We actually do not pass the coordinates to the flux and the eigenvalues kernels, too.
18 kernel files need to be changed.
Workaround
----------
Specify the coordinates or a coefficient depending on the coordinates as material parameter.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/286Observed issues using AMR with erase2019-08-02T16:52:06+02:00m.tavelliObserved issues using AMR with eraseI have observe a couple of issue that can be reproduced with the current version of application SFDI:
1. Only one refine level and erase lead to the error reported below. This happen with applications:
SFDI, parfile, after step 121, ti...I have observe a couple of issue that can be reproduced with the current version of application SFDI:
1. Only one refine level and erase lead to the error reported below. This happen with applications:
SFDI, parfile, after step 121, time=0.0100127:
[SFDI_2D_1refissue.exahype](/uploads/1bc47bc22d0e688354210422775a3eae/SFDI_2D_1refissue.exahype)
GPRDR, parfile:
[GPRDR_2D.exahype](/uploads/d1bce4d670a633a4b76abe519ceb102a/GPRDR_2D.exahype)
101.299009 [hilbert],rank:0, core:30, tid:13 error exahype::solvers::ADERDGSolver::surfaceIntegral(...) Riemann solve was not performed on all faces of cell= (solverNumber:0,neighbourMergePerformed:[,,,],hasCompletedLastStep:0,parentIndex:3492,hasVirtualChildren:0,type:Cell,refinementEvent:None,level:6,offset:[1.07173,0.244726],size:[0.0168776,0.0168776],previousTimeStamp:inf,previousTimeStepSize:inf,timeStepSize:0.000121198,timeStamp:0.0100127,solutionIndex:91946,solutionAveragesIndex:91949,solutionCompressedIndex:-...
2. After a certain time it continue to run but the time does not increase, see the log below. The same issue appears also with a coarsest mesh ("maximum_mesh_size": 1.0) but later:
parfile:
[SFDI_2D_issue1.exahype](/uploads/c31d7ac41db8ae6e36dac320299a233a/SFDI_2D_issue1.exahype)
```
48460.082966 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48460.082980 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 20 iterations
48460.082992 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) finalise mesh refinement (if applicable)
48460.083004 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) reinitialise cells and send data to neigbours (if applicable)
48460.194436 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) recompute solution locally (if applicable) and compute new time step size
48460.385842 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) step 15681 t_min =0.298657
48460.385869 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) dt_min =0.000111607
48460.385885 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner cells/inner unrefined cells=48460/25257
48460.385902 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner max/min mesh width=[0.16,0.16]/[0.0177778,0.0177778]
48460.385915 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) max level=6
48460.385929 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) outer cells/outer unrefined cells=590/456
48460.385976 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runAsMaster(...) memoryUsage =4693 MB
48460.664413 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runTimeStepsWithFusedAlgorithmicSteps(...) local recomputation requested by at least one solver
48460.711840 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) pre-spreading of limiter status
48461.121757 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) perform mesh refinement
48461.121796 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.265934 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.265964 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.266009 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.266022 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.413310 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.413336 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.413380 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.413393 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.560601 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.560629 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.560671 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.560684 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.707636 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.707664 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #4, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.707708 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.707722 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.853066 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.853091 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #5, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.853132 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.853144 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.999806 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48461.999834 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #6, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48461.999879 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48461.999924 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48462.147141 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48462.147168 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #7, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.147211 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.147224 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48462.293720 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48462.293745 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #8, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.293787 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.293800 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48462.440954 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=3, all solvers attained stable state=1, max level=6
48462.440980 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #9, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.441020 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.441032 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=4, all solvers attained stable state=1, max level=6
48462.588360 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=5, all solvers attained stable state=1, max level=6
48462.588385 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #10, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.588425 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.588438 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=6, all solvers attained stable state=1, max level=6
48462.735102 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=1, max level=6
48462.735127 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #11, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.735167 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.735179 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48462.884632 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48462.884659 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #12, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48462.884701 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48462.884715 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.035295 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.035321 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #13, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48463.035361 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.035374 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.184642 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48463.184669 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #14, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48463.184711 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.184724 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48463.334715 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.334742 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #15, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48463.334783 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.334797 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.484046 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.484072 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #16, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48463.484112 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.484125 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48463.632638 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48463.632665 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #17, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48463.632705 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.632718 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48463.781876 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #18, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=0
48463.781928 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48463.781941 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 18 iterations
48463.781953 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) finalise mesh refinement (if applicable)
48463.781964 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) reinitialise cells and send data to neigbours (if applicable)
48463.894936 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) recompute solution locally (if applicable) and compute new time step size
48464.081949 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) step 15682 t_min =0.298657
48464.081988 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) dt_min =0.000111607
48464.082004 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner cells/inner unrefined cells=48460/25257
48464.082021 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner max/min mesh width=[0.16,0.16]/[0.0177778,0.0177778]
48464.082034 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) max level=6
48464.082046 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) outer cells/outer unrefined cells=590/456
48464.082093 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runAsMaster(...) memoryUsage =4693 MB
48464.369570 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runTimeStepsWithFusedAlgorithmicSteps(...) mesh update requested by at least one solver
48464.418752 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) pre-spreading of limiter status
48464.827970 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) perform mesh refinement
48464.828009 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48464.974119 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48464.974148 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48464.974190 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48464.974202 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.122067 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.122095 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.122138 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.122153 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.269962 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.269989 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.270062 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.270076 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.417461 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.417487 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #4, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.417526 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.417537 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.564058 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.564084 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #5, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.564126 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.564139 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.711640 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.711667 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #6, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.711708 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.711720 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.859318 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48465.859344 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #7, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48465.859385 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48465.859398 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.007531 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.007558 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #8, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.007601 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.007614 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.155125 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.155152 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #9, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.155194 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.155207 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.301976 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48466.302002 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #10, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.302044 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.302056 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48466.449749 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=3, all solvers attained stable state=1, max level=6
48466.449774 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #11, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.449810 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.449823 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=4, all solvers attained stable state=1, max level=6
48466.598445 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=5, all solvers attained stable state=1, max level=6
48466.598474 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #12, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.598521 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.598533 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=6, all solvers attained stable state=1, max level=6
48466.749009 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=1, max level=6
48466.749042 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #13, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.749093 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.749106 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48466.902091 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48466.902119 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #14, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48466.902161 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48466.902173 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.053397 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.053424 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #15, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48467.053463 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.053475 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.204646 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48467.204672 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #16, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48467.204710 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.204723 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48467.354426 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.354451 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #17, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48467.354489 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.354502 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.503538 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.503563 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #18, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48467.503600 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.503613 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48467.653651 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48467.653677 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #19, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48467.653716 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.653729 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48467.803491 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #20, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=0
48467.803575 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48467.803589 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 20 iterations
48467.803602 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) finalise mesh refinement (if applicable)
48467.803614 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) reinitialise cells and send data to neigbours (if applicable)
48467.915869 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) recompute solution locally (if applicable) and compute new time step size
48468.101707 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) step 15683 t_min =0.298657
48468.101733 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) dt_min =0.000111607
48468.101750 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner cells/inner unrefined cells=48460/25257
48468.101766 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner max/min mesh width=[0.16,0.16]/[0.0177778,0.0177778]
48468.101779 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) max level=6
48468.101801 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) outer cells/outer unrefined cells=590/456
48468.101850 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runAsMaster(...) memoryUsage =4693 MB
48468.378415 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runTimeStepsWithFusedAlgorithmicSteps(...) local recomputation requested by at least one solver
48468.427412 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) pre-spreading of limiter status
48468.837452 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) perform mesh refinement
48468.837492 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48468.982716 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48468.982745 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48468.982791 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48468.982803 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.129704 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.129732 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.129777 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.129790 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.278224 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.278251 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.278294 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.278307 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.426106 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.426132 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #4, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.426172 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.426184 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.572144 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.572172 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #5, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.572212 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.572226 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.720586 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.720612 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #6, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.720654 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.720697 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.868148 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48469.868175 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #7, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48469.868215 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48469.868227 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48470.016876 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48470.016902 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #8, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.016943 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.016955 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48470.164718 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=3, all solvers attained stable state=1, max level=6
48470.164743 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #9, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.164785 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.164797 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=4, all solvers attained stable state=1, max level=6
48470.312895 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=5, all solvers attained stable state=1, max level=6
48470.312921 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #10, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.312960 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.312974 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=6, all solvers attained stable state=1, max level=6
48470.460639 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=1, max level=6
48470.460664 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #11, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.460704 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.460716 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48470.610736 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48470.610762 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #12, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.610801 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.610814 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48470.761530 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48470.761557 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #13, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.761597 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.761609 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48470.911284 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48470.911310 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #14, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48470.911350 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48470.911363 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48471.061626 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48471.061652 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #15, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48471.061692 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48471.061705 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48471.211935 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48471.211961 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #16, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48471.212001 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48471.212014 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48471.361122 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=1, all solvers attained stable state=1, max level=6
48471.361147 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #17, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48471.361187 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48471.361206 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=0, initial refinement=0, stable iterations in a row=2, all solvers attained stable state=1, max level=6
48471.510989 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #18, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=0
48471.511041 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48471.511055 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 18 iterations
48471.511068 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) finalise mesh refinement (if applicable)
48471.511079 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) reinitialise cells and send data to neigbours (if applicable)
48471.623254 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) recompute solution locally (if applicable) and compute new time step size
48471.812402 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) step 15684 t_min =0.298657
48471.812430 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) dt_min =0.000111607
48471.812446 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner cells/inner unrefined cells=48460/25257
48471.812463 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) inner max/min mesh width=[0.16,0.16]/[0.0177778,0.0177778]
48471.812476 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) max level=6
48471.812490 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::printTimeStepInfo(...) outer cells/outer unrefined cells=590/456
48471.812551 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runAsMaster(...) memoryUsage =4693 MB
48472.101873 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::runTimeStepsWithFusedAlgorithmicSteps(...) mesh update requested by at least one solver
48472.150552 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) pre-spreading of limiter status
48472.559120 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::updateMeshAndSubdomains(...) perform mesh refinement
48472.559160 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.703099 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.703128 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48472.703174 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48472.703186 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.847218 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.847247 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
48472.847293 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid(...) memoryUsage =4693 MB
48472.847307 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.996567 [hilbert],rank:0, core:4, tid:0 info exahype::State::continueToConstructGrid(...) grid construction not yet finished. grid balanced=1, grid stationary=1, still in refining mode=1, initial refinement=0, stable iterations in a row=0, all solvers attained stable state=0, max level=6
48472.996596 [hilbert],rank:0, core:4, tid:0 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=6, idle-nodes=0, vertical solver communication=0, continue to construct grid=1
```Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/288Separate Variables from Parameters2019-07-23T10:40:15+02:00Jean-Matthieu GallardSeparate Variables from ParametersProblem: Currently parameters are stored right after the variables in luh, Q and its related arrays (lQi, lQhi, lQhbnd). This make it difficult to provide vectorized PDE options and make some operations where the parameter have to be ski...Problem: Currently parameters are stored right after the variables in luh, Q and its related arrays (lQi, lQhi, lQhbnd). This make it difficult to provide vectorized PDE options and make some operations where the parameter have to be skipped less intuitive and slower with SIMD
Solution: Introduce a new array like luh to store the parameters. This array is to be passed to the user functions using parameters (nullptr if no parameters) like luh or Q is. The linear space time predictor with split_ck already do this locally.
Issue: Will change the signatures of some user functions and formulation if they are indeed using parameters.
@ga96nuvDominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/289Luh padding and alternative layout2019-07-23T10:59:21+02:00Jean-Matthieu GallardLuh padding and alternative layoutCurrently the optimized kernels do not change luh. This was done for historical reasons (the non temp arrays had a set size independently of the kernels, now changed) and to ensure that the plotter, limiter and AMR would work no matter t...Currently the optimized kernels do not change luh. This was done for historical reasons (the non temp arrays had a set size independently of the kernels, now changed) and to ensure that the plotter, limiter and AMR would work no matter the kernels.
However being able to work with a padded luh or even further an SoS-layout luh (x,n,y,z coordinates instead of the current n,x,y,z) could provide a speedup. Currently the linear vectorized split_ck implementation does the back and forth transpositions to SoS layout and the cost of these is non negligible (even if more than compensated by the vectorization speedup).
The Limiter projections+tests and AMR can now be done using optimized kernels too so the compatibility issue here could easily be solved by generating the right kernels.
Regarding the plotters a preprocessing step to translate back luh locally could be added in case luh is non standard.
Is there a good way to do this?
Open questions: would this impact other domain of the code
@domcha @leo.rannabauer
Occurrences of luh:
-------------------
Search Term: ``getSolution()``
**exahype plotters**
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Carpet/ADERDG2Carpet.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Carpet/FiniteVolume2Carpet.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/CSV/ADERDG2LegendreCSV.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/CSV/Patch2CSV.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FlashHDF5/ADERDG2FlashHDF5.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/ADERDG2CartesianPeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/ADERDG2LegendrePeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/PeanoFileFormat/FiniteVolumes2PeanoPatchFileFormat.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/Tecplot/ExaHyPE2Tecplot.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LegendreDivergenceVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LegendreVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/ADERDG2LobattoVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/FiniteVolumes2VTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/LimitingADERDG2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/LimitingADERDGSubcells2CartesianVTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/VTK/Patch2VTK.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/ADERDG2ProbeAscii.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/ADERDG2UserDefined.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FiniteVolumes2ProbeAscii.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/FiniteVolumes2UserDefined.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/plotters/LimitingADERDG2UserDefined.cpp
**exhaype/solvers**
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_FusedTimeStepJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_PredictionJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/ADERDGSolver.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/FiniteVolumesSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver_LocalRecomputationJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver_UpdateJob.cpp
/ExaHyPE-Engine_upstream/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cppJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/290Allow periodic boundary conditions for single-node runs2019-07-23T16:34:47+02:00Dominic Etienne CharrierAllow periodic boundary conditions for single-node runsThis allows to simply realise periodic boundary conditions via a lookup table.
Step 0: Initial data is known, i.e. solution at boundary is known, too. Impose this data.
Start building up the lookup table.
Step 1: Have the looku...This allows to simply realise periodic boundary conditions via a lookup table.
Step 0: Initial data is known, i.e. solution at boundary is known, too. Impose this data.
Start building up the lookup table.
Step 1: Have the lookup table. Use it.Dominic Etienne CharrierDominic Etienne Charrierhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/293Nans for gll nodes2019-08-14T15:37:26+02:00Leonhard RannabauerNans for gll nodesWe observe nans for the recently implemented gll nodes as soon as we work with amr on supermucng.
We start with a series of tests:
| Basis | Mesh | AMR | Arch | Opt | MPI | Works |
| ------ | ------ | ------ | ------ | ------ | ------ ...We observe nans for the recently implemented gll nodes as soon as we work with amr on supermucng.
We start with a series of tests:
| Basis | Mesh | AMR | Arch | Opt | MPI | Works |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| GLL | 25 | 1 | hsw| vect | No | yes |
| GLL | 7 | 2 | hsw| vect | No | yes |
| GL | 25 | 1 | skx| vect | Yes |yes |
| GLL | 25 | 1 | skx| vect | Yes |yes |
| GL | 25 | 2 | skx| vect | Yes | **nans** |
| GLL | 25 | 2 | skx| vect | Yes | **nans** |
| GL | 25 | 2 | skx| gen | Yes | **nans** |
| GLL | 25 | 2 | skx| gen | Yes | **nans** |
| GLL | 79 | 1 | skx| gen | Yes | yes |
| GLL | 79 | 1 | skx| gen | Yes | yes |
| GLL | 239 | 1 | skx| gen | Yes | yes |
As GL as well as GLL nodes seem to fail repeat the GL 2 AMR level test on the master branch:
| Basis | Mesh | AMR | Arch | Opt | MPI | ExaSeis | Works |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| GL | 25 | 2 | skx| gen | Yes | master | **nans** |
Which shows the same behaviour for default old cases.Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/295Local Time Stepping Plan2019-09-06T14:51:45+02:00Dominic Etienne CharrierLocal Time Stepping PlanWe could rather quickly implement a local time steppging version a la SeisSol
where we store the full space-time face data only for faces of coarse grid cells
along mesh resolution transitions.
Steps (in German):
* [ ] Aufbrechen des ...We could rather quickly implement a local time steppging version a la SeisSol
where we store the full space-time face data only for faces of coarse grid cells
along mesh resolution transitions.
Steps (in German):
* [ ] Aufbrechen des Extrapolations-Operators (Cell Boundary -> Single Face)
* [ ] Aufbrechen der Face Data Felder. We need one pointer/index per face.
* [ ] Mithilfe meiner Marker fuer gewisse Faces space-time Face Data speichern
* [ ] Den space Prolongationsoperator mit einem space-time Operator ersetzen (Face Prolongation -> "Volume" Prolongation (vorhanden))
* [ ] Counter im State einfuehren und dann entsprechend des Wertes nur bestimmte Levels an* [ ] und ausschalten, ginge
mit Hilfe Deiner level Information in den MappingSpecifications
* [ ] Nur am Anfang eines Batches die alte Loesung sichern (implementiert)
* [ ] Batchgroesse entsprechend des aktuellen tiefsten Verfeinerungslevels berechnen
* [ ] Rollback zu Anfang des Batches, falls das Gitter nicht gepasst hat.
Some implementational aspects
-----------------------------
Some preparational work has already been done.
* No changes to the current faceIntegral procedure need to be performed
as we already skip the integral where a Leaf cell shares an interface with a Refined ("Parent") cell.
```
const int dofsPerFace = getBndFluxSize();
for (int direction=0; direction<DIMENSIONS; direction++) {
for (int orientation=0; orientation<2; orientation++) {
const int faceIndex=2*direction+orientation;
if ( cellDescription.getFacewiseAugmentationStatus(faceIndex)<MaximumAugmentationStatus ) { // ignore Ancestors
double* const lFhbnd = static_cast<double*>(cellDescription.getFluctuation()) + dofsPerFace * faceIndex;
faceIntegral(output,lFhbnd,direction,orientation,0/*implicit conversion*/,0,cellDescription.getSize(),
cellDescription.getTimeStepSize(),addToUpdate);
}
}
}
```
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/296GradQ format2019-09-21T15:09:46+02:00Jean-Matthieu GallardGradQ formatThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*...The format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/300Job failed with nodes=82019-09-20T15:30:57+02:00Duo Lid.li@lmu.deJob failed with nodes=8The same job works with nodes=2 on NG but failed with more nodes e.g. nodes=8.
OMP=8 and --ntasks-per-node=6
60.199790 [i01r01c04s06.sng.lrz.de],rank:40, core:16, tid:6 error exahype::solvers::ADERDGSolver::surfaceIntegral(......The same job works with nodes=2 on NG but failed with more nodes e.g. nodes=8.
OMP=8 and --ntasks-per-node=6
60.199790 [i01r01c04s06.sng.lrz.de],rank:40, core:16, tid:6 error exahype::solvers::ADERDGSolver::surfaceIntegral(...) Riemann solve was not performed on all faces of cell= (solverNumber:0,neighbourMergePerformed:[^A,^A,^A,^A,^@,^A],parentIndex:-1,type:Leaf,parentType:undefined,level:4,offset:[4,14,14],size:[1,1,1],previousTimeStamp:0,previousTimeStepSize:0.002805,timeStepSize:0.002805,timeStamp:0.002805,solutionIndex:288,solutionAveragesIndex:291,solutionCompressedIndex:-1,solution:0x7f2c50061600,solutionAverages:0x7f2c500d0e00,solutionCompressed:0,previousSolutionIndex:289,previousSolutionAveragesIndex:290,previousSolutionCompressedIndex:-1,previousSolution:0x7f2c50060100,previousSolutionAverages:0x7f2c500d0f00,previousSolutionCompressed:0,updateIndex:292,updateAveragesIndex:293,updateCompressedIndex:-1,update:0x7f2c50066080,updateAverages:0x7f2c500d7380,updateCompressed:0,extrapolatedPredictorIndex:294,extrapolatedPredictorAveragesIndex:295,extrapolatedPredictorCompressedIndex:-1,extrapolatedPredictor:0x7f2c4f9b7300,extrapolatedPredictorAverages:0x7f2c5008d800,extrapolatedPredictorCompressed:0,extrapolatedPredictorGradientIndex:-1,extrapolatedPredictorGradient:0,fluctuationIndex:296,fluctuationAveragesIndex:297,fluctuationCompressedIndex:-1,fluctuation:0x7f2c4f9bbb00,fluctuationAverages:0x7f2c5006b400,fluctuationCompressed:0,solutionMinIndex:298,solutionMaxIndex:299,solutionMin:0x7f2c50081c00,solutionMax:0x7f2c50081a00,facewiseAugmentationStatus:[2,2,2,1,3,1],augmentationStatus:2,facewiseCommunicationStatus:[2,2,2,2,1,2],communicationStatus:2,facewiseRefinementStatus:[0,0,0,0,0,0],refinementStatus:0,previousRefinementStatus:0,refinementFlag:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1,creation:UniformRefinement) (file:/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver.cpp,line:1188)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/302error when use dynamic AMR in LimitedADERDG2020-11-20T15:10:48+01:00Duo Lid.li@lmu.deerror when use dynamic AMR in LimitedADERDGparameter tarch::parallel::Node::getInstance().getRank(): 17
parameter "if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before": if the neighbour data buffer is emp...parameter tarch::parallel::Node::getInstance().getRank(): 17
parameter "if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before": if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before
ExaHyPE-Diffuse: /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./Peano/peano/../peano/heap/BoundaryDataExchanger.cpph:282: void peano::heap::BoundaryDataExchanger<Data, SendReceiveTaskType, VectorContainer>::freeAllMessagesInDeployBuffer() [with Data = char, SendReceiveTaskType = peano::heap::SendReceiveTask<char>, VectorContainer = std::vector<char, std::allocator<char>>]: Assertion `false' failed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/304Merge with Latest Peano to resolve debug logging errors2020-01-15T11:36:58+01:00Samuel TootleMerge with Latest Peano to resolve debug logging errorsWhen compiling with MODE=Debug, various errors arise related to Peano logging. I checked the Peano master repo and discovered that this issue has been resolved, however, this version has not yet been synched with Exahype sub-modules. T...When compiling with MODE=Debug, various errors arise related to Peano logging. I checked the Peano master repo and discovered that this issue has been resolved, however, this version has not yet been synched with Exahype sub-modules. This was tested using updateSubmodules.sh which results in reverting to the old git hash even after manually pulling Peano/master.