ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2018-03-02T11:23:39+01:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/214Faster mesh refinement2018-03-02T11:23:39+01:00Ghost UserFaster mesh refinementProblem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If...Problem 1 (Costly operation is performed while no parallelism available)
------------------------------------------------------------------------
Peano's shared-memory parallelism is based on identifying regular subgrids in the
tree. If a new cells is introduced to the tree it might not be yet be identified
as part of a regular subgrid and the operations performed on the cell
are thus parallelised.
Solution: If imposing initial conditions or evaluating a refinement criterion is too
expensive we could let the user choose to perform it as background task.
This might lead to more mesh setup iteration but potentially to a better
exploitation of the available cores. It should also benefit the hiding of MPI communication
during the mesh setup.
Problem 2 (Overall concurrency)
-------------------------------
ExaHyPE's regular shared-memory parallelism during the mesh setup is currently further limited as
multiple cells might write to the same vertex in order to set refinement events.
Solution: We should be able to solve this by inverting the control. The
vertex checks in touchVertexLastTime if any cell has set a refinement event,
and refines if that is the case. This would increase the concurrency of the
enterCell operations.
Problem 3 (Memory)
-------------------------------
ExaHyPE's initial mesh setup is performed at the beginning by a single rank.
Gradually more and more ranks are added. In order to prevent that
any of the ranks runs out of memory during the initial mesh setup,
it might make sense to only temporarily allocate memory, impose initial
conditions, evaluate the refinement criterion, and then free the memory again (better:
recylce it).
After the initial mesh setup, we would then allocate memory on all ranks
and impose initial conditions.
Problem 4 (Load Balancing)
-------------------------------
The load-balancing does currently only count the number of cells.
It does not take the different cell types in ExaHyPE's grid into account.
The helper cell types Descendant and Ancestor have way less work to do than
the compute cells of type Cell.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/213Named variables in VTK plots2018-09-10T19:38:52+02:00Ghost UserNamed variables in VTK plotsI want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for p...I want to have them. I will implement this tonight via the plotter mapping class as an optional virtual function (isn't this already there? HDF5 uses it) and pass the data straight forwards throught VTK. This will result in a patch for peano.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/212Number of Plotters is not detected2018-09-10T19:43:27+02:00Ghost UserNumber of Plotters is not detectedWhen you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even a...When you compile an ExaHyPE application with a Specfile with *n* plotters but run it with *m* plotters (say the types of the first *m-n* plotters are the same), there is no detection that `m != n`. This leads to weird errors which even are hardly understandable in DEBUG mode:
```
0.0310583 15:50:15 [nils]rank:0 debug exahype::parser::Parser::getIdentifierForPlotter() found token notoken (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:999)
assertion in file /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp, line 1001 failed: token.compare(_noTokenFound) != 0
parameter token: notoken
parameter solverNumber: 0
parameter plotterNumber: 3
ExaHyPE-GRMHD: /home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp:1001: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> exahype::parser::Parser::getIdentifierForPlotter(int, int) const: Assertion `false' failed.
Abgebrochen
```
but completely nonunderstandable in Release mode:
```
0.00535548 15:32:20 [nils]rank:0 error exahype::parser::Parser::getFirstSnapshotTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'time' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1046)
0.00538875 15:32:20 [nils]rank:0 error exahype::parser::Parser::getRepeatTimeForPlotter() 'GRMHDSolver_FV' - plotter 3: 'repeat' value must be a float. (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/parser/Parser.cpp,line:1067)
```
note that in this example, `n=4` and `m=3`, so especially plotter 3 looked allright where the error message tried to complain actually about the nonexisting plotter 4.
This is very bad. We need some better enforcement of this rule.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/211Multicore-ise LoadBalancing mapping2019-09-20T15:46:28+02:00Ghost UserMulticore-ise LoadBalancing mappingThis affects the concurrency level within the grid setup iterations.This affects the concurrency level within the grid setup iterations.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/210Efficent output format for large simulations2018-03-02T13:23:47+01:00Leonhard RannabauerEfficent output format for large simulationsAs soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TU...As soon as output reaches a certain size plotting is our major bottleneck. (a mesh of ~40 mio dofs with 14 unknowns for pvtu on supermuc takes 1h per plot)
We should look into different approaches like the ASYNC lib https://github.com/TUM-I5/ASYNC which sacrifices a singe thread per rank on output.
For supermuc: We should also start to look into parallel file systems like LUSTRE https://en.wikipedia.org/wiki/Lustre_(file_system).Leonhard RannabauerLeonhard Rannabauerhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/209Toolkit should accept command line arguments for finetuning2019-04-15T12:40:31+02:00Ghost UserToolkit should accept command line arguments for finetuning> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always...> **Note:** This is a **minor feature suggestion**. Something which should be done on the **long term** to improve the users daily life. It is **not important right now**
Currently, the toolkit has a way of overwriting some files always while creating some files only if they do not yet exist.
Sometimes, especially when the API or toolkit changes, one needs to recreate everything. The typical workflow is then to call `rm Abstract*Solver* KernelCalls*`. Instead, it would be nice if the toolkit supports some option `--overwrite` or similar to always recreate the files. It could also offer an option `--compare` to print whether generated files differ from existing files.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/208Random initial conditions for LimitingADERDGSolver pose a problem2019-04-15T12:39:44+02:00Ghost UserRandom initial conditions for LimitingADERDGSolver pose a problemCurrently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy...Currently, random initial conditions likely lead to undefined
behaviour for the LimitingADERDGSolver.
Background
-----------
The LimitingADERDGSolver replaces the ADER-DG solution locally by a FV solution
if the former does not satisfy certain conditions (PAD, DMP).
During the imposition of initial conditions, we then allocate a new FV patch and
impose the initial conditions on its FV solution degrees of freedom.
Random initial conditions
-------------------------
We have to ensure that FV and ADER-DG solution are consistent.
This is currently only ensured if we have deterministic initial conditions.
In case of random initial conditions, the imposition of initial conditions
on the ADER-DG solution and on the FV solution might lead to two different
outcomes.
A cell might encounter rough initial conditions while imposing the initial conditions
on the ADER-DG solution but it then might encounter smooth initial conditions
for the FV solution.
But more severely, both differ from each other.
A solution?
-----------
In case a cell allocates an FV patch and has a limiter status such that it will
compute with Finite Volumes in the first iteration, we will then treat this FV solution as
the correct one and will project it back on the ADER-DG solution.
This way ADER-DG and FV solution are consistent.
On the other hand, if the limiter status of the cell is such that the cell
will compute with ADER-DG in the first iteration, we will treat the ADER-DG solution as the correct
one, we will project the ADER-DG solution onto the FV solution. We will not impose initial
conditions on the FV solution.
FV and ADER-DG solution are consistent in this case as well.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/207"error: Neighbours cannot communicate." (Minimal working example Application)2018-02-12T15:25:32+01:00Ghost User"error: Neighbours cannot communicate." (Minimal working example Application)In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria....In commit https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/baacbecadd7321cb25aec130707bd8b6dc08b11a I added the new LimitingADERDG application `ApplicationExamples/Experiments/GridDemonstrator` where I test different limiter criteria.
In the submitted example, the code crashes after the first timestep with **error: Neighbours cannot communicate**.
```
12.1608 info memoryUsage =333 MB
12.5276 info grid setup iteration #40, idle-nodes=1, vertical solver communication=0
12.5276 info memoryUsage =333 MB
12.8925 info grid setup iteration #41, idle-nodes=1, vertical solver communication=0
12.8925 info memoryUsage =333 MB
13.2603 info grid setup iteration #42, idle-nodes=1, vertical solver communication=0
13.2603 info memoryUsage =333 MB
13.2604 info finished grid setup after 42 iterations
13.2604 info finalise mesh refinement and compute first time step size
13.4246 info initialised all data and computed first time step size
13.7191 info plotted initial solution (if specified) and computed first predictor
13.7191 info step 0 t_min =0
13.7192 info dt_min =0.0125926
13.7192 info memoryUsage =333 MB
13.7192 info plot
13.78 error Neighbours cannot communicate.
cell1=(solverNumber:0,neighbourMergePerformed:[1,0,1,0],isInside:[1,1,1,1],parentIndex:920,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.428,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:42627,solutionAverages:42630,solutionCompressed:-1,previousSolution:42626,previousSolutionAverages:42629,previousSolutionCompressed:-1,update:42628,updateAverages:42631,updateCompressed:-1,extrapolatedPredictor:42632,extrapolatedPredictorAverages:42634,extrapolatedPredictorCompressed:-1,fluctuation:42633,fluctuationAverages:42635,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[1,0,1,0],helperStatus:2,facewiseLimiterStatus:[2,0,1,0],limiterStatus:2,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36444784,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1287913656)
.cell2=(solverNumber:0,neighbourMergePerformed:[0,0,1,0],isInside:[1,1,1,1],parentIndex:933,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:6,offset:[12.5926,8.47737],size:[0.164609,0.164609],previousCorrectorTimeStamp:0,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0.0125926,correctorTimeStamp:0,predictorTimeStepSize:0.0125926,predictorTimeStamp:0.0125926,solution:8218,solutionAverages:8221,solutionCompressed:-1,previousSolution:8217,previousSolutionAverages:8220,previousSolutionCompressed:-1,update:8219,updateAverages:8222,updateCompressed:-1,extrapolatedPredictor:8223,extrapolatedPredictorAverages:8225,extrapolatedPredictorCompressed:-1,fluctuation:8224,fluctuationAverages:42960,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,previousAugmentationStatus:0,facewiseHelperStatus:[0,0,1,0],helperStatus:2,facewiseLimiterStatus:[0,0,0,0],limiterStatus:0,previousLimiterStatus:0,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:36491056,bytesPerDoFInSolution:0,bytesPerDoFInUpdate:8015768,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:-1285749088) (file:/home/sven/numrel/exahype/Engine-ExaHyPE/ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp,line:1570)
terminate called without an active exception
Abgebrochen
```
The specfile describes a PDE with 1 scalar field (doing nothing), the setup runs on a notebook in <10 seconds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/206CCZ4: Run BH until T=10002018-02-09T22:50:28+01:00Ghost UserCCZ4: Run BH until T=1000We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical t...We need some setup of ExaHyPE,CCZ4+Limiter on a reasonable grid (L>25, dx<0.1). This ticket shall report the progress. Just describe your runs in the comments.
*Explanation of notation: The unit "M/h" means "simulation time / physical time in hours". M means "Mass" and roughly indicates for us that we refer to the simulation time.*https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/205CCZ4: Run BH with2018-01-20T20:50:14+01:00Ghost UserCCZ4: Run BH withhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/204Guidebook should not be written in LaTeX2018-01-26T11:39:44+01:00Ghost UserGuidebook should not be written in LaTeX> Note: This issue is really low-importance. Do not continue reading if you care about your time.
The guidebook is currently written in LaTeX, with a primary focus on printing and much efforts put into aligning figures and styling text...> Note: This issue is really low-importance. Do not continue reading if you care about your time.
The guidebook is currently written in LaTeX, with a primary focus on printing and much efforts put into aligning figures and styling texts. IMHO this is the wrong focus. Instead, the guidebook should *focus on content, not on layout*. It should be written in some simple markup language which allows rendering to a web page and a PDF similarly. This is useful for easier deep-linking and reading on screen. It also encourages a more uniform style accross the book.
There are several instances of such language. One very common in the Python world is [Shinx](http://www.sphinx-doc.org), it is very widespread due to its simple syntax in Markdown. I recently stumbled over the [Visit documentation](http://visit-sphinx-user-manual.readthedocs.io) which was rewritten with Sphinx and renders at the same time to a comprehensive website and a [400 page book/pdf](https://media.readthedocs.org/pdf/visit-sphinx-user-manual/latest/visit-sphinx-user-manual.pdf) with a table of contents, numbered images and all that. An example page is http://visit-sphinx-user-manual.readthedocs.io/en/latest/Quantitative/Expressions.html which is generated by the very readable restructuredtext source http://visit-sphinx-user-manual.readthedocs.io/en/latest/_sources/Quantitative/Expressions.rst.txt
Translating our current LaTeX guidebook to something like Restructuredtext or Markdown is a <1h job, therefore switching the publication system is not a mammoth task at all. It is more something people have to agree on. I wonder if Tobias will?https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/203Peano: Geometrical comparisons must always use relative tolerance2018-02-02T14:12:26+01:00Ghost UserPeano: Geometrical comparisons must always use relative toleranceThe tarch::la:: routines use usually an absolute tolerance (default: machine precision)
for comparing floating point numbers.
This is problematic for numbers with large absolute values.
Fix: These numbers must be normalised and scaled t...The tarch::la:: routines use usually an absolute tolerance (default: machine precision)
for comparing floating point numbers.
This is problematic for numbers with large absolute values.
Fix: These numbers must be normalised and scaled to range [-1,1] before
comparing with respect to the tolerance.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/202Segfault in some VTK plotters if variables = 02019-03-21T10:48:34+01:00Ghost UserSegfault in some VTK plotters if variables = 0https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/201Clean up vertical MPI communication2019-09-20T15:46:26+02:00Ghost UserClean up vertical MPI communicationThere is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.There is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/200Cell-wise plotter2017-11-29T17:45:59+01:00Ghost UserCell-wise plotterCreate a VTK plotter which only plots a single cell. Plot then:
* LimitingStatus
* Which MPI rank hosts the cell
This also should support VTU (for MPI).Create a VTK plotter which only plots a single cell. Plot then:
* LimitingStatus
* Which MPI rank hosts the cell
This also should support VTU (for MPI).https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/199Extend batching to global time stepping2017-11-21T14:41:53+01:00Ghost UserExtend batching to global time steppingI am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll ba...I am currently working on extending the batching from "globalfixed" to "global" time stepping.
The idea is here to only reduce the time step size and other data in the last iteration of the batch.
If something went wrong, we then roll back to the state at the beginning of the batch.
This WiP issue is for tracking my progress and issues I ran into.
Reduction of time step size
---------------------------
While we will not update the currently used CFL time step size in every single iteration
in the planned batched global time stepping scheme,
we will search the minimum CFL time step size over all batch iterations.
This allows us to check if the used time step size did violate the CFL constraint
during the execution of the batch.
The consequence might then be to use the newly determined minimum batch time step size as
time step size for a rerun of the whole batch.
Basic modifications to the fused time stepping algorithm
--------------------------------------------------------
- We will store the previous solution only at the beginning of a batch.
The same happens with the previous time stamp and time step size.
- TBC...
Rollbacks:
----------
TBC...
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/198Increase robustness of neighbour merging routines2017-11-21T13:43:49+01:00Ghost UserIncrease robustness of neighbour merging routinesIssue
-----
During the mesh refinement iterations, we have observed that cell description link maps stored in
each vertex sometimes hold wrong but not invalid adjacency indices.
This was observed on a new worker directly after a fork has...Issue
-----
During the mesh refinement iterations, we have observed that cell description link maps stored in
each vertex sometimes hold wrong but not invalid adjacency indices.
This was observed on a new worker directly after a fork has been finished.
Solution
--------
Use geometry information to determine if a patch is really adjacent to
a vertex at a given face.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/197Let tarch VTK plotters pass file creation errors to ExaHyPE plotters2018-03-02T12:46:18+01:00Ghost UserLet tarch VTK plotters pass file creation errors to ExaHyPE plottersThis is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetH...This is something where I want to patch Peano and subsequently send the patch to Tobias:
Currently it's annoying that some ExaHyPE plotters stop the program when they cannot open output files (the ASCII/CSV writers as well as my CarpetHDF5 writer) while other silently ignore this issue (Tobias tarch-based VTK writers). Clearly, we want the user to be able to control whether problems at output shall be severe or not.
I can easily implement this in the ExaHyPE codebase, but I don't control peano, so I have to make sure Peano passes such errors into exahype's domain.
=> TODO @ Sven.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/196Fix musclhancock scheme for certain patch sizes2018-02-22T09:45:39+01:00Ghost UserFix musclhancock scheme for certain patch sizesFor me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applicatio...For me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applications (GRMHD,CCZ4).
I will do this, ticket just for book keeping.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/195Additional refinement along MPI boundaries2018-03-20T16:13:38+01:00Ghost UserAdditional refinement along MPI boundariesPeano's spacetree traversal is inverted every second iteration.
This poses a problem for ExaHyPE's master-worker communication
if the master rank has local subtrees.
Example:
--------
Imagine a uniform grid where a fork was performed on...Peano's spacetree traversal is inverted every second iteration.
This poses a problem for ExaHyPE's master-worker communication
if the master rank has local subtrees.
Example:
--------
Imagine a uniform grid where a fork was performed on Level 2. 3^d - 1 workers of our
master rank have been introduced. They and the master rank hold 1/3^d of the computational domain.
We call the portion belonging to the master rank, the local subtree of the master (rank).
In the first iteration, we correctly kick off all workers on the coarse grid before we descend into
the local subtree of the master.
In the second iteration, we start within the local subtree of the master and perform computations.
As soon as we reach the coarse grid, we kick off all workers.
The workers had to wait while we performed our computations on the master's local subtree.
Now finally, the workers will start their computations.
The whole tree traversal might take twice as long as assumed.
This becomes even worse if there are more Master-Worker boundaries.
The red bars in the plot below indicate such scenarios.
Tobias proposes to add additional refinement along the MPI boundary to prevent the need of vertical
communication. At least from the master to the worker. This might work.
I wonder however if it is really beneficial for ExaHyPE to change the traversal order in every second iteration.
In a MPI setting where we perform asynchronous communication, an inversion of the
traversal order is especially ill-suited since we then have to wait till the last message was received
by the neighbour. Instead, we would need to wait and (block) until all messages for the currently
touched vertex are received.
**Update:** The traversal order is hardwired into Peano. It is necessary to run it forward and backward.
![master_worker_synchronisation](/uploads/e1fc83f445841cb3763e0bfd7d8ec701/master_worker_synchronisation.jpg)
Additional Refinement along the MPI boundary
----------------------------------------------
This can be accomplished by continuously refining the top most parent patch (which is of type Cell) of every cell
of type Descendant which is at a Master-Worker boundary.
This has to be done until we end up with a patch of type Cell at the Master-Worker boundary.
At this point, we then need to send the solution values of the Master cell to the worker.
We further might need to impose initial conditions.
We further need to perform status flag merges in prepareSendToWorker.
**Problems with this approach:** It might introduce rippling refinings around
the artificially refined cell.
Introduce a no-operation traversal
----------------------------
We could further introduce a no-operation traversal before we perform
reductions and broadcasts which would rewind Peano's streams
but does not perform any computations and communication.
In this case, we would always follow the top-down traversal.
The observed Master-Worker synchronisation would not appear.
**Problems with this approach:**
- ~~Batching is currently not possible with~~
~~multiple adapters. We could maybe perform no operation in every second iteration.~~
~~However, we would then have still a Master Worker synchronisation. Or would we not?~~
We could have a single empty traversal in front of a batch. That would
work.
- The BoundaryDataExchanger of the heaps always assumes an inversion of the traversal in
every iteration.
To alter this behaviour, we would need to change the receive methods
in Peano's AbstractHeap,DoubleHeap, and BoundaryDataExchanger methods.
We are required to add a bool "assumeForwardTraversal" (defaults to false) to the signature.
We are further required to update ExaHyPE's solver implementations:
Whenever we receive boundary data after we have run the no-operation traversal,
we need to set this new flag to true when calling receiveData.
If we run of batch iteration, this must be done only
in the first iteration of the batch.
Have both
----------------------------
For optimal performance, it might be useful to employ both techniques.
We might use "loop padding", i.e. insert empty traversals, in order to end up with the forward traversal
any time we need to broadcast / reduce something.
In generally it might useful to handle broadcasts and reductions outside of the mapping.
Therefore, we would however need to plug into both, runAsMaster and runAsWorker.