ExaHyPE issueshttps://gitlab.lrz.de/groups/exahype/-/issues2019-03-07T19:39:00+01:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/152Dynamic plotter registration2019-03-07T19:39:00+01:00Ghost UserDynamic plotter registrationUsers shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plot...Users shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plotter registration function (semi pseudocode)
```c++
Writer* kernels::getNamedWriter(std::string name, Solver& solver) {
if(name == "ConservedWriter") return new GRMHD::ConservedWriter(*static_cast<exahype::solvers::LimitingADERDGSolver*>(solver));
if(name == "usw") return new GRMHD::SomeOtherWriter(*static_cast<exahype::solvers::ADERDGSolver*>(solver));
if(name == ...
else
failure: Dont now this plotter type.
}
```
We then can replace the generated current section
```c++
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,0,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,1,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,2,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,3,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,4,parser,new GRMHD::IntegralsWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
```
with a non-generated section (pseudocode)
```c+++
for(const int& solvernum : Parser->getSolvers()) {
int plotternum=0;
for( const string& plottername : Parser->getPlotterNamesForSolver(solvernum)) {
exahype::plotters::RegisteredPlotters.push_back(new exahype::plotters::Plotter(solvernum,plotternum++,parser,getNamedWriter(plottername, exahype::solvers::RegisteredSolvers[solvernum]));
}
}
```
This is something I can do on my own. If the Parser gives the neccessary data...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/92Multi-solvers / Parameter Studies / Sensitivity Analysis2019-09-20T15:46:45+02:00Ghost UserMulti-solvers / Parameter Studies / Sensitivity Analysis### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG so...### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG solver.
### Issues/Features
* 30/12/16: Parameter studies require using the same solver with different initial conditions and/or source terms.
The problem and discretisation (order,variables,plotters,...) of the solver does not change in these studies.
I plan to enable such studies with an annotation {<studyNumber>}, e.g., "solver SolverType MySolver{10}". that is interpreted by the Toolkit
which then creates a constructor that passes the number of the study (0 to 9 in the above example), and
further adds the required number of solvers to the registry.
The current Plotter infrastructure does not support this idea yet because of its dependence on exahype::Parser.
* ~~30/12/16: Limiting-ADER-DG: Limiter domain seems to be required to often while the actual limiter domain does not change per solver.~~ Fixed.
* ~~30/12/16: MPI crashes for multiple ADER-DG/Limiting ADER-DG solvers (seg-fault.)~~ Fixed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/151Single-node MPI strong scaling differences between 2d and 3d2019-09-20T15:46:36+02:00Ghost UserSingle-node MPI strong scaling differences between 2d and 3dI am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* A...I am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* All plotters are turned off
* To exclude interconnect effects, all experiments are performed on a single-node.
In my experiments, I switch the master-worker communication (M/W)
on or off as well as the neighbour communication (N).
## Only Peano communication (M/W=off,N=off)
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 37.07 | 1.27828 | 316.768 | 10.923 |
3 | ADERDGTimeStep | 29 | 36.25 | 1.25 | 305.752 | 10.5432 |
12 | ADERDGTimeStep | 29 | 27.04 | 0.932414 | 206.142 | 7.10834 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.4012 | 0.841421 |
## M/W=on, N=off
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 |ADERDGTimeStep | 29 | 37.58 | 1.29586 | 316.044 | 10.8981 |
3 | ADERDGTimeStep | 29 | 36.3 | 1.25172 | 306.977 | 10.5854 |
12 | ADERDGTimeStep | 29 | 27.21 | 0.938276 | 207.078 | 7.14064 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.5317 | 0.845921 |
## M/W=off, N=on
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 39.52 | 1.36276 | 337.709 | 11.6451
3 | ADERDGTimeStep | 29 | 99.04 | 3.41517 | 995.378
12 | ADERDGTimeStep | 29 | 121.76 | 4.19862 | 1106.45 | 38.1534
28 | ADERDGTimeStep | 29 | 18.24 | 0.628966 | 105.858 | 3.65027
## M/W=on, N=on
Slightly worse than M/W=off,N=on.
# Insights:
* Rank 3 and 12 performance are load balancing issues.
* For the 28 rank run, the LB only deploys 10 ranks. This is actually a well-balanced setup for 10 ranks. (If we set 10 ranks, we have a load balancing issue again.)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/69Do memory precision tests2019-03-07T19:23:10+01:00Ghost UserDo memory precision testsTobias schreibt:
>
ich braeuchte mal Deine Hilfe (ja, natuerlich schnell, weil das in die Praesentation rein soll, aber das haste eh vermutet ;-). Die neue ExaHyPE-Version von gestern Nacht kann mit Zahldarstellungen <IEEE double Standa...Tobias schreibt:
>
ich braeuchte mal Deine Hilfe (ja, natuerlich schnell, weil das in die Praesentation rein soll, aber das haste eh vermutet ;-). Die neue ExaHyPE-Version von gestern Nacht kann mit Zahldarstellungen <IEEE double Standard arbeiten. Das ist teuer (erste Schritte suggerieren einen Faktor 5 in der Laufzeit, aber das koennte auch am Chip liegen, d.h. ich muss die Russenmaschine testen), aber es reduziert halt den Speicherbedarf einer Simulation etwas. Ich bekomm 10% schon fuer den einfachen Euler 2d mit Ordnung 3 - hoffe aber, dass das bei anderen Setups deutlich signifikanter ist. Einschalten laesst sich das Feature, indem man
>
double-compression = 0.000001
>
auf etwas zwischen 0 und 1 setzt. Das andere Flag spawn-double-compression-as-background-thread kannste vergessen, das ist meine Baustelle. Meine Frage/Bitte ist nun: Hast Du einen sinnvollen Benchmark zur Hand aus der Astrophysik und kannst Du mal
double-compression = 0.0
double-compression = 0.0001
double-compression = 0.000001
double-compression = 0.000000000001
>
damit ausprobieren und mir sagen, ob Werte grosser 0 die Loesung qualitativ verschlechtern? Ich hab keine Ahnung, was Ihr Euch typischerweise anseht (Ankunftszeiten/Amplituden/...?) deswegen brauch ich da ein qualifiziertes Auge. Wenn Du parallel noch drauf schaust, was er als memoryUsage ausgibt (macht er jetzt automatisch, wenn man nicht mit MPI uebersetzt), dann wuerde mir das sehr helfen, da eine Einschaetzung zu bekommen.
>
Nachtrag: So saehe dann eine Optimisation Sectino aus
```
optimisation
fuse-algorithmic-steps = on
fuse-algorithmic-steps-factor = 0.99
timestep-batch-factor = 0.0
skip-reduction-in-batched-time-steps = on
disable-amr-if-grid-has-been-stationary-in-previous-iteration = off
double-compression = 0.000001
spawn-double-compression-as-background-thread = off
end optimisation
```
@svenk: Machen.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/58Reduce memory footprint2019-09-20T15:46:51+02:00Ghost UserReduce memory footprint# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish b...# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish between cell and face data since we have helper cells that do not allocate cell data but face data.
* We can remove prediction and volumeFlux fields completely if we perform also the time integration
in the volume integral and the boundary extrapolation routines.
This would further make it easier to switch between global and local time stepping.
Here, we would just load a different kernel for the boundary extrapolation and allocate space-time face data
if the user switches local time stepping on.
* Allocate all the temporary arrays like the rhs, lQi_old, etc. only once per Thread and not dynamically
during the kernel calls. **(This could be done easily now in each solver!)**
* Create one "big" ADERDGTimeStep function in kernels/solver. This might help the compiler/is more Cache-friendly.
# Done
I think the following has been Tobias' idea originally;
It is not necessary to store temporary data on the heap for every cell description.
We need to analyse which ADER-DG fields are temporary and which
need to be stored persistently on the heap.
From my point of view, the following variables are temporary:
* spaceTimePredictor
* predictor
* spaceTimeVolumeFlux (includes sources)
* volumeFlux (includes sources)
The spacetime fields have a massive memory footprint.
They scale with (N+1)^{d+1} and d*(N+1)^{d+1}.
I thus propose that we assign each thread its own spaceTimePredictor spaceTimeVolumeFlux, predictor, and volume flux fields
and remove the fields from the heap cell descriptions.
This would reduce the memory footprint of the ADER-DG method dramatically (and might further lead to more cache-friendly code ?).
In a second step, we should kick out the volumeFlux field completely, don't do the time integration of the spaceTimeVolumeFlux,
and directly perform the volume integral with the spaceTimeVolumeFlux.
Implementation details
* Allocate arrays in Prediction mapping for each threadhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/148LimitingADERDGSolver alsos performs limiter status spreading for user refinement2019-09-20T15:46:35+02:00Ghost UserLimitingADERDGSolver alsos performs limiter status spreading for user refinementPotential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.Potential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/57Finite volumes solver / Limiter2019-09-20T15:46:56+02:00Ghost UserFinite volumes solver / Limiter# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to g...# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to get rid of the "update" field if we directly
add a weighted (by dt and quad weight) update to the "solution" field
from the volume integral and surface integral evaluation.
# Status of the implementation of the limiting ADER-DG scheme (unordered notes)
* The limiter workflow works now for uniform meshes with Intel's TBBs.
[sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/b71d24c7b82f805c604f31c1b322fcae/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi)
* Found and fixed some more bugs. Now everything looks fine. The limiter has a local characteristic and only
requires a rollback/reallocation of memory in every fifth time step (49 out of 255) in the experiment
shown [sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/1e065c7c5c2a52aefb551ef0a03f0ac5/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi), where we ran an experiment with P=3 order ADER-DG approx, \delta_0=1e-2, \epsilon=1e-3.
As long as the limiter domain does not change, we do not need to reallocate new memory and do a
recomputation of certain cells in our novel implementation.
* Find another simulation using a 9-th order ADER-DG approximation and parameters \delta_0=10^-4 , \epsilon=10^-3 here:
[sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi](/uploads/fb466322f7df5339790aac4afca9ce6c/sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi)
~~We observee a strange x velocity in this benchmark. Can't be seen in the video. The both profile of the Shock-Tube differs signifantly
in vicinity of the contact discontinuity.~~This is actually the momentum density. Everything is fine.
* Explosion problem with P=5 and delta_0=1e-4 and epsilon=1e-3:
[explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi](/uploads/4ace50cdcbfcbc1706432d0cc180fcea/explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi). Here the limiter domain must be adjusted after most of the time steps
* I performed some further optimisations of the DMP evaluation. I found that we can easily perform a
"loop" fusion of the min and max computation and the DMP.
* Implemented the physical admissibility detection (PAD) now as well and use it to
detect non-physical oscillations in the initial conditions. Here, I found that we can directly
pass it the solution min and max we computed as part of the DMP calculation. No need
to loop over the whole degrees of freedom and the Lobatto nodes again.
* ~~There is still an issue with the discrete maximum principle which detects too many
cells to be troubled.~~ Was resolved by tuning the DMP parameters.
* AMR and Limiting need to be combined. This is in principle "a simple" wiring of FV patches
along the ADER-DG tree. We further need spatial averaging and interpolation operators.
Then, we can follow the AMR timestepping implementation of the ADER-DG scheme.
# Writing a Godunov type first order method
Due to the issues I encountered while [rewriting](#issueswithrewritingthefinitevolumessolver) the
pseudo-TVD donor cell type finite volumes solver, Tobias and me decided
to write a simple first order Godunov type finite volumes solver that only exchanges volume averages/fluxes
between direct (face) neighbours.
Replacement of this simple method by a more complex one will be tackled in later stages of the project -- if necessary.
# Issues with higher order finite volumes solvers
* Higher-order methods have a reconstruction stencil which is larger than the
Riemann solver stencil (simple star).
* Reconstruction and time evolution of boundary extrapolated values at one face of a patch can only be performed
after all volume averages from the direct neighbour and corner neighbour patches are available.
I have identified the following phases of the FVM solver now:
* Gather: Just get the neighbour values (arithmetic intensity = 0). This will move into the Merging mapping.
* Spatial reconstruction: Compute spatial slopes in the cells or something similar. This will move into the SolutionUpdate mapping.
* Temporal evolution of boundary-extrapolated values: This will move into the SolutionUpdate mapping.
* Riemann solve: This will move into the SolutionUpdate mapping.
* Solution update; This will move into SolutionUpdate mapping.
* Extrapolation of volume averages: This will send layers of the volume averages to the neighbours.
We send layers instead of a single layer in order to only have one data exchange instead of two.
With the above strategy, we can merge FVM solvers into the current framework. The difference to the ADER-DG solver
is that the computational intensity of the neighbour merging operation is zero.
We will further need to consider neighbour data exchange over edges (3-d) and corners in our
merging scheme. Neighbour data exchange over edges in 3-d will require additional falgs.
Exchange over corners does not require additional flags.
# Issues with rewriting the finite volumes solver
I tried to decompose our pseudo-TVD donor cell type finite volumes
solver into a Riemann solve and a solution update part,
and ran into the following issues.
## Open issues with the donor cell type pseudo-TVD finite volumes solver implementation:
* The current solver uses updated solution values in the solution update.
We have to distinguish between old values and new values or
have to introduce an update.
* We need to consider two layers of the neighbour to compute all extrapolated boundary
values (wLx,wRx,wLy,wRy). Currently only one layer ist considered. This means
that the boundary extrapolated values and thus the face fluxes at the outermost faces
are computed wrongly.
* To tackle the above problem, we either need to exchange a single cell of
the diagonal neighbours or we need to rely on two data exchanges.
Tobias told me there exists however a trick to circumvent this:
Reconstructing the diagonal neighbours' contributions with values from
the direct neighbours.
## Limitations of the donor cell type pseudo-TVD finite volumes solver:
* The solver is not TVD in multiple dimensions.
* We do neither perform corner correction nor (dimensional) operator splitting. We should
thus observe loss of mass conservation, large dispersion errors, and a reduced CFL stability limit.
The reduced CFL stability limit was taken into account in our implementation.
This is also a problem of the multi-dim. Godunov method in the unsplit form.
## Follow up: Low-order Euler-DG patch with direct coupling to ADER-DG method?
## Dissemination
For the following benchmarks I always used copy boundary conditions which are equal to
outflow/inflow boundary conditions as long as everything flows out of the domain.
* Euler (Compressible Hydrodynamics)
* Explosion Problem with 27^2 grid, P=3, and N=2*3+1 FV subgrid:
![lim-aderdg_explosion](/uploads/6459f9b4a775a482cdfec24e129c56ae/lim-aderdg_explosion.png)
[lim-aderdg_explosion.avi](/uploads/6a38fa00a6bf41fe3aefbee380a03396/lim-aderdg_explosion.avi)
* Sod Shock Tube with 27^2 grid, P=3 and N=2*3+1 FV subgrid:
![lim-aderdg_euler_sod-shock-tube](/uploads/eb2a5e57ca8f37665c2dffe48dbf1e22/lim-aderdg_euler_sod-shock-tube.png)
[lim-aderdg_euler_sod-shock-tube.avi](/uploads/61dfedf7db4ead996d7fd6b9796ba5c0/lim-aderdg_euler_sod-shock-tube.avi)
![hires_antialias](/uploads/26843a0f50e52ded919cfc12b777360b/draft2_hires_antialias.png)
* SRMHD (Special Relativistic Magnetohydrodynamics; basically: Special Relativistic Euler+Maxwell)
* Blast Wave setup as in 10.1016/j.cpc.2014.03.018 with 27^2 grid, P=5, and N=2*5+1 FV subgrid:
![lim-aderg_mhd__blast-wave](/uploads/feb026167c7c2d882c132f3b775c37ac/lim-aderg_mhd__blast-wave.png)
[lim-aderdg_mhd_blast-wave.avi] (/uploads/4d715c47a03bd3850d3c0398e39da58c/lim-aderdg_mhd_blast-wave.avi)
* Rotor setup as in http://adsabs.harvard.edu/abs/2004MSAIS...4...36D with 9^2 grid, P=9, and N=2*9+1 FV subgrid:
![lim-aderdg_mhd_rotor_9x9_P9](/uploads/b35e7dc2dc137068e7416a1952d02a1a/lim-aderdg_mhd_rotor_9x9_P9.png)
[lim-aderdg_mhd_rotor_9x9_P9.avi](/uploads/c5ee8ff29f18a7201be66a1a26b664dc/lim-aderdg_mhd_rotor_9x9_P9.avi)
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/145SegFault occurs if I turn conservative flux off2018-06-15T15:13:07+02:00Ghost UserSegFault occurs if I turn conservative flux offProject: DIM_LimitingADERDGProject: DIM_LimitingADERDGhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/37Precision issues2019-04-15T12:37:23+02:00Ghost UserPrecision issuesI observed that we can only apply the restriction and prolongation operations
a certain number of times until the projected constant values in my test are not equal anymore
to the reference values (with respect to a tolerance of 1e-12)....I observed that we can only apply the restriction and prolongation operations
a certain number of times until the projected constant values in my test are not equal anymore
to the reference values (with respect to a tolerance of 1e-12).
For more infos see the todos in 2D and 3D definitions of
GenericEulerKernelTest::testFaceUnknownsProjection(),
and
GenericEulerKernelTest::testVolumeUnknownsProjection()https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/143New template-less pure virtual API2019-03-07T19:23:41+01:00Ghost UserNew template-less pure virtual APITODO: Implement this new API.
![usersolver_layout](/uploads/b314b4b3a6196966cec01c80918fb0c9/usersolver_layout.png)
Tobias favours it. Don't know how it goes with optimized kernels. SMall patch for the piucture from Tobias:
```
/* N...TODO: Implement this new API.
![usersolver_layout](/uploads/b314b4b3a6196966cec01c80918fb0c9/usersolver_layout.png)
Tobias favours it. Don't know how it goes with optimized kernels. SMall patch for the piucture from Tobias:
```
/* NEW: */
kernels::aderdg::generic::c::spaceTimePredictorLinear(BasisSolverAPI& solver, other parameters ... );
```
In textual form [Sketch.h](/uploads/8bcfc8c2920e7532f73e1dac5f8f5a07/Sketch.h)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/142ADERDG, inverseDX2018-06-19T11:17:55+02:00Jean-Matthieu GallardADERDG, inverseDX* Problem: most kernel use 1/dx instead of dx. Slow operation that could easily be optimized away.
* Solution:
- Peano implement a cellDescription.getInverseSize()
- ADERDGSolver use it
- ADERDG Kernel adapted
* Already done:
- Op...* Problem: most kernel use 1/dx instead of dx. Slow operation that could easily be optimized away.
* Solution:
- Peano implement a cellDescription.getInverseSize()
- ADERDGSolver use it
- ADERDG Kernel adapted
* Already done:
- Optimized kernel adapted, generate the inverseDx in the ADERDGSolver (code isolated with preprocessor)
@di25coxJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/140Intel compiler bug on certain large C++ files ("exahype/repositories/Reposito...2019-08-25T11:41:31+02:00Ghost UserIntel compiler bug on certain large C++ files ("exahype/repositories/Repository...")With Intel 17 (Intel 15/16 apparently not affected) aka `COMPILER=Intel, SHAREDMEM=TBB, MODE=Debug, DISTRIBUTEDMEM=None.` there is a bug when compiling files like
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstant...With Intel 17 (Intel 15/16 apparently not affected) aka `COMPILER=Intel, SHAREDMEM=TBB, MODE=Debug, DISTRIBUTEDMEM=None.` there is a bug when compiling files like
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstantiation4LimiterStatusMergingAndSpreadingMPI.o
and
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstantiation4GridErasing.o
and similar. This is known to Tobias and should be reported to Nicolay Hammer at LRZ or similar.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/1362D / 3D build check is not performed correctly2019-03-18T21:55:01+01:00Ghost User2D / 3D build check is not performed correctlyWe have a startup check in the code that makes sure that the specfile has some same `const` build constants as for instance the ADERDG polynomial order. However, this test fails for checking that for instance a 2D build is also runned wi...We have a startup check in the code that makes sure that the specfile has some same `const` build constants as for instance the ADERDG polynomial order. However, this test fails for checking that for instance a 2D build is also runned with a 2D grid specification. Thus we get strange errors as in https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/10b1261b34e2e239f9a4fe1e2c42f0e38d85a207 :
```
0.0459649 error Invalid simulation end-time: notoken
(file:/home/koeppel/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/Parser.cpp,line:377)
` ``
And that not after the beginning but after one timestep.
There should be a well-speaking check instead at the beginning.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/130Guidebook: Fix recommendation for MPI Tags for Reductions2019-03-07T19:24:19+01:00Ghost UserGuidebook: Fix recommendation for MPI Tags for ReductionsWhen implementing an own MPI Reductions plotter, the guidebook tells us we should write something like
```
void finishRow() {
#ifdef Parallel
// Question: Do we really reserve a free tag and release on each function call?
c...When implementing an own MPI Reductions plotter, the guidebook tells us we should write something like
```
void finishRow() {
#ifdef Parallel
// Question: Do we really reserve a free tag and release on each function call?
const int reductionTag = tarch::parallel::Node::getInstance().reserveFreeTag(
std::string("TimeSeriesReductions(") + filename + ")::finishRow()" );
if(master) {
double recieved[LEN];
for (int rank=1; rank<tarch::parallel::Node::getInstance().getNumberOfNodes(); rank++) {
if(!tarch::parallel::NodePool::getInstance().isIdleNode(rank)) {
MPI_Recv( &recieved[0], LEN, MPI_DOUBLE, rank, reductionTag, tarch::parallel::Node::getInstance().getCommunicator(), MPI_STATUS_IGNORE );
addValues(recieved);
}
}
} else {
for (int rank=1; rank<tarch::parallel::Node::getInstance().getNumberOfNodes(); rank++) {
if(!tarch::parallel::NodePool::getInstance().isIdleNode(rank)) {
MPI_Send( &data[0], LEN, MPI_DOUBLE, tarch::parallel::Node::getGlobalMasterRank(), reductionTag, tarch::parallel::Node::getInstance().getCommunicator());
}
}
}
tarch::parallel::Node::getInstance().releaseTag(reductionTag);
#endif
if(master) {
TimeSeriesReductions::finishRow();
writeRow();
}
}
```
However I assume this is not that good, looking at the output:
![mpi-reservefreetags](/uploads/de93f1e5c48d10c62101531611e044cd/mpi-reservefreetags.png)
*First question*: Does every plotter need it's own tag? In principal only if they would be called in parallel, isn't it?
In any case, I think the tags should'nt be created so frequently, isn't it? Instead, only once on startup time?
This should be fixed in the guidebook and my code :Dhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/129Access to neighbours for which kernels?2018-06-15T15:13:07+02:00Ghost UserAccess to neighbours for which kernels?Finite Volumes Solver:
- source; ex: bathymetry in SWEFinite Volumes Solver:
- source; ex: bathymetry in SWEhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/127Revise time stepping notation2019-09-20T15:46:39+02:00Ghost UserRevise time stepping notationI have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little comp...I have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little complicated and the docu just gives an partial overview.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/126CFL factor per solver as const int the specfile2019-04-15T12:37:03+02:00Ghost UserCFL factor per solver as const int the specfileThis reduces inconsistencies. Can now be easily generated as constexpr double into the AbstractMySolver class.This reduces inconsistencies. Can now be easily generated as constexpr double into the AbstractMySolver class.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/124Toolkit - Move from replaceAll to true template engine2018-09-21T12:09:36+02:00Jean-Matthieu GallardToolkit - Move from replaceAll to true template engineThe templates used by the Toolkit start to require some logic. For example the NamingScheme tag requires to expand a Set to multiple lines.
Currently we put that logic in the java class and use a replaceAll but it would be better to put...The templates used by the Toolkit start to require some logic. For example the NamingScheme tag requires to expand a Set to multiple lines.
Currently we put that logic in the java class and use a replaceAll but it would be better to put that logic in the template itself using a true template engine. For example the NamingScheme tag would be replace with a foreach loop that expand the Set in the template. Also this would help factorizing the code by putting common tag and their value into a more global context set for the template engine
Switching should be simple as we are already using standard template tags for our variable.
Ofc the template engine needs to be light and fully available in a jar or any format requiring no installation to avoid adding new dependencies. For example https://github.com/HubSpot/jinjava or http://jtwig.org/
This is a "nice to have" feature so we should do it when we have more time.
Concerned: @di25cox , @gi26det , @svenk , @ga96nuvJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/117Support material parameters2019-09-20T15:46:49+02:00Ghost UserSupport material parametersTODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument swit...TODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument switch to the constexpr for variables,parameters,and
order/basisSize.~~
* ~~MyElasticWaveSolver: Realised that we have to rethink our material approach
a little. Material parameters are not available from Q during the space-time predictor
computation which uses the space-time predictor or predictor DoF.
We need either:~~
* store the material values in solution(+previousSolution),
predictor,and space-time predictor
* ~~or pass the material values as separate argument
to flux,eigenvalues,ncp,matrixb etc.
We should then also pass them as separate arguments
to adjustedSolutionValues.~~
~~Follow up:
It might wise then to split the material parameters from the variables.~~ We have chosen the first option.
* ~~The Nonlinear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Nonlinear 3D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Linear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~ The Linear3D ADER-DG C/C++ kernels do not support materials yet.~~
* The Fortran ADER-DG kernels do not support materials yet.
* The 2D FV kernels do not support materials yet.
* The 3D FV kernels do not support materials yet.
* The Linear 2D+3D ADER-DG C/C++ kernels need to consider material parameters in the AMR operators.
* (Optional) Transform all generic kernels to templates and use constexpr for variables,parameters and order/basisSize.
Questions:
* What happens at the boundary for the Finite Volumes method? Currently, we hand no material parameters in at the boundary.
* Are there no time averaged source terms in the spaceTimePredictorLinear2d/3d? Yes, I consider them now at least in the striding.
* The linear ADER-DG kernels work with time derivatives. Does it make sense to let the user set the nodal values?
* Why wasn't tempPointForceSources merged into tempSpaceTimeUnknowns. The API change wasn't necessary.
Follow ups:
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.
* Distinguish more clearly between temporary array sizes and dof array sizes.
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/110Compression2018-06-15T15:13:07+02:00Ghost UserCompression## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedom## TODO
* ~~add "hint-size" to the relevant compression fields.~~
* support the Finite Volumes solver degrees of freedom