ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2020-11-20T17:30:31+01:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/297FiniteVolumeSolver error about time step2020-11-20T17:30:31+01:00Duo Lid.li@lmu.deFiniteVolumeSolver error about time stepWhen running FV, there is this error. I use CFL=0.7. How to fix the warning or how does it matter to the code.
```cpp
if ( !tarch::la::equals(cellDescription.getTimeStepSize(), 0.0) && tarch::la::smaller(admissibleTimeStepSize,cellDesc...When running FV, there is this error. I use CFL=0.7. How to fix the warning or how does it matter to the code.
```cpp
if ( !tarch::la::equals(cellDescription.getTimeStepSize(), 0.0) && tarch::la::smaller(admissibleTimeStepSize,cellDescription.getTimeStepSize()) ) {
logWarning("updateSolution(...)","Finite volumes solver time step size harmed CFL condition. dt="<<
cellDescription.getTimeStepSize()<<", dt_adm=" << admissibleTimeStepSize << ". cell=" <<cellDescription.toString());
}
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/302error when use dynamic AMR in LimitedADERDG2020-11-20T15:10:48+01:00Duo Lid.li@lmu.deerror when use dynamic AMR in LimitedADERDGparameter tarch::parallel::Node::getInstance().getRank(): 17
parameter "if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before": if the neighbour data buffer is emp...parameter tarch::parallel::Node::getInstance().getRank(): 17
parameter "if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before": if the neighbour data buffer is empty, you have perhaps forgotten to call releaseMessages() on the heap in the traversal before
ExaHyPE-Diffuse: /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./Peano/peano/../peano/heap/BoundaryDataExchanger.cpph:282: void peano::heap::BoundaryDataExchanger<Data, SendReceiveTaskType, VectorContainer>::freeAllMessagesInDeployBuffer() [with Data = char, SendReceiveTaskType = peano::heap::SendReceiveTask<char>, VectorContainer = std::vector<char, std::allocator<char>>]: Assertion `false' failed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/304Merge with Latest Peano to resolve debug logging errors2020-01-15T11:36:58+01:00Samuel TootleMerge with Latest Peano to resolve debug logging errorsWhen compiling with MODE=Debug, various errors arise related to Peano logging. I checked the Peano master repo and discovered that this issue has been resolved, however, this version has not yet been synched with Exahype sub-modules. T...When compiling with MODE=Debug, various errors arise related to Peano logging. I checked the Peano master repo and discovered that this issue has been resolved, however, this version has not yet been synched with Exahype sub-modules. This was tested using updateSubmodules.sh which results in reverting to the old git hash even after manually pulling Peano/master.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/299Compilation error2019-10-16T10:05:11+02:00Duo Lid.li@lmu.deCompilation errorI compiled with mpi.intel/2017 and tbb/2017, after update Exahype and submodules.
Here attached the errors:
/import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver_ProlongationJob.cpp(42): error: argument...I compiled with mpi.intel/2017 and tbb/2017, after update Exahype and submodules.
Here attached the errors:
/import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver_ProlongationJob.cpp(42): error: argument of type "double *" is incompatible with parameter of type "const char *"
_mm_prefetch(lQhbnd, _MM_HINT_NTA);
^
/import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver_ProlongationJob.cpp(43): error: argument of type "double *" is incompatible with parameter of type "const char *"
_mm_prefetch(lFhbnd, _MM_HINT_NTA);
^
compilation aborted for /import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver_ProlongationJob.cpp (code 2)
/import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/Makefile:626: recipe for target '/import/deadlock-data/dli/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/ADERDGSolver_ProlongationJob.o’ failedhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/303error in compilation2019-10-15T09:38:50+02:00Duo Lid.li@lmu.deerror in compilationwhen compiling master Exahype, i have a bunch of errors with GaussLobattoBasis.hsnippet:
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from ...when compiling master Exahype, i have a bunch of errors with GaussLobattoBasis.hsnippet:
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(1): error: expected a declaration
namespace kernels {
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(22): warning #12: parsing restarts here after previous syntax error
extern double nodes_1[1+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(23): error: invalid storage class for a class member
extern double nodes_2[2+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(24): error: invalid storage class for a class member
extern double nodes_3[3+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(25): error: invalid storage class for a class member
extern double nodes_4[4+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(26): error: invalid storage class for a class member
extern double nodes_5[5+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(27): error: invalid storage class for a class member
extern double nodes_6[6+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(28): error: invalid storage class for a class member
extern double nodes_7[7+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(29): error: invalid storage class for a class member
extern double nodes_8[8+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(30): error: invalid storage class for a class member
extern double nodes_9[9+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(31): error: invalid storage class for a class member
extern double nodes_10[10+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(32): error: invalid storage class for a class member
extern double nodes_11[11+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(33): error: invalid storage class for a class member
extern double nodes_12[12+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(34): error: invalid storage class for a class member
extern double nodes_13[13+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(35): error: invalid storage class for a class member
extern double nodes_14[14+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(36): error: invalid storage class for a class member
extern double nodes_15[15+1];
^
In file included from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/GaussLobattoBasis.h(20),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/../Curvilinear/CurvilinearTransformation.h(6),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/include/Context/ContextCurvilinear.h(11),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.h(28),
from /hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ApplicationExamples/ExaSeis/DynamicRupture/TPV29/SlipWeakeningSolver.cpp(9):
/hppfs/work/pr63qo/di52lak2/ExaHyPE_deadlock/ExaHyPE-Engine/./ExaHyPE/kernels/generated/GaussLobattoBasis.hsnippet(37): error: invalid storage class for a class member
extern double* nodes[15+1];
does any one have any idea?https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/296GradQ format2019-09-21T15:09:46+02:00Jean-Matthieu GallardGradQ formatThe format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*...The format of gradQ (used by NCP and ViscousFlux) is not consistent
* Linear => like F for the flux PDE so gradQ[nDim][nVar]
* Nonlinear => all in one dimension gradQ[nDim\*nVar]
* FV => all in one dimension but with param gradQ[nDim\*nData]
Changing this will change the user API and break some application, however a simple adapter could allow them to run until a more in depth correction is done.
Do we need the parameter in FV?
This would also be a good time to maybe change BgradQ similarly if possible (I don't know it it would make sense) so that linear and nonlinear have the same API, even if it introduce a slight inefficiency in nonlinear/FV
@reinarz @domcha @ga24dib @leo.rannabauerJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/57Finite volumes solver / Limiter2019-09-20T15:46:56+02:00Ghost UserFinite volumes solver / Limiter# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to g...# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to get rid of the "update" field if we directly
add a weighted (by dt and quad weight) update to the "solution" field
from the volume integral and surface integral evaluation.
# Status of the implementation of the limiting ADER-DG scheme (unordered notes)
* The limiter workflow works now for uniform meshes with Intel's TBBs.
[sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/b71d24c7b82f805c604f31c1b322fcae/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi)
* Found and fixed some more bugs. Now everything looks fine. The limiter has a local characteristic and only
requires a rollback/reallocation of memory in every fifth time step (49 out of 255) in the experiment
shown [sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/1e065c7c5c2a52aefb551ef0a03f0ac5/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi), where we ran an experiment with P=3 order ADER-DG approx, \delta_0=1e-2, \epsilon=1e-3.
As long as the limiter domain does not change, we do not need to reallocate new memory and do a
recomputation of certain cells in our novel implementation.
* Find another simulation using a 9-th order ADER-DG approximation and parameters \delta_0=10^-4 , \epsilon=10^-3 here:
[sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi](/uploads/fb466322f7df5339790aac4afca9ce6c/sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi)
~~We observee a strange x velocity in this benchmark. Can't be seen in the video. The both profile of the Shock-Tube differs signifantly
in vicinity of the contact discontinuity.~~This is actually the momentum density. Everything is fine.
* Explosion problem with P=5 and delta_0=1e-4 and epsilon=1e-3:
[explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi](/uploads/4ace50cdcbfcbc1706432d0cc180fcea/explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi). Here the limiter domain must be adjusted after most of the time steps
* I performed some further optimisations of the DMP evaluation. I found that we can easily perform a
"loop" fusion of the min and max computation and the DMP.
* Implemented the physical admissibility detection (PAD) now as well and use it to
detect non-physical oscillations in the initial conditions. Here, I found that we can directly
pass it the solution min and max we computed as part of the DMP calculation. No need
to loop over the whole degrees of freedom and the Lobatto nodes again.
* ~~There is still an issue with the discrete maximum principle which detects too many
cells to be troubled.~~ Was resolved by tuning the DMP parameters.
* AMR and Limiting need to be combined. This is in principle "a simple" wiring of FV patches
along the ADER-DG tree. We further need spatial averaging and interpolation operators.
Then, we can follow the AMR timestepping implementation of the ADER-DG scheme.
# Writing a Godunov type first order method
Due to the issues I encountered while [rewriting](#issueswithrewritingthefinitevolumessolver) the
pseudo-TVD donor cell type finite volumes solver, Tobias and me decided
to write a simple first order Godunov type finite volumes solver that only exchanges volume averages/fluxes
between direct (face) neighbours.
Replacement of this simple method by a more complex one will be tackled in later stages of the project -- if necessary.
# Issues with higher order finite volumes solvers
* Higher-order methods have a reconstruction stencil which is larger than the
Riemann solver stencil (simple star).
* Reconstruction and time evolution of boundary extrapolated values at one face of a patch can only be performed
after all volume averages from the direct neighbour and corner neighbour patches are available.
I have identified the following phases of the FVM solver now:
* Gather: Just get the neighbour values (arithmetic intensity = 0). This will move into the Merging mapping.
* Spatial reconstruction: Compute spatial slopes in the cells or something similar. This will move into the SolutionUpdate mapping.
* Temporal evolution of boundary-extrapolated values: This will move into the SolutionUpdate mapping.
* Riemann solve: This will move into the SolutionUpdate mapping.
* Solution update; This will move into SolutionUpdate mapping.
* Extrapolation of volume averages: This will send layers of the volume averages to the neighbours.
We send layers instead of a single layer in order to only have one data exchange instead of two.
With the above strategy, we can merge FVM solvers into the current framework. The difference to the ADER-DG solver
is that the computational intensity of the neighbour merging operation is zero.
We will further need to consider neighbour data exchange over edges (3-d) and corners in our
merging scheme. Neighbour data exchange over edges in 3-d will require additional falgs.
Exchange over corners does not require additional flags.
# Issues with rewriting the finite volumes solver
I tried to decompose our pseudo-TVD donor cell type finite volumes
solver into a Riemann solve and a solution update part,
and ran into the following issues.
## Open issues with the donor cell type pseudo-TVD finite volumes solver implementation:
* The current solver uses updated solution values in the solution update.
We have to distinguish between old values and new values or
have to introduce an update.
* We need to consider two layers of the neighbour to compute all extrapolated boundary
values (wLx,wRx,wLy,wRy). Currently only one layer ist considered. This means
that the boundary extrapolated values and thus the face fluxes at the outermost faces
are computed wrongly.
* To tackle the above problem, we either need to exchange a single cell of
the diagonal neighbours or we need to rely on two data exchanges.
Tobias told me there exists however a trick to circumvent this:
Reconstructing the diagonal neighbours' contributions with values from
the direct neighbours.
## Limitations of the donor cell type pseudo-TVD finite volumes solver:
* The solver is not TVD in multiple dimensions.
* We do neither perform corner correction nor (dimensional) operator splitting. We should
thus observe loss of mass conservation, large dispersion errors, and a reduced CFL stability limit.
The reduced CFL stability limit was taken into account in our implementation.
This is also a problem of the multi-dim. Godunov method in the unsplit form.
## Follow up: Low-order Euler-DG patch with direct coupling to ADER-DG method?
## Dissemination
For the following benchmarks I always used copy boundary conditions which are equal to
outflow/inflow boundary conditions as long as everything flows out of the domain.
* Euler (Compressible Hydrodynamics)
* Explosion Problem with 27^2 grid, P=3, and N=2*3+1 FV subgrid:
![lim-aderdg_explosion](/uploads/6459f9b4a775a482cdfec24e129c56ae/lim-aderdg_explosion.png)
[lim-aderdg_explosion.avi](/uploads/6a38fa00a6bf41fe3aefbee380a03396/lim-aderdg_explosion.avi)
* Sod Shock Tube with 27^2 grid, P=3 and N=2*3+1 FV subgrid:
![lim-aderdg_euler_sod-shock-tube](/uploads/eb2a5e57ca8f37665c2dffe48dbf1e22/lim-aderdg_euler_sod-shock-tube.png)
[lim-aderdg_euler_sod-shock-tube.avi](/uploads/61dfedf7db4ead996d7fd6b9796ba5c0/lim-aderdg_euler_sod-shock-tube.avi)
![hires_antialias](/uploads/26843a0f50e52ded919cfc12b777360b/draft2_hires_antialias.png)
* SRMHD (Special Relativistic Magnetohydrodynamics; basically: Special Relativistic Euler+Maxwell)
* Blast Wave setup as in 10.1016/j.cpc.2014.03.018 with 27^2 grid, P=5, and N=2*5+1 FV subgrid:
![lim-aderg_mhd__blast-wave](/uploads/feb026167c7c2d882c132f3b775c37ac/lim-aderg_mhd__blast-wave.png)
[lim-aderdg_mhd_blast-wave.avi] (/uploads/4d715c47a03bd3850d3c0398e39da58c/lim-aderdg_mhd_blast-wave.avi)
* Rotor setup as in http://adsabs.harvard.edu/abs/2004MSAIS...4...36D with 9^2 grid, P=9, and N=2*9+1 FV subgrid:
![lim-aderdg_mhd_rotor_9x9_P9](/uploads/b35e7dc2dc137068e7416a1952d02a1a/lim-aderdg_mhd_rotor_9x9_P9.png)
[lim-aderdg_mhd_rotor_9x9_P9.avi](/uploads/c5ee8ff29f18a7201be66a1a26b664dc/lim-aderdg_mhd_rotor_9x9_P9.avi)
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/58Reduce memory footprint2019-09-20T15:46:51+02:00Ghost UserReduce memory footprint# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish b...# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish between cell and face data since we have helper cells that do not allocate cell data but face data.
* We can remove prediction and volumeFlux fields completely if we perform also the time integration
in the volume integral and the boundary extrapolation routines.
This would further make it easier to switch between global and local time stepping.
Here, we would just load a different kernel for the boundary extrapolation and allocate space-time face data
if the user switches local time stepping on.
* Allocate all the temporary arrays like the rhs, lQi_old, etc. only once per Thread and not dynamically
during the kernel calls. **(This could be done easily now in each solver!)**
* Create one "big" ADERDGTimeStep function in kernels/solver. This might help the compiler/is more Cache-friendly.
# Done
I think the following has been Tobias' idea originally;
It is not necessary to store temporary data on the heap for every cell description.
We need to analyse which ADER-DG fields are temporary and which
need to be stored persistently on the heap.
From my point of view, the following variables are temporary:
* spaceTimePredictor
* predictor
* spaceTimeVolumeFlux (includes sources)
* volumeFlux (includes sources)
The spacetime fields have a massive memory footprint.
They scale with (N+1)^{d+1} and d*(N+1)^{d+1}.
I thus propose that we assign each thread its own spaceTimePredictor spaceTimeVolumeFlux, predictor, and volume flux fields
and remove the fields from the heap cell descriptions.
This would reduce the memory footprint of the ADER-DG method dramatically (and might further lead to more cache-friendly code ?).
In a second step, we should kick out the volumeFlux field completely, don't do the time integration of the spaceTimeVolumeFlux,
and directly perform the volume integral with the spaceTimeVolumeFlux.
Implementation details
* Allocate arrays in Prediction mapping for each threadhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/117Support material parameters2019-09-20T15:46:49+02:00Ghost UserSupport material parametersTODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument swit...TODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument switch to the constexpr for variables,parameters,and
order/basisSize.~~
* ~~MyElasticWaveSolver: Realised that we have to rethink our material approach
a little. Material parameters are not available from Q during the space-time predictor
computation which uses the space-time predictor or predictor DoF.
We need either:~~
* store the material values in solution(+previousSolution),
predictor,and space-time predictor
* ~~or pass the material values as separate argument
to flux,eigenvalues,ncp,matrixb etc.
We should then also pass them as separate arguments
to adjustedSolutionValues.~~
~~Follow up:
It might wise then to split the material parameters from the variables.~~ We have chosen the first option.
* ~~The Nonlinear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Nonlinear 3D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Linear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~ The Linear3D ADER-DG C/C++ kernels do not support materials yet.~~
* The Fortran ADER-DG kernels do not support materials yet.
* The 2D FV kernels do not support materials yet.
* The 3D FV kernels do not support materials yet.
* The Linear 2D+3D ADER-DG C/C++ kernels need to consider material parameters in the AMR operators.
* (Optional) Transform all generic kernels to templates and use constexpr for variables,parameters and order/basisSize.
Questions:
* What happens at the boundary for the Finite Volumes method? Currently, we hand no material parameters in at the boundary.
* Are there no time averaged source terms in the spaceTimePredictorLinear2d/3d? Yes, I consider them now at least in the striding.
* The linear ADER-DG kernels work with time derivatives. Does it make sense to let the user set the nodal values?
* Why wasn't tempPointForceSources merged into tempSpaceTimeUnknowns. The API change wasn't necessary.
Follow ups:
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.
* Distinguish more clearly between temporary array sizes and dof array sizes.
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/92Multi-solvers / Parameter Studies / Sensitivity Analysis2019-09-20T15:46:45+02:00Ghost UserMulti-solvers / Parameter Studies / Sensitivity Analysis### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG so...### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG solver.
### Issues/Features
* 30/12/16: Parameter studies require using the same solver with different initial conditions and/or source terms.
The problem and discretisation (order,variables,plotters,...) of the solver does not change in these studies.
I plan to enable such studies with an annotation {<studyNumber>}, e.g., "solver SolverType MySolver{10}". that is interpreted by the Toolkit
which then creates a constructor that passes the number of the study (0 to 9 in the above example), and
further adds the required number of solvers to the registry.
The current Plotter infrastructure does not support this idea yet because of its dependence on exahype::Parser.
* ~~30/12/16: Limiting-ADER-DG: Limiter domain seems to be required to often while the actual limiter domain does not change per solver.~~ Fixed.
* ~~30/12/16: MPI crashes for multiple ADER-DG/Limiting ADER-DG solvers (seg-fault.)~~ Fixed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/98Scalability Tests (MPI,TBB,MPI+TBB)2019-09-20T15:46:43+02:00Ghost UserScalability Tests (MPI,TBB,MPI+TBB)This is my spreadsheet.
## Tests
Testing scalability with parameters
* dim=2,3,
* p=3,5,7,9
* h=???
## Systems
| system | processor | cores per processor | cores per node | memory per node|
|----------|--------------|-...This is my spreadsheet.
## Tests
Testing scalability with parameters
* dim=2,3,
* p=3,5,7,9
* h=???
## Systems
| system | processor | cores per processor | cores per node | memory per node|
|----------|--------------|-------------------------|-------------------|---------------------- |
| phi1 | 2 Intel Xeon E5-2650 2.0 GHz | 8 cores per processor | 16 cores per node | 64 GByte RAM |
### ADER-DG
Additional parameters:
* ts=fused,standard,
Runs;
* **TBB**
* Strong scaling
* [EulerFlow,2D,phi1,Gnu](/uploads/c4c454dc3c41fe7ee0295a07320db9c7/EulerFlow_TBB_Gnu.tar.gz)
* [Z4,3D,phi1,Gnu](/uploads/4bac6f1c44aa732c4c6c06116f540754/Z4_TBB_Gnu.tar.gz)
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
### Godunov FV
Additional parameters:
Comment: Fused time stepping results here in a single sweep scheme.
* ts=fused,standard
Runs;
* **TBB**
* Strong scaling
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
### Limiting ADER-DG (ADER-DG + FV)
Comment: Fused time stepping is not supported yet for the
limiting ADER-DG solver.
Additional parameters:
* ts=standard,fused
Runs;
* **TBB**
* Strong scaling
* [EulerFlow,Explosion(t=0.4),2D,phi1,Gnu](/uploads/27c9c32a34c46c75e7d918578972cc7b/EulerFlow_LimitingADERDG_TBB_Gnu.tar.gz)
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
## Setting up experiments on Hamilton (Durham's HPC)
Here I will document an (more or less) efficient approach to run the above experiments
on Hamilton.
### Installing the latest ExaHyPE and Peano snapshots
* **Installing ExaHyPE**: Hamilton has git version 1.7.1 installed. Unfortunately, I had login issues while I was trying to clone the ExaHyPE-Engine reppository.
I am thus required to perform a two stage process to get ExaHyPE on my login node.
1. Check ExaHyPE out on my laptop.
2. Optional: scp to mira. Login to mira.
3. scp to hamilton.
* **Installing Peano**: Hamilton has svn version 1.6.11 installed.
* Obtaining Peano is very convenient. It is done via:
``svn checkout http://svn.code.sf.net/p/peano/code/trunk peano``
### Building ExaHyPE
...
### Job scripts
We have currently three different job script "frameworks" available in the ExaHyPE-Engine
repository.
While Dominics job scripts are only suited for TBB on Hamilton and SuperMUC,
Tobias' job script suite is suited to perform MPI and MPI+TBB benchmarks on Hamilton (DUR).
Vasco's python based job script framework can basically run anything on SuperMUC.
I will try to merge the three approaches into a python version with different backends based on the
supercomputer.
* **Tobias'** bash job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/hamilton``.
* **Dominics'** bash job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/scaling``.
* **Vasco's** python job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/``.
### Performance analysis scripts
* **Analysing domain decomposition**: ...
* **Plotting speedups**: ...
### Benchmark March 2017
* Dominic's MPI and MPI+TBB job scripts for Hamilton (SLURM) and SuperMUC (LoadLeveler) [Benchmark_Mar2017.tar.gz](/uploads/1966ff2a9b7d28d12037444fca09e213/Benchmark_Mar2017.tar.gz)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/127Revise time stepping notation2019-09-20T15:46:39+02:00Ghost UserRevise time stepping notationI have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little comp...I have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little complicated and the docu just gives an partial overview.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/151Single-node MPI strong scaling differences between 2d and 3d2019-09-20T15:46:36+02:00Ghost UserSingle-node MPI strong scaling differences between 2d and 3dI am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* A...I am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* All plotters are turned off
* To exclude interconnect effects, all experiments are performed on a single-node.
In my experiments, I switch the master-worker communication (M/W)
on or off as well as the neighbour communication (N).
## Only Peano communication (M/W=off,N=off)
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 37.07 | 1.27828 | 316.768 | 10.923 |
3 | ADERDGTimeStep | 29 | 36.25 | 1.25 | 305.752 | 10.5432 |
12 | ADERDGTimeStep | 29 | 27.04 | 0.932414 | 206.142 | 7.10834 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.4012 | 0.841421 |
## M/W=on, N=off
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 |ADERDGTimeStep | 29 | 37.58 | 1.29586 | 316.044 | 10.8981 |
3 | ADERDGTimeStep | 29 | 36.3 | 1.25172 | 306.977 | 10.5854 |
12 | ADERDGTimeStep | 29 | 27.21 | 0.938276 | 207.078 | 7.14064 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.5317 | 0.845921 |
## M/W=off, N=on
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 39.52 | 1.36276 | 337.709 | 11.6451
3 | ADERDGTimeStep | 29 | 99.04 | 3.41517 | 995.378
12 | ADERDGTimeStep | 29 | 121.76 | 4.19862 | 1106.45 | 38.1534
28 | ADERDGTimeStep | 29 | 18.24 | 0.628966 | 105.858 | 3.65027
## M/W=on, N=on
Slightly worse than M/W=off,N=on.
# Insights:
* Rank 3 and 12 performance are load balancing issues.
* For the 28 rank run, the LB only deploys 10 ranks. This is actually a well-balanced setup for 10 ranks. (If we set 10 ranks, we have a load balancing issue again.)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/148LimitingADERDGSolver alsos performs limiter status spreading for user refinement2019-09-20T15:46:35+02:00Ghost UserLimitingADERDGSolver alsos performs limiter status spreading for user refinementPotential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.Potential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/186Turn mapping events off on coarse levels2019-09-20T15:46:32+02:00Ghost UserTurn mapping events off on coarse levels* We basically will go from here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::...* We basically will go from here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
```
To here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
if (level < exahype::solvers::getCoarsestMeshLevelOfAllSolvers()) {
return peano::MappingSpecification(
peano::MappingSpecification::Nop,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
```
* Second, we should set the alterState bool only to true in mappings where this is necessary, i.e.,
where we perform reductions or use temporary variables.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/190Seg Fault in MPI Build if distributed-memory section is missing2019-09-20T15:46:30+02:00Ghost UserSeg Fault in MPI Build if distributed-memory section is missingThis occurs in Runner::initHPCEnvironment(...) and has
nothing to do with other Seg Faults occuring in builds using Alignment.This occurs in Runner::initHPCEnvironment(...) and has
nothing to do with other Seg Faults occuring in builds using Alignment.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/211Multicore-ise LoadBalancing mapping2019-09-20T15:46:28+02:00Ghost UserMulticore-ise LoadBalancing mappingThis affects the concurrency level within the grid setup iterations.This affects the concurrency level within the grid setup iterations.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/201Clean up vertical MPI communication2019-09-20T15:46:26+02:00Ghost UserClean up vertical MPI communicationThere is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.There is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/255Add toolkit option for specifying thread stack size2019-09-20T15:46:25+02:00Ghost UserAdd toolkit option for specifying thread stack sizeChanges affect "shared_memory" section in JSON schema, the C++ Parser, and the Runner's "initSharedMem..."
routine.Changes affect "shared_memory" section in JSON schema, the C++ Parser, and the Runner's "initSharedMem..."
routine.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/248Dynamic AMR Infinite Loop Scenario2019-09-20T15:46:24+02:00Ghost UserDynamic AMR Infinite Loop ScenarioFound a situation where Ancestors are not able to erase their children and the mesh refinement ends up
in an infinite loop.
A few affected cells
--------------------
```
47.8326 info (solverNumber:0,neighbourMergePerform...Found a situation where Ancestors are not able to erase their children and the mesh refinement ends up
in an infinite loop.
A few affected cells
--------------------
```
47.8326 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:41,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.481481],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,0,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8328 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:41,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.222222,0.481481],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,0,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.833 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:41,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.222222,0.518519],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,0,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8332 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:41,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.518519],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,0,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.857 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:36,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.222222,0.407407],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,1,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8572 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:36,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.407407],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,0,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8575 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:36,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.37037],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[1,0,1,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8577 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:36,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.222222,0.37037],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,3,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[0,0,2,0],communicationStatus:1,facewiseRefinementStatus:[-2,-2,-1,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.858 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:36,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.333333],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[2,0,1,0],communicationStatus:1,facewiseRefinementStatus:[-1,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8736 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:9,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.37037,0.222222],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,3,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[2,0,0,0],communicationStatus:1,facewiseRefinementStatus:[-1,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8739 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:9,parentCellLevel:-1,parentOffset:[-1,-1],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.333333,0.259259],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,3,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[1,0,2,0],communicationStatus:1,facewiseRefinementStatus:[-2,-2,-1,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8747 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:62,parentCellLevel:3,parentOffset:[0.222222,0.222222],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.296296,0.296296],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[4,4,4,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[1,0,1,0],communicationStatus:0,facewiseRefinementStatus:[-2,-2,-2,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8748 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:62,parentCellLevel:3,parentOffset:[0.222222,0.222222],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.259259,0.296296],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,3,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[2,0,2,1],communicationStatus:1,facewiseRefinementStatus:[-1,-2,-1,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
47.8752 info (solverNumber:0,neighbourMergePerformed:[0,0,0,0],hasCompletedTimeStep:1,parentIndex:62,parentCellLevel:3,parentOffset:[0.222222,0.222222],hasVirtualChildren:0,type:Ancestor,refinementEvent:ErasingChildrenRequested,level:4,offset:[0.296296,0.259259],size:[0.037037,0.037037],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:0,correctorTimeStepSize:0,correctorTimeStamp:0.140328,predictorTimeStepSize:0,predictorTimeStamp:0.140328,solution:-1,solutionAverages:-1,solutionCompressed:-1,previousSolution:-1,previousSolutionAverages:-1,previousSolutionCompressed:-1,update:-1,updateAverages:-1,updateCompressed:-1,extrapolatedPredictor:-1,extrapolatedPredictorAverages:-1,extrapolatedPredictorCompressed:-1,fluctuation:-1,fluctuationAverages:-1,fluctuationCompressed:-1,solutionMin:-1,solutionMax:-1,facewiseAugmentationStatus:[3,4,3,4],augmentationStatus:4,previousAugmentationStatus:4,facewiseCommunicationStatus:[2,1,2,0],communicationStatus:1,facewiseRefinementStatus:[-1,-2,-1,-2],refinementFlag:0,refinementStatus:-2,previousRefinementStatus:-1,iterationsToCureTroubledCell:0,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:-1,bytesPerDoFInSolution:-1,bytesPerDoFInUpdate:-1,bytesPerDoFInExtrapolatedPredictor:-1,bytesPerDoFInFluctuation:-1)
```
Spec file for reproducing the error
-----------------------------------
```
exahype-project Euler
peano-kernel-path const = ./Peano
exahype-path const = ./ExaHyPE
output-directory const = ./ApplicationExamples/Euler/Euler_ADERDG
architecture const = noarch
computational-domain
dimension const = 2
width = 1.0, 1.0, 1.0
offset = 0.0, 0.0, 0.0
end-time = 1.001
end computational-domain
shared-memory
identifier = dummy
configure = {background-tasks:4}
cores = 8
properties-file = sharedmemory.properties
end shared-memory
distributed-memory
identifier = static_load_balancing
configure = {hotspot,fair,ranks_per_node:4}
buffer-size = 64
timeout = 240
end distributed-memory
global-optimisation
fuse-algorithmic-steps = on
fuse-algorithmic-steps-factor = 0.99
spawn-predictor-as-background-thread = on
spawn-amr-background-threads = off
/* 0.0 und 0.8 sind schon mal zwei Faktoren */
disable-vertex-exchange-in-time-steps = on
time-step-batch-factor = 1.0
disable-metadata-exchange-in-batched-time-steps = on
double-compression = 0.0
spawn-double-compression-as-background-thread = off
end global-optimisation
solver ADER-DG EulerSolver_ADERDG
variables const = rho:1,j:3,E:1
order const = 3
/* 27 points: 0.05, 9 points: 0.15 */
maximum-mesh-size = 0.4
maximum-mesh-depth = 3
halo-cells = 1
time-stepping = globalfixed
type const = nonlinear
terms const = flux
optimisation const = generic,usestack
language const = C
constants = reference:entropywave
plot vtu::Cartesian::vertices::ascii ErrorPlotter
// abserrorl1percell[nvar],q[nvar],qanalytical[nvar]
variables const = 15
time = 0.0
repeat = 1e-2
output = ./errors
end plot
end solver
end exahype-project
```