ExaHyPE-Engine issueshttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues2017-07-13T12:35:21+02:00https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/161AMR+Limiting leads to dispersion errors2017-07-13T12:35:21+02:00Ghost UserAMR+Limiting leads to dispersion errors![amr_limiter_error](/uploads/4f1591cfd8d0db8a2fbc76675deec3a6/amr_limiter_error.png)
* Shock is not at the correct position.
* Problem appears for both, Godunov and MUSCL-Hancock methods.
* Problem affects MUSCL-Hancock more
P...![amr_limiter_error](/uploads/4f1591cfd8d0db8a2fbc76675deec3a6/amr_limiter_error.png)
* Shock is not at the correct position.
* Problem appears for both, Godunov and MUSCL-Hancock methods.
* Problem affects MUSCL-Hancock more
Problem is not solved yet
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/162Combine adapters SolutionUpdate and TimeStepSizeComputation2017-08-16T13:06:55+02:00Ghost UserCombine adapters SolutionUpdate and TimeStepSizeComputation......https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/166Peano Heaps and Intel MPI do not work properly together2017-08-25T11:26:38+02:00Ghost UserPeano Heaps and Intel MPI do not work properly together(Open MPI seems to work.)
Single-node MPI tests show:
* MPI is currently not working correctly.
* TBB seems to work correctly.
It's always fun to debug MPI...
# 1 MPI+TBB
## 1.1
```
Euler_ADERDG-no-output-gen-fused...(Open MPI seems to work.)
Single-node MPI tests show:
* MPI is currently not working correctly.
* TBB seems to work correctly.
It's always fun to debug MPI...
# 1 MPI+TBB
## 1.1
```
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t1-c24.out
8.68453 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
8.68455 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
8.68459 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
9.77685 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
9.77688 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
9.7769 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
10.7874 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000540765
10.7875 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269703
10.7875 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
11.8094 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000810468
11.8094 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269363
11.8095 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
12.8234 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00107983
12.8234 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269194
12.8235 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 5 t_min =0.00134902
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269109
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
14.8835 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 6 t_min =0.00161813
14.8835 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269066
14.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
15.8789 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 7 t_min =0.0018872
15.879 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269045
15.879 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 8 t_min =0.00215624
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269034
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 9 t_min =0.00242528
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269029
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
18.9628 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 10 t_min =0.00269431
18.9629 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269026
18.9629 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
19.9945 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 11 t_min =0.00296333
19.9945 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269025
19.9955 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
21.0106 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00323236
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269024
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 0
```
## 1.2
```
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t24-c1.out
5.84491 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 17 iterations
6.1414 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runAsMaster(...) initialised all data and computed first time step size
8.1216 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runAsMaster(...) plotted initial solution (if specified) and computed first predictor
8.12165 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
8.12167 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
8.12171 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
11.0395 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
13.2524 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
13.2525 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000251415 !!! DIFFERENCE !!!
13.2525 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.445 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
18.9004 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000521798
18.9005 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000242749
18.9005 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
22.8658 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
25.8135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000764547
25.8136 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000235312
25.8136 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
30.957 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.000999859
30.9571 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000235312
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 13 t_min =0.00315762
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000242922
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 6
```
## 1.3
```
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t4-c6.out
16.3129 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
16.3129 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
16.313 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
25.2424 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
31.3638 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
31.3638 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000255681 !!! DIFFERENCE !!!
31.3639 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
40.3852 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000526063
40.3853 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000255681
40.3853 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
49.7392 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
57.1134 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000781745
57.1135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000248662
57.1135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
67.3553 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00103041
67.3554 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000248662
...
...
...
175.134 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00311021
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.00026858
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 2
```
# 2 MPI+None
## 2.1
```
Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t1-c24.out
17.7097 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
17.7097 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
17.7098 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
28.1517 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
28.1518 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
28.1518 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000540765
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269703
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
48.7899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000810468
48.79 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269363
48.79 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
59.1023 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00107983
59.1023 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269194
...
...
...
130.907 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00323236
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269024
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 0
```
## 2.2
```
Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t24-c1.out
14.0067 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.5843 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
19.2036 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
19.2036 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000212189
19.2037 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000482572
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000212189
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
25.2896 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000694761
25.2896 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000223607
25.2897 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
29.067 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.000918368
29.0671 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000229512
...
...
...
109.009 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 13 t_min =0.00303507
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000243652
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 1
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/167Grid cells are not erased properly2017-08-16T20:03:29+02:00Ghost UserGrid cells are not erased properly... only the cell description is deleted. Empty remains in the grid.
Furthermore, we do not erase all cells and cell descriptions at the end of the simulation.
We rely on the OS's process manager to free the application memory.... only the cell description is deleted. Empty remains in the grid.
Furthermore, we do not erase all cells and cell descriptions at the end of the simulation.
We rely on the OS's process manager to free the application memory.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/168Metadata-merging currently not done properly2017-08-14T12:53:04+02:00Ghost UserMetadata-merging currently not done properlyMy latest changes introduced a bug into codes using AMR or LimitingADERDGSolver:
There is only a merge of local metadata performed in the first iteration.My latest changes introduced a bug into codes using AMR or LimitingADERDGSolver:
There is only a merge of local metadata performed in the first iteration.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/170Clear out the private member variables in the solvers2017-11-21T13:40:26+01:00Ghost UserClear out the private member variables in the solversMany of the private member variables in in the ADERDGSolver,FiniteVolumesSolver,
LimitingADERDGSolver can be computed. This includes cardinalities.
Only store the order of approximation, number of variables and number of parameters.
Comp...Many of the private member variables in in the ADERDGSolver,FiniteVolumesSolver,
LimitingADERDGSolver can be computed. This includes cardinalities.
Only store the order of approximation, number of variables and number of parameters.
Compute all other cardinalities; provide getter functions.
- It would be better if the optimised solver would dismiss the "getBnd.." functions and would
overwrite the existing, now virtual, "getUnknowns..." and "getData.." functions.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/171Bug in LimitingADERDGSolver MPI implementation2018-03-20T16:14:23+01:00Ghost UserBug in LimitingADERDGSolver MPI implementationMin and max is not send correctly to neighbour if Heap neighbour comm. is
configured as non-blocking (CreateCopiesOfSentData=false).Min and max is not send correctly to neighbour if Heap neighbour comm. is
configured as non-blocking (CreateCopiesOfSentData=false).https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/174Generic kernels: Select fluxes,ncp similar as for optimised kernels?2017-10-04T16:06:01+02:00Ghost UserGeneric kernels: Select fluxes,ncp similar as for optimised kernels?This issue is open for discussion.
Issue
-----
Working with new ExaHyPE users has revealed that they often are not familar
with the object-oriented programming concept of inheritance.
Especially, they do not know how to overriding vir...This issue is open for discussion.
Issue
-----
Working with new ExaHyPE users has revealed that they often are not familar
with the object-oriented programming concept of inheritance.
Especially, they do not know how to overriding virtual functions
of the AbstractMySolver class.
They are further not familiar with the keywords "virtual" and "override".
This issue makes it difficult for them to select the
right PDE kernels (flux,ncp,...) for their application.
Even worse: The code might even compile and run but it will not perform
the expected calculations.
Such an error is very hard to detect in practice.
Especially for a new ExaHyPE user.
Toolkit-based solution (open for discussion)
--------------------------------------------
JM has moved the selection of the
PDE kernels to the toolkit by requiring the
user to specify the kernels in the following way:
```
kernels const = optimised::fluxes::nonlinear // flux only
```
or
```
kernels const = optimised::fluxes::ncp::nonlinear // flux and ncp
```
or
```
kernels const = optimised::fluxes::ncp::source::nonlinear // flux and ncp, source
```
In my opinion, this is the better approach.
The "const" modifier of "kernels" indicates that the user has to
rerun the toolkit everytime he selects different PDE-kernels
The toolkit will then update the AbstractSolver Header file.
The compiler will deal with any inconsistencies between
the files:
* The compiler will tell you if you have not implemented a
kernel you have specified - not an assertion
as it is the case right now.
(Users often do not even know about the Asserts Mode.)
* The compiler will tell you if you have implemented
a method which is not called. You should then
comment out the implementation or remove it.
What is your opinion?
---------------------
Please comment below.Jean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/176Bug in optimised kernel generation for limiting ader-dg solver2017-09-06T13:58:22+02:00Ghost UserBug in optimised kernel generation for limiting ader-dg solverInclude file is wrong. Is <MySolver>_ADERDG.h for Limiting-ADER-DG solver.
```
/ddn/home/jdmd33/dev/ExaHyPE-Engine/./Benchmarks/hamilton/Euler/kernels/EulerSolver/stableTimeStepSize.cpp(7): catastrophic error: cannot open source file "E...Include file is wrong. Is <MySolver>_ADERDG.h for Limiting-ADER-DG solver.
```
/ddn/home/jdmd33/dev/ExaHyPE-Engine/./Benchmarks/hamilton/Euler/kernels/EulerSolver/stableTimeStepSize.cpp(7): catastrophic error: cannot open source file "EulerSolver.h"
#include "EulerSolver.h"
^
```Jean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/179Split TimeStepSizeComputation and merge parts with SolutionUpdate and LocalRe...2017-10-04T17:11:38+02:00Ghost UserSplit TimeStepSizeComputation and merge parts with SolutionUpdate and LocalRecomputation- TimeStepSizeComputation will be reduced to a simple
time step size computation function.
SolutionUpdate and LocalRecomputation will also
compute a time step size and will further advance in time.
This will reduce logic.
- Next step wi...- TimeStepSizeComputation will be reduced to a simple
time step size computation function.
SolutionUpdate and LocalRecomputation will also
compute a time step size and will further advance in time.
This will reduce logic.
- Next step will be a fusion of multiple algorithmic phases of the
ADER-DG and Limiting ADER-DG schemes in a single solver function.
This will hopefully make it easier to optimise for the compiler
and easier for the processor cache to decide what to hold or drop.
- We might be able to get rid of mapping FusedTimeSteppingInitialisation
with the above split.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/181Get rid of TemporaryVariables2018-03-20T16:13:53+01:00Ghost UserGet rid of TemporaryVariables- We still have to debate if this makes sense.- We still have to debate if this makes sense.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/182Aligned heaps throw segmentation fault upon deleteData invocation2017-10-13T12:16:05+02:00Ghost UserAligned heaps throw segmentation fault upon deleteData invocation- This is especially an issue for adaptive simulations.
- Adaptive simulations are currently also not possible with the optimised kernels
since there are no optimised variants of the prolongation and restriction
kernels available yet...- This is especially an issue for adaptive simulations.
- Adaptive simulations are currently also not possible with the optimised kernels
since there are no optimised variants of the prolongation and restriction
kernels available yet.
TODO
- I will add a test for the heap allocation. Hopefully this will minimise the complexity
of the problem.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/183MPI FV doesnt compile2017-10-02T16:30:46+02:00Ghost UserMPI FV doesnt compileProblem occurs in a line with TODO(Dominic), so @di25cox :
The FiniteVolumeCellDescription does not contain a method getAdjacentToRemoteRank(), so building fails:
```
/home/sven/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/solvers/...Problem occurs in a line with TODO(Dominic), so @di25cox :
The FiniteVolumeCellDescription does not contain a method getAdjacentToRemoteRank(), so building fails:
```
/home/sven/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp: In member function ‘virtual void exahype::solvers::FiniteVolumesSolver::preProcess(int, int) const’:
/home/sven/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp:2152:22: error: ‘exahype::solvers::FiniteVolumesSolver::CellDescription {aka class exahype::records::FiniteVolumesCellDescription}’ has no member named ‘getAdjacentToRemoteRank’
!cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here?
^
/home/sven/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp: In member function ‘virtual void exahype::solvers::FiniteVolumesSolver::postProcess(int, int)’:
/home/sven/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp:2168:24: error: ‘exahype::solvers::FiniteVolumesSolver::CellDescription {aka class exahype::records::FiniteVolumesCellDescription}’ has no member named ‘getAdjacentToRemoteRank’
!cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here?
^
```
This workaround works for me:
```
diff --git a/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp b/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp
index ddba006..949ba38 100644
--- a/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp
+++ b/ExaHyPE/exahype/solvers/FiniteVolumesSolver.cpp
@@ -2149,7 +2149,7 @@ void exahype::solvers::FiniteVolumesSolver::preProcess(
cellDescription.getType()==CellDescription::Type::Cell
#ifdef Parallel
&&
- !cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here?
+ 1 // !cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here? // TODO FIX THIS LINE
#endif
) {
uncompress(cellDescription);
@@ -2165,7 +2165,7 @@ void exahype::solvers::FiniteVolumesSolver::postProcess(
cellDescription.getType()==CellDescription::Type::Cell
#ifdef Parallel
&&
- !cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here?
+ 1 // !cellDescription.getAdjacentToRemoteRank() // TODO(Dominic): What is going on here? // TODO FIX THIS LINE
#endif
&&
CompressionAccuracy>0.0
```
I pushed this to https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/1d435a25b782e5cc30d10b27db30d8b8e6609c7d on the master. Should have used a merge request instead. Is a workaround anyway. Please fix it.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/184Decompose mapping Merging into two mappings2017-11-21T13:39:29+01:00Ghost UserDecompose mapping Merging into two mappingsMapping merging does currently perform neighbour merges (touchVertexFirstTime etc.) as well as the
merging of the time step data from the master with the worker.
Often only a merging of time step data is necessary. The touchVertexFirstTi...Mapping merging does currently perform neighbour merges (touchVertexFirstTime etc.) as well as the
merging of the time step data from the master with the worker.
Often only a merging of time step data is necessary. The touchVertexFirstTime merges
then reduce to a nop. In a shared memory context, this means that we add unnecessary overhead.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/185[Toolkit] Makefile generation with template engine + remove ARCHITECTURE2017-10-06T16:28:14+02:00Jean-Matthieu Gallard[Toolkit] Makefile generation with template engine + remove ARCHITECTURETODO
* Generate the Makefile using the template engine
* Remove the ARCHITECTURE parameter export, use the one from the spec fileTODO
* Generate the Makefile using the template engine
* Remove the ARCHITECTURE parameter export, use the one from the spec fileJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/186Turn mapping events off on coarse levels2019-09-20T15:46:32+02:00Ghost UserTurn mapping events off on coarse levels* We basically will go from here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::...* We basically will go from here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
```
To here:
```
peano::MappingSpecification
exahype::mappings::XYZ::enterCellSpecification(int level) const {
if (level < exahype::solvers::getCoarsestMeshLevelOfAllSolvers()) {
return peano::MappingSpecification(
peano::MappingSpecification::Nop,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
return peano::MappingSpecification(
peano::MappingSpecification::WholeTree,
peano::MappingSpecification::RunConcurrentlyOnFineGrid,true);
}
```
* Second, we should set the alterState bool only to true in mappings where this is necessary, i.e.,
where we perform reductions or use temporary variables.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/187No time averaging and kernel's signature2017-10-24T15:25:13+02:00Jean-Matthieu GallardNo time averaging and kernel's signatureHi,
Small design decision. The no time averaging (NTA) option has the effect that the SpaceTimePredictor and VolumeIntegral don't take the same argument with or without the option.
In the case of the SpaceTimePredictor it's a few data ...Hi,
Small design decision. The no time averaging (NTA) option has the effect that the SpaceTimePredictor and VolumeIntegral don't take the same argument with or without the option.
In the case of the SpaceTimePredictor it's a few data storage that aren't required (the one storing the time averaged data) so it's not a bid deal as I can just pass the pointer that happen to be nullptr if NTA is enabled.
In the case of the VolumeIntegral however it's a bit dirtier: it needs either lFhi (time averaged flux) or lFi. Currently I have chosen to always gives the kernel both storage and let it use the right one, ignoring the other (lFhi might be nullptr, lFi is always correct). Another possibility since both lFi and lFhi are double* would be to use the same signature and just pass the correct one, which I could detect by checking if lFhi is nullptr but it would be a bit unclean because it would mean assuming it has to be nullptr when using NTA (which it currently is, but I don't like having such hidden relationship in the code).
Do you agree with the current design (giving everything to the kernel, possibly nullptr, and letting it do the correct choice based on its template parameters) or do you have another design preference ?
Best,
JM
@svenk @di25cox @gi26detJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/189Bug in Solver mesh level computation if bounding box is virtually expanded2017-10-11T16:47:02+02:00Ghost UserBug in Solver mesh level computation if bounding box is virtually expandedThere is a bug in the mesh level computation if the boundy box is virtually expanded:
```
_coarsestMeshLevel =
exahype::solvers::Solver::computeMeshLevel(_maximumMeshSize,domainSize[0]);
```
Then, ``_domainSize != _boundingBoxSize``.There is a bug in the mesh level computation if the boundy box is virtually expanded:
```
_coarsestMeshLevel =
exahype::solvers::Solver::computeMeshLevel(_maximumMeshSize,domainSize[0]);
```
Then, ``_domainSize != _boundingBoxSize``.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/190Seg Fault in MPI Build if distributed-memory section is missing2019-09-20T15:46:30+02:00Ghost UserSeg Fault in MPI Build if distributed-memory section is missingThis occurs in Runner::initHPCEnvironment(...) and has
nothing to do with other Seg Faults occuring in builds using Alignment.This occurs in Runner::initHPCEnvironment(...) and has
nothing to do with other Seg Faults occuring in builds using Alignment.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/194Erasing of Peano grid vertices2018-03-20T16:13:04+01:00Ghost UserErasing of Peano grid verticesCurrently, I remove only my data from the heap when I erase a cell.
The Peano grid structure still persists.
I tried a variety of tricks to remove the grid structure on the fly during
the mesh refinement but always run into problems:
- ...Currently, I remove only my data from the heap when I erase a cell.
The Peano grid structure still persists.
I tried a variety of tricks to remove the grid structure on the fly during
the mesh refinement but always run into problems:
- Inconsistent adjacency indices
- Spurious erases of neighbours
- Data races with TBB
- ...
I think now it does make more sense to erase Peano vertices after the
mesh refinement. Here the refined grid is fixed. Refinement does not
interfere with erasing.
I think of either:
- Erase Peano vertices on the fly during the time stepping iterations
however I do not know the overhead and impact on the parallelisation.
- Run a few iterations of a dedicated mesh erasing adapter after the
refinement.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/195Additional refinement along MPI boundaries2018-03-20T16:13:38+01:00Ghost UserAdditional refinement along MPI boundariesPeano's spacetree traversal is inverted every second iteration.
This poses a problem for ExaHyPE's master-worker communication
if the master rank has local subtrees.
Example:
--------
Imagine a uniform grid where a fork was performed on...Peano's spacetree traversal is inverted every second iteration.
This poses a problem for ExaHyPE's master-worker communication
if the master rank has local subtrees.
Example:
--------
Imagine a uniform grid where a fork was performed on Level 2. 3^d - 1 workers of our
master rank have been introduced. They and the master rank hold 1/3^d of the computational domain.
We call the portion belonging to the master rank, the local subtree of the master (rank).
In the first iteration, we correctly kick off all workers on the coarse grid before we descend into
the local subtree of the master.
In the second iteration, we start within the local subtree of the master and perform computations.
As soon as we reach the coarse grid, we kick off all workers.
The workers had to wait while we performed our computations on the master's local subtree.
Now finally, the workers will start their computations.
The whole tree traversal might take twice as long as assumed.
This becomes even worse if there are more Master-Worker boundaries.
The red bars in the plot below indicate such scenarios.
Tobias proposes to add additional refinement along the MPI boundary to prevent the need of vertical
communication. At least from the master to the worker. This might work.
I wonder however if it is really beneficial for ExaHyPE to change the traversal order in every second iteration.
In a MPI setting where we perform asynchronous communication, an inversion of the
traversal order is especially ill-suited since we then have to wait till the last message was received
by the neighbour. Instead, we would need to wait and (block) until all messages for the currently
touched vertex are received.
**Update:** The traversal order is hardwired into Peano. It is necessary to run it forward and backward.
![master_worker_synchronisation](/uploads/e1fc83f445841cb3763e0bfd7d8ec701/master_worker_synchronisation.jpg)
Additional Refinement along the MPI boundary
----------------------------------------------
This can be accomplished by continuously refining the top most parent patch (which is of type Cell) of every cell
of type Descendant which is at a Master-Worker boundary.
This has to be done until we end up with a patch of type Cell at the Master-Worker boundary.
At this point, we then need to send the solution values of the Master cell to the worker.
We further might need to impose initial conditions.
We further need to perform status flag merges in prepareSendToWorker.
**Problems with this approach:** It might introduce rippling refinings around
the artificially refined cell.
Introduce a no-operation traversal
----------------------------
We could further introduce a no-operation traversal before we perform
reductions and broadcasts which would rewind Peano's streams
but does not perform any computations and communication.
In this case, we would always follow the top-down traversal.
The observed Master-Worker synchronisation would not appear.
**Problems with this approach:**
- ~~Batching is currently not possible with~~
~~multiple adapters. We could maybe perform no operation in every second iteration.~~
~~However, we would then have still a Master Worker synchronisation. Or would we not?~~
We could have a single empty traversal in front of a batch. That would
work.
- The BoundaryDataExchanger of the heaps always assumes an inversion of the traversal in
every iteration.
To alter this behaviour, we would need to change the receive methods
in Peano's AbstractHeap,DoubleHeap, and BoundaryDataExchanger methods.
We are required to add a bool "assumeForwardTraversal" (defaults to false) to the signature.
We are further required to update ExaHyPE's solver implementations:
Whenever we receive boundary data after we have run the no-operation traversal,
we need to set this new flag to true when calling receiveData.
If we run of batch iteration, this must be done only
in the first iteration of the batch.
Have both
----------------------------
For optimal performance, it might be useful to employ both techniques.
We might use "loop padding", i.e. insert empty traversals, in order to end up with the forward traversal
any time we need to broadcast / reduce something.
In generally it might useful to handle broadcasts and reductions outside of the mapping.
Therefore, we would however need to plug into both, runAsMaster and runAsWorker.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/196Fix musclhancock scheme for certain patch sizes2018-02-22T09:45:39+01:00Ghost UserFix musclhancock scheme for certain patch sizesFor me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applicatio...For me, for certain patch sizes, our 2nd order FV scheme in ExaHyPE (musclhancock) fails. Godunov runs fine. This has to be debugged (it certainly depends on the patch size: Some work, some not).
2nd order is crucial for some applications (GRMHD,CCZ4).
I will do this, ticket just for book keeping.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/159DMP is illegally called for ordinary ADERDG-Solver2017-11-02T18:15:15+01:00Ghost UserDMP is illegally called for ordinary ADERDG-SolverThis are actually two problems:
1. For an ordinary ADERDG-Solver (not Limiter) where the variable `dmp-observables` is not given, the default value is **not** `0` but instead just a random number (NaN or MaX or whatever for int). ...This are actually two problems:
1. For an ordinary ADERDG-Solver (not Limiter) where the variable `dmp-observables` is not given, the default value is **not** `0` but instead just a random number (NaN or MaX or whatever for int). This is very bad but solvable in the Parser for me.
2. The method `mapDiscreteMaximumPrincipleObservables` in the abstract ADERDG solver is called. This should not happen at all! Why does it happen?
![dmp](/uploads/fd5837af72fe6f92e871426712399023/dmp.png)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/158AMR+LimitingADERDGSolver crashes for certain limiter status changes2017-11-02T18:15:15+01:00Ghost UserAMR+LimitingADERDGSolver crashes for certain limiter status changes# 1. Issue:
There is another issue with the MUSCL-Hancock solver which crashes in the min and max
determination after a global recomputation.
```
147.062 info exahype::runners::Runner::updateMeshFusedTimeStepping(......# 1. Issue:
There is another issue with the MUSCL-Hancock solver which crashes in the min and max
determination after a global recomputation.
```
147.062 info exahype::runners::Runner::updateMeshFusedTimeStepping(...) recompute solution locally (if applicable) and compute new time step size
assertion in file /home/dominic/dev/codes/c/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp, line 1111 failed: *(observablesMin+i)<std::numeric_limits<double>::max()
parameter i: 1
parameter solverPatch.toString(): (solverNumber:0,neighbourMergePerformed:[1,1,1,1],isInside:[1,1,0,1],parentIndex:69,isAugmented:0,newlyCreated:0,type:Cell,refinementEvent:None,level:5,offset:[0.518519,0],size:[0.0123457,0.0123457],previousCorrectorTimeStamp:1.79769e+308,previousCorrectorTimeStepSize:1.79769e+308,correctorTimeStepSize:0.000497671,correctorTimeStamp:0.0909671,predictorTimeStepSize:0,predictorTimeStamp:0.0914648,solution:2838,solutionAverages:2843,solutionCompressed:-1,previousSolution:2839,previousSolutionAverages:2841,previousSolutionCompressed:-1,update:2840,updateAverages:2842,updateCompressed:-1,extrapolatedPredictor:2844,extrapolatedPredictorAverages:2846,extrapolatedPredictorCompressed:-1,fluctuation:2845,fluctuationAverages:2847,fluctuationCompressed:-1,solutionMin:2848,solutionMax:2849,facewiseAugmentationStatus:[0,0,0,0],augmentationStatus:0,facewiseHelperStatus:[2,2,2,2],helperStatus:2,facewiseLimiterStatus:[0,0,0,0],limiterStatus:0,previousLimiterStatus:3,iterationsToCureTroubledCell:10,compressionState:Uncompressed,bytesPerDoFInPreviousSolution:305,bytesPerDoFInSolution:-296662368,bytesPerDoFInUpdate:32765,bytesPerDoFInExtrapolatedPredictor:0,bytesPerDoFInFluctuation:0)
ExaHyPE-Euler: /home/dominic/dev/codes/c/ExaHyPE/ExaHyPE-Engine/./ExaHyPE/exahype/solvers/LimitingADERDGSolver.cpp:1111: void exahype::solvers::LimitingADERDGSolver::determineSolverMinAndMax(exahype::solvers::LimitingADERDGSolver::SolverPatch&): Assertion `false' failed.
```
I have to gather more information about this first.
# 2. Issue
Cell of type 3 or 4 (FV->DG) changes to Troubled.
Neighbour is of Type 1 or 2 (DG->FV).
Example:
Before:
![before](/uploads/0a30bada69ece5db04146246ff72c5c1/before.png)
After:
![after](/uploads/fcf89aaaa40453a52ead9622bd283033/after.png)
* Fix 1: Need to stop iterations if situation detected in neighbour merging
-> write stable values, i.e. own values to ghost layer
-> Perform rollback in affected cells (=>irregular
limiter domain chage; requires local recomputation)
* Fix 2: Use less dissipative FV methods or higher order
ADERDG => finer subcell resolution
Or set parameter steps to cure troubled cells to a higher value
# 3. Found bugs:
* The whole global recomputation thing is more sophisticated than previously thought. I
have to be careful how I go back in time. This is based on the previous limiter status.
I am only allowed to delete patches after the recomputation.
# 4. Ways to increase stability
* Do not change Local Recomputation to Global Recomputation if limiter based mesh refinement is also necessary.
# 5. Algorithms:
(Stuff above is outdated. Keep it for reference.)
## Local Recomputation
1. limiter status spreading
2. local reinitialisation
3. local recomputation + local predictor computation
## Global Recomputation
1. limiter status spreading
2. global rollback (keep new limiter status) <-ensures we adjust the previous solution during mesh refinement
3. mesh refinement according to new limiter status
4. overwrite new limiter status with previous values
5. recompute time step size
6. reinitialise fused time stepping and recompute predictor
Problems:
* TimeStepSizeComputation does update the time stamps (solved)
* Need additional adapter for global rollback (global reinitialisation)
* I am not allowed to deallocate limiter patches during limiter status spreading
* Have to keep in mind to overwrite the limiter status in finalise mesh refinement.
## Mesh Refinement
1. mesh refinement according to ref. crit.
4. recompute time step size
5. reinitialise fused time stepping and recompute predictorhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/156LimitingADERDGSolver currently crashing with TBB and AMR2017-11-02T18:15:15+01:00Ghost UserLimitingADERDGSolver currently crashing with TBB and AMR* ~~Uniform grids: LimitingADERDGSolver fails assertion if TBB is switched on.
Probably something with the status spreading with TBB.~~
Seems to be solved now.
* Adaptive grids: LimitingADERDGSolver mesh update iterates foreve...* ~~Uniform grids: LimitingADERDGSolver fails assertion if TBB is switched on.
Probably something with the status spreading with TBB.~~
Seems to be solved now.
* Adaptive grids: LimitingADERDGSolver mesh update iterates forever
* Adaptive grids: LimitingADERDGSolver fails assertion.
Probably something because of the new domain boundary treatment.
(Not sure if this is still an issue.)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/155Bounding Box Scaling (virtually-expand-domain) Does Not Work2017-11-02T18:15:15+01:00Ghost UserBounding Box Scaling (virtually-expand-domain) Does Not WorkVirtually expanding the bounding box around the computational domain
is a Peano trick to shut off neighbour communication with the global master (rank 0).
Virtually expanding the bounding box does currently lead to problems in ExaHyP...Virtually expanding the bounding box around the computational domain
is a Peano trick to shut off neighbour communication with the global master (rank 0).
Virtually expanding the bounding box does currently lead to problems in ExaHyPE:
* It enables scenarios where a coarse grid vertex is on the boundary/outside of the domain,
but a fine grid vertex (and its h environment) is inside of the domain.
* It confuses the isFaceInside function in class Cell.
# Virtually Expanding the Domain
Some background on the virtually expand domain flag:
* Peano does only consider inside and boundary vertices for the (MPI) neighbour
merging. Outside vertices are ignored for this purpose.
* Since rank 1 is placed into a centre of 3^d child cells belonging to rank 0,
it will perform neighbour merging with rank 0 as long as those vertices are
either inside or directly at the boundary of the domain.
* Switching off neighbour merging directly at the domain boundary (``vertex.isBoundary()``) does not
make sense. The reason is that refinement at the boundary will introduce hanging nodes.
Boundary nodes should however be persistent. (Tobias' reasoning. Have to ask further why this is bad.)
* Virtually expanding the domain does place the nodes located at the remote boundary to
rank 0 outside of the domain
* From the above points, it is clear that virtually expanding the boundary is mandatory for reasonable MPI
scalability especially in 3d. This is exactly what we have observed in our 3D MPI experiments.
* This will be a little inconvenient for people who prescribe initial conditions at certain boundaries by means of (x,t).
(Seismic people know about this. Leonhard knows how to deal with it.)
# Remarks
* Virtually expanding the bounding box usually leads to a shrinking of the actual computational domain since
only inside cells are considered as within the computational domain.
ExaHyPE should thus tell the user what the shrinked domain looks like.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/150Buildinfo infrastructure does not update always2017-11-02T18:15:16+01:00Ghost UserBuildinfo infrastructure does not update alwaysThe `buildinfo.h` file is generated at every make call, but the macros are evaluated in `main.cpp` only once, ie. the main file compilation is not retriggered.
To avoid this, we could must have the buildinfo an own compilation context, ...The `buildinfo.h` file is generated at every make call, but the macros are evaluated in `main.cpp` only once, ie. the main file compilation is not retriggered.
To avoid this, we could must have the buildinfo an own compilation context, ie some `buildinfo.cpp` going along with the `buildinfo.h`. That's the only option to get runtime build information.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/149MUSCL-Hancock BC are not imposed correctly?2017-11-02T18:15:16+01:00Ghost UserMUSCL-Hancock BC are not imposed correctly?I observe weird perturbations at the boundary when I use the MUSCL-Hancock FV limiter.
![boundary_effects](/uploads/108aab3f64a1c8d8ac541b212c9d4c39/boundary_effects.png)I observe weird perturbations at the boundary when I use the MUSCL-Hancock FV limiter.
![boundary_effects](/uploads/108aab3f64a1c8d8ac541b212c9d4c39/boundary_effects.png)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/147Dynamic AMR crashes if we switch to gloabl recomputation branch.2017-11-02T18:15:16+01:00Ghost UserDynamic AMR crashes if we switch to gloabl recomputation branch.[webmize](/uploads/db32921836ea432a6ce41060cfb87f97/webmize)Dynamic AMR crashes everytime the solver switches to the global
recomputation branch.
Potential reason:
* ~~We reuse the ADERDGTimeStep adapter which does update the limite...[webmize](/uploads/db32921836ea432a6ce41060cfb87f97/webmize)Dynamic AMR crashes everytime the solver switches to the global
recomputation branch.
Potential reason:
* ~~We reuse the ADERDGTimeStep adapter which does update the limiter
status again. This should not happen.~~
* ~~Erasing is triggered before the limiter status spreading has finished.
Add inertia for erasing requests.~~This is now realised by also taking the previous limiter status into account.
Problems seem to be solved:
![sod_shock_tube](/uploads/2bdedd63eadf6eb0bde5d093e34dbf88/sod_shock_tube.webm)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/198Increase robustness of neighbour merging routines2017-11-21T13:43:49+01:00Ghost UserIncrease robustness of neighbour merging routinesIssue
-----
During the mesh refinement iterations, we have observed that cell description link maps stored in
each vertex sometimes hold wrong but not invalid adjacency indices.
This was observed on a new worker directly after a fork has...Issue
-----
During the mesh refinement iterations, we have observed that cell description link maps stored in
each vertex sometimes hold wrong but not invalid adjacency indices.
This was observed on a new worker directly after a fork has been finished.
Solution
--------
Use geometry information to determine if a patch is really adjacent to
a vertex at a given face.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/200Cell-wise plotter2017-11-29T17:45:59+01:00Ghost UserCell-wise plotterCreate a VTK plotter which only plots a single cell. Plot then:
* LimitingStatus
* Which MPI rank hosts the cell
This also should support VTU (for MPI).Create a VTK plotter which only plots a single cell. Plot then:
* LimitingStatus
* Which MPI rank hosts the cell
This also should support VTU (for MPI).https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/201Clean up vertical MPI communication2019-09-20T15:46:26+02:00Ghost UserClean up vertical MPI communicationThere is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.There is too much data sent around.
I further use too many different structures and methods
(MeshUpdateFlags,SolverFlags,solver->sendDataToWorker/Master...).
This should be cleaned up.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/203Peano: Geometrical comparisons must always use relative tolerance2018-02-02T14:12:26+01:00Ghost UserPeano: Geometrical comparisons must always use relative toleranceThe tarch::la:: routines use usually an absolute tolerance (default: machine precision)
for comparing floating point numbers.
This is problematic for numbers with large absolute values.
Fix: These numbers must be normalised and scaled t...The tarch::la:: routines use usually an absolute tolerance (default: machine precision)
for comparing floating point numbers.
This is problematic for numbers with large absolute values.
Fix: These numbers must be normalised and scaled to range [-1,1] before
comparing with respect to the tolerance.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/204Guidebook should not be written in LaTeX2018-01-26T11:39:44+01:00Ghost UserGuidebook should not be written in LaTeX> Note: This issue is really low-importance. Do not continue reading if you care about your time.
The guidebook is currently written in LaTeX, with a primary focus on printing and much efforts put into aligning figures and styling text...> Note: This issue is really low-importance. Do not continue reading if you care about your time.
The guidebook is currently written in LaTeX, with a primary focus on printing and much efforts put into aligning figures and styling texts. IMHO this is the wrong focus. Instead, the guidebook should *focus on content, not on layout*. It should be written in some simple markup language which allows rendering to a web page and a PDF similarly. This is useful for easier deep-linking and reading on screen. It also encourages a more uniform style accross the book.
There are several instances of such language. One very common in the Python world is [Shinx](http://www.sphinx-doc.org), it is very widespread due to its simple syntax in Markdown. I recently stumbled over the [Visit documentation](http://visit-sphinx-user-manual.readthedocs.io) which was rewritten with Sphinx and renders at the same time to a comprehensive website and a [400 page book/pdf](https://media.readthedocs.org/pdf/visit-sphinx-user-manual/latest/visit-sphinx-user-manual.pdf) with a table of contents, numbered images and all that. An example page is http://visit-sphinx-user-manual.readthedocs.io/en/latest/Quantitative/Expressions.html which is generated by the very readable restructuredtext source http://visit-sphinx-user-manual.readthedocs.io/en/latest/_sources/Quantitative/Expressions.rst.txt
Translating our current LaTeX guidebook to something like Restructuredtext or Markdown is a <1h job, therefore switching the publication system is not a mammoth task at all. It is more something people have to agree on. I wonder if Tobias will?https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/205CCZ4: Run BH with2018-01-20T20:50:14+01:00Ghost UserCCZ4: Run BH withhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/211Multicore-ise LoadBalancing mapping2019-09-20T15:46:28+02:00Ghost UserMulticore-ise LoadBalancing mappingThis affects the concurrency level within the grid setup iterations.This affects the concurrency level within the grid setup iterations.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/215User Defined plotters API: Pass the information from the specfile2019-04-15T12:36:16+02:00Ghost UserUser Defined plotters API: Pass the information from the specfileHow can we access:
* Name of output file
* Full information about cell (Limiting status, etc.)
I think the plotting API is hiding too much information. The UserDefinedADERDG plotter should pass more information to the user.
This is so...How can we access:
* Name of output file
* Full information about cell (Limiting status, etc.)
I think the plotting API is hiding too much information. The UserDefinedADERDG plotter should pass more information to the user.
This is something I can do :)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/216Issue during the compilation2018-04-10T18:36:23+02:00Ghost UserIssue during the compilationHi, I get the following error during the compilation with the last updates:
/WORK/maurizio_exa/ExaHyPE-Engine/ExaHyPE/exahype/solvers/ADERDGSolver.cpp: In member function ‘virtual void exahype::solvers::ADERDGSolver::mergeWithNeighbourD...Hi, I get the following error during the compilation with the last updates:
/WORK/maurizio_exa/ExaHyPE-Engine/ExaHyPE/exahype/solvers/ADERDGSolver.cpp: In member function ‘virtual void exahype::solvers::ADERDGSolver::mergeWithNeighbourData(int, const HeapEntries&, int, int, const tarch::la::Vector<2, int>&, const tarch::la::Vector<2, int>&, const tarch::la::Vector<2, double>&, int)’:
/WORK/maurizio_exa/ExaHyPE-Engine/ExaHyPE/exahype/solvers/ADERDGSolver.cpp:3566:27: error: ‘s’ was not declared in this scope
lFhbnd,dofPerFace,s
^https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/217MPI bug2019-09-20T15:43:08+02:00Ghost UserMPI bugHi, with the last updates I'm not able to run the code using MPI, in parcitular I get the message reported below. This can be reproduced with the GPR application that is in the repository and even if I turn off all the plotters. The seri...Hi, with the last updates I'm not able to run the code using MPI, in parcitular I get the message reported below. This can be reproduced with the GPR application that is in the repository and even if I turn off all the plotters. The serial version seems to work.
```
0.643946 [CERVINO],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
0.643966 [CERVINO],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =6.5714e-05
0.643983 [CERVINO],rank:0 info exahype::runners::Runner::runTimeStepsWithFusedAlgorithmicSteps(...) plot
[CERVINO:07241] *** Process received signal ***
[CERVINO:07241] Signal: Segmentation fault (11)
[CERVINO:07241] Signal code: Address not mapped (1)
[CERVINO:07241] Failing at address: (nil)
[CERVINO:07241] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13150)[0x7f24fae22150]
[CERVINO:07241] [ 1] ./ExaHyPE-GPR(+0x48e949)[0x557f27611949]
[CERVINO:07241] [ 2] ./ExaHyPE-GPR(+0x494e77)[0x557f27617e77]
[CERVINO:07241] [ 3] ./ExaHyPE-GPR(+0x4a7170)[0x557f2762a170]
[CERVINO:07241] [ 4] ./ExaHyPE-GPR(+0x4a741e)[0x557f2762a41e]
[CERVINO:07241] [ 5] ./ExaHyPE-GPR(+0x2bbd7d)[0x557f2743ed7d]
[CERVINO:07241] [ 6] ./ExaHyPE-GPR(+0x2beaa6)[0x557f27441aa6]
[CERVINO:07241] [ 7] ./ExaHyPE-GPR(+0x32f84b)[0x557f274b284b]
[CERVINO:07241] [ 8] ./ExaHyPE-GPR(+0x365f95)[0x557f274e8f95]
[CERVINO:07241] [ 9] ./ExaHyPE-GPR(+0x3a67f6)[0x557f275297f6]
[CERVINO:07241] [10] ./ExaHyPE-GPR(+0x3a76f7)[0x557f2752a6f7]
[CERVINO:07241] [11] ./ExaHyPE-GPR(+0x3a68bd)[0x557f275298bd]
[CERVINO:07241] [12] ./ExaHyPE-GPR(+0x3a76f7)[0x557f2752a6f7]
[CERVINO:07241] [13] ./ExaHyPE-GPR(+0x3a68bd)[0x557f275298bd]
[CERVINO:07241] [14] ./ExaHyPE-GPR(+0x3a76f7)[0x557f2752a6f7]
[CERVINO:07241] [15] ./ExaHyPE-GPR(+0x3ca149)[0x557f2754d149]
[CERVINO:07241] [16] ./ExaHyPE-GPR(+0x3cc690)[0x557f2754f690]
[CERVINO:07241] [17] ./ExaHyPE-GPR(+0x306521)[0x557f27489521]
[CERVINO:07241] [18] ./ExaHyPE-GPR(+0x25419c)[0x557f273d719c]
[CERVINO:07241] [19] ./ExaHyPE-GPR(+0x25474e)[0x557f273d774e]
[CERVINO:07241] [20] ./ExaHyPE-GPR(+0x25671e)[0x557f273d971e]
[CERVINO:07241] [21] ./ExaHyPE-GPR(+0x375b0)[0x557f271ba5b0]
[CERVINO:07241] [22] ./ExaHyPE-GPR(+0x37b09)[0x557f271bab09]
[CERVINO:07241] [23] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f24faa501c1]
[CERVINO:07241] [24] ./ExaHyPE-GPR(+0x45a5a)[0x557f271c8a5a]
[CERVINO:07241] *** End of error message ***
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/221Deeply check the public repository for CCZ42019-03-07T19:41:06+01:00Ghost UserDeeply check the public repository for CCZ4It is at https://github.com/exahype/exahype and we don't want to have the CCZ4 system in the code or any old commits. Make this sure by inspecting the code.
Cannot do it now since the repo is 70MB in size and I'm in a train with a bad w...It is at https://github.com/exahype/exahype and we don't want to have the CCZ4 system in the code or any old commits. Make this sure by inspecting the code.
Cannot do it now since the repo is 70MB in size and I'm in a train with a bad wifi.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/222Fix the M/h computation in the CCZ4/Writers/TimingStatisticsWriter.cpph2019-04-15T12:41:12+02:00Ghost UserFix the M/h computation in the CCZ4/Writers/TimingStatisticsWriter.cpphFrom a mail at Luke:
>
I just noticed there is something wrong in the M/h determination:
>
As explained in my last mail to Tobias and you, I just divide these
two numbers. However, the time in the first column of stdout measures
the tim...From a mail at Luke:
>
I just noticed there is something wrong in the M/h determination:
>
As explained in my last mail to Tobias and you, I just divide these
two numbers. However, the time in the first column of stdout measures
the time since program start.
>
In ExaHyPE, the grid setup sometimes takes a considerable amount --
like 10 minutes. If you measure the M/h straight after the first
timesteps after these 10 minutes, you get of course totally wrong
numbers. However, if you measure after 1000 minutes runtime, the 10
minutes grid setup do not change the result so much.
>
It is not hard to substract the time the grid setup needs in order to
improve the correctness of the number. You can do this either by hand
(just look up when the first time step started) or we can do this in
code (CCZ4/Writers/TimingStatisticsWriter.cpph).https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/223Implement new Space time predictor without (variable number of) picard loops2019-04-15T12:34:30+02:00Ghost UserImplement new Space time predictor without (variable number of) picard loopsThe task is relatively easy: Diff Dumbsers fortran prototype (in repository) and check out the changes in the generic kernels.
From Dumbser, 11. März 2018 um 11:20:
> Wäre natürlich super, wenn Du meinen Fortran Code in C übersetzen kö...The task is relatively easy: Diff Dumbsers fortran prototype (in repository) and check out the changes in the generic kernels.
From Dumbser, 11. März 2018 um 11:20:
> Wäre natürlich super, wenn Du meinen Fortran Code in C übersetzen könntest.
> Wenn es Probleme gibt,
> machen wir wieder eine Skype session.
> Das Format der Schleifen und der nötigen Berechnungen ist quasi dasselbe wie
> für alle anderen Rechnungen
> im space-time predictor, d.h. da kann man sehr viel übernehmen. Nur, dass
> man den Zeitindex nicht mehr
> mitschleppen muss, sondern nur im Raum arbeiten kann. Der einzige Kernel der
> geändert werden muss ist der
> space-time predictor (in 2D und 3D).
>
> Ich würde nur den second und third order initial guess implementieren, siehe
> den Code in
> SpaceTimePredictor.f90 unter
>
> #ifdef SECOND_ORDER_INITIAL_GUESS
>
> #ifdef THIRD_ORDER_INITIAL_GUESS
>
> Ich habe mich diese Woche auf Scaling und den Vergleich Runge-Kutta DG /
> ADER-DG konzentriert,
> d.h. an dem 2D FO-fCCZ4 habe ich nicht mehr weitergearbeitet. Ich will erst
> das GRMHD Paper vom
> Tisch haben.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/224Numerical details how to evolve CCZ4 with FD2019-04-15T12:34:55+02:00Ghost UserNumerical details how to evolve CCZ4 with FDCollection of what was written by Dumbser in various E-Mails in order to proceed in the Runge Kutta - Finte Differencing code (Cactus/Antelope/Okapi):
### Dumbser, 3. April 2018 um 11:03: FO-CCZ4 with Finite differencing works
> since S...Collection of what was written by Dumbser in various E-Mails in order to proceed in the Runge Kutta - Finte Differencing code (Cactus/Antelope/Okapi):
### Dumbser, 3. April 2018 um 11:03: FO-CCZ4 with Finite differencing works
> since Sven has reported some difficulties with the implementation of FO-CCZ4 in the Einstein toolkit last week, and since I wanted to understand the potential problems in depth, I have simply written my own finite difference code for the Einstein equations, based on central finite differences in space and Runge-Kutta time integration. According to Sven's and Elias' description, this is exactly what you are also doing in Cactus, right? The implementation is straightforward, since FD schemes are extremely simple.
>
>
>
> To do my tests, I have just copy-pasted my Fortran subroutine PDEFusedSrcNCP into the finite difference code, and I then insert FD point values of Q and central FD approximations for the first spatial derivatives. To save CPU time, I have done all computations in 1D so far.
>
>
>
> Please find attached the results that I have obtained for the Gauge wave with amplitude A=0.01 and A=0.1 until a final time of t=1000. I have used sixth order central FD in space and a classical third order Runge-Kutta scheme in time. For FO-CCZ4, I have set all damping coefficients to zero (kappa1=kappa2=kappa3=0), and I use c=0 with e=2. Zero shift (sk=0) and harmonic lapse. CFL number based on e is set to CFL=0.5 at the moment. Now the important points:
>
>
>
> 1. Runge-Kutta O3 is for now preferrable over Runge-Kutta O4, since it is intrinsically dissipative. The reason is that the fourth order time derivative term in the Taylor series on the left hand side remains with RK3, while it cancels with RK4, and when moving the term q_tttt to the right hand side and after Cauchy-Kowalevsky procedure, it becomes a fourth order spatial derivative term with negative sign, which is good for stability (second spatial derivatives must have positive sign on the right hand side, fourth spatial derivatives must have negative sign for stability. this is easy to check via Fourier series and the dispersion relation of the PDE).
>
>
>
> 2. The RK3 alone is not enough to stabilize the scheme for the larger amplitude A=0.1 of the Gauge Wave, but it is sufficient for A=0.01. I therefore explicitly needed to subtract a term of the type - dt/dx*u_xxxx, which is essentially a fourth order Kreiss-Oliger-type dissipation with appropriately chosen viscosity coefficient.
>
>
>
> I will now replace the Kreiss-Oliger dissipation which I do not like with the numerical dissipation that you would have obtained with a Rusanov flux in a fourth order accurate finite volume scheme. In the end, the dissipation operator will again be written as a finite difference, but there I know at least exactly what is going on and we will exactly know the amount of dissipation to be put (it can only be a function of the largest eigenvalue). So there will be NO parameter to be tuned. I will keep you updated on this.
>
>
>
> From my own results I can conclude that everything is working as expected, i.e. you must have at least one bug in your implementation of FO-CCZ4 in Cactus; or you have run the tests with the wrong parameters (please use CFL=0.5 for the moment, Kreiss-Oliger dissipation with a viscosity factor so that you get -dt/dx*u_xxxx on the right hand side in the end, please set kappa1=kappa2=kappa3=0 and set e=2 and c=0 in FO-CCZ4). If your code still does not run, I can send you my finite difference Fortran code to help you with the debugging.
>
>
>
> While running and implementing my FD code for FO-CCZ4 I have also been working on the vectorization of FO-CCZ4 in ADER-DG. The interesting news: the finite difference code requires more than 20 microseconds per FD point update, and ADER-DG with the good new initial guess for the space-time polynomial and proper vectorization needs also about 20 microseconds per DOF update for FO-CCZ4, i.e. ADER-DG is indeed becoming competitive with FD, who would have every believed this last year :-) On the new 512bit vector machines of RSC in Moscow, we expect the PDE to run even twice as fast, since the vector registers are twice as large as the current state of the art. We are aiming at a time per DOF update of about 10 microseconds. I will keep you informed.
### Dumbser, 4. April 2018 um 10:51:
>
However, my latest experiments show that you can also use RK4 together with a finite-volume type dissipative operator, which is very simple to
implement and which does not require any parameters to be tuned. It will just replace the Kreiss-Oliger dissipation. And by the way: in this setting,
the scheme can be run with CFL=0.9, which is what we want. I will send around more details later.
### Dumbser, 4. April 2018 um 18:23
> there are again good news from the finite difference for FO-CCZ4 front. Instead of your classical Kreiss-Oliger dissipation, I suggest to use the following dissipation operator, which should simply be "added" to the time derivatives of
>
> all quantities on the right hand side, i.e.:
>
>
> ```
> dudt(:,i,j,k) = dudt(:,i,j,k) - 1.0/dx(1)* 3.0/256.0* smax * ( -15.0*u(:,i+1,j,k)-u(:,i+3,j,k)-15.0*u(:,i-1,j,k)+6.0*u(:,i+2,j,k)+20.0*u(:,i,j,k)+6.0*u(:,i-2,j,k)-u(:,i-3,j,k) )
> ```
>
>
> where dudt(:,i,j,k) is the time derivative of the discrete solution computed by the existing Fortran function PDEFusedSrcNCP and smax is the maximum eigenvalue in absolute value. This operator derives from
>
>
> ```
> - 1/dx(1)*( fp - fm ),
> ```
>
>
> where the dissipative flux fp is defined as
>
>
> ```
> fp = - 1/2 * smax * ( uR - uL ),
> ```
>
>
> and uR and uL are the central high order polynomial reconstructions of u evaluated at the cell interface x_i+1/2. The flux fm is the same, but on the left interface x_i-1/2.
>
>
>
> Please find attached the new results for the Gauge Wave with A=0.1 amplitude. Everything looks fine, i.e., the ADM constraints as well as the waveform at the final time. Note that this simulation was now run with the fourth order
> Runge-Kutta scheme in time and using a CFL number of CFL=0.9 based on the maximum eigenvalue.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/225Provide vectorized user functions in the optimized kernels2019-03-07T19:41:41+01:00Ghost UserProvide vectorized user functions in the optimized kernelsThis is something Jean Matthieu should do.
Then we can immediately test a couple of PDEs, such as Euler, GRMHD or CCZ4. Dumbser shared his vectorized code also somewhere.This is something Jean Matthieu should do.
Then we can immediately test a couple of PDEs, such as Euler, GRMHD or CCZ4. Dumbser shared his vectorized code also somewhere.Jean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/227Reflection for the parameter system2019-03-07T20:43:51+01:00Ghost UserReflection for the parameter systemCurrently, using runtime parameters requires to parse them somewhere, leading to code like
```c++
void GeometricBallLimiting::readParameters(const mexa::mexafile& para) {
radius = para["radius"].as_double();
std::string where = para...Currently, using runtime parameters requires to parse them somewhere, leading to code like
```c++
void GeometricBallLimiting::readParameters(const mexa::mexafile& para) {
radius = para["radius"].as_double();
std::string where = para["where"].as_string();
toLower(where);
if(where == "inside") limit_inside = true;
else if(where == "outside") limit_inside = false;
else {
logError("readParameters()", "Valid values for where are 'inside' and 'outside'. Keeping default.");
}
logInfo("readParameters()", "Limiting " << (limit_inside ? "within" : "outside of") << " a ball with radius=" << radius);
}
```
associated to a structure where the parameters are stored,
```c++
struct GeometricBallLimiting : public LimitingCriterionCode {
double radius; ///< Radius of the ball (default -1 = no limiting)
bool limit_inside; ///< Whether to limit inside or outside (default inside)
GeometricBallLimiting() : radius(-1), limit_inside(true) {}
bool isPhysicallyAdmissible(IS_PHYSICALLY_ADMISSIBLE_SIGNATURE) const override;
void readParameters(const mexa::mexafile& parameters) override;
};
```
This is lot's of overhead and really redundant.
In Cactus, the user can declare parameters *including their description/meaning and valid values* in a nice language, they are then made available as a structure by the glue code, all the parsing is abstracted away. Example of a Cactus parameter file (CCL file):
```
real eta "Damping coefficient for the Gamma Driver" STEERABLE=always
{
0:* :: "should be 1-2/M"
}0.2
KEYWORD evol_type "Which set of equations to evolve"
{
"BSSN" :: "traditional BSSN"
"Z4c" :: "Z4c"
"CCZ4" :: "(Covariant) and conformal Z4"
"FOCCZ4" :: "First order formulation of the CCZ4"
}"Z4c"
boolean include_theta_source "Only FO-CCZ4: set to false to remove the algebraic source terms of the type -2*Theta" STEERABLE=always
{
} yes
```
In the code, one then just has something like
```c++
struct parameters {
double eta;
std::string evol_type;
boolean include_theta_source;
}
```
which is already filled nicely with values.
While at least I certainly don't want to code such a glue code for ExaHyPE, in contrast with minimal effort we can get much better then the current MEXA system. In fact, it would be nice to use *OOP reflection* to automatically register class attributes for common data types (int/double/bool/string). That means I would be fine with writing
```c++
struct GeometricBallLimiting {
double radius;
enum class limit_at { inside, outside };
REGISTER_PARAM(radius, DEFAULT(-1), "Radius of the ball");
REGISTER_PARAM(limit_at, DEFAULT(limit_at::inside), "Where to limit");
}
```
In fact, something like this is possible some macro magic.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/157Plotter variables are not constants2018-09-10T19:41:33+02:00Ghost UserPlotter variables are not constantsActually the ExaHyPE runtime treats the number of writtenUnknowns per plotter as a runtime variable, ie.
```
plot hdf5::flash ConservedWriter
variables = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/c...Actually the ExaHyPE runtime treats the number of writtenUnknowns per plotter as a runtime variable, ie.
```
plot hdf5::flash ConservedWriter
variables = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/conserved
end plot
```
one can change `variables = x` without recompiling. However, the toolkit wants this to be a constant:
```
plot hdf5::flash ConservedWriter
variables const = 19
time = 0.0
repeat = 0.00001
output = ./hdf5-flash/conserved
end plot
```
otherwise it says `ERROR: eu.exahype.parser.ParserException: [71,17] expecting: 'const'`
This should not happen, ie. the toolkit grammar should accept without `const`.
As always, there is assumably no case when somebody wants to do this **except benchmarking plotter file formats which is exactly what I'm doing now** and the typical generated code by the toolkit is not aware of a non-constexpr number of variables, but all the `ExaHyPE/exahype/plotters/` code actually treats the number as runtime constant and I see no reason why to artificially introduce something constexpr here.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/153Constants in specfiles2018-09-10T19:42:06+02:00Ghost UserConstants in specfilesI have too much constants that a specfile syntax
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size...I have too much constants that a specfile syntax
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = initial_data:tovstar,boundary_x_lower:reflective,boundary_y_lower:reflective,boundary_z_upper:outgoing,tovstar-mass:1.234,tovstar-rl-ratio:2.345
```
would make sense. Tobias thinks in such a case a user would do something like
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = configuration-file:foobar.txt
```
and outsource the configuration in a language the user likes. However, this breaks with the idea of a single specfile for a single run. Therefore, I see two options
## Add a DATA section after the specfile
This is what many script languages do, for instance perl ([the DATA syntax in perl](https://stackoverflow.com/questions/13463509/the-data-syntax-in-perl)). The idea is just that the parsers ignore what goes after the end of the specfile (ie. the line containing `end exahype-project`). Users could dump there any content in their favourite language. I would vote for this as it is super easy to implement, allows file concatenation and flexibility.
## Allow user constant section
We could also just allow users to do something like
```
solver Limiting-ADER-DG GRMHDSolver
variables const = rho:1,vel:3,E:1,B:3,psi:1,lapse:1,shift:3,gij:6
order const = 4
maximum-mesh-size = 6.0009
maximum-mesh-depth = 0
time-stepping = global
kernel const = generic::fluxes::nonlinear
language const = C
constants = parameters:appended
limiter-kernel const = generic::musclhancock
limiter-language const = C
dmp-observables = 2
dmp-relaxation-parameter = 1e-2
dmp-difference-scaling = 1e-3
steps-till-cured = 0
simulation-parameters
foo = bar
baz = bar
blo = bar
blu = bar
etc.
end simulation-parameters
plot vtk::Cartesian::vertices::limited::binary ConservedWriter
variables const = 19
time = 0.0
repeat = 0.00166667
output = ./vtk-output/conserved
end plot
...
```
This would go well with the specfile syntax. In order to implement, we need
* Such a section with any key-value pairs added to the grammar, so the toolkit does not complain
* Support in the `Parser.cpp` (which is not too hard to add)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/152Dynamic plotter registration2019-03-07T19:39:00+01:00Ghost UserDynamic plotter registrationUsers shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plot...Users shall be able to add, comment out or remove plotters at runtime without recompiling. The plotter ordering should not be fixed.
This is not too hard to obtain, just requires changes at `KernelCalls.cpp` with having a generated plotter registration function (semi pseudocode)
```c++
Writer* kernels::getNamedWriter(std::string name, Solver& solver) {
if(name == "ConservedWriter") return new GRMHD::ConservedWriter(*static_cast<exahype::solvers::LimitingADERDGSolver*>(solver));
if(name == "usw") return new GRMHD::SomeOtherWriter(*static_cast<exahype::solvers::ADERDGSolver*>(solver));
if(name == ...
else
failure: Dont now this plotter type.
}
```
We then can replace the generated current section
```c++
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,0,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,1,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,2,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,3,parser,new GRMHD::ConservedWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
exahype::plotters::RegisteredPlotters.push_back( new exahype::plotters::Plotter(0,4,parser,new GRMHD::IntegralsWriter( *static_cast<exahype::solvers::LimitingADERDGSolver*>(exahype::solvers::RegisteredSolvers[0])) ));
```
with a non-generated section (pseudocode)
```c+++
for(const int& solvernum : Parser->getSolvers()) {
int plotternum=0;
for( const string& plottername : Parser->getPlotterNamesForSolver(solvernum)) {
exahype::plotters::RegisteredPlotters.push_back(new exahype::plotters::Plotter(solvernum,plotternum++,parser,getNamedWriter(plottername, exahype::solvers::RegisteredSolvers[solvernum]));
}
}
```
This is something I can do on my own. If the Parser gives the neccessary data...https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/92Multi-solvers / Parameter Studies / Sensitivity Analysis2019-09-20T15:46:45+02:00Ghost UserMulti-solvers / Parameter Studies / Sensitivity Analysis### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG so...### Progress:
* Multi-solver infrastructure is implemented for all solvers (ADER-DG,Godunov FV,Limiting ADER-DG).
Further tests and more debugging are necessary for MPI and TBB - especially with respect to the limiting ADER-DG solver.
### Issues/Features
* 30/12/16: Parameter studies require using the same solver with different initial conditions and/or source terms.
The problem and discretisation (order,variables,plotters,...) of the solver does not change in these studies.
I plan to enable such studies with an annotation {<studyNumber>}, e.g., "solver SolverType MySolver{10}". that is interpreted by the Toolkit
which then creates a constructor that passes the number of the study (0 to 9 in the above example), and
further adds the required number of solvers to the registry.
The current Plotter infrastructure does not support this idea yet because of its dependence on exahype::Parser.
* ~~30/12/16: Limiting-ADER-DG: Limiter domain seems to be required to often while the actual limiter domain does not change per solver.~~ Fixed.
* ~~30/12/16: MPI crashes for multiple ADER-DG/Limiting ADER-DG solvers (seg-fault.)~~ Fixed.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/151Single-node MPI strong scaling differences between 2d and 3d2019-09-20T15:46:36+02:00Ghost UserSingle-node MPI strong scaling differences between 2d and 3dI am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* A...I am investigating the strong scaling behaviour of the 2d and 3d versions of ExaHyPE.
While the 2d version shows reasonable scalability, the 3d version does not.
* Experiments are performed on a single-node of SuperMUC Phase 2.
* All plotters are turned off
* To exclude interconnect effects, all experiments are performed on a single-node.
In my experiments, I switch the master-worker communication (M/W)
on or off as well as the neighbour communication (N).
## Only Peano communication (M/W=off,N=off)
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 37.07 | 1.27828 | 316.768 | 10.923 |
3 | ADERDGTimeStep | 29 | 36.25 | 1.25 | 305.752 | 10.5432 |
12 | ADERDGTimeStep | 29 | 27.04 | 0.932414 | 206.142 | 7.10834 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.4012 | 0.841421 |
## M/W=on, N=off
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 |ADERDGTimeStep | 29 | 37.58 | 1.29586 | 316.044 | 10.8981 |
3 | ADERDGTimeStep | 29 | 36.3 | 1.25172 | 306.977 | 10.5854 |
12 | ADERDGTimeStep | 29 | 27.21 | 0.938276 | 207.078 | 7.14064 |
28 | ADERDGTimeStep | 29 | 10.27 | 0.354138 | 24.5317 | 0.845921 |
## M/W=off, N=on
ranks | adapter name | iterations | total CPU time [t]=s | average CPU time [t]=s | total user time [t]=s | average user time [t]=s |
----------|-----------------------|-------------------|-----------------------------------|---------------------------------------|----------------------------------|---------------------------------------|
2 | ADERDGTimeStep | 29 | 39.52 | 1.36276 | 337.709 | 11.6451
3 | ADERDGTimeStep | 29 | 99.04 | 3.41517 | 995.378
12 | ADERDGTimeStep | 29 | 121.76 | 4.19862 | 1106.45 | 38.1534
28 | ADERDGTimeStep | 29 | 18.24 | 0.628966 | 105.858 | 3.65027
## M/W=on, N=on
Slightly worse than M/W=off,N=on.
# Insights:
* Rank 3 and 12 performance are load balancing issues.
* For the 28 rank run, the LB only deploys 10 ranks. This is actually a well-balanced setup for 10 ranks. (If we set 10 ranks, we have a load balancing issue again.)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/69Do memory precision tests2019-03-07T19:23:10+01:00Ghost UserDo memory precision testsTobias schreibt:
>
ich braeuchte mal Deine Hilfe (ja, natuerlich schnell, weil das in die Praesentation rein soll, aber das haste eh vermutet ;-). Die neue ExaHyPE-Version von gestern Nacht kann mit Zahldarstellungen <IEEE double Standa...Tobias schreibt:
>
ich braeuchte mal Deine Hilfe (ja, natuerlich schnell, weil das in die Praesentation rein soll, aber das haste eh vermutet ;-). Die neue ExaHyPE-Version von gestern Nacht kann mit Zahldarstellungen <IEEE double Standard arbeiten. Das ist teuer (erste Schritte suggerieren einen Faktor 5 in der Laufzeit, aber das koennte auch am Chip liegen, d.h. ich muss die Russenmaschine testen), aber es reduziert halt den Speicherbedarf einer Simulation etwas. Ich bekomm 10% schon fuer den einfachen Euler 2d mit Ordnung 3 - hoffe aber, dass das bei anderen Setups deutlich signifikanter ist. Einschalten laesst sich das Feature, indem man
>
double-compression = 0.000001
>
auf etwas zwischen 0 und 1 setzt. Das andere Flag spawn-double-compression-as-background-thread kannste vergessen, das ist meine Baustelle. Meine Frage/Bitte ist nun: Hast Du einen sinnvollen Benchmark zur Hand aus der Astrophysik und kannst Du mal
double-compression = 0.0
double-compression = 0.0001
double-compression = 0.000001
double-compression = 0.000000000001
>
damit ausprobieren und mir sagen, ob Werte grosser 0 die Loesung qualitativ verschlechtern? Ich hab keine Ahnung, was Ihr Euch typischerweise anseht (Ankunftszeiten/Amplituden/...?) deswegen brauch ich da ein qualifiziertes Auge. Wenn Du parallel noch drauf schaust, was er als memoryUsage ausgibt (macht er jetzt automatisch, wenn man nicht mit MPI uebersetzt), dann wuerde mir das sehr helfen, da eine Einschaetzung zu bekommen.
>
Nachtrag: So saehe dann eine Optimisation Sectino aus
```
optimisation
fuse-algorithmic-steps = on
fuse-algorithmic-steps-factor = 0.99
timestep-batch-factor = 0.0
skip-reduction-in-batched-time-steps = on
disable-amr-if-grid-has-been-stationary-in-previous-iteration = off
double-compression = 0.000001
spawn-double-compression-as-background-thread = off
end optimisation
```
@svenk: Machen.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/58Reduce memory footprint2019-09-20T15:46:51+02:00Ghost UserReduce memory footprint# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish b...# Open issues
* We have consecutive heap indices for volume data and face data. We thus need to store one index for each
and can get the others by incrementing the index up to the bound we know of as developers.
We distinguish between cell and face data since we have helper cells that do not allocate cell data but face data.
* We can remove prediction and volumeFlux fields completely if we perform also the time integration
in the volume integral and the boundary extrapolation routines.
This would further make it easier to switch between global and local time stepping.
Here, we would just load a different kernel for the boundary extrapolation and allocate space-time face data
if the user switches local time stepping on.
* Allocate all the temporary arrays like the rhs, lQi_old, etc. only once per Thread and not dynamically
during the kernel calls. **(This could be done easily now in each solver!)**
* Create one "big" ADERDGTimeStep function in kernels/solver. This might help the compiler/is more Cache-friendly.
# Done
I think the following has been Tobias' idea originally;
It is not necessary to store temporary data on the heap for every cell description.
We need to analyse which ADER-DG fields are temporary and which
need to be stored persistently on the heap.
From my point of view, the following variables are temporary:
* spaceTimePredictor
* predictor
* spaceTimeVolumeFlux (includes sources)
* volumeFlux (includes sources)
The spacetime fields have a massive memory footprint.
They scale with (N+1)^{d+1} and d*(N+1)^{d+1}.
I thus propose that we assign each thread its own spaceTimePredictor spaceTimeVolumeFlux, predictor, and volume flux fields
and remove the fields from the heap cell descriptions.
This would reduce the memory footprint of the ADER-DG method dramatically (and might further lead to more cache-friendly code ?).
In a second step, we should kick out the volumeFlux field completely, don't do the time integration of the spaceTimeVolumeFlux,
and directly perform the volume integral with the spaceTimeVolumeFlux.
Implementation details
* Allocate arrays in Prediction mapping for each threadhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/148LimitingADERDGSolver alsos performs limiter status spreading for user refinement2019-09-20T15:46:35+02:00Ghost UserLimitingADERDGSolver alsos performs limiter status spreading for user refinementPotential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.Potential optimisation:
* LimitingADERDGSolver alsos performs limiter status spreading for user refinement.
This can potentially be turned off. Requires an enum return type instead of a bool.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/57Finite volumes solver / Limiter2019-09-20T15:46:56+02:00Ghost UserFinite volumes solver / Limiter# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to g...# Treating NaNs in the update vector
If NaNs occur in the update vector, a rollback
using the update vector is not possible anymore.
We thus need to introduce a "previousSolution" field
to the ADER-DG solution.
We might be able to get rid of the "update" field if we directly
add a weighted (by dt and quad weight) update to the "solution" field
from the volume integral and surface integral evaluation.
# Status of the implementation of the limiting ADER-DG scheme (unordered notes)
* The limiter workflow works now for uniform meshes with Intel's TBBs.
[sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/b71d24c7b82f805c604f31c1b322fcae/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi)
* Found and fixed some more bugs. Now everything looks fine. The limiter has a local characteristic and only
requires a rollback/reallocation of memory in every fifth time step (49 out of 255) in the experiment
shown [sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi](/uploads/1e065c7c5c2a52aefb551ef0a03f0ac5/sod-shock-tube_limiting-aderdg-P_3_Godunov-N_7_dmp-only.avi), where we ran an experiment with P=3 order ADER-DG approx, \delta_0=1e-2, \epsilon=1e-3.
As long as the limiter domain does not change, we do not need to reallocate new memory and do a
recomputation of certain cells in our novel implementation.
* Find another simulation using a 9-th order ADER-DG approximation and parameters \delta_0=10^-4 , \epsilon=10^-3 here:
[sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi](/uploads/fb466322f7df5339790aac4afca9ce6c/sod-shock-tube_limiting-aderdg-P_9_Godunov-N_19_dmp_pad.avi)
~~We observee a strange x velocity in this benchmark. Can't be seen in the video. The both profile of the Shock-Tube differs signifantly
in vicinity of the contact discontinuity.~~This is actually the momentum density. Everything is fine.
* Explosion problem with P=5 and delta_0=1e-4 and epsilon=1e-3:
[explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi](/uploads/4ace50cdcbfcbc1706432d0cc180fcea/explosion_limiting-aderdg-P_5_Godunov-N_11_dmp_pad.avi). Here the limiter domain must be adjusted after most of the time steps
* I performed some further optimisations of the DMP evaluation. I found that we can easily perform a
"loop" fusion of the min and max computation and the DMP.
* Implemented the physical admissibility detection (PAD) now as well and use it to
detect non-physical oscillations in the initial conditions. Here, I found that we can directly
pass it the solution min and max we computed as part of the DMP calculation. No need
to loop over the whole degrees of freedom and the Lobatto nodes again.
* ~~There is still an issue with the discrete maximum principle which detects too many
cells to be troubled.~~ Was resolved by tuning the DMP parameters.
* AMR and Limiting need to be combined. This is in principle "a simple" wiring of FV patches
along the ADER-DG tree. We further need spatial averaging and interpolation operators.
Then, we can follow the AMR timestepping implementation of the ADER-DG scheme.
# Writing a Godunov type first order method
Due to the issues I encountered while [rewriting](#issueswithrewritingthefinitevolumessolver) the
pseudo-TVD donor cell type finite volumes solver, Tobias and me decided
to write a simple first order Godunov type finite volumes solver that only exchanges volume averages/fluxes
between direct (face) neighbours.
Replacement of this simple method by a more complex one will be tackled in later stages of the project -- if necessary.
# Issues with higher order finite volumes solvers
* Higher-order methods have a reconstruction stencil which is larger than the
Riemann solver stencil (simple star).
* Reconstruction and time evolution of boundary extrapolated values at one face of a patch can only be performed
after all volume averages from the direct neighbour and corner neighbour patches are available.
I have identified the following phases of the FVM solver now:
* Gather: Just get the neighbour values (arithmetic intensity = 0). This will move into the Merging mapping.
* Spatial reconstruction: Compute spatial slopes in the cells or something similar. This will move into the SolutionUpdate mapping.
* Temporal evolution of boundary-extrapolated values: This will move into the SolutionUpdate mapping.
* Riemann solve: This will move into the SolutionUpdate mapping.
* Solution update; This will move into SolutionUpdate mapping.
* Extrapolation of volume averages: This will send layers of the volume averages to the neighbours.
We send layers instead of a single layer in order to only have one data exchange instead of two.
With the above strategy, we can merge FVM solvers into the current framework. The difference to the ADER-DG solver
is that the computational intensity of the neighbour merging operation is zero.
We will further need to consider neighbour data exchange over edges (3-d) and corners in our
merging scheme. Neighbour data exchange over edges in 3-d will require additional falgs.
Exchange over corners does not require additional flags.
# Issues with rewriting the finite volumes solver
I tried to decompose our pseudo-TVD donor cell type finite volumes
solver into a Riemann solve and a solution update part,
and ran into the following issues.
## Open issues with the donor cell type pseudo-TVD finite volumes solver implementation:
* The current solver uses updated solution values in the solution update.
We have to distinguish between old values and new values or
have to introduce an update.
* We need to consider two layers of the neighbour to compute all extrapolated boundary
values (wLx,wRx,wLy,wRy). Currently only one layer ist considered. This means
that the boundary extrapolated values and thus the face fluxes at the outermost faces
are computed wrongly.
* To tackle the above problem, we either need to exchange a single cell of
the diagonal neighbours or we need to rely on two data exchanges.
Tobias told me there exists however a trick to circumvent this:
Reconstructing the diagonal neighbours' contributions with values from
the direct neighbours.
## Limitations of the donor cell type pseudo-TVD finite volumes solver:
* The solver is not TVD in multiple dimensions.
* We do neither perform corner correction nor (dimensional) operator splitting. We should
thus observe loss of mass conservation, large dispersion errors, and a reduced CFL stability limit.
The reduced CFL stability limit was taken into account in our implementation.
This is also a problem of the multi-dim. Godunov method in the unsplit form.
## Follow up: Low-order Euler-DG patch with direct coupling to ADER-DG method?
## Dissemination
For the following benchmarks I always used copy boundary conditions which are equal to
outflow/inflow boundary conditions as long as everything flows out of the domain.
* Euler (Compressible Hydrodynamics)
* Explosion Problem with 27^2 grid, P=3, and N=2*3+1 FV subgrid:
![lim-aderdg_explosion](/uploads/6459f9b4a775a482cdfec24e129c56ae/lim-aderdg_explosion.png)
[lim-aderdg_explosion.avi](/uploads/6a38fa00a6bf41fe3aefbee380a03396/lim-aderdg_explosion.avi)
* Sod Shock Tube with 27^2 grid, P=3 and N=2*3+1 FV subgrid:
![lim-aderdg_euler_sod-shock-tube](/uploads/eb2a5e57ca8f37665c2dffe48dbf1e22/lim-aderdg_euler_sod-shock-tube.png)
[lim-aderdg_euler_sod-shock-tube.avi](/uploads/61dfedf7db4ead996d7fd6b9796ba5c0/lim-aderdg_euler_sod-shock-tube.avi)
![hires_antialias](/uploads/26843a0f50e52ded919cfc12b777360b/draft2_hires_antialias.png)
* SRMHD (Special Relativistic Magnetohydrodynamics; basically: Special Relativistic Euler+Maxwell)
* Blast Wave setup as in 10.1016/j.cpc.2014.03.018 with 27^2 grid, P=5, and N=2*5+1 FV subgrid:
![lim-aderg_mhd__blast-wave](/uploads/feb026167c7c2d882c132f3b775c37ac/lim-aderg_mhd__blast-wave.png)
[lim-aderdg_mhd_blast-wave.avi] (/uploads/4d715c47a03bd3850d3c0398e39da58c/lim-aderdg_mhd_blast-wave.avi)
* Rotor setup as in http://adsabs.harvard.edu/abs/2004MSAIS...4...36D with 9^2 grid, P=9, and N=2*9+1 FV subgrid:
![lim-aderdg_mhd_rotor_9x9_P9](/uploads/b35e7dc2dc137068e7416a1952d02a1a/lim-aderdg_mhd_rotor_9x9_P9.png)
[lim-aderdg_mhd_rotor_9x9_P9.avi](/uploads/c5ee8ff29f18a7201be66a1a26b664dc/lim-aderdg_mhd_rotor_9x9_P9.avi)
https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/37Precision issues2019-04-15T12:37:23+02:00Ghost UserPrecision issuesI observed that we can only apply the restriction and prolongation operations
a certain number of times until the projected constant values in my test are not equal anymore
to the reference values (with respect to a tolerance of 1e-12)....I observed that we can only apply the restriction and prolongation operations
a certain number of times until the projected constant values in my test are not equal anymore
to the reference values (with respect to a tolerance of 1e-12).
For more infos see the todos in 2D and 3D definitions of
GenericEulerKernelTest::testFaceUnknownsProjection(),
and
GenericEulerKernelTest::testVolumeUnknownsProjection()https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/143New template-less pure virtual API2019-03-07T19:23:41+01:00Ghost UserNew template-less pure virtual APITODO: Implement this new API.
![usersolver_layout](/uploads/b314b4b3a6196966cec01c80918fb0c9/usersolver_layout.png)
Tobias favours it. Don't know how it goes with optimized kernels. SMall patch for the piucture from Tobias:
```
/* N...TODO: Implement this new API.
![usersolver_layout](/uploads/b314b4b3a6196966cec01c80918fb0c9/usersolver_layout.png)
Tobias favours it. Don't know how it goes with optimized kernels. SMall patch for the piucture from Tobias:
```
/* NEW: */
kernels::aderdg::generic::c::spaceTimePredictorLinear(BasisSolverAPI& solver, other parameters ... );
```
In textual form [Sketch.h](/uploads/8bcfc8c2920e7532f73e1dac5f8f5a07/Sketch.h)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/142ADERDG, inverseDX2018-06-19T11:17:55+02:00Jean-Matthieu GallardADERDG, inverseDX* Problem: most kernel use 1/dx instead of dx. Slow operation that could easily be optimized away.
* Solution:
- Peano implement a cellDescription.getInverseSize()
- ADERDGSolver use it
- ADERDG Kernel adapted
* Already done:
- Op...* Problem: most kernel use 1/dx instead of dx. Slow operation that could easily be optimized away.
* Solution:
- Peano implement a cellDescription.getInverseSize()
- ADERDGSolver use it
- ADERDG Kernel adapted
* Already done:
- Optimized kernel adapted, generate the inverseDx in the ADERDGSolver (code isolated with preprocessor)
@di25coxJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/140Intel compiler bug on certain large C++ files ("exahype/repositories/Reposito...2019-08-25T11:41:31+02:00Ghost UserIntel compiler bug on certain large C++ files ("exahype/repositories/Repository...")With Intel 17 (Intel 15/16 apparently not affected) aka `COMPILER=Intel, SHAREDMEM=TBB, MODE=Debug, DISTRIBUTEDMEM=None.` there is a bug when compiling files like
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstant...With Intel 17 (Intel 15/16 apparently not affected) aka `COMPILER=Intel, SHAREDMEM=TBB, MODE=Debug, DISTRIBUTEDMEM=None.` there is a bug when compiling files like
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstantiation4LimiterStatusMergingAndSpreadingMPI.o
and
./ExaHyPE/exahype/repositories/RepositoryExplicitGridTemplateInstantiation4GridErasing.o
and similar. This is known to Tobias and should be reported to Nicolay Hammer at LRZ or similar.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/1362D / 3D build check is not performed correctly2019-03-18T21:55:01+01:00Ghost User2D / 3D build check is not performed correctlyWe have a startup check in the code that makes sure that the specfile has some same `const` build constants as for instance the ADERDG polynomial order. However, this test fails for checking that for instance a 2D build is also runned wi...We have a startup check in the code that makes sure that the specfile has some same `const` build constants as for instance the ADERDG polynomial order. However, this test fails for checking that for instance a 2D build is also runned with a 2D grid specification. Thus we get strange errors as in https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/10b1261b34e2e239f9a4fe1e2c42f0e38d85a207 :
```
0.0459649 error Invalid simulation end-time: notoken
(file:/home/koeppel/numrel/exahype/Engine-ExaHyPE/./ExaHyPE/exahype/Parser.cpp,line:377)
` ``
And that not after the beginning but after one timestep.
There should be a well-speaking check instead at the beginning.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/130Guidebook: Fix recommendation for MPI Tags for Reductions2019-03-07T19:24:19+01:00Ghost UserGuidebook: Fix recommendation for MPI Tags for ReductionsWhen implementing an own MPI Reductions plotter, the guidebook tells us we should write something like
```
void finishRow() {
#ifdef Parallel
// Question: Do we really reserve a free tag and release on each function call?
c...When implementing an own MPI Reductions plotter, the guidebook tells us we should write something like
```
void finishRow() {
#ifdef Parallel
// Question: Do we really reserve a free tag and release on each function call?
const int reductionTag = tarch::parallel::Node::getInstance().reserveFreeTag(
std::string("TimeSeriesReductions(") + filename + ")::finishRow()" );
if(master) {
double recieved[LEN];
for (int rank=1; rank<tarch::parallel::Node::getInstance().getNumberOfNodes(); rank++) {
if(!tarch::parallel::NodePool::getInstance().isIdleNode(rank)) {
MPI_Recv( &recieved[0], LEN, MPI_DOUBLE, rank, reductionTag, tarch::parallel::Node::getInstance().getCommunicator(), MPI_STATUS_IGNORE );
addValues(recieved);
}
}
} else {
for (int rank=1; rank<tarch::parallel::Node::getInstance().getNumberOfNodes(); rank++) {
if(!tarch::parallel::NodePool::getInstance().isIdleNode(rank)) {
MPI_Send( &data[0], LEN, MPI_DOUBLE, tarch::parallel::Node::getGlobalMasterRank(), reductionTag, tarch::parallel::Node::getInstance().getCommunicator());
}
}
}
tarch::parallel::Node::getInstance().releaseTag(reductionTag);
#endif
if(master) {
TimeSeriesReductions::finishRow();
writeRow();
}
}
```
However I assume this is not that good, looking at the output:
![mpi-reservefreetags](/uploads/de93f1e5c48d10c62101531611e044cd/mpi-reservefreetags.png)
*First question*: Does every plotter need it's own tag? In principal only if they would be called in parallel, isn't it?
In any case, I think the tags should'nt be created so frequently, isn't it? Instead, only once on startup time?
This should be fixed in the guidebook and my code :Dhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/127Revise time stepping notation2019-09-20T15:46:39+02:00Ghost UserRevise time stepping notationI have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little comp...I have to write down carefully how I handle the time step sizes for
* fused time stepping
* standard time stepping
and what I do during mesh refinement for
* fused time stepping
* standard time stepping
This is getting a little complicated and the docu just gives an partial overview.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/126CFL factor per solver as const int the specfile2019-04-15T12:37:03+02:00Ghost UserCFL factor per solver as const int the specfileThis reduces inconsistencies. Can now be easily generated as constexpr double into the AbstractMySolver class.This reduces inconsistencies. Can now be easily generated as constexpr double into the AbstractMySolver class.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/124Toolkit - Move from replaceAll to true template engine2018-09-21T12:09:36+02:00Jean-Matthieu GallardToolkit - Move from replaceAll to true template engineThe templates used by the Toolkit start to require some logic. For example the NamingScheme tag requires to expand a Set to multiple lines.
Currently we put that logic in the java class and use a replaceAll but it would be better to put...The templates used by the Toolkit start to require some logic. For example the NamingScheme tag requires to expand a Set to multiple lines.
Currently we put that logic in the java class and use a replaceAll but it would be better to put that logic in the template itself using a true template engine. For example the NamingScheme tag would be replace with a foreach loop that expand the Set in the template. Also this would help factorizing the code by putting common tag and their value into a more global context set for the template engine
Switching should be simple as we are already using standard template tags for our variable.
Ofc the template engine needs to be light and fully available in a jar or any format requiring no installation to avoid adding new dependencies. For example https://github.com/HubSpot/jinjava or http://jtwig.org/
This is a "nice to have" feature so we should do it when we have more time.
Concerned: @di25cox , @gi26det , @svenk , @ga96nuvJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/117Support material parameters2019-09-20T15:46:49+02:00Ghost UserSupport material parametersTODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument swit...TODO:
* ~~Make Generic C/C++ kernels support parameters x {FV,ADERDG}~~
* ~~Make Generic fortran kernels support parameters.
( solutionUpdate is here the problem.)~~
* ~~In all kernels using the solver template argument switch to the constexpr for variables,parameters,and
order/basisSize.~~
* ~~MyElasticWaveSolver: Realised that we have to rethink our material approach
a little. Material parameters are not available from Q during the space-time predictor
computation which uses the space-time predictor or predictor DoF.
We need either:~~
* store the material values in solution(+previousSolution),
predictor,and space-time predictor
* ~~or pass the material values as separate argument
to flux,eigenvalues,ncp,matrixb etc.
We should then also pass them as separate arguments
to adjustedSolutionValues.~~
~~Follow up:
It might wise then to split the material parameters from the variables.~~ We have chosen the first option.
* ~~The Nonlinear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Nonlinear 3D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~The Linear 2D ADER-DG C/C++ kernels do not support materials yet.~~
* ~~ The Linear3D ADER-DG C/C++ kernels do not support materials yet.~~
* The Fortran ADER-DG kernels do not support materials yet.
* The 2D FV kernels do not support materials yet.
* The 3D FV kernels do not support materials yet.
* The Linear 2D+3D ADER-DG C/C++ kernels need to consider material parameters in the AMR operators.
* (Optional) Transform all generic kernels to templates and use constexpr for variables,parameters and order/basisSize.
Questions:
* What happens at the boundary for the Finite Volumes method? Currently, we hand no material parameters in at the boundary.
* Are there no time averaged source terms in the spaceTimePredictorLinear2d/3d? Yes, I consider them now at least in the striding.
* The linear ADER-DG kernels work with time derivatives. Does it make sense to let the user set the nodal values?
* Why wasn't tempPointForceSources merged into tempSpaceTimeUnknowns. The API change wasn't necessary.
Follow ups:
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.
* Distinguish more clearly between temporary array sizes and dof array sizes.
* Add a boolean to the AbstractMySolver class indicating if we consider Linear or Nonlinear kernels.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/98Scalability Tests (MPI,TBB,MPI+TBB)2019-09-20T15:46:43+02:00Ghost UserScalability Tests (MPI,TBB,MPI+TBB)This is my spreadsheet.
## Tests
Testing scalability with parameters
* dim=2,3,
* p=3,5,7,9
* h=???
## Systems
| system | processor | cores per processor | cores per node | memory per node|
|----------|--------------|-...This is my spreadsheet.
## Tests
Testing scalability with parameters
* dim=2,3,
* p=3,5,7,9
* h=???
## Systems
| system | processor | cores per processor | cores per node | memory per node|
|----------|--------------|-------------------------|-------------------|---------------------- |
| phi1 | 2 Intel Xeon E5-2650 2.0 GHz | 8 cores per processor | 16 cores per node | 64 GByte RAM |
### ADER-DG
Additional parameters:
* ts=fused,standard,
Runs;
* **TBB**
* Strong scaling
* [EulerFlow,2D,phi1,Gnu](/uploads/c4c454dc3c41fe7ee0295a07320db9c7/EulerFlow_TBB_Gnu.tar.gz)
* [Z4,3D,phi1,Gnu](/uploads/4bac6f1c44aa732c4c6c06116f540754/Z4_TBB_Gnu.tar.gz)
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
### Godunov FV
Additional parameters:
Comment: Fused time stepping results here in a single sweep scheme.
* ts=fused,standard
Runs;
* **TBB**
* Strong scaling
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
### Limiting ADER-DG (ADER-DG + FV)
Comment: Fused time stepping is not supported yet for the
limiting ADER-DG solver.
Additional parameters:
* ts=standard,fused
Runs;
* **TBB**
* Strong scaling
* [EulerFlow,Explosion(t=0.4),2D,phi1,Gnu](/uploads/27c9c32a34c46c75e7d918578972cc7b/EulerFlow_LimitingADERDG_TBB_Gnu.tar.gz)
* Weak scaling
* **MPI**
* Strong scaling
* Weak scaling
* **MPI+TBB**
* Strong scaling
* Weak scaling
## Setting up experiments on Hamilton (Durham's HPC)
Here I will document an (more or less) efficient approach to run the above experiments
on Hamilton.
### Installing the latest ExaHyPE and Peano snapshots
* **Installing ExaHyPE**: Hamilton has git version 1.7.1 installed. Unfortunately, I had login issues while I was trying to clone the ExaHyPE-Engine reppository.
I am thus required to perform a two stage process to get ExaHyPE on my login node.
1. Check ExaHyPE out on my laptop.
2. Optional: scp to mira. Login to mira.
3. scp to hamilton.
* **Installing Peano**: Hamilton has svn version 1.6.11 installed.
* Obtaining Peano is very convenient. It is done via:
``svn checkout http://svn.code.sf.net/p/peano/code/trunk peano``
### Building ExaHyPE
...
### Job scripts
We have currently three different job script "frameworks" available in the ExaHyPE-Engine
repository.
While Dominics job scripts are only suited for TBB on Hamilton and SuperMUC,
Tobias' job script suite is suited to perform MPI and MPI+TBB benchmarks on Hamilton (DUR).
Vasco's python based job script framework can basically run anything on SuperMUC.
I will try to merge the three approaches into a python version with different backends based on the
supercomputer.
* **Tobias'** bash job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/hamilton``.
* **Dominics'** bash job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/scaling``.
* **Vasco's** python job scripts can be found at ``~/dev/codes/c/ExaHyPE/ExaHyPE-Engine/Miscellaneous/JobScripts/``.
### Performance analysis scripts
* **Analysing domain decomposition**: ...
* **Plotting speedups**: ...
### Benchmark March 2017
* Dominic's MPI and MPI+TBB job scripts for Hamilton (SLURM) and SuperMUC (LoadLeveler) [Benchmark_Mar2017.tar.gz](/uploads/1966ff2a9b7d28d12037444fca09e213/Benchmark_Mar2017.tar.gz)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/137The GRMHD, SRMHD, and EulerFlow error plotters seem to not work correctly (Th...2018-06-15T15:13:42+02:00Ghost UserThe GRMHD, SRMHD, and EulerFlow error plotters seem to not work correctly (The ADERDG correctness is broken (GRMHD AccretionDisk 2D/3D) )The errors in the AccretionDisk3D, GRMHD application, looked very good (`max mesh ref = 0.5` on a domain `0.0 .. 2.0` in x,y,z) one week ago:
```
sven@nils:~/numrel/exahype/Engine-ExaHyPE/ApplicationExamples/GRMHD$ cat output/error-r...The errors in the AccretionDisk3D, GRMHD application, looked very good (`max mesh ref = 0.5` on a domain `0.0 .. 2.0` in x,y,z) one week ago:
```
sven@nils:~/numrel/exahype/Engine-ExaHyPE/ApplicationExamples/GRMHD$ cat output/error-rho.asc
plotindex time l1norm l2norm max min avg
1 0.000000e+00 5.669120e-12 7.324244e-11 3.750836e-09 1.676159e-13 8.555365e-13
2 1.333333e-02 1.852645e-06 2.236011e-06 3.977681e-05 2.178258e-13 2.698931e-07
3 2.000000e-02 2.860199e-06 3.615806e-06 6.110038e-05 2.300382e-13 4.156812e-07
4 2.666667e-02 3.469631e-06 4.545670e-06 7.338015e-05 2.300382e-13 5.023163e-07
5 3.333333e-02 3.877169e-06 5.212832e-06 8.077556e-05 2.300382e-13 5.578673e-07
6 4.000000e-02 4.180962e-06 5.713450e-06 8.532972e-05 2.300382e-13 5.966567e-07
7 4.666667e-02 4.433994e-06 6.101790e-06 8.813653e-05 2.300382e-13 6.268654e-07
8 5.333333e-02 4.662388e-06 6.411162e-06 8.981804e-05 2.300382e-13 6.524981e-07
9 6.000000e-02 4.880226e-06 6.663393e-06 9.074876e-05 2.300382e-13 6.756477e-07
```
The solution was stationary and the errors should always stay in the same order of magnitude. They did.
Now, we experience something different *both in 2D and 3D ADERDG* and probably also in the FV scheme (but this could have another source).
This ticket shall track these problems.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/160Support a Tecplot-compatible output format2018-06-15T15:13:42+02:00Ghost UserSupport a Tecplot-compatible output formatThis is really low priority, but just to not forget about:
TecPlot360 is a proprietary but amazing visualization software which feels (in my hands) much better then Visit or ParaView. However, it does not (even) load the smalles common ...This is really low priority, but just to not forget about:
TecPlot360 is a proprietary but amazing visualization software which feels (in my hands) much better then Visit or ParaView. However, it does not (even) load the smalles common denominator VTK. Instead, a number of file formats are supported in-house:
```
• CGNS Loader
• DEM Loader
• DXF Loader
• EnSight Loader
• Excel Loader
• FEA Loader
• FLOW-3D Loader
• FLUENT Loader
• General Text Loader
• HDF Loader
• HDF5 Loader
• Kiva Loader
• PLOT3D Loader
• PLY Loader
• Tecplot-Format Loader
• Text Spreadsheet Loader
```
See also: http://home.ustc.edu.cn/~cbq/360_data_format_guide.pdf with meaningful pictures as this one:
![beautiful-tecplot-nodal-values](/uploads/c4ea3adb5281c5c2f713199792c50f53/beautiful-tecplot-nodal-values.png)
Instead of Trentos code, we don't want to implement a writer which writes only the tecplot-specific format but instead use some of these widespread formats, for instance "FLUENT" or so.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/154Min and max is not send to master/worker2018-06-15T15:13:43+02:00Ghost UserMin and max is not send to master/worker* Currently, the min and max is not send between master and worker ranks. This issue does only affect certain AMR+MPI builds
where a cell is at a master/worker boundary but at the same at a remote neighbour boundary.
* Furthermore, ...* Currently, the min and max is not send between master and worker ranks. This issue does only affect certain AMR+MPI builds
where a cell is at a master/worker boundary but at the same at a remote neighbour boundary.
* Furthermore, I have to discuss the whole forking process performed by Peano with Tobias.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/109An idea to solve the double parsing and compile/runtime configuration problem2018-06-15T15:13:43+02:00Ghost UserAn idea to solve the double parsing and compile/runtime configuration problemSimilar to #107, this ticket proposes an idea which solves two problems we currently have in ExaHyPE.
## Problem 1: Double Parsing problem
The ExaHyPE specification files are described by the [exahype.grammar](https://gitlab.lrz.de/exa...Similar to #107, this ticket proposes an idea which solves two problems we currently have in ExaHyPE.
## Problem 1: Double Parsing problem
The ExaHyPE specification files are described by the [exahype.grammar](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/master/Toolkit/exahype.grammar). For the Java toolkit, there is a parser generator (SableCC) which allows a neat way to create the application *during* parsing the specification file. But since the specification file also holds runtime constants, the ExaHyPE binary expects a valid and compatible spec file as it's first argument, ie. call with
```cmd> ExaHyPE-Euler.exe ../EulerFlow.exahype```
In order to parse the specfile at (C++) runtime, there is that [Parser.h](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/master/ExaHyPE/exahype/Parser.h) code. It works with regular expressions and knows nothing about `exahype.grammar`. Thus we sometimes expect that the two parsers don't parse the same spec file similarly or with similar error messages when having problems.
## Problem 2: No clear seperation off compiletime (ie. computing) and runtime (ie. physics) parameters
The current specfile format combines two semantically different concepts of parameters:
* *Computing-related* parameters, ie. project names, source code paths, architecture and compiler specific stuff, optimization parameters, all the PDE properties, all the basic properties about generated code (ie. number and names of classes of plotters)
* *Run-related* parameters, ie. the grid extend, parallelization parameters (number of cores, hybrid setups), plotting properties (frequencies, slicing, ...), physics/application/user parameters (ie. Initial Data, Boundary Conditions)
The user has absolutely no idea which parameter is a compile time constant (and probably even needs a rerun of the toolkit after change) and which parameter is a runtime constant, ie. can be chosen freely when submitting a job at a HPC queue.
Actually this problem is even deeper: There are further constants which can be chosen when running `make`, ie. *after* code generation by the toolkit. These are especially options about the parallelization, ie. whether to build in TBB and MPI support or not. However, these options are also specified in the toolkit.
## A seperation approach
I suggest to keep the specification files for the toolkit runs. I will demonstrate it by splitting the [current EulerFlow.exahype](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/3c8a41ad41a55cab990b0b452fa3cc6e0369b8da/ApplicationExamples/EulerFlow.exahype) into two files. The toolkit specfile would look like:
```
exahype-project Euler
peano-kernel-path = ./Peano
exahype-path = ./ExaHyPE
output-directory = ./ApplicationExamples/EulerFlow
architecture = noarch
computational-domain
dimension = 2
end computational-domain
shared-memory
identifier = dummy
end shared-memory
distributed-memory
configure = {whatever-needs-to-be-generated-here?}
end distributed-memory
solver ADER-DG MyEulerSolver
variables = rho:1,j:3,E:1
order = 3
kernel = generic::fluxes::nonlinear
language = C
plot vtk::Cartesian::cells::ascii ConservedQuantitiesWriter
variables = 5
end plot
plot vtk::Cartesian::vertices::ascii ComputeGlobalIntegrals
variables = 0
end plot
plot vtk::Cartesian::vertices::ascii PrimitivesWriter
variables = 5
end plot
plot vtk::Cartesian::vertices::ascii ExactPrimitivesWriter
variables = 5
end plot
plot probe::ascii ProbeWriter
variables = 5
end plot
end solver
end exahype-project
```
In contrast, we could now introduce a very simple parameter file format where an accurate parser can be written in C++ very simply and which is handed as command line argument to the ExaHyPE runs:
```
# Only single line comments like this
log-file = mylogfile.log
computational-domain::width = 1.0, 1.0
computational-domain::offset = 0.0, 0.0
computational-domain::end-time = 0.12
shared-memory::identifier = dummy
shared-memory::cores = 1 # NB: This should REALLY be driven by an ENV parameter like OMP_NUM_THREADS
distributed-memory::buffer-size = 64
distributed-memory::timeout = 60
optimisation::fuse-algorithmic-steps = off
# 0.0 und 0.8 sind schon mal zwei Faktoren
optimisation::fuse-algorithmic-steps-factor = 0.99
optimisation::timestep-batch-factor = 0.0
optimisation::skip-reduction-in-batched-time-steps = on
optimisation::disable-amr-if-grid-has-been-stationary-in-previous-iteration = off
optimisation::double-compression = 0.0
optimisation::spawn-double-compression-as-background-thread = off
# when this is a runtime constant...
MyEulerSolver::order = 3
# I would love to specify the maximum-mesh-level = 3 instead
MyEulerSolver::maximum-mesh-size = 0.05
# Plotter example
ConservedQuantitiesWriter::time = 0.0
ConservedQuantitiesWriter::repeat = 0.05
ConservedQuantitiesWriter::output = ./conserved
ConservedQuantitiesWriter::select = x:0.0,y:0.0
ComputeGlobalIntegrals::time = 0.0
ComputeGlobalIntegrals::repeat = 0.05
ComputeGlobalIntegrals::output = ./output/these-files-should-not-be-there
ComputeGlobalIntegrals::select = x:0.0,y:0.0
# etc.
# own parameters simple as hell
ML_CCZ4::timelevels = 3
ML_CCZ4::harmonicN = 1.0 # 1+log
ML_CCZ4::harmonicF = 2.0 # 1+log
ML_CCZ4::BetaDriver = 0.4 # ~1/M (\eta)
ML_CCZ4::advectLapse = 1
ML_CCZ4::advectShift = 1
ML_CCZ4::evolveA = 0
ML_CCZ4::evolveB = 1
ML_CCZ4::shiftGammaCoeff = 0.75
ML_CCZ4::shiftFormulation = 0 # Gamma driver
ML_CCZ4::fixAdvectionTerms = 1
ML_CCZ4::dampk1 = 0.05
ML_CCZ4::dampk2 = 0.0
ML_CCZ4::GammaShift = 0.5
ML_CCZ4::MinimumLapse = 1.0e-8
ML_CCZ4::conformalMethod = 1 # 1 for W
ML_CCZ4::dt_lapse_shift_method = "noLapseShiftAdvection"
ML_CCZ4::initial_boundary_condition = "extrapolate-gammas"
ML_CCZ4::rhs_boundary_condition = "NewRad"
Boundary::radpower = 2
ML_CCZ4::ML_log_confac_bound = "none"
ML_CCZ4::ML_metric_bound = "none"
ML_CCZ4::ML_Gamma_bound = "none"
ML_CCZ4::ML_trace_curv_bound = "none"
ML_CCZ4::ML_curv_bound = "none"
ML_CCZ4::ML_lapse_bound = "none"
ML_CCZ4::ML_dtlapse_bound = "none"
ML_CCZ4::ML_shift_bound = "none"
ML_CCZ4::ML_dtshift_bound = "none"
AHFinderDirect::N_horizons = 1
AHFinderDirect::find_every = 64
AHFinderDirect::output_h_every = 0
AHFinderDirect::max_Newton_iterations__initial = 50
AHFinderDirect::max_Newton_iterations__subsequent = 50
AHFinderDirect::max_allowable_Theta_growth_iterations = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars = "order=4"
AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars = "order=4"
AHFinderDirect::verbose_level = "physics details"
AHFinderDirect::move_origins = "yes"
AHFinderDirect::origin_x[1] = 0.0
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = 0.0
AHFinderDirect::initial_guess__coord_sphere__y_center[1] = 0.0
AHFinderDirect::initial_guess__coord_sphere__z_center[1] = 0.0
AHFinderDirect::initial_guess__coord_sphere__radius[1] = 3.0
AHFinderDirect::which_surface_to_store_info[1] = 0
AHFinderDirect::set_mask_for_individual_horizon[1] = "no"
AHFinderDirect::reset_horizon_after_not_finding[1] = "no"
AHFinderDirect::find_after_individual_time[1] = 2000.0
AHFinderDirect::max_allowable_horizon_radius[1] = 5.0
# ...
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/135FV + MPI crashes2018-06-15T15:13:43+02:00Ghost UserFV + MPI crashesApplication: `GRMHD_FV` with current grid as in repository, start with
$ `mpirun -np 4 ./ExaHyPE-GRMHD ../GRMHD_FV.exahype`
and you get
```
assertion in file /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/Time...Application: `GRMHD_FV` with current grid as in repository, start with
$ `mpirun -np 4 ./ExaHyPE-GRMHD ../GRMHD_FV.exahype`
and you get
```
assertion in file /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/TimeStepSizeComputation.cpp, line 229 failed: solver->getNextMaxCellSize()>0
parameter solver->getNextMaxCellSize(): -1.00000000000000000000e+00
parameter _maxCellSizes[solverNumber]: -1.79769313486231570815e+308
parameter _minCellSizes[solverNumber]: 5.3047 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 3 (199 message(s))
5.30473 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 2 (78 message(s))
5.30475 [sveinn],rank:1 info peano::parallel::SendReceiveBufferAbstractImplementation::releaseSentMessages() sent all messages belonging to node 0 (42 message(s))
1.79769313486231570815e+308
ExaHyPE-GRMHD: /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/exahype/mappings/TimeStepSizeComputation.cpp:229: void exahype::mappings::TimeStepSizeComputation::endIteration(exahype::State&): Assertion `false' failed.
```
* Voller Log: [mpi.log](/uploads/e598f84752224c16f03ebee9b0d646ba/mpi.log)
* Gepackter Coredump: [core.gz](/uploads/bd610012c8e810aaac9fa6f9ccd9214d/core.gz)
* Version information: [exa.version](/uploads/e30e199b8ef94ee4d5839ecb3ab08843/exa.version) (MPI, kein TBB, GCC)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/146Static AMR SharedMemory-Memory Leaks2018-06-15T15:13:43+02:00Ghost UserStatic AMR SharedMemory-Memory LeaksI observe
* a very slightly increasing memory consumption
of the ADERDGSolver
![memory_leak_slow](/uploads/edfb85be1570b74f2da985597a17f76b/memory_leak_slow.png)
* plus an increase of the memory consumption of the ADERDGSo...I observe
* a very slightly increasing memory consumption
of the ADERDGSolver
![memory_leak_slow](/uploads/edfb85be1570b74f2da985597a17f76b/memory_leak_slow.png)
* plus an increase of the memory consumption of the ADERDGSolver
by large chunks.
![memory_leak_fast](/uploads/9d464f1c3b545dbe93a2beeb2ab94bd6/memory_leak_fast.png)
Both can probably be traced back to a problem with allocating
deallocating temporary variables in the mappings.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/87Let's get rid of the "Code" directory2018-06-15T15:13:43+02:00Ghost UserLet's get rid of the "Code" directoryIn accordance with #86 this proposal suggests that instead of having a repository layout as
```
ExaHyPE-Engine/
├── Code
│ ├── ApplicationExamples
│ ├── Applications
│ ├── CodeGenerator
│ ├── ExaHyPE
│ ├── ExternalLibraries
│ ...In accordance with #86 this proposal suggests that instead of having a repository layout as
```
ExaHyPE-Engine/
├── Code
│ ├── ApplicationExamples
│ ├── Applications
│ ├── CodeGenerator
│ ├── ExaHyPE
│ ├── ExternalLibraries
│ ├── Miscellaneous
│ ├── Peano
│ ├── Toolkit
│ └── UserCodes
├── LICENSE.txt
├── README.md
└── RUN.sh
```
we should have this simpler one:
```
ExaHyPE-Engine/
├── ApplicationExamples
├── CodeGenerator
├── ExaHyPE
├── ExternalLibraries
├── LICENSE.txt
├── Miscellaneous
├── Peano
├── README.md
├── RUN.sh
├── Toolkit
└── UserCodes
```
The idea is when having not a `Code` repository, people don't even plan to create something non-Code-ish in the repository. Which is a good idea.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/141Pictures in Wiki missing2018-06-15T15:13:43+02:00Ghost UserPictures in Wiki missingFor instance at https://gitlab.lrz.de/exahype/ExaHyPE-Engine/wikis/Eclipse_import
Sven: Try to find them somewhere and upload.
They were most likely removed at the repository transition.For instance at https://gitlab.lrz.de/exahype/ExaHyPE-Engine/wikis/Eclipse_import
Sven: Try to find them somewhere and upload.
They were most likely removed at the repository transition.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/86Let's get rid of the "Applications" folder2018-06-15T15:13:43+02:00Ghost UserLet's get rid of the "Applications" folderThis message goes mostly to @di25cox and @ga96nuv as they have worked in folders in https://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/master/Code/Applications in the last seven days:
As Vasco and me figured out in the last days at ht...This message goes mostly to @di25cox and @ga96nuv as they have worked in folders in https://gitlab.lrz.de/exahype/ExaHyPE-Engine/tree/master/Code/Applications in the last seven days:
As Vasco and me figured out in the last days at https://gitlab.lrz.de/exahype/ExaHyPE-Engine/issues/67#note_38114,
> * `ApplicationExamples` currently holds the public applications
> * `Applications` currently holds the private applications
However, the only "private" applications so far (Z4) moved to an own repository. I would hence suggest to **merge the folders `Applications` and `ApplicationExamples`**.
Please let me know if you have any objectives against.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/139Kernel activation functions2018-06-15T15:13:43+02:00Ghost UserKernel activation functionsSome notes we might want to pickup later on:
* ``useNCP`` and ``useMatrixB`` are always used as a pair since the Matrix B is fed into the ncp kernel.
* ``useNCP``, ``useMatrixB``, and ``useFlux`` are set for all cells globally. It does ...Some notes we might want to pickup later on:
* ``useNCP`` and ``useMatrixB`` are always used as a pair since the Matrix B is fed into the ncp kernel.
* ``useNCP``, ``useMatrixB``, and ``useFlux`` are set for all cells globally. It does not make
sense to turn them off or on locally in contrast to source terms and point sources.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/108Toolkit: Parser errors with faulty specfile2018-06-15T15:13:43+02:00Ghost UserToolkit: Parser errors with faulty specfileSo the specfile introduced new attributes (properties, or however you call them) and the first what happens is that the toolkit does not run any more with old specfiles. Today: Trying to parse [SRMHD.exahype](/uploads/02379c4f0485dad9d1a...So the specfile introduced new attributes (properties, or however you call them) and the first what happens is that the toolkit does not run any more with old specfiles. Today: Trying to parse [SRMHD.exahype](/uploads/02379c4f0485dad9d1a69748d0ca7445/SRMHD.exahype) as it is in the [current](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/commit/3c8a41ad41a55cab990b0b452fa3cc6e0369b8da) repository:
```
ERROR: eu.exahype.parser.ParserException: [66,5] expecting: 'buffer-size'
```
I don't get this error message. If you open the [SRMHD.exahype](/uploads/02379c4f0485dad9d1a69748d0ca7445/SRMHD.exahype) there is nothing happening at this line.
So I tried out:
* Inserting the new `log-file = mylogfile.log` - didnt change a thing
* Inserting `variables = rho:1,j:3,E:1,B:3,damping:1` - didnt change a thing
* Commenting out `/* constants = {initialdata:alfven} */` gave me this error : ```ERROR: eu.exahype.lexer.LexerException: [61,41] Unknown token: *``` **WHY THE HELL CAN'T I JUST COMMENT OUT STUFF**??
* It turned out that I had to **REMOVE** the constants to get the toolkit running throught. **COMMENTING OUT IS NOT ENOUGH. WHY?**https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/133Variables in wrong class2018-06-15T15:13:43+02:00Ghost UserVariables in wrong classJust found out with my clang static source analyzer: We have
```
class Z4::AbstractZ4Solver: public exahype::solvers::ADERDGSolver {
public:
static constexpr int NumberOfVariables = 54;
static constexpr int NumberOfParameters...Just found out with my clang static source analyzer: We have
```
class Z4::AbstractZ4Solver: public exahype::solvers::ADERDGSolver {
public:
static constexpr int NumberOfVariables = 54;
static constexpr int NumberOfParameters = 0;
static constexpr int Order = 3;
class Variables;
class ReadOnlyVariables;
class Fluxes;
...
```
but then
```
class Z4::Z4Solver::ReadOnlyVariables {
private:
const double* const _Q;
public:
static constexpr int SizeVariables
...
```
However, it should be `Z4::AbstractZ4Solver::ReadOnlyVariables` or the forward declarations have to change classes.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/138Fix Fortran module detection Makefile again2018-06-15T15:13:43+02:00Ghost UserFix Fortran module detection Makefile againNote for @svenk.
It's extremely nerving to restart builds all the time, especially when using out-of-tree builds.Note for @svenk.
It's extremely nerving to restart builds all the time, especially when using out-of-tree builds.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/85Peano broken once again2018-06-15T15:13:43+02:00Ghost UserPeano broken once againAnd since we only have the tarball in the repository, I cannot do anything else than posting the code that repairs it:
```
sven@nils:~/numrel/exahype/master/Code/Peano/peano/datatraversal/autotuning$ head -n100 OracleForOnePhase.cpp Me...And since we only have the tarball in the repository, I cannot do anything else than posting the code that repairs it:
```
sven@nils:~/numrel/exahype/master/Code/Peano/peano/datatraversal/autotuning$ head -n100 OracleForOnePhase.cpp MethodTrace.cpp
==> OracleForOnePhase.cpp <==
#include "peano/datatraversal/autotuning/OracleForOnePhase.h"
#include "tarch/Assertions.h"
std::string peano::datatraversal::autotuning::toString( const MethodTrace& methodTrace ) {
return "<error>";
}
==> MethodTrace.cpp <==
#include "peano/datatraversal/autotuning/MethodTrace.h"
#include "tarch/Assertions.h"
peano::datatraversal::autotuning::MethodTrace peano::datatraversal::autotuning::toMethodTrace(const std::string& identifier) {
return MethodTrace::NumberOfDifferentMethodsCalling;
}
peano::datatraversal::autotuning::MethodTrace peano::datatraversal::autotuning::toMethodTrace(int identifier) {
return MethodTrace::NumberOfDifferentMethodsCalling;
```
Then it compiles again. Refering to latest commit https://gitlab.lrz.de/gi26det/ExaHyPE/commit/5794462e9dfb71ec2375c95b1ba5ac369c5a8661https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/132MySolver::flux parameter F (double**) contiguous or not ?2018-06-15T15:13:43+02:00Jean-Matthieu GallardMySolver::flux parameter F (double**) contiguous or not ?Here is the signature of the flux function:
`void GRMHD::GRMHDSolver::flux(const double* const Q,double** F)`
When using Fortran user code, it is assumed by @svenk that F is a contiguous array. This is true with the generic kernel but...Here is the signature of the flux function:
`void GRMHD::GRMHDSolver::flux(const double* const Q,double** F)`
When using Fortran user code, it is assumed by @svenk that F is a contiguous array. This is true with the generic kernel but false with the optimized one due to changing the data layout of LFi from (t, z, y, x, nDim + 1 for Source, nVar) to (nDim + 1 for Source,t, z,y, x, , nVar_padded)
The assumption that F is contiguous was never explicitly given so my question:
Should F be assumed to be contiguous ?
If yes I need to adapt the optimized kernel, if not then the Fortran user code should at least explicitly mention this and if possible do the conversion himself.
Concerned: @svenk @ga96nuv @di25cox @gi26detJean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/1313D Legendre Plotting fails2018-06-15T15:13:43+02:00Ghost User3D Legendre Plotting failsWhen I do a 3D simulation and plot with `vtk::Cartesian::vertices::ascii`, I clearly see the cube and all elements nicely with their 3D coordinate:
![plotters-vertex](/uploads/dc054456b5c6915882dfa3480aa91638/plotters-vertex.png)
Inste...When I do a 3D simulation and plot with `vtk::Cartesian::vertices::ascii`, I clearly see the cube and all elements nicely with their 3D coordinate:
![plotters-vertex](/uploads/dc054456b5c6915882dfa3480aa91638/plotters-vertex.png)
Instead, when I switch to `vtk::Legendre::vertices::ascii` to see the actual degree of fredom, all elements get **the same z value**:
![plot-legendre-3d](/uploads/37af4b51f820a020abe876b1c855f32c/plot-legendre-3d.png)
The elements are still there, the VTK files have the same size, but the z coordinate is missing.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/83Testing MHD by solving with Two kernels at the same time2018-06-15T15:13:43+02:00Ghost UserTesting MHD by solving with Two kernels at the same timeA proposal by Vasco
2016-11-21 10:16 GMT+01:00 Vasco Varduhn <varduhn@tum.de>:
>
> Hi,
>
>
>
> I had an idea in for testing the C kernels i.e. comparing them with the FORTRAN kernels. In the MySomethingSolver_generated.cpp are the ca...A proposal by Vasco
2016-11-21 10:16 GMT+01:00 Vasco Varduhn <varduhn@tum.de>:
>
> Hi,
>
>
>
> I had an idea in for testing the C kernels i.e. comparing them with the FORTRAN kernels. In the MySomethingSolver_generated.cpp are the calls to the kernels.
>
>
>
> In there you could copy the input data to additional arrays, call both the C and the FORTRAN kernels on the different copies, and then compare the output data entry by entry and throw an error, if the values differ (too much).
>
>
>
> This would allow for an element-wise comparison of all kernels.
>
>
>
> Best,
>
> Vasco
>
>
As a followup from gi26det/ExaHyPE#72https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/107An idea to solve both the toolkit overwriting and file location problem2018-06-15T15:13:43+02:00Ghost UserAn idea to solve both the toolkit overwriting and file location problemThe Java toolkit creates C++ files in the application directory ("output Directory"). This creates (amongst others) two problems:
1. In large applications, users cannot freely structure their code in subdirectories
2. Users cannot u...The Java toolkit creates C++ files in the application directory ("output Directory"). This creates (amongst others) two problems:
1. In large applications, users cannot freely structure their code in subdirectories
2. Users cannot understand which files are overwritten and which not
To address problem (1), I recently introduced an heuristic approach, [FileSearch.java](https://gitlab.lrz.de/exahype/ExaHyPE-Engine/blob/6e7e9552fb3d65b4c5feb5c70e36038be06499ad/Toolkit/src/eu/exahype/FileSearch.java). This allows to put a file *with a similar name* in any subdirectory in the output directory. Thus, for instance, we can have an application to be structured in folders like
```
SRMHD
├── C2P-MHD.f90
├── C2P-MHD.h
├── C2PRoutines.f90 -> ../SRHD/C2PRoutines.f90
├── ExaHyPE-MHDSolver-p2
├── extended.log
├── InitialDataAdapter.cpp
├── InitialDataAdapter.h
├── InitialData.f90
├── KernelCalls.cpp
├── Makefile
├── make-p2.log
├── MHDSolver.cpp
├── MHDSolver_generated.cpp
├── MHDSolver.h
├── Parameters.f90
├── parameters.mod
├── PDE.f90
├── PDE.h
├── run_withEnv.sh
└── Writers
├── ConservedWriter.cpp
├── ConservedWriter.h
├── ErrorWriter.cpp
├── ErrorWriter.h
├── ExactPrimitivesWriter.cpp
├── ExactPrimitivesWriter.h
├── IntegralsWriter.cpp
├── IntegralsWriter.h
├── PrimitivesWriter.cpp
├── PrimitivesWriter.h
├── RelativeErrorWriter.cpp
├── RelativeErrorWriter.h
└── TimeSeriesReductions.h -> ../../EulerFlow/TimeSeriesReductions.h
```
ie. note the `Writers` subdirectory.
Actually, this ticket shall propose an idea which solves the problems (1) and (2) altogether.
## Defining a header
My idea proposes to define a magic line at any place (ie. header or footer) of the generated `.h` or `.cpp` files, containing something like:
```
/** ExaHyPE.jar: autogenerated at Fr 3. Feb 19:08:51 CET 2017 **/
/** ExaHyPE.jar: File identifier: Plotter[i].HeaderCode **/ <- this is the identifier line
/** ExaHyPE.jar: Rebuild options: [x] Rebuild on every toolkit call [ ] Manually disable rebuild [ ] Rebuild on structural change **/ <- This can be changed by user
```
The identifier line solves the problem to find an appropriate file. It allows users to rename them as he wants. The toolkit has to inspect all possible files with a `grep` like search, but this is not too bad. We don't really expect too big applications...
The options line allows the user both to understand the default setting and to enforce his own rules. This is very helpful when working at ExaHyPE internals and to stop this long lasting conflict with the toolkit overwriting stuff. It also allows users to request an overwrite of a file. The specific syntax is open to discussion, ie. one could also choose something better machine readable here.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/128Dynamic AMR race conditions possible2018-06-15T15:13:43+02:00Ghost UserDynamic AMR race conditions possibleGradient based refinement criteria can introduce a race condition
in my current dynamic AMR implementation.
The user has to make sure here that the fine grid cells are not requesting
erasing while the coarse grid cell requests refi...Gradient based refinement criteria can introduce a race condition
in my current dynamic AMR implementation.
The user has to make sure here that the fine grid cells are not requesting
erasing while the coarse grid cell requests refinement.
I properly have to introduce a flag/cell type that notifies the fine grid cells that the
parent has requested refinement during this grid setup.
I was able to implement stable refinement criteria solely based on the magnitude of the target quantity.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/106Named Variables in toolkit also as strings2018-06-15T15:13:43+02:00Ghost UserNamed Variables in toolkit also as stringsDominic new named variables stuff is nice (there is an ongoing email discussion), but @Svenk, implement some stringification stuff afterwards so we can get the name of the variables also easily and loop over them in the plotters and so.Dominic new named variables stuff is nice (there is an ongoing email discussion), but @Svenk, implement some stringification stuff afterwards so we can get the name of the variables also easily and loop over them in the plotters and so.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/125Compiling on the cluster, notes2018-06-15T15:13:43+02:00Ghost UserCompiling on the cluster, notesFirst: SVN is old and breaks generate-buildinfo.sh, almost fixed.
Second: MPI/TBB, all that, is ugly:
```
mpiicpc -DALIGNMENT=16 -DnoMultipleThreadsMayTriggerMPICalls -DDim2 -fstrict-aliasing -std=c++0x -restrict -xHost -O3 -no-ipo -ip...First: SVN is old and breaks generate-buildinfo.sh, almost fixed.
Second: MPI/TBB, all that, is ugly:
```
mpiicpc -DALIGNMENT=16 -DnoMultipleThreadsMayTriggerMPICalls -DDim2 -fstrict-aliasing -std=c++0x -restrict -xHost -O3 -no-ipo -ip -DSharedTBB -DParallel -DMPICH_IGNORE_CXX_SEEK /usr/include/tbb -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./ApplicationExamples/EulerFlow -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/mpibalancing/.. -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/sharedmemoryoracles/.. -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/multiscalelinkedcell/.. -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/peano/.. -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/tarch/.. -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./ExaHyPE -I/home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./ApplicationExamples/EulerFlow -c /home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/peano/grid/TraversalOrderOnTopLevel.cpp -o /home/astro/koeppel/ExaHyPE/Engine-ExaHyPE/./Peano/peano/grid/TraversalOrderOnTopLevel.o
icpc: error #10236: File not found: '/usr/include/tbb'
```
on LOEWE after
```
$ module list
Currently Loaded Modulefiles:
1) slurm/2.6.3 3) mpi/mvapich2/intel-17.0.1/2.2 5) intel-tbb-oss/intel64/40_20120613oss
2) intel/compiler/64/17.0.1 4) mpi/intel/4.1.0.024
```
I think the `tbb` path is in some `ENV` var, should not be assumed at `/usr/include/tbb`.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/105[Design decision][CodeGenerator] Optimizing unused feature2018-06-15T15:13:43+02:00Jean-Matthieu Gallard[Design decision][CodeGenerator] Optimizing unused feature@di25cox @svenk @gi26det
I'm currently rewriting the SpaceTimePredictor Code Generator and have the following issue:
The source terms for the SpaceTimePredictor and VolumeIntegral (in memory: 4th block of tempFluxUnknowns = lFhi a...@di25cox @svenk @gi26det
I'm currently rewriting the SpaceTimePredictor Code Generator and have the following issue:
The source terms for the SpaceTimePredictor and VolumeIntegral (in memory: 4th block of tempFluxUnknowns = lFhi and tempSpaceTimeFluxUnknowns[0] = lFi in 3D, tempSpaceTimeFluxUnknowns[1] = gradQ and tempStateSizedVector = BgradQ) are totally unused in Euler and therefore weren't implemented at all in the optimized kernel.
I could implement them like the generic kernel do (always active) but it would be highly inefficient as a lot of tensor product and function call for nothing would be required, also the generic kernel should be optimized to be able to chose to disable this feature altogether if not required.
So I'll need to force the user to chose to activate the use of source term or not (at least for the optimized kernel, the generic ones can later be adapted the same way). I see 3 way of doing this efficiently:
* Write it somewhere in the spec file so that the Code Generator can generate code without source term if required
* Use preprocessor instruction and tell the user to modify the local makefile or a new compile configuration file to enable/disable it and then mark the source term related code with preprocessor branching
* Use templating and template metaprogramming to generate two or more version of the code, then let the user chose which one to use by changing a boolean that will be read in the MySolver_generated to choose the correct implementation
Example of solution 3:
```
template <boolean useSource>
void test() {
if(useSource) {
printf("Source term code used");
} else {
printf("No Source term code");
}
}
// [...] in Mysolver_generated
if(useSourceLocalBoolean) {
test<true>();
} else {
test<false>();
}
```
The advantage is that the branching optimization is done at compile time with 2 version of the code generated and the branching evaluation will be very cheap (once per cell per timestep). The drawback is that the code become more complex.
I would prefer solution 1 or 2 but we can discuss it as it is not urgent matter (I'm implementing without source at first).Jean-Matthieu GallardJean-Matthieu Gallardhttps://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/123Derivative plotter, part 22018-06-15T15:13:43+02:00Ghost UserDerivative plotter, part 2We wrote a plotter which access one cell ("patch") and computes the local derivatives, but it cannot plot to VTK but only reductions. We need to extend this to VTK, too.
Application context: Z4, we need to see the constraints.
This is ...We wrote a plotter which access one cell ("patch") and computes the local derivatives, but it cannot plot to VTK but only reductions. We need to extend this to VTK, too.
Application context: Z4, we need to see the constraints.
This is something I can try to do.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/76Hand Gauss point index to helper functions of generic kernels2018-06-15T15:13:43+02:00Ghost UserHand Gauss point index to helper functions of generic kernels# Rationale
This will enable us to implement the point source like
solution adjustements required by the seismology people.
# Proposed signatures
* ``static void eigenvalues(double* const x,int* const ix,double t,int it,double* const Q,...# Rationale
This will enable us to implement the point source like
solution adjustements required by the seismology people.
# Proposed signatures
* ``static void eigenvalues(double* const x,int* const ix,double t,int it,double* const Q,int normalNonZeroIndex,double* lambda);``
* ``static void flux(double* const x,int* const ix,double t,int it,double* const Q,double** F);``
* ``static void source(double* const x,int* const ix,double t,int it,double* const Q,double* S);``
* ``static void boundaryValues(double* const x,double* const ix,double t,int it,const int faceIndex,int normalNonZero,double * const fluxIn,double* const stateIn,double *fluxOut, double* stateOut);``
* ``static void adjustedSolutionValues(double* const x,int* const ix,double t,int it,double* Q);``
``
``ix`` is the spatial Gauss point index (multiindex of size ``DIMENSIONS``).
``it`` is the temporal Gauss point index.
# Boundary values
We decided to sample pointwise and perform a time integration afterwards (global time stepping)
or directly add the face's flux contribution to the cell's update.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/122FV kernels NCP / gradQ2018-06-15T15:13:43+02:00Ghost UserFV kernels NCP / gradQNote also that the Godunov kernels still lack support for NCP and gradQ and all that.
It's something I could try work on.Note also that the Godunov kernels still lack support for NCP and gradQ and all that.
It's something I could try work on.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/74Do nightly convergence tests2018-06-15T15:13:43+02:00Ghost UserDo nightly convergence testsAs an initiative for proving and monitoring the correctness of ExaHyPE, we should come up with a series of very simple tests which demonstrate how we debug the code and also give convergence rates.
This is something Sven and Vasco can d...As an initiative for proving and monitoring the correctness of ExaHyPE, we should come up with a series of very simple tests which demonstrate how we debug the code and also give convergence rates.
This is something Sven and Vasco can do.
A problem is that almost all tests require periodic BC, thought.
## Collection of problems
We could solve:
* Advection Equations
* With the Initial Data:
* Low order polynomials (`Q(:) = 1 + v0 * x(0) - v0 * t`). Intermediate solver steps are always analytically known
* No polynomials (`Q(:) = ICA * SIN(2*pi*(x(0) - t))` et al)
* For random matter distributions
* Implemented as conservative equation (`F(Q) = Q`)
* Implemented as nonconservative equation (`F=0, BgradQ = (1,0,0), S=0`)
* EulerFlow, SRHD, MHD: See the [List of Benchmarks](https://gitlab.lrz.de/gi26det/ExaHyPE/wikis/list%20of%20benchmarks)
See also: gi26det/ExaHyPE#73 and gi26det/ExaHyPE#64.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/121Grid Setup takes forever for aspect-ratio<0.4 domains.2018-06-15T15:13:43+02:00Ghost UserGrid Setup takes forever for aspect-ratio<0.4 domains.This was confirmed for the Finite Volumes, ADER-DG and Limiting ADER-DG solver
**Update**: width=1.0,0.4 works but not width = 1.0,0.3 and smaller aspect ratios width_y/width_x.
This might be a Peano bug since the if-body in the sn...This was confirmed for the Finite Volumes, ADER-DG and Limiting ADER-DG solver
**Update**: width=1.0,0.4 works but not width = 1.0,0.3 and smaller aspect ratios width_y/width_x.
This might be a Peano bug since the if-body in the snippet below
is not entered anymore after the first grid setup iteration.
```
void exahype::mappings::MeshRefinement::refineVertexIfNecessary(
exahype::Vertex& fineGridVertex,
const tarch::la::Vector<DIMENSIONS, double>& fineGridH,
bool isCalledByCreationalEvent
) const {
for (const auto& p : exahype::solvers::RegisteredSolvers) {
if (
fineGridVertex.getRefinementControl() == Vertex::Records::Unrefined
&&
tarch::la::allGreater(fineGridH,p->getMaximumMeshSize())
) {
logInfo("refineVertexIfNecessary(...)",fineGridH);
#ifdef Parallel
if (isCalledByCreationalEvent) {
fineGridVertex.enforceRefine();
}
else {
fineGridVertex.refine();
}
#else
fineGridVertex.refine();
#endif
}
}
}
```
Output:
```
0.00654268 info peano::utils::UserInterface::writeHeader() Application based upon the PDE framework Peano - 3rd Generation
0.0065589 info peano::utils::UserInterface::writeHeader() revision: 2509
0.00657392 info peano::utils::UserInterface::writeHeader() build: dim=2
0.00658727 info peano::utils::UserInterface::writeHeader() optimisations: d-loop persistent-attributes packed opt-static-subtrees recursion-unrolling persistent-regular-subtrees
0.00660133 info peano::utils::UserInterface::writeHeader() (C) 2005 - 2016 www.peano-framework.org
0.00661516 info peano::utils::UserInterface::writeHeader() processes: 1, threads: 1
0.00667906 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [1,1]
0.00671291 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [1,1]
0.00695658 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00709009 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00744748 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00749516 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00753284 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.333333,0.333333]
0.00765061 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00768328 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00771189 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00837231 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00840974 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00844097 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00893974 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00898051 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.00901055 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0099957 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106082 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106461 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0106778 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0154214 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.015456 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0154729 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0201931 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0202355 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0202694 info exahype::mappings::MeshRefinement::refineVertexIfNecessary(...) [0.111111,0.111111]
0.0283382 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0284777 info exahype::runners::Runner::createGrid() grid setup iteration #1, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:208,numberOfBoundaryVertices:154,numberOfOuterVertices:352,numberOfInnerCells:144,numberOfOuterCells:306,numberOfInnerLeafVertices:104,numberOfBoundaryLeafVertices:64,numberOfOuterLeafVertices:176,numberOfInnerLeafCells:135,numberOfOuterLeafCells:266,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0285568 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.041122 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.04128 info exahype::runners::Runner::createGrid() grid setup iteration #2, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:208,numberOfBoundaryVertices:154,numberOfOuterVertices:1106,numberOfInnerCells:144,numberOfOuterCells:729,numberOfInnerLeafVertices:104,numberOfBoundaryLeafVertices:64,numberOfOuterLeafVertices:534,numberOfInnerLeafCells:135,numberOfOuterLeafCells:642,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:0,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.041399 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0546217 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0550416 info exahype::runners::Runner::createGrid() grid setup iteration #3, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1550,numberOfInnerCells:225,numberOfOuterCells:972,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:744,numberOfInnerLeafCells:135,numberOfOuterLeafCells:858,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:0,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0551374 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0675743 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0676742 info exahype::runners::Runner::createGrid() grid setup iteration #4, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:358,numberOfBoundaryVertices:94,numberOfOuterVertices:1026,numberOfInnerCells:225,numberOfOuterCells:702,numberOfInnerLeafVertices:179,numberOfBoundaryLeafVertices:34,numberOfOuterLeafVertices:498,numberOfInnerLeafCells:135,numberOfOuterLeafCells:618,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0677187 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0781031 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0782623 info exahype::runners::Runner::createGrid() grid setup iteration #5, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1106,numberOfInnerCells:225,numberOfOuterCells:729,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:534,numberOfInnerLeafCells:135,numberOfOuterLeafCells:642,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:0,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:0,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0783317 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.0897303 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0898073 info exahype::runners::Runner::createGrid() grid setup iteration #6, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:358,numberOfBoundaryVertices:94,numberOfOuterVertices:1550,numberOfInnerCells:225,numberOfOuterCells:972,numberOfInnerLeafVertices:179,numberOfBoundaryLeafVertices:34,numberOfOuterLeafVertices:744,numberOfInnerLeafCells:135,numberOfOuterLeafCells:858,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:0,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:0)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
0.0898409 info exahype::runners::Runner::createGrid(...) memoryUsage =29 MB
0.098608 info exahype::mappings::LoadBalancing::endIteration(State) memoryUsage =29 MB
0.0987325 info exahype::runners::Runner::createGrid() grid setup iteration #7, max-level=4, state=(mergeMode:MergeNothing,sendMode:ReduceAndMergeTimeStepData,reinitTimeStepData:0,stabilityConditionOfOneSolverWasViolated:0,timeStepSizeWeightForPredictionRerun:0,limiterDomainHasChanged:0,minMeshWidth:[0.037037,0.037037],maxMeshWidth:[0.037037,0.037037],numberOfInnerVertices:374,numberOfBoundaryVertices:150,numberOfOuterVertices:1026,numberOfInnerCells:225,numberOfOuterCells:702,numberOfInnerLeafVertices:187,numberOfBoundaryLeafVertices:62,numberOfOuterLeafVertices:498,numberOfInnerLeafCells:135,numberOfOuterLeafCells:618,maxLevel:4,hasRefined:1,hasTriggeredRefinementForNextIteration:1,hasErased:1,hasTriggeredEraseForNextIteration:0,hasChangedVertexOrCellState:1,hasModifiedGridInPreviousIteration:1,isTraversalInverted:1)[smallest-regular-tree-that-is-held-persistently(currently/previous iteration)=0/0], idle-nodes=1
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/73ShuVortex convergence tests interpretation2018-06-15T15:13:43+02:00Ghost UserShuVortex convergence tests interpretationAt 28. Sept Michael Dumbser wrote about the convergence rates of the Euler ShuVortex simulation:
>
I think the convergence results are not bad at all. The "systematic artefacts" are not artefacts, but we clearly understand where they co...At 28. Sept Michael Dumbser wrote about the convergence rates of the Euler ShuVortex simulation:
>
I think the convergence results are not bad at all. The "systematic artefacts" are not artefacts, but we clearly understand where they come from and the code would be wrong if we could not see them :-) To get a clean and proper convergence table for Exahype without periodic boundary conditions, you must simulate the ShuVortex
>
1. only until a rather small final time, say t=0.5 or t=1.0. The vortex must not touch the boundary at all, and since the vortex is an exponential function,
the high order DG scheme will even see very small errors of the order 1e-7 or 1e-8, hence the Gaussian must really be perfectly flat on the boundary, well below this accuracy. Not only at the initial, but also at the final time. In numerical analysis, it is not common practice to report convergence order over simulation time. Usually, one reports the error in a certain norm (L1, L2, Linf) on a given mesh for a fixed output time, which satisfies all the necessary theoretical criteria for getting high order (sufficiently smooth solution). If convergence order drops due to boundary effects, at least one of the necessary criteria (regularity) is violated, so it is clear that no convergence at the expexted rate is observed.
>
2. If you want larger simulation times, you must increase the domain size (even bigger than the canonical [0,10]^2 domain, say [-10,20]^2), in order to make sure that
the Gaussian is perfectly flat on the boundary, up to the desired error threshold.
>
3. A nice test case which works for arbitrarily large times and without periodic boundaries is the large Amplitude Alfven wave for SRMHD. Olindo can send you the setup. The exact solution is a travelling sine wave, and if you impose the exact solution on the boundaries, you should be able to simulate the problem even without periodic boundaries and without any artefact.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/104EulerFlow and SRMHD Convergence2018-06-15T15:13:43+02:00Ghost UserEulerFlow and SRMHD ConvergenceThere was a mistake in the EulerFlow and SRMHD error computation routines:
The analytical solution expressed in conserved variables was compared vs.
the numerical solution expressed in primitive variables.
I will report the errors ...There was a mistake in the EulerFlow and SRMHD error computation routines:
The analytical solution expressed in conserved variables was compared vs.
the numerical solution expressed in primitive variables.
I will report the errors for EulerFlow Moving Gauss 2D and SRMHD Alfven Wave 2D after the fix to make sure
we do not get the computation wrong again.
I further added a Gauss-Legendre quadrature based error computation
to the EulerFlow project in order to validate the Riemann quadrature values. Indeed, there is only a small
difference. Convergence is observed in both cases .
(I did not calculate the rates yet. Sven should have a script for that.)
Some debugging tips:
* **We need to make sure that always matching variables (cons.,prim.) are compared against eacht other -- also for the SRMHD convergence tests.)**
* **Even if the solver might get inaccurate in the long term, the initial data should always be better resolved on a finer mesh. We should always see
convergence in projecting the initial data!**
## EulerFlow moving Gauss 2D-- rho errors
settings: fused-algorithmic-steps=on,timestepping=globalfixed
### Gauss-Legendre quadrature:
maximum-mesh-size = 0.5
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.750382e-13 8.925639e-13 1.311617e-12 6.208367e-13 8.950533e-13
2 5.459310e-02 1.038389e-03 2.920092e-03 1.570339e-02 1.460953e-06 8.411925e-04
3 1.013872e-01 1.410669e-03 4.361094e-03 2.551770e-02 2.776317e-06 1.122370e-03
4 1.559803e-01 1.239346e-03 3.639293e-03 2.137039e-02 8.062877e-07 1.042290e-03
5 2.027744e-01 8.993649e-04 2.033884e-03 9.321796e-03 4.938160e-06 8.758699e-04
6 2.573675e-01 8.112109e-04 2.558304e-03 2.066562e-02 1.188077e-06 8.807274e-04
7 3.041615e-01 1.171406e-03 3.257146e-03 2.334808e-02 1.116484e-06 1.177089e-03
8 3.509556e-01 1.247573e-03 3.321656e-03 2.104360e-02 8.826953e-07 1.255401e-03
9 4.055487e-01 9.415592e-04 2.712697e-03 2.152044e-02 4.344998e-08 9.937017e-04
```
maximum-mesh-size = 0.15
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.772611e-13 8.949179e-13 1.507683e-12 6.206147e-13 8.976664e-13
2 5.199343e-02 1.824504e-04 1.173739e-03 1.840355e-02 1.319400e-11 1.656369e-04
3 1.013872e-01 1.463228e-04 8.135031e-04 1.585928e-02 9.383494e-12 1.586405e-04
4 1.507809e-01 1.511930e-04 7.129781e-04 7.640309e-03 6.468159e-13 1.499519e-04
5 2.001747e-01 1.799638e-04 1.039565e-03 1.571490e-02 7.177703e-12 1.593651e-04
6 2.521681e-01 2.472488e-04 1.460660e-03 2.225137e-02 1.960077e-11 2.191721e-04
7 3.015619e-01 2.123867e-04 9.078946e-04 1.156214e-02 2.549394e-11 2.037580e-04
8 3.509556e-01 1.996272e-04 1.025265e-03 1.816118e-02 3.995360e-11 2.028881e-04
9 4.003494e-01 2.200875e-04 1.407165e-03 2.126848e-02 6.046564e-10 1.847641e-04
```
maximum-mesh-size = 0.05
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.775711e-13 8.953091e-13 1.799227e-12 6.206147e-13 8.981400e-13
2 5.026031e-02 8.648108e-06 1.024604e-04 2.880816e-03 1.421085e-14 8.765676e-06
3 1.005206e-01 1.310812e-05 1.404085e-04 4.134064e-03 1.211253e-13 1.270370e-05
4 1.507809e-01 1.204385e-05 1.501490e-04 7.148238e-03 2.442491e-14 1.169658e-05
5 2.001747e-01 1.309551e-05 1.688337e-04 5.761659e-03 3.804734e-13 1.238790e-05
6 2.504350e-01 1.609669e-05 1.985765e-04 7.332591e-03 1.674216e-13 1.489681e-05
7 3.006953e-01 1.583313e-05 2.188010e-04 9.998774e-03 3.019807e-14 1.503079e-05
8 3.500891e-01 1.771167e-05 2.384444e-04 8.503132e-03 7.386314e-13 1.616133e-05
9 4.003494e-01 1.941804e-05 2.599394e-04 1.078235e-02 2.096101e-13 1.797866e-05
```
### Riemann quadrature:
maximum-mesh-size = 0.5
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.208791e-04 2.760006e-03 1.152340e-02 1.857846e-09 8.208791e-04
2 5.459310e-02 1.136775e-03 2.987134e-03 1.768960e-02 1.300473e-06 1.136775e-03
3 1.013872e-01 1.339213e-03 3.890847e-03 2.752230e-02 1.402174e-07 1.339213e-03
4 1.559803e-01 1.298609e-03 3.445100e-03 2.234650e-02 3.597158e-09 1.298609e-03
5 2.027744e-01 1.040780e-03 2.383341e-03 9.564163e-03 1.192050e-09 1.040780e-03
6 2.573675e-01 1.125659e-03 2.468552e-03 8.197815e-03 7.159261e-10 1.125659e-03
7 3.041615e-01 1.645793e-03 3.824891e-03 1.758752e-02 3.099943e-09 1.645793e-03
8 3.509556e-01 1.710023e-03 4.109781e-03 1.967037e-02 6.597448e-09 1.710023e-03
9 4.055487e-01 1.238782e-03 2.674109e-03 8.709187e-03 5.247924e-09 1.238782e-03
```
maximum-mesh-size = 0.15
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.976664e-13 9.187753e-13 1.507683e-12 6.206147e-13 8.976664e-13
2 5.199343e-02 1.656369e-04 1.006196e-03 1.840355e-02 1.319400e-11 1.656369e-04
3 1.013872e-01 1.586405e-04 8.668425e-04 1.585928e-02 9.383494e-12 1.586405e-04
4 1.507809e-01 1.499519e-04 7.055692e-04 7.640309e-03 6.468159e-13 1.499519e-04
5 2.001747e-01 1.593651e-04 8.780007e-04 1.571490e-02 7.177703e-12 1.593651e-04
6 2.521681e-01 2.191721e-04 1.259805e-03 2.225137e-02 1.960077e-11 2.191721e-04
7 3.015619e-01 2.037580e-04 8.841593e-04 1.156214e-02 2.549394e-11 2.037580e-04
8 3.509556e-01 2.028881e-04 1.060100e-03 1.816118e-02 3.995360e-11 2.028881e-04
9 4.003494e-01 1.847641e-04 1.146441e-03 2.126848e-02 6.046564e-10 1.847641e-04
```
maximum-mesh-size = 0.05
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 8.981400e-13 9.193660e-13 1.799227e-12 6.206147e-13 8.981400e-13
2 5.026031e-02 8.765676e-06 1.038935e-04 2.880816e-03 1.421085e-14 8.765676e-06
3 1.005206e-01 1.270370e-05 1.360760e-04 4.134064e-03 1.211253e-13 1.270370e-05
4 1.507809e-01 1.169658e-05 1.486293e-04 7.148238e-03 2.442491e-14 1.169658e-05
5 2.001747e-01 1.238790e-05 1.582185e-04 5.761659e-03 3.804734e-13 1.238790e-05
6 2.504350e-01 1.489681e-05 1.849682e-04 7.332591e-03 1.674216e-13 1.489681e-05
7 3.006953e-01 1.503079e-05 2.125449e-04 9.998774e-03 3.019807e-14 1.503079e-05
8 3.500891e-01 1.616133e-05 2.143398e-04 8.503132e-03 7.386314e-13 1.616133e-05
9 4.003494e-01 1.797866e-05 2.425857e-04 1.078235e-02 2.096101e-13 1.797866e-05
```
## SRMHD Alfven wave 2D -- vely errors
settings: fused-algorithmic-steps=off,timestepping=global
maximum-mesh-size = 0.5
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 1.547407e-03 2.445560e-03 4.573094e-03 2.602050e-05 1.547407e-03
2 6.015317e-02 1.111526e-03 1.657690e-03 3.769028e-03 3.275033e-06 1.111526e-03
3 1.002207e-01 1.398944e-03 1.869334e-03 4.319537e-03 2.836757e-04 1.398944e-03
```
maximum-mesh-size = 0.15
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 2.302968e-05 3.590189e-05 7.108230e-05 1.139464e-07 2.302968e-05
2 5.344862e-02 2.056201e-05 3.057057e-05 7.954177e-05 3.314441e-07 2.056201e-05
3 1.002171e-01 2.300635e-05 3.160379e-05 7.990042e-05 5.301505e-07 2.300635e-05
```
maximum-mesh-size = 0.05
```
plotindex time l1norm l2norm max min avg
1 0.000000e+00 2.913558e-07 4.523600e-07 9.036162e-07 4.721517e-10 2.913558e-07
2 5.122013e-02 3.221749e-07 4.326605e-07 1.331551e-06 5.221886e-09 3.221749e-07
3 1.002133e-01 3.809067e-07 5.525386e-07 1.662368e-06 1.168645e-09 3.809067e-07
```https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/120Cleanup generic kernels2018-06-15T15:13:43+02:00Ghost UserCleanup generic kernelsIts producing a lot of warnings, ie.
```
const int dofStartIndex = nodeIndex * numberOfVariables;
^
In file included from /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels....Its producing a lot of warnings, ie.
```
const int dofStartIndex = nodeIndex * numberOfVariables;
^
In file included from /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h:222:0,
from /media/storage/numrel/exahype/ExaHyPE-Engine/./ApplicationExamples/GRMHD/AbstractGRMHDSolver.cpp:13:
/media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/c/3d/solutionAdjustment.cpph: In instantiation of ‘void kernels::aderdg::generic::c::solutionAdjustment(SolverType&, double*, const tarch::la::Vector<3, double>&, const tarch::la::Vector<3, double>&, double, double) [with SolverType = GRMHD::GRMHDSolver]’:
/media/storage/numrel/exahype/ExaHyPE-Engine/./ApplicationExamples/GRMHD/AbstractGRMHDSolver.cpp:84:115: required from here
/media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/c/3d/solutionAdjustment.cpph:49:19: warning: unused variable ‘dofStartIndex’ [-Wunused-variable]
const int dofStartIndex = nodeIndex * numberOfVariables;
^
In file included from /media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/Kernels.h:227:0,
from /media/storage/numrel/exahype/ExaHyPE-Engine/./ApplicationExamples/GRMHD/AbstractGRMHDSolver.cpp:13:
/media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/c/3d/dummyK_Kernel.cpph: In instantiation of ‘void kernels::aderdg::generic::c::dummyK_Kernel(SolverType&, double, double, const tarch::la::Vector<3, double>&, const tarch::la::Vector<3, double>&, int, int, int, double*) [with SolverType = GRMHD::GRMHDSolver]’:
/media/storage/numrel/exahype/ExaHyPE-Engine/./ApplicationExamples/GRMHD/AbstractGRMHDSolver.cpp:115:209: required from here
/media/storage/numrel/exahype/ExaHyPE-Engine/./ExaHyPE/kernels/aderdg/generic/c/3d/dummyK_Kernel.cpph:30:10: warning: unused variable ‘x0’ [-Wunused-variable]
double x0[3];
```
also get rid of `malloc`, use stack.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/72MHD crashes after few timsteps2018-06-15T15:13:43+02:00Ghost UserMHD crashes after few timstepsThis was urgently raised last week by Michael Dumbser: The MHD AlfenWave is still crashing very quickly. We have to find the reason for that, otherwise there will be no convergence result next week.This was urgently raised last week by Michael Dumbser: The MHD AlfenWave is still crashing very quickly. We have to find the reason for that, otherwise there will be no convergence result next week.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/119DummyK2018-06-15T15:13:43+02:00Ghost UserDummyKQuestions:
* Do we consider point sources here?
* If so, why do we need an additional isDummyK() boolean getter here?
* If so, why is a point source not related to normal sources and needs its extra memory (see mapping Prediction?)
...Questions:
* Do we consider point sources here?
* If so, why do we need an additional isDummyK() boolean getter here?
* If so, why is a point source not related to normal sources and needs its extra memory (see mapping Prediction?)
We already allocate memory for the source terms in the temporary variables for space-time and spatial fluxes.https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/70Discussion topics for a discussion ("Coding") day at Oct 23., 2016 in Munich2018-06-15T15:13:43+02:00Ghost UserDiscussion topics for a discussion ("Coding") day at Oct 23., 2016 in Munich* Kernel situation
* Trento will not be able to implement new numerics into C++ "nested loop" kernels
* We invested two weeks of finding bugs porting Fortran Z4 to ExaHyPE -- this is a dead end!
* We either need modern Fortran kern...* Kernel situation
* Trento will not be able to implement new numerics into C++ "nested loop" kernels
* We invested two weeks of finding bugs porting Fortran Z4 to ExaHyPE -- this is a dead end!
* We either need modern Fortran kernels or a good C++ linear algebra library (there are plenty of them)
* (Generic) research kernels might be in a different shape than optimized kernels!
* What is ExaHyPE?
* If it shall be a standalone *application*, then it should provide the best support for parameters, etc. as possible
* If it shall serve as a "utility", end users have to come up with individual solutions on theirselves. Both is fine, but there should be a common idea everybody is aware of.
* Workflow
* Durham cannot provide implementation support (code monkeying) for ExaHyPE applications in this intensity anymore
* Frankfurt cannot provide support for running benchmarks and (parameter) test in this intensity anymore
* Fortran situation in Applications
* MHD was converted to C++ with surprising results in stability and speed
* What's the joint aim for the team? There is more and more Fortran code (Applications get bigger), cf. the `Z4` application
* API and Code Generation
* Kernel calls should be method calls instead of templated static function calls
* Toolkit only needs to provide initial "scaffold" code. In 90% of applications, generated code is now in the repo, edited after generation and gets overwritten regularly.
* Toolkit should not distribute constants in the generated code but collect them in a unique file (`GeneratedConstants.h`). Paradigm: Generate maintainable code.
* Specification file format
* There is a lot to say here. We should aggree on a whishlist of features
* We could also agree to keep the spec files small and let the problem be solved by the users
* Treatment of external libraries and the build system
* Today is already tomorrow -- I have secretly introduced dependencies on GSL, BLAS due to the Astrophysics Initial Data codes.
* A decision should also affect the treatment of Peano.
* IO
* VTK Fileformat (alternatives)
* Does it make sense to define a tailored ExaHyPE output format ourselves?
* Efficient batch processing 3D VTK to movie plotting
* Unit tests
* Mathematically/Physically, there is little point in testing the numerical scheme with random data at every ExyHyPE `DEBUG or Asserts` run.
* Instead, simple but well understood tests with analytic solutions should be run (ie. via Jenkins). We should come up with a list of these tests (ie: Advection, ShuVortex, MHD Wave, Gauge Wave, etc.)
*Please just add points to this list as you whish*
# Assignments in Discussion at 25. Oct 2016
At Oct 25, in Garching there were Tobias, Vasco, Dominik, Alejandro and Sven. We discussed these points and even more and concluded these assignments:
* Fortran support (VV)
* Call linear kernels with temp arrays (DC)
* Non-conservative parts (SK) - done, cf. https://gitlab.lrz.de/gi26det/ExaHyPE/commit/1723512d16acbf118f43465d78d252cda55e9bc6 and similar
* Split of the repositories (SK) - tracked at gi26det/ExaHyPE#55
* Start the release page & remove binaries from GIT (VV) - as followup for gi26det/ExaHyPE#55, right?
* Find new Jour Fix time (incl. Dominic and Trento) (VV)
* Nightly convergence tests (SK+VV) - tracked at gi26det/ExaHyPE#74
* Period BCs: Organise small coding week in Durham (VV+TW)
* Replace static function + templates with std::function (SK+TW) - tracked at gi26det/ExaHyPE#75
* Extend Solver::solutionAdjustment to handle point sources (DC+VV)https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/103Run long EulerFlow simulation to understand constant plotting issue and inclu...2018-06-15T15:13:43+02:00Ghost UserRun long EulerFlow simulation to understand constant plotting issue and include CSV dt outputzweiter. Diskussionspunkt: "Zeitschritte" im Sinne des "Constantly Plottings" bzw. "Output every nth iteration", verhalten sich anders als erwartet. Wir vergleichen das reale Erstellen des Plots ab Simulationsstart mit dem erwarteten ide...zweiter. Diskussionspunkt: "Zeitschritte" im Sinne des "Constantly Plottings" bzw. "Output every nth iteration", verhalten sich anders als erwartet. Wir vergleichen das reale Erstellen des Plots ab Simulationsstart mit dem erwarteten idealen Erstellen des Plots. Die Messung nehmen wir von exahype::plotters::Plotter::UserOnTheFlyPostProcessing::startPlotting(double time) ab.
Zwei Beobachtungen:
* (i) die Kurve tIdeal-tReal ist mal <0, mal >0. Messfehler?
* (ii) Die Kurve tIdeal-tReal weist einen Trend propertional zur Simulationszeit auf. Fehler in ExaHyPE?
2a) Tobias schlägt vor, die Simulation länger laufen zu lassen, um mehr Daten zu haben. Kann ich mit EulerFlow/ShuVortex + TBB schlecht machen weil sich der 2D-Vortex in der Diagonalen bewegt, Simulationsdomäne damit quadratisch zur Zeit und ich keinen Benchmark mehr machen will der einen Tag lang läuft. => Benötige MPI-Reductions. Wird gemacht.
**2b) Einbau eines CSV-Plotters für Zeitschritte direkt in den Code, damit ich nicht die stdout-Ausgaben parsen muss. Kann ich machen.**https://gitlab.lrz.de/exahype/ExaHyPE-Engine/-/issues/118Grid extends are wrong in 3D2018-06-15T15:13:43+02:00Ghost UserGrid extends are wrong in 3DLook at this screenshot, the Z range specified in the specfile is not correctly interpreted.
![exahype-grid-size](/uploads/fcc13d63a1b6e1fa14c54b72d27500e1/exahype-grid-size.png)Look at this screenshot, the Z range specified in the specfile is not correctly interpreted.
![exahype-grid-size](/uploads/fcc13d63a1b6e1fa14c54b72d27500e1/exahype-grid-size.png)