Peano Heaps and Intel MPI do not work properly together
(Open MPI seems to work.)
Single-node MPI tests show:
-
MPI is currently not working correctly.
-
TBB seems to work correctly.
It's always fun to debug MPI...
1 MPI+TBB
1.1
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t1-c24.out
8.68453 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
8.68455 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
8.68459 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
9.77685 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
9.77688 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
9.7769 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
10.7874 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000540765
10.7875 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269703
10.7875 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
11.8094 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000810468
11.8094 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269363
11.8095 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
12.8234 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00107983
12.8234 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269194
12.8235 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 5 t_min =0.00134902
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269109
13.8676 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
14.8835 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 6 t_min =0.00161813
14.8835 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269066
14.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
15.8789 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 7 t_min =0.0018872
15.879 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269045
15.879 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 8 t_min =0.00215624
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269034
16.9168 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 9 t_min =0.00242528
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269029
17.9286 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
18.9628 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 10 t_min =0.00269431
18.9629 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269026
18.9629 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
19.9945 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 11 t_min =0.00296333
19.9945 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269025
19.9955 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
21.0106 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00323236
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269024
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
21.0107 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 0
1.2
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t24-c1.out
5.84491 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::createGrid(Repository) finished grid setup after 17 iterations
6.1414 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runAsMaster(...) initialised all data and computed first time step size
8.1216 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runAsMaster(...) plotted initial solution (if specified) and computed first predictor
8.12165 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
8.12167 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
8.12171 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
11.0395 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
13.2524 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
13.2525 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000251415 !!! DIFFERENCE !!!
13.2525 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.445 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
18.9004 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000521798
18.9005 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000242749
18.9005 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
22.8658 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
25.8135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000764547
25.8136 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000235312
25.8136 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
30.957 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.000999859
30.9571 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000235312
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 13 t_min =0.00315762
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000242922
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
156.472 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 6
1.3
Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t4-c6.out
16.3129 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
16.3129 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
16.313 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
25.2424 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
31.3638 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
31.3638 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000255681 !!! DIFFERENCE !!!
31.3639 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
40.3852 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000526063
40.3853 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000255681
40.3853 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
49.7392 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
57.1134 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000781745
57.1135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000248662
57.1135 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
67.3553 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00103041
67.3554 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000248662
...
...
...
175.134 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00311021
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.00026858
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
193.899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 2
2 MPI+None
2.1
Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t1-c24.out
17.7097 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 0 t_min =0
17.7097 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
17.7098 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
28.1517 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
28.1518 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000270382
28.1518 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000540765
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269703
38.5223 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
48.7899 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000810468
48.79 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269363
48.79 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
59.1023 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.00107983
59.1023 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269194
...
...
...
130.907 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 12 t_min =0.00323236
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000269024
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
141.16 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 0
2.2
Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t24-c1.out
14.0067 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
16.5843 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) recompute space-time predictor
19.2036 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 1 t_min =0.000270382
19.2036 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000212189
19.2037 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 2 t_min =0.000482572
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000212189
21.8836 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
25.2896 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 3 t_min =0.000694761
25.2896 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000223607
25.2897 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
29.067 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 4 t_min =0.000918368
29.0671 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000229512
...
...
...
109.009 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) step 13 t_min =0.00303507
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::startNewTimeStep(...) dt_min =0.000243652
125.061 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of mesh refinements = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of local recomputations = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of global recomputations = 0
125.062 [cn7063.hpc.dur.ac.uk],rank:0 info exahype::runners::Runner::printStatistics(...) number of predictor reruns = 1