Peano Heaps and Intel MPI do not work properly together

(Open MPI seems to work.)

Single-node MPI tests show:

• MPI is currently not working correctly.

• TBB seems to work correctly.

It's always fun to debug MPI...

1 MPI+TBB

1.1

Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t1-c24.out

 8.68453      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 0 t_min          =0
 8.68455      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 8.68459      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 9.77685      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 1 t_min          =0.000270382
 9.77688      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 9.7769       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 10.7874      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 2 t_min          =0.000540765
 10.7875      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269703
 10.7875      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 11.8094      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 3 t_min          =0.000810468
 11.8094      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269363
 11.8095      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 12.8234      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 4 t_min          =0.00107983
 12.8234      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269194
 12.8235      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 13.8676      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 5 t_min          =0.00134902
 13.8676      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269109
 13.8676      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 14.8835      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 6 t_min          =0.00161813
 14.8835      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269066
 14.8836      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 15.8789      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 7 t_min          =0.0018872
 15.879       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269045
 15.879       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 16.9168      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 8 t_min          =0.00215624
 16.9168      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269034
 16.9168      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 17.9286      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 9 t_min          =0.00242528
 17.9286      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269029
 17.9286      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 18.9628      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 10        t_min          =0.00269431
 18.9629      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269026
 18.9629      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 19.9945      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 11        t_min          =0.00296333
 19.9945      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269025
 19.9955      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 21.0106      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 12        t_min          =0.00323236
 21.0107      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269024
 21.0107      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of mesh refinements      = 0
 21.0107      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of local recomputations  = 0
 21.0107      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of global recomputations = 0
 21.0107      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of predictor reruns      = 0

1.2

Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t24-c1.out

 5.84491      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::createGrid(Repository)        finished grid setup after 17 iterations
 6.1414       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runAsMaster(...)              initialised all data and computed first time step size
 8.1216       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runAsMaster(...)              plotted initial solution (if specified) and computed first predictor
 8.12165      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 0 t_min          =0
 8.12167      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 8.12171      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 11.0395      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 13.2524      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 1 t_min          =0.000270382
 13.2525      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000251415 !!! DIFFERENCE !!!
 13.2525      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 16.445       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 18.9004      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 2 t_min          =0.000521798
 18.9005      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000242749
 18.9005      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 22.8658      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 25.8135      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 3 t_min          =0.000764547
 25.8136      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000235312
 25.8136      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 30.957       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 4 t_min          =0.000999859
 30.9571      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000235312

 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 13        t_min          =0.00315762
 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000242922
 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of mesh refinements      = 0
 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of local recomputations  = 0
 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of global recomputations = 0
 156.472      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of predictor reruns      = 6

1.3

Euler_ADERDG-no-output-gen-fused-regular-0-p5-TBB-Intel-n1-t4-c6.out

 16.3129      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 0 t_min          =0
 16.3129      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 16.313       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 25.2424      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 31.3638      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 1 t_min          =0.000270382
 31.3638      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000255681 !!! DIFFERENCE !!!
 31.3639      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 40.3852      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 2 t_min          =0.000526063
 40.3853      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000255681
 40.3853      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 49.7392      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 57.1134      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 3 t_min          =0.000781745
 57.1135      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000248662
 57.1135      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 67.3553      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 4 t_min          =0.00103041
 67.3554      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000248662
...
...
...
 175.134      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 12        t_min          =0.00311021
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.00026858
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of mesh refinements      = 0
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of local recomputations  = 0
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of global recomputations = 0
 193.899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of predictor reruns      = 2

2 MPI+None

2.1

Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t1-c24.out 

 17.7097      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 0 t_min          =0
 17.7097      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 17.7098      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 28.1517      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 1 t_min          =0.000270382
 28.1518      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000270382
 28.1518      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 38.5223      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 2 t_min          =0.000540765
 38.5223      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269703
 38.5223      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 48.7899      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 3 t_min          =0.000810468
 48.79        [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269363
 48.79        [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 59.1023      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 4 t_min          =0.00107983
 59.1023      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269194
...
...
...
 130.907      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 12        t_min          =0.00323236
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000269024
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of mesh refinements      = 0
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of local recomputations  = 0
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of global recomputations = 0
 141.16       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of predictor reruns      = 0

2.2

Euler_ADERDG-no-output-gen-fused-regular-0-p5-None-Intel-n1-t24-c1.out

 14.0067      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 16.5843      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...)                  recompute space-time predictor
 19.2036      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 1 t_min          =0.000270382
 19.2036      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000212189
 19.2037      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 21.8836      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 2 t_min          =0.000482572
 21.8836      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000212189
 21.8836      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 25.2896      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 3 t_min          =0.000694761
 25.2896      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000223607
 25.2897      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 29.067       [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 4 t_min          =0.000918368
 29.0671      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000229512
...
...
...
 109.009      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::runOneTimeStepWithFusedAlgorithmicSteps(...) run 1 iterations with fused algorithmic steps
 125.061      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)         step 13        t_min          =0.00303507
 125.061      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::startNewTimeStep(...)                dt_min         =0.000243652
 125.061      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of mesh refinements      = 0
 125.062      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of local recomputations  = 0
 125.062      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of global recomputations = 0
 125.062      [cn7063.hpc.dur.ac.uk],rank:0 info         exahype::runners::Runner::printStatistics(...)          number of predictor reruns      = 1