Bounding Box Scaling (virtually-expand-domain) Does Not Work

Virtually expanding the bounding box around the computational domain is a Peano trick to shut off neighbour communication with the global master (rank 0).

Virtually expanding the bounding box does currently lead to problems in ExaHyPE:

• It enables scenarios where a coarse grid vertex is on the boundary/outside of the domain, but a fine grid vertex (and its h environment) is inside of the domain.
• It confuses the isFaceInside function in class Cell.

Virtually Expanding the Domain

Some background on the virtually expand domain flag:

• Peano does only consider inside and boundary vertices for the (MPI) neighbour merging. Outside vertices are ignored for this purpose.
• Since rank 1 is placed into a centre of 3^d child cells belonging to rank 0, it will perform neighbour merging with rank 0 as long as those vertices are
  either inside or directly at the boundary of the domain.
• Switching off neighbour merging directly at the domain boundary (vertex.isBoundary()) does not make sense. The reason is that refinement at the boundary will introduce hanging nodes. Boundary nodes should however be persistent. (Tobias' reasoning. Have to ask further why this is bad.)
• Virtually expanding the domain does place the nodes located at the remote boundary to rank 0 outside of the domain
• From the above points, it is clear that virtually expanding the boundary is mandatory for reasonable MPI scalability especially in 3d. This is exactly what we have observed in our 3D MPI experiments.
• This will be a little inconvenient for people who prescribe initial conditions at certain boundaries by means of (x,t). (Seismic people know about this. Leonhard knows how to deal with it.)

Remarks

• Virtually expanding the bounding box usually leads to a shrinking of the actual computational domain since only inside cells are considered as within the computational domain. ExaHyPE should thus tell the user what the shrinked domain looks like.