02a_Demonstrators-Workflow.tex 12.4 KB
Newer Older
1
\section{High-level overview of \exahype\ workflow}
2
\exahype\ is designed in such a way that most of the implementation details are hidden away from the user in the heart of an engine, i.e. \texttt{ExaHyPE} directory. A connecting piece between the user defined application \texttt{UserApplication} and \texttt{ExaHyPE} engine is the \texttt{Toolkit} utility. \texttt{Toolkit} fulfils the purpose of the 'configure' step present in some tools and thus comes before the compilation. \texttt{Toolkit} is responsible for decoding of \textit{configuration file} and generating \textit{glue code} and \textit{make system} for your \exahype\ applications. The so called \textit{glue code} establishes all appropriate connections between \texttt{UserApplication} and \texttt{ExaHyPE} intrinsics to make sure \texttt{UserApplication} can run smoothly. Among files generated by the \texttt{Toolkit}, some are meant for you to interact with, i.e. you need to provide in these files your implementation of a system of PDEs to be solved. Once it is done, you are ready to compile the application using a \texttt{Makefile} generated by the \texttt{Toolkit}. This will result in an executable \texttt{ExaHyPE-UserApplication} appearing in your project's directory. At this point you can run your application from \texttt{UserApplication} directory by typing in \texttt{./ExaHyPE-UserApplication} and passing your \textit{configuration file} as an argument. In terms of command line the process is realised as shown in the code snippet below:
3
4
5
6
7
8
9
10
\begin{code}
> cd ExaHyPE-Engine
> ./Toolkit/toolkit.sh UserApplication/*.exahype
> cd UserApplication
> make
> ./ExaHyPE-UserApplication *.exahype
\end{code}
\noindent
11
12
13
14
15
16
17
18
19
20
21
22
23
Depending on your system, you might have to change some environment variables. \exahype\ by default for example wants to use an Intel compiler and builds a release mode, i.e. a highly optimised code variant. To change this, type in before \texttt{make}:
\begin{code}
export MODE=Asserts
export COMPILER=GNU 
\end{code}
\noindent
All these environment variables are enlisted by the toolkit, together with recommended settings for your system as specification file co-determines which variant of
the \exahype\ engine is actually built. You can always check / reconstruct these settings by passing in \texttt{--version} to your executable instead of the configuration file:
\begin{code}
./ExaHyPE-UserApplicaiton --version
\end{code}
\noindent
The high-level overview of the \exahype\ workflow has been visualised in Figure \ref{fig:workflow}.
24

25
\afterpage{\clearpage}
Anne Reinarz's avatar
Anne Reinarz committed
26
27
%\begin{landscape}
\begin{figure}
28
29
30
31
\centering
\includegraphics[width=.8\textwidth]{tikz/demonstrators-workflow.pdf}
\includegraphics[width=.8\textwidth]{tikz/demonstrators-legend.pdf}
\caption[Sketch (tikz): High-level overview of \exahype\ workflow]{High-level overview of \exahype\ workflow and files generated by the application. \label{fig:workflow}}
Anne Reinarz's avatar
Anne Reinarz committed
32
33
\end{figure}
%\end{landscape}
34
\newpage
35
\subsection{Configuration file}
36
37
38
39
40
41

An \exahype\ configuration / specification is a text file where you specify exactly what you want to solve in which way. This file\footnote{Some examples can be found in the Toolkit directory or on the \exahype\ webpage. They have the extension \texttt{.exahype}.} is handed over to our \exahype\ \texttt{Toolkit}, a small python 3 application generating all required code. This code (also linking to all other required resources such as parameter sets or input files) then is handed over to a compiler. Depending on the option used, see \ref{chapter:a-toolkit}, you end up with an executable that may run without the python toolkit or any additional sources from \exahype. It however rereads the specification file again for input files or some
parameters. We could have worked with two different files, a specification file used to generate code and a config file, but decided to have everything in one place.\\
\vspace{0.1cm}
\noindent
\textit{Configuration file is hence a centrepiece of the workflow.} As mentioned, it is used for two purposes:
42
43
\begin{enumerate}
\vspace{0.1cm}
44
\item to provide \textbf{configuration} for \texttt{Toolkit} so that the correct context can be generated
45
\vspace{0.1cm}
46
\item to provide \textbf{specification} to \texttt{ExaHyPE-UserApplication} executable at runtime
47
\end{enumerate}
48
\vspace{0.2cm}
49
\noindent
50
51
52
53
54
55
56
57
58
An \exahype\ specification file acts on the one hand as classic configuration files passed into the executable for a run. It holds all the constants (otherwise often passed via command lines),
output file names, core numbers, and so forth. On the other hand, a specification file defines the solver's characteristics such as numerical scheme used, polynomial order, used postprocessing steps, and so forth. Specification files translate the \exahype\ engine into an \exahype\ application. As they realise a single point-of-contact policy, \exahype\ fosters reproducability. We minimise the number of files required to uniquely define and launch an application. As the specification files also describe the used machine (configurations), they enable \exahype\ to work with machine-tailored engine instances. The engine can, at hands of the specification, optimise aggressively. If in doubt, \exahype\ prefers to run the compiler rather than to work with few generic executables.
\begin{design}
\exahype\ users typically write one code specification per file experiment plus machine plus setup combination.  
\end{design}

\subsubsection{Supported configuration file formats}

For now it should not be any reason to worry, but it is a good place to mention, that \exahype\ currently supports two main configuration file formats, i.e.:
59
60
61
62
63
\begin{enumerate}
\vspace{0.1cm}
\item \textbf{\texttt{*.exahype}}\\
\textit{legacy specification file format}
\vspace{0.1cm}
64
\item \textbf{\texttt{*.exahype2 (*.json / *.hjson}})\\
65
66
\textit{recommended new specification file format}
\end{enumerate}
67
\vspace{0.1cm}
68
\noindent
69
The need to develop support for a new configuration file format was dictated by the technical limitations of the previous version of the \texttt{Toolkit} written in \texttt{Java}, which put some restrictions on configuration files in terms of their size and number of variables that could be processed. A new \texttt{Toolkit} has been developed in \texttt{Python}, among other reasons, also in order to reduce the number of \exahype\ dependencies and becasue it provides conveniant means of handling popular configuration file formats.\\
70
\noindent
71
72
73
74
Typically, when coming into contact with a new programme, users often need to familiarise themselves with an application specific syntax for writing configuration files, as was the case with \texttt{*.exahype} configuration files. This makes the learning curve for the new application steeper. However, there exist some well established configuration file languages, which are commonly used across many \textit{open source} projects. One of such configuration file languages is \texttt{JSON}. Because \texttt{Python} provides \texttt{JSON} support and there is a high chance that many of the new \exahype\ developers and users are already familiar with the syntax, \texttt{JSON} has been chosen as the new configuration file format.\\
\noindent
For now, the \texttt{Toolkit} still supports both flavours of configuation files, i.e. the legacy \texttt{*.exahype} as well as the new and preferred \texttt{*.exahype2}. For each of the configuration file formats, \texttt{Toolkit} provides a separate \textit{reader}, which interprets the syntax and returns a data structure in the form of a dictionary. This structure is subsequently validated against a defined \texttt{JSON}-Schema and enriched with default values from the schema, if needed. In other words, internally, the \texttt{*.exahype} syntax is translated to \texttt{*.exahype2} anyway before the validation step is performed. Hence, \texttt{*.exahype2} should be seen as the preferred configuration file format and it can be expected that in future \exahype\ releases support for the legacy configuration file format will be gone!

75
\begin{info}
76
\small For details regarding the \textit{legacy} configuration file format \texttt{*.exahype}, users can refer to provided \texttt{JINJA} template. It can be found under the following path: \texttt{ExaHyPE-Engine/Toolkit/exahype/} \texttt{specfiles/templates/} \texttt{specfile1.exahype.template}.
77
78
\end{info}
\begin{info}
79
\small For details regarding the \textit{new recommended} configuration file format \texttt{*.exahype2}, users can have a look at a provided \texttt{JSON}-Schema. It can be found under the following path: \texttt{ExaHyPE-Engine/} \texttt{Toolkit/exahype-specfile.schema.json}.
80
81
\end{info}

82
83
84
\subsubsection{Exploring the syntax}

As an example illustrating the basic syntax, users might want to analyse a \texttt{TrivialProject} configuration / specification file to be found in \texttt{Demonstrators} subdirectory of \exahype. Most parameters should be rather self-explanatory. For your own projects, you might have to adopt the paths \footnote{You may specify pathes of individual components in more detail (cmp.~Section \ref{chapter:finetuning}), but for most applications working with the three pathes above should be sufficient.}. Note that \exahype\ supports both two- and three-dimensional setups.
85
\begin{code}
86
87
88
89
90
91
92
93
94
95
96
97
exahype-project  TrivialProject
  peano-kernel-path const    = ./Peano
  exahype-path const         = ./ExaHyPE
  output-directory const     = ./Demonstrators/TrivialProject
  
  computational-domain
    dimension const          = 2
    width                    = 1.0, 1.0
    offset                   = 0.0, 0.0
    end-time                 = 10.0
  end computational-domain
end exahype-project  
98
\end{code}
99
100
101
102
103
104
105
\begin{design}
The \texttt{*.exahype} specification file parameters that are interpreted by the toolkit are marked as \texttt{const}. If constant parameters or the structure of the file (all regions terminated by an \texttt{end}) change, you have to rerun the toolkit. All other variables are read at runtime, i.e.~you may alter them without a rerun of the toolkit or a re-compile.
\end{design}
\vspace{0.1cm}
\noindent
If you run various experiments, you start from one specification file and build up your application from hereon. You can either pass the original specification file to your excutable, or you can pass other variants of it as well. That is, you may work with sets of specification files all feeding into one executable. However, the specification files have to match each other in terms of const
variables and their structure. \exahype\ itself checks for consistent files in many places and, for many invalid combinations, complains.
106
107

\noindent
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
Due to the fact that many users have their cases already set up using the legacy configuration file format, \exahype\ provides a means of translating \texttt{*.exahype} to \texttt{*.exahype2} format. In particular, all legacy configuration files with extension \texttt{*.exahype} can be converted into \texttt{*.exahype2} format using the \texttt{-j} flag of the \texttt{Toolkit}. Using it will write a \texttt{JSON} file with a similar filename and \texttt{*.exahype2} extention, if  legacy specification file is read. To keep things simple and expolore some of the syntactic differences between the old and new configuration file format, you might want to first translate \texttt{TrivialProject} configuration file into the new format as follows:
\begin{code}
> cd ExaHyPE-Engine
> ./Toolkit/toolkit.sh -j Demonstrators/TrivialProject/TrivialProject.exahype
\end{code}
\noindent
You should now be able to find \texttt{TrivialProject.exahype2} file in \texttt{TrivialProject} subdirectory. The file should have the following content:
\begin{code}
{
  "project_name": "TrivialProject",
  "paths": {
    "peano_kernel_path": "./Peano",
    "exahype_path": "./ExaHyPE",
    "output_directory": "./Demonstrators/TrivialProject"
  },
  "computational_domain": {
    "dimension": 2,
    "end_time": 10.0,
    "offset": [
      0.0,
      0.0
    ],
    "width": [
      1.0,
      1.0
    ]
  },
  "solvers": []
}
\end{code}
\noindent
More details regarding specification file setup and all the parameters can be found in Section \ref{sec:solver-configuration}. It might still refer to the legacy specification file. However, there is planty of information available in the Internet regarding syntax of \texttt{JSON} configuration file language. If you take into consideration that all dashes ''-'' in legacy parameter names have now been changed into underscres ''\_'' in \texttt{JSON} syntax and that the \texttt{const} indicators are gone, you should be able to find your way and use the new syntax easily.