02a_Demonstrators-Workflow.tex 12.4 KB
 Olga Glogowska committed Aug 21, 2019 1 \section{High-level overview of \exahype\ workflow}  2 \exahype\ is designed in such a way that most of the implementation details are hidden away from the user in the heart of an engine, i.e. \texttt{ExaHyPE} directory. A connecting piece between the user defined application \texttt{UserApplication} and \texttt{ExaHyPE} engine is the \texttt{Toolkit} utility. \texttt{Toolkit} fulfils the purpose of the 'configure' step present in some tools and thus comes before the compilation. \texttt{Toolkit} is responsible for decoding of \textit{configuration file} and generating \textit{glue code} and \textit{make system} for your \exahype\ applications. The so called \textit{glue code} establishes all appropriate connections between \texttt{UserApplication} and \texttt{ExaHyPE} intrinsics to make sure \texttt{UserApplication} can run smoothly. Among files generated by the \texttt{Toolkit}, some are meant for you to interact with, i.e. you need to provide in these files your implementation of a system of PDEs to be solved. Once it is done, you are ready to compile the application using a \texttt{Makefile} generated by the \texttt{Toolkit}. This will result in an executable \texttt{ExaHyPE-UserApplication} appearing in your project's directory. At this point you can run your application from \texttt{UserApplication} directory by typing in \texttt{./ExaHyPE-UserApplication} and passing your \textit{configuration file} as an argument. In terms of command line the process is realised as shown in the code snippet below:  Olga Glogowska committed Aug 21, 2019 3 4 5 6 7 8 9 10 \begin{code} > cd ExaHyPE-Engine > ./Toolkit/toolkit.sh UserApplication/*.exahype > cd UserApplication > make > ./ExaHyPE-UserApplication *.exahype \end{code} \noindent  11 12 13 14 15 16 17 18 19 20 21 22 23 Depending on your system, you might have to change some environment variables. \exahype\ by default for example wants to use an Intel compiler and builds a release mode, i.e. a highly optimised code variant. To change this, type in before \texttt{make}: \begin{code} export MODE=Asserts export COMPILER=GNU \end{code} \noindent All these environment variables are enlisted by the toolkit, together with recommended settings for your system as specification file co-determines which variant of the \exahype\ engine is actually built. You can always check / reconstruct these settings by passing in \texttt{--version} to your executable instead of the configuration file: \begin{code} ./ExaHyPE-UserApplicaiton --version \end{code} \noindent The high-level overview of the \exahype\ workflow has been visualised in Figure \ref{fig:workflow}.  Olga Glogowska committed Aug 21, 2019 24   25 \afterpage{\clearpage}  Anne Reinarz committed Oct 01, 2019 26 27 %\begin{landscape} \begin{figure}  28 29 30 31 \centering \includegraphics[width=.8\textwidth]{tikz/demonstrators-workflow.pdf} \includegraphics[width=.8\textwidth]{tikz/demonstrators-legend.pdf} \caption[Sketch (tikz): High-level overview of \exahype\ workflow]{High-level overview of \exahype\ workflow and files generated by the application. \label{fig:workflow}}  Anne Reinarz committed Oct 01, 2019 32 33 \end{figure} %\end{landscape}  34 \newpage  Olga Glogowska committed Aug 21, 2019 35 \subsection{Configuration file}  36 37 38 39 40 41  An \exahype\ configuration / specification is a text file where you specify exactly what you want to solve in which way. This file\footnote{Some examples can be found in the Toolkit directory or on the \exahype\ webpage. They have the extension \texttt{.exahype}.} is handed over to our \exahype\ \texttt{Toolkit}, a small python 3 application generating all required code. This code (also linking to all other required resources such as parameter sets or input files) then is handed over to a compiler. Depending on the option used, see \ref{chapter:a-toolkit}, you end up with an executable that may run without the python toolkit or any additional sources from \exahype. It however rereads the specification file again for input files or some parameters. We could have worked with two different files, a specification file used to generate code and a config file, but decided to have everything in one place.\\ \vspace{0.1cm} \noindent \textit{Configuration file is hence a centrepiece of the workflow.} As mentioned, it is used for two purposes:  Olga Glogowska committed Aug 21, 2019 42 43 \begin{enumerate} \vspace{0.1cm}  44 \item to provide \textbf{configuration} for \texttt{Toolkit} so that the correct context can be generated  Olga Glogowska committed Aug 21, 2019 45 \vspace{0.1cm}  46 \item to provide \textbf{specification} to \texttt{ExaHyPE-UserApplication} executable at runtime  Olga Glogowska committed Aug 21, 2019 47 \end{enumerate}  48 \vspace{0.2cm}  Olga Glogowska committed Aug 21, 2019 49 \noindent  50 51 52 53 54 55 56 57 58 An \exahype\ specification file acts on the one hand as classic configuration files passed into the executable for a run. It holds all the constants (otherwise often passed via command lines), output file names, core numbers, and so forth. On the other hand, a specification file defines the solver's characteristics such as numerical scheme used, polynomial order, used postprocessing steps, and so forth. Specification files translate the \exahype\ engine into an \exahype\ application. As they realise a single point-of-contact policy, \exahype\ fosters reproducability. We minimise the number of files required to uniquely define and launch an application. As the specification files also describe the used machine (configurations), they enable \exahype\ to work with machine-tailored engine instances. The engine can, at hands of the specification, optimise aggressively. If in doubt, \exahype\ prefers to run the compiler rather than to work with few generic executables. \begin{design} \exahype\ users typically write one code specification per file experiment plus machine plus setup combination. \end{design} \subsubsection{Supported configuration file formats} For now it should not be any reason to worry, but it is a good place to mention, that \exahype\ currently supports two main configuration file formats, i.e.:  Olga Glogowska committed Aug 21, 2019 59 60 61 62 63 \begin{enumerate} \vspace{0.1cm} \item \textbf{\texttt{*.exahype}}\\ \textit{legacy specification file format} \vspace{0.1cm}  64 \item \textbf{\texttt{*.exahype2 (*.json / *.hjson}})\\  Olga Glogowska committed Aug 21, 2019 65 66 \textit{recommended new specification file format} \end{enumerate}  67 \vspace{0.1cm}  Olga Glogowska committed Aug 21, 2019 68 \noindent  69 The need to develop support for a new configuration file format was dictated by the technical limitations of the previous version of the \texttt{Toolkit} written in \texttt{Java}, which put some restrictions on configuration files in terms of their size and number of variables that could be processed. A new \texttt{Toolkit} has been developed in \texttt{Python}, among other reasons, also in order to reduce the number of \exahype\ dependencies and becasue it provides conveniant means of handling popular configuration file formats.\\  Olga Glogowska committed Aug 21, 2019 70 \noindent  71 72 73 74 Typically, when coming into contact with a new programme, users often need to familiarise themselves with an application specific syntax for writing configuration files, as was the case with \texttt{*.exahype} configuration files. This makes the learning curve for the new application steeper. However, there exist some well established configuration file languages, which are commonly used across many \textit{open source} projects. One of such configuration file languages is \texttt{JSON}. Because \texttt{Python} provides \texttt{JSON} support and there is a high chance that many of the new \exahype\ developers and users are already familiar with the syntax, \texttt{JSON} has been chosen as the new configuration file format.\\ \noindent For now, the \texttt{Toolkit} still supports both flavours of configuation files, i.e. the legacy \texttt{*.exahype} as well as the new and preferred \texttt{*.exahype2}. For each of the configuration file formats, \texttt{Toolkit} provides a separate \textit{reader}, which interprets the syntax and returns a data structure in the form of a dictionary. This structure is subsequently validated against a defined \texttt{JSON}-Schema and enriched with default values from the schema, if needed. In other words, internally, the \texttt{*.exahype} syntax is translated to \texttt{*.exahype2} anyway before the validation step is performed. Hence, \texttt{*.exahype2} should be seen as the preferred configuration file format and it can be expected that in future \exahype\ releases support for the legacy configuration file format will be gone!  Olga Glogowska committed Aug 21, 2019 75 \begin{info}  76 \small For details regarding the \textit{legacy} configuration file format \texttt{*.exahype}, users can refer to provided \texttt{JINJA} template. It can be found under the following path: \texttt{ExaHyPE-Engine/Toolkit/exahype/} \texttt{specfiles/templates/} \texttt{specfile1.exahype.template}.  Olga Glogowska committed Aug 21, 2019 77 78 \end{info} \begin{info}  79 \small For details regarding the \textit{new recommended} configuration file format \texttt{*.exahype2}, users can have a look at a provided \texttt{JSON}-Schema. It can be found under the following path: \texttt{ExaHyPE-Engine/} \texttt{Toolkit/exahype-specfile.schema.json}.  Olga Glogowska committed Aug 21, 2019 80 81 \end{info}  82 83 84 \subsubsection{Exploring the syntax} As an example illustrating the basic syntax, users might want to analyse a \texttt{TrivialProject} configuration / specification file to be found in \texttt{Demonstrators} subdirectory of \exahype. Most parameters should be rather self-explanatory. For your own projects, you might have to adopt the paths \footnote{You may specify pathes of individual components in more detail (cmp.~Section \ref{chapter:finetuning}), but for most applications working with the three pathes above should be sufficient.}. Note that \exahype\ supports both two- and three-dimensional setups.  Olga Glogowska committed Aug 21, 2019 85 \begin{code}  86 87 88 89 90 91 92 93 94 95 96 97 exahype-project TrivialProject peano-kernel-path const = ./Peano exahype-path const = ./ExaHyPE output-directory const = ./Demonstrators/TrivialProject computational-domain dimension const = 2 width = 1.0, 1.0 offset = 0.0, 0.0 end-time = 10.0 end computational-domain end exahype-project  Olga Glogowska committed Aug 21, 2019 98 \end{code}  99 100 101 102 103 104 105 \begin{design} The \texttt{*.exahype} specification file parameters that are interpreted by the toolkit are marked as \texttt{const}. If constant parameters or the structure of the file (all regions terminated by an \texttt{end}) change, you have to rerun the toolkit. All other variables are read at runtime, i.e.~you may alter them without a rerun of the toolkit or a re-compile. \end{design} \vspace{0.1cm} \noindent If you run various experiments, you start from one specification file and build up your application from hereon. You can either pass the original specification file to your excutable, or you can pass other variants of it as well. That is, you may work with sets of specification files all feeding into one executable. However, the specification files have to match each other in terms of const variables and their structure. \exahype\ itself checks for consistent files in many places and, for many invalid combinations, complains.  Olga Glogowska committed Aug 21, 2019 106 107  \noindent  108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 Due to the fact that many users have their cases already set up using the legacy configuration file format, \exahype\ provides a means of translating \texttt{*.exahype} to \texttt{*.exahype2} format. In particular, all legacy configuration files with extension \texttt{*.exahype} can be converted into \texttt{*.exahype2} format using the \texttt{-j} flag of the \texttt{Toolkit}. Using it will write a \texttt{JSON} file with a similar filename and \texttt{*.exahype2} extention, if legacy specification file is read. To keep things simple and expolore some of the syntactic differences between the old and new configuration file format, you might want to first translate \texttt{TrivialProject} configuration file into the new format as follows: \begin{code} > cd ExaHyPE-Engine > ./Toolkit/toolkit.sh -j Demonstrators/TrivialProject/TrivialProject.exahype \end{code} \noindent You should now be able to find \texttt{TrivialProject.exahype2} file in \texttt{TrivialProject} subdirectory. The file should have the following content: \begin{code} { "project_name": "TrivialProject", "paths": { "peano_kernel_path": "./Peano", "exahype_path": "./ExaHyPE", "output_directory": "./Demonstrators/TrivialProject" }, "computational_domain": { "dimension": 2, "end_time": 10.0, "offset": [ 0.0, 0.0 ], "width": [ 1.0, 1.0 ] }, "solvers": [] } \end{code} \noindent More details regarding specification file setup and all the parameters can be found in Section \ref{sec:solver-configuration}. It might still refer to the legacy specification file. However, there is planty of information available in the Internet regarding syntax of \texttt{JSON} configuration file language. If you take into consideration that all dashes ''-'' in legacy parameter names have now been changed into underscres ''\_'' in \texttt{JSON} syntax and that the \texttt{const} indicators are gone, you should be able to find your way and use the new syntax easily.