2.12.2021, 9:00 - 11:00: Due to updates GitLab may be unavailable for some minutes between 09:00 and 11:00.

Commit 25aac7be authored by eckhart's avatar eckhart
Browse files

Step by step guide extended

parent c1bd21ce
......@@ -24,7 +24,7 @@ import os
import sys
from DHParser.compile import compile_source
from DHParser.dsl import compileDSL, compile_on_disk
from DHParser.dsl import compileDSL, compile_on_disk, recompile_grammar
from DHParser.ebnf import get_ebnf_grammar, get_ebnf_transformer, get_ebnf_compiler
from DHParser.log import logging
from DHParser.toolkit import re
......@@ -208,8 +208,12 @@ def create_project(path: str):
GRAMMAR_TEST_TEMPLATE.format(name=name, dhparserdir=dhparserdir))
create_file('example.dsl', 'Life is but a walking shadow\n')
os.chmod('tst_%s_grammar.py' % name, 0o755)
# The following is left to the user as an exercise
# print('Creating file "%s".' % (name + 'Compiler.py'))
# recompile_grammar(name + '.ebnf', force=True)
print('\nNow generate a DSL compiler from the EBNF-grammar by running\n'
'\n python tst_%s_gramar.py\n' % name)
os.chdir(curr_dir)
print('ready.\n')
def selftest() -> bool:
......
......@@ -61,20 +61,109 @@ files and directories for sure, but those will not concern us for now::
LICENSE.txt - DHParser's license. It's open source (hooray!)
Introduction.md - An introduction and appetizer for DHParser
In order to verify that the installation works, you can simply run the
"dhparser.py" script and, when asked, chose "3" for the self-test. If the
self-test runs through without error, the installation has succeded.
Staring a new DHParser project
------------------------------
In order to setup a new DHParser project, you run the ``dhparser.py``-script
with the name of the new project. For the sake of the example, let's type::
$ python dhparser.py experimental/poetry
Creating new DHParser-project "poetry".
Creating file "grammar_tests/01_test_word.ini".
Creating file "grammar_tests/02_test_document.ini".
Creating file "poetry.ebnf".
Creating file "README.md".
Creating file "tst_poetry_grammar.py".
Creating file "example.dsl".
ready.
$ cd experimental/poetry
This creates a new DHParser-project with the name "poetry" in directory with the same name within the subdirectory "experimental".
\ No newline at end of file
This creates a new DHParser-project with the name "poetry" in directory with
the same name within the subdirectory "experimental". This new directory
contains the following files::
README.md - a stub for a readme-file explaiing the project
poetry.ebnf - a trivial demo grammar for the new project
example.dsl - an example file written in this grammar
tst_poetry_grammar.py - a python script ("test-script") that re-compiles
the grammar (if necessary) and runs the unit tests
grammar_tests/01_test_word.ini - a demo unit test
grammar_tests/02_test_document.ini - another unit test
Now, if you look into the file "example.dsl" you will find that it contains a
simple sequence of words, namely "Life is but a walking shadow". In fact, the
demo grammar that comes with a newly created project is nothing but simple
grammar for sequences of words separated by whitespace. Now, since we alread
have unit tests, our first exercise will be to run the unit tests by starting
the script "tst_poetry_grammar.py"::
$ python tst_poetry_grammar.py
This will run through the unit-tests in the grammar_tests directory and print
their success or failure on the screen. If you check the contents of your
project directory after running the script, you might notice that there now
exists a new file "poetryCompiler.py" in the project directory. This is an
auto-generated compiler-script for our DSL. You can use this script to compile
any source file of your DSL, like "example.dsl". Let's try::
$ python poetryCompiler.py example.dsl
The output is a block of pseudo-XML, looking like this::
<document>
<:ZeroOrMore>
<WORD>
<:RegExp>Life</:RegExp>
<:Whitespace> </:Whitespace>
</WORD>
<WORD>
<:RegExp>is</:RegExp>
<:Whitespace> </:Whitespace>
</WORD>
...
Now, this does not look too helpful yet, partly, because it is cluttered with
all sorts of seemingly superflous pseudo-XML-tags like "<:ZeroOrMore>".
However, you might notice that it contains the original sequence of words
"Life is but a walkting shadow" in a structured form, where each word is
(among other things) surrounded by <WORD>-tags. In fact, the output of the
compiler script is a pseudo-XML-representation of the *contrete syntax tree* of our "example.dsl"-document according the grammar specified in "poetry.ebnf"
(which we haven't looked into yet, but we will do so soon).
If you see the pseudo-XML on screen, the setup of the new DHParser-project
has been successful.
Understanding how compilation of DSL-documents with DHParser works
------------------------------------------------------------------
Generally speaking, the compilation process consists of three stages:
1. Parsing a document. This yields a *concrete syntax tree* (CST) of the
document.
2. Transforming. This transforms the CST into the much more concise *abstract
syntax tree* (AST) of the document.
3. Compiling. This turns the AST into anything you'd like, for example, an
XML-representation or a relational database record.
Now, DHParser can fully automize the generation of a parser from a syntax-description in EBNF-form, like our "poetry.ebnf", but it cannot automize the transformation from the concrete into the abstract syntax tree
(which for the sake of brevity we will simply call "AST-Transformation" in the following), and neither can it automize the compilation of the abstract
syntax tree into something more useful. Therefore, the AST-Transformation
in the autogenerated compile-script is simply left empty, while the compiling stage simply converts the syntax tree into a pseudo-XML-format.
The latter to stages have to be coded into the compile-script by hand, using
the existing templates within this script. If the grammar of the DSL is
changed - as it will be frequently during the development of a DSL - the
parser-part of this script will be regenerated by the testing-script before
the unit tests are run. The script will notice if the grammar has changed.
This also means that the parser part of this script will be overwritten and
should never be edited by hand. The other two stages can and should be edited
by hand. Stubs for theses parts of the compile-script will only be generated
if the compile-script does not yet exist, that is, on the very first calling
of the test-srcipt.
Usually, if you have adjusted the grammar, you will want
to run the unit tests anyway. Therefore, the regeneration of the parser-part
of the compile-script is triggered by the test-script.
The development workflow for DSLs
---------------------------------
[TO BE CONTINUED...]
\ No newline at end of file
......@@ -36,7 +36,7 @@ from DHParser.dsl import grammar_provider, DHPARSER_IMPORTS
class TestInfiLoopsAndRecursion:
def test_direct_left_recursion1(self):
minilang ="""
minilang ="""
expr = expr ("+"|"-") term | term
term = term ("*"|"/") factor | factor
factor = /[0-9]+/~
......@@ -99,7 +99,7 @@ class TestInfiLoopsAndRecursion:
class TestFlowControl:
def setup(self):
self.t1 = """
All work and no play
All work and no play
makes Jack a dull boy
END
"""
......@@ -363,7 +363,7 @@ class TestPopRetrieve:
codeblock = delimiter { text | (!:delimiter delimiter_sign) } ::delimiter
delimiter = delimiter_sign # never use delimiter between capture and pop except for retrival!
delimiter_sign = /`+/
text = /[^`]+/
text = /[^`]+/
"""
mini_lang2 = """
@braces_filter=counterpart
......@@ -379,7 +379,7 @@ class TestPopRetrieve:
env = (specialtag | opentag) text [closespecial | closetag]
opentag = "<" name ">"
specialtag = "<" /ABC/ !name ">"
closetag = close_slash | close_star
closetag = close_slash | close_star
close_slash = "<" ::name "/>"
close_star = "<" ::name "*>"
closespecial = "<" /ABC/~ ">"
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment