Offer of help
I am developing a GPL template processor. I think it could be useful
for generating configuration files from templates. It works, but it is not
finished yet. I plan to release it soon, but if some people on this list are
interested I would like to share it with you now. Please let me know either
privately or through this list.
This message has two parts: background and technical info.
Background:
An ancient time ago (July 97) I was involved in an unfortunate flame war
in this list. We were discussing the design of a config tool based on
templates. I wanted the tool to include parsers too. I regret to have
helped the design stall, even though the main reason was that people
wanted to wait for COAS, which at that time looked promising.
I still did some thinking about all this from time to time. As time passed,
it became clear to me that the first step in an admin tool must be the
creation of a template processor. Once the template processor is done, all the
tools can use the abstract view of the system provided by the database. What I
did not like about templates is that all implementations I knew about fell
into one of two groups: either they were based on a heavyweight scripting
language (like eperl) or they lacked the necessary functionality (like m4.)
The former had the risk that people would end up writing perl scripts instead
of templates, throwing away the clear distinction between templates and the
database which I think is essential to a good admintool design. The latter
could be used as a base on which to add the missing functionality.
Therefore, I started looking at m4. I was soon convinced that it was not a good
base to start with. It was full of global variables, which was bad because I
wanted the template processor to be a library and I found it difficult to add
hooks for the database lookup. So, finally I started from scratch, even though
I have used some parts of the m4 sources.
I hope I can get to the parser part someday, but right now I want to finish
the template processor first, and then maybe some database utility library
to interact with the database. I do not like writing GUI stuff, so any
help in that area is very welcome. Basically, programs which read the database,
let the user edit its values and write them to the database again. Finally,
some glue code must be written to have the system interact with other tools
(like dpkg and friends.)
Technical info:
The template processor is a general purpose macro interpreter somewhat similar
to m4. However, it has some unique features which make it suitable for
generating files from templates:
- It has a built-in database engine based on Patricia trees. This allows
keys to be of arbitrary length without a performance penalty and
encourages the use of hierarchical name spaces.
- It has hooks to add database drivers. Currently only one driver is
implemented: plain text file (which I called "MDF"). A GDBM driver is
planned next.
- The format of the "MDF" file is the same as Debian's Packages files. My
program parses that file correctly and can use its fields in templates.
- Like m4, the program can perform an infinitely recursive macro expansion.
- Unlike m4, no quoting is necessary to prevent rescanning. Optionally,
only \escaped words are expanded and only at top level. This default
behavior can be altered by \*unescaping a word at any level.
- A tiny tiny (I know...) macro language greatly extends m4 functionality.
Besides the usual $<digit>, $0 and $# directives the following directives
are defined: $:(label), $-(jump), $?(conditional jump) and a few
specialized other (totaling 22 directives.)
- It has a table processor which runs through all records in a database
and applies a macro transformation to each of them.
- Most m4 functions are either implemented or emulated.
- PERL regular expressions are used, including the "split" command.
This is implemented via the libpcre library.
- Arguments can be given in two styles: (arg1, arg2, ..., arg-n) and
{arg1}{arg2}...{arg-n} (with a different behavior: {} eats all whitespace
afterwards and does not expand macros inside it)
- A built-in R5RS Scheme evaluator is included. Currently, it is based on
"tinyscheme", but this is subject to change in the future. I chose
Scheme because of its functional style, which prevents it from eating up
the whole template (unlike Perl.) For example, the "incr" built-in macro
from m4 is emulated this way: \define{incr}{$_(+ $1 1)}, where $_ is
the directive for calling the Scheme evaluator.
- It can be built as a library, so its powerful macro and database engines
are available to any kind of program.
- Lexical translation is fully configurable, so the meaning of all
characters can be modified at run time.
Example:
This is an example of a template. As you can see, the macro directives
don't get too much in the way. This is a working example. All necessary
features are already implemented.
The same file can generate either /etc/profile or /etc/login.csh, depending
on the value of SHELLTYPE. (Run the program with -DSHELLTYPE=<desired type>)
The tables are defined in "MDF" format. For example, the
"System-Shells-Aliases" is:
------- BEGIN table -------------
ALIAS: dir
VALUE: ls -lAF
ALIAS: dirh
VALUE: ls -lAt | head
ALIAS: x11
VALUE: cd ; startx
------- END table -------------
------- BEGIN template -------------
\ifelse{\*SHELLTYPE}
{BSH}{\define{shell_ifeq}{if [ $1 = $2 ]; then}
\define{shell_endif}{fi}
\define{shellvar}{$1=$2\printf(\n)export $1}
\define{shell_alias}{alias $1='$2'}}
{CSH}{\define{shell_ifeq}{if ( $1 == $2 ) then}
\define{shell_endif}{endif}
\define{shellvar}{setenv $1 $2}
\define{shell_alias}{alias $1 '$2'}}
\ifelse{\*SHELLTYPE}
{BSH}{# /etc/profile: system-wide .profile file for bash(1).}
{CSH}{# /etc/csh.login: system-wide .login file for csh(1)/tcsh(1)}`'
# Generate the path from the system table "System-Shells-Paths".
\dnl The only complication is that we don't want a : after the last entry.
\dnl The empty `' is to preserve the newline following it
\shellvar{PATH}
{"\systable{System-Shells-Paths}
{PATH$=($record,$records,skip_colon):$:skip_colon}"}`'
umask 002
/usr/bin/check-sendfile
# Systemwide Aliases are stored in the table "System-Shells-Aliases"
# systable_nl is like systable, but it generates a newline after each record
\systable_nl(System-Shells-Aliases, \shell_alias(ALIAS,VALUE))
# Shell variables
# Use the pager defined in the system variable "System-Utils-Pager" or,
# if not defined, use "more"
\shellvar(PAGER, \sysvar(System-Utils-Pager, more))
# Pre-template setup had to hardcode the following
# xterm-debian does not work in remote non-Linux systems...
#if [ $TERM = xterm-debian ]; then
# TERM=xterm
#fi
\shell_ifeq("x$DISPLAY", "x")
\shellvar(TERM, \sysvar(Hardware-Console-DefaultType,vt100))
else
\shellvar(TERM, \sysvar(System-X11-DefaultTerminal,xterm))
\shell_endif
cd
------- END template -------------
The machinery necessary for this to work is hidden in the initialization
file. This file needs not be modified by template writers, only by admintool
developers:
\define{basedir}{$ /usr/local/adm/db}
\define{_init_}{$ \readtable(SYSVAR,\*basedir/System-Variables)}
\define{systable}{\readtable(_T_,\*basedir/$1)\runtable(_T_,$2)\deltable(_T_)}
\define{systable_nl}{\systable($1,$2\printf(\n))}
\define{sysvar}
{$=($field(SYSVAR,N,$1,V),,skip_it)\field(SYSVAR,N,$1,V)$-(exit)$:skip_it$2}
\define{syscom}{\field(SYSVAR,N,$1,C)}
\define{sysdef}{$=($field(SYSVAR,N,$1,V),,skip_it)$2$:skip_it}
\_init_
The "System-Variables" table is currently just another "MDF" file with the
format:
N: Name of variable
V: Value of variable
C: Comment/description of variable
D: Default value
T: Timestamp
G: generator
R: trigger
The last 4 fields are not used right now, but maybe used in the future,
especially in connection with the GUI tools.
Thanks,
Fernando
Reply to: