Invenzzia... in English - Open Power Template

New expression parser for OPT 2.1

Zyx — Sat, 23 Jan 2010 13:00:00 +0100

The incoming Open Power Template 2.1 will have a new, more powerful expression parser. Basically speaking, expressions are those pieces of language that evaluate something, producing a value, i.e. variables and operators. Both OPT 1.1 and 2.0 use a nice, manually-written parser that compiles them to a PHP code, validating their syntax and performing some extra transformations. While it works well, I think I reached the end of possible ways to extend it. It would be simply too complicated to implement new features to it, so I decided I must rewrite it into a formal grammar. Today, the main works on it are finished.

Formal grammars

The new expression parser is defined using a formal grammar. It is a set of rules describing, how the tokens, the smallest units of the language, can be connected to produce something useful. In the example below, we have produced a simple grammar that allows adding and multiplying numbers:

expr -> expr PLUS expr
expr -> expr MUL expr
expr -> LEFT_PARENTHESIS expr RIGHT_PARENTHESIS
expr -> NUMBER

Of course we have to define the operator precedence to make it fully correct, but the basic idea is simple. We write a set of rules (called 'productions') saying, how to produce the more and more complex parts of the expressions from the smallest units. For a number of grammar classes, there are computer algorithms that are able to produce a complete parser code for them, and the most popular one for programming languages is called LALR(1). Now all we have to do is to write the grammar in the format accepted by the parser generator and simply run it to produce a ready-to-use code.

For Open Power Template, I chose PHP Parser Generator, a clone of Lemon parser generator rewritten from C to PHP. The same stuff is used by the guys that develop Smarty 3, but they have applied it to parse the whole template. In case of OPT it is not necessary, as it uses an ordinary XML parser. Below, you can see a part of the grammar file that defines function calls:

calculated(res)		::= function_call(fc).		{	res = fc;	}
calculated(res)		::= method_call(oc).	{	res = oc;	}

function_call(res)	::= functional(fun).		{	res = $this->_expr->_makeFunction(fun);	}

functional(f)	::= IDENTIFIER(s) L_BRACKET argument_list(a) R_BRACKET.	{	f = $this->_expr->_makeFunctional(s, a); }
functional(f)	::= IDENTIFIER(s) L_BRACKET container_def(a) R_BRACKET.	{	f = $this->_expr->_makeFunctional(s, array($this->_expr->_containerValue(a, Opt_Expression_Standard::CONTAINER_WEIGHT)));	}
functional(f)	::= IDENTIFIER(s) L_BRACKET R_BRACKET.	{	f = $this->_expr->_makeFunctional(s, array()); }

As you can see, for each production we can define a PHP code snippet that may perform some action, if the specified sequence of symbols is found. This eases the process of defining a language very much.

New features of the OPT expression language

The new parser allowed me to implement lots of new and useful stuff. The most lacking feature in OPT 2.0 was the inability to create a container in a template, like arrays in PHP. We could not write:

{url(container('controller' => 'foo', 'action' => 'bar'))}

The routing path must have been defined either in a script or saved as a string. In OPT 2.1 this limitation is finally removed. We can create new containers dynamically, using a convenient syntax designed for using within XML documents:

xml
{url( [ 'controller': 'foo', 'action': 'bar'] )}
{url('controller': 'foo', 'action': 'bar')}

The shortened version (syntactic sugar) for functions is also possible, as we can see. The two lines above do exactly the same thing. The container call syntax was also extended. Now it is possible to select the container index dynamically:

$container.($index).foo

However, you should be aware that some data formats may not support it and it will not always work.

A lot of effort has been made to develop new operators which you should find very comfortable and convenient, especially with the longer expressions:

$number is between 5 and 10
$number is not between 5 and 10
$number is either 5 or 10
$number is neither 5 nor 10

$container contains 'foo'
$container contains either 'foo' or 'bar'
$container contains neither 'foo' nor 'bar'
$container contains both 'foo' and 'bar'

'foo' is in $container
'foo' is not in $container
'foo' is either in $foo or $bar
'foo' is neither in $foo nor $bar
'foo' is both in $foo and $bar

The first group of operators is a shortened form for expressions like $number > 5 and $number < 10. The second grup tests the contents of containers, allowing to check if they contain one or more specified values. In the last group, the situation is reversed: we check if the element exists in one or more containers. The advantage of the new operators is that they will be able to be reprogrammed with data formats, hiding even more implementation details from the template designer and improving the template portability.

There is only one feature removed. In OPT 2.0 it is possible to write eq eq eq and the parser correctly recognizes, which "eq" is an operator and which is a string. Unfortunately, such trick is not possible with ordinary LALR(1) grammars so I was forced not to implement it in order not to produce a mess.

The last change concerning the expressions is that the direct access to PHP arrays and objects is disabled by default: $foo'bar', $foo::field. Using them instead of containers and data formats is considered as a bad programming practice in OPT, and reduces the template portability. The idea is to encourage the programmers to learn what containers are and why they are better in templates than PHP objects and arrays.

Conclusion

There is still a lot of work to do in Open Power Template 2.1. I hope that after the exams at university I will find more time to finish it. Anyway, I'll do my best to make OPT 2.1 the best template engine ever.

Open Power Template 2.1 - a closer look

Zyx — Sat, 28 Nov 2009 11:35:00 +0100

In this note, I would like to show some new features of the incoming Open Power Template 2.1. The new branch, which is currently under development, will bring many new tools and improvements to the template language syntax and the library API. Some of them are already implemented, but there is still a lot of work to do. It is hard to say when it is going to be released, but I'll do my best, so that you would not have to wait another year for it.

Reasons for the changes

From the technical point of view, OPT is a pioneer project. Many features were never implemented in any existing template engine and at the first time it was sometimes hard to say what potential they actually offer. We were learning the library during the development process. Unfortunately, the improvements could not be invented and applied forever. One day we had to stop and say: `OK, now it is the time to make it stable'. The result is quite satisfactory, but the list of the things that could be done better was getting longer and longer. Some of the features were suggested by the users, the rest comes from us. OPT 2.1 is going to be the next step in the template engine evolution.

Compiler changes

In OPT 2.1, the compiler becomes a universal framework for implementing template languages, primarily based on XML. It provides the compilation process structure, interfaces and ports for many features, such as data formats. However, the details of the syntax processing are moved to external parts:

Custom parsers - their task is to parse the template source code and construct an Abstract Syntax Tree (AST).
Custom AST nodes - currently, the AST nodes match the specific OPT-XML needs and it is impossible to write new ones without changing the compiler code. In OPT 2.1 it becomes possible.
Custom expression parsers - they parse expressions such as $a+$b. Since OPT 2.1 it will be possible to write new ones.

In other words, OPT is going to bring the compilation structure, whereas the programmer matches it to his or her needs. Personally, I hope that someone will use it and port Template Attribute Language to OPT. However, most programmers will not be interested in the details of compilation process, but rather the default language it offers. Here, OPT 2.1 language will be very similar to the current language. It will be just ported to the new API.

New parsers

By default, OPT 2.1 will offer three parsers that could be used by the programmers. They will replace the current `compiler modes':

XML - completely new parser based on the XMLReader extension. Its primary feature is 100% sure full compatibility with XML standards, including such details, as xmlns attribute. However, the parser cannot be scaled to ignore some syntax requirements.
HTML - the current parser, customizable with many options.
Quirks - the quirks mode parser.

New dynamic attribute value syntax

As OPT is based on XML, there occurs a problem, how to put an expression value into an attribute. In the early development stage, we decided to make use of namespaces to indicate the attributes which have dynamic values. The life shown that it was not the best idea. It is not scalable and causes problems, if the attribute already contains a namespace. Redesigning this feature from scratch was one of our priorities. In OPT 2.1, the value type will be encoded in the attribute value, not in the name. It is illustrated by the following examples:

<p class="text">Static attribute value: text</p>
<p class="parse:$text">Dynamic attribute value read from a variable</p>
<p class="null:parse:tekst">Static attribute value: parse:text</p>

It can be also used to select the custom expression syntax:

<p class="foo:##mysteriousSyntax">text</p>

And in curly brackets, too:

<p>Mysterious syntax: {foo:##mysteriousSyntax}</p>
<p>A bit nonsense example ("parse" is the default): {parse:$variable}</p>
<p>Even more nonsense example, but still possible: {str:$variable}</p>

And in instruction attributes:

<opt:include file="parse:$file" />
<opt:attribute name="str:theName" value="str:value" />

Extended expression modifiers

In OPT 2.0, there were two hardcoded modifiers used for the HTML escaping control: e: and u: used as follows: {e:$expression}. Since OPT 2.1, it will be possible to create custom modifiers and use them in the attributes. Furthermore, there will be three predefined modifiers that could be overwritten:

To enable the escaping (by default e:)
To disable the escaping (by default u:)
To enable the escaping in attributes (by default a: - the new default modifier)

The reason to introduce the new default modifier were the needs of HTML escaping and XSS protection for attribute values that must be a bit more complex than for normal tag body. You will be allowed to use the more complex escaping function for attributes, and the simpler, but faster - for ordinary texts.

Some of you may have noticed that the modifiers use the same syntax, as the expression type chooser. Nothing more further from the truth! Thanks to some simple conventions everything is unambiguous. First of all, to specify both the expression type and the modifier, we specify the parser, and later the modifier: {parse:e:$variable}. Secondly, the modifier name is always one character long. At at last, in the code like this: {e:$variable}, the compiler matches the modifier first and then the expression parser.

Extended data format support

Data formats are probably the most brilliant feature of OPT and in the next branch this element will be greatly improved. Let's take a look at instructions first:

<opt:section name="foo">
 ...
</opt:section>

To specify the data format for thi section, we have to link it with a variable (in this case: $foo), but it is not always possible. For example, the variable may be specified by the datasource attribute or the section could have nothing to do with variables. Such situation occured while developing Open Power Forms. Some sections must be connected to the form and have no equivalent in template variables. And what if we would like to write a new implementation of opt:include? In OPT 2.0 it is impossible to pass the information about the data format to it.

The problem will be solved by introducing a special, optional attribute called id. Obviously, it will allow to define an unique identifier for the instruction which can be used to find the data format definition or modify any other aspect of the behaviour. It leads to the extension of the list of the available data formats. Some of the new ones are:

Include - the default data format for opt:include
System - the $system special variable handling, moved from the compiler code to the data format.
Scalar - the scalar data (text, numbers, logical values) that does not allow the container, array and object access. It will be impossible to use them as section data sources which will improve the template code security and error resistance.
RuntimeIterator - if you do not want to write a custom data format and Objective does not meet your requirements, but you still want to use sections with your own objects and their own logic, this format will allow this. It will work with a special interface that needs to be implemented in your class that provides the whole section functionality.

Snippet arguments

Snippets resemble macros from such languages, as C. A new interesting addition in OPT 2.1 will be the possibility of giving a snippet the arguments. I have not invented a syntax for this element yet, so please treat the following code as the illustration of an idea:

<opt:snippet name="snippet">
 <p>{$info.foo}</p>
 <p>{$info.bar}</p>
</opt:snippet>
 
<opt:section name="foo">
<div>
 <p>Some content</p>
 <opt:insert snippet="snippet" info="$foo" />
</div>

Now the snippet variable $info will be treated as $foo, keeping its data format and features. It will make the snippets much more portable.

Snippet loading

Currently, the shared snippets can be included in our template in two ways:

Template inheritance
opt:root attribute include

The last way will be extended by a special tag opt:load which will allow to load multiple templates with snippets:

<opt:root>
  <opt:load template="snippets_1.tpl" />
  <opt:load template="snippets_2.tpl" />

  <!-- the code here -->
</opt:root>

Template functions

Snippets have one important disadvantage: they are static and exist only during the compilation. It is impossible to select the snippet name from a variable or something like that. Many people ask for such a feature, and the problem became known as "dynamic snippets". I did a long research looking for the possible ways of adding this feature, but due to the PHP language construction and the snippet features, it seems to be impossible. The snippets cannot be implemented as functions or class methods, because:

The access to the template variables {@variable} will be lost.
The access to some internal compilation code variables will be lost, which could lead to serious problems with data formats.
The snippets would be strictly connected with one, particular data format, whereas they should match the current data format in the place of insertion.

This is why I decided to introduce a new language element which does not have an official name yet, and during the development process we call it "template functions". Why? Because it is a reimplementation of the classic PHP function features. They are created once and physically called, intead of being copy-pasted during the compilation, like in snippets. What is more, they will support a dynamic selection of the called function.

My plan is to implement them so that they could be used in many places, where the snippets can be currently applied:

Section bodies
Components
Blocks

Of course some of the locations will be unavailable for them and moreover, they will be strictly connected to a single data format. An interesting feature is that they could be injected into the component objects, like (please remember that the syntax is not invented yet):

<!-- define a layout of the checkbox list for the checkbox component -->
<opt:function name="checkboxListLayout" list="required">
  <ul>
  <opt:section name="list" datasource="$list">
    <li>{u:$list.checkboxCode}</li>
  </opt:section>
  </ul>
</opt:function>

<form:checkbox-list name="list">
  <label for="parse:$system.component.id">Select something:</label>

  <!-- dynamically select the layout to inject -->
  <opt:display inject="$system.component.layout" />
</form>

Improved section syntax

Together with the current syntax with opt:show and the empty section tag, in OPT 2.1 it will be possible to use the new style presented by the example below:

<opt:section name="foo">
	<opt:body>
		<ol>
			<opt:item>
			<li>Dodood</li>
			</opt:item>
		</ol>
	</opt:body>
	<opt:else>
		<p>Hi universe!</p>
	</opt:else>
</opt:section>

As we can see, in place of opt:show we write the section tag, and in the body we use opt:body to indicate the section body and opt:item to specify a place which will be iterated. The current syntax will not be removed, as they are complementary to each other, mostly when it comes to the cooperation with snippets. I was also asked for the possibility of having the section body enclosed in an extra tag which is quite useful if we have an extra opt:else at the same place, and want the body to be loaded from a snippet.

Moreover, the small changes will be applied to the opt:selector instruction:

<opt:selector name="foo">
	<opt:select item="fff">ssdf</opt:select>
	<opt:select item="ssddf">ssdf</opt:select>
	<opt:select item="abc">ssdf</opt:select>
</opt:selector>

In other words, it will be no longer necessary to create thousands of new tags for our choices.

Switch instruction

OPT 2.1 will introduce the new instruction: opt:switch. Contrary to the switch statements from the classic programming languages, it will be a real swiss knife. Take a look:

<!-- classic switch -->
<opt:switch test="$condition">
 <opt:equals value="value1">Body...</opt:equals>
 <opt:equals value="value2">Body...</opt:equals>
 <opt:equals value="value3">Body...</opt:equals>
 
 <!-- an equivalent of playing with "break" -->
 <opt:equals value="value4">
   Some content...
   <opt:equals value="value5">
     Some content...
   </opt:equals>
 </opt:equals>
 <opt:default>...</opt:default>
</opt:switch>

If you like the power of case... break construct in PHP, you will surely enjoy this stuff:

 <opt:equals value="value4">
   Some content...
   <opt:equals value="value5">
     Some content...
   </opt:equals>
   Some content...
   <opt:equals value="value6">
     More optional content
   </opt:equals>
   The ending
 </opt:equals>

If the tested condition equals value5, we will see just the contents of the value5 tag. But if it equals value4, we will see its contents, together with value5 and value6. But this is not the end. OPT switch will work with... containers:

<opt:switch test="$container">
  <opt:contains value="foo">body...</opt:contans>
  <opt:contains value="bar">body...</opt:contans>
  <opt:contains value="joe">body...</opt:contans>
</opt:switch>

In this case, OPT will be able to display the values of more than one choice. opt:contains could also be nested, similarly to opt:equals. And what would you say, if you see that they can be mixed?

<opt:switch test="$condition">
  <opt:contains value="foo">body...</opt:contans>
  <opt:contains value="bar">body...</opt:contans>
  <opt:contains value="joe">body...</opt:contans>
  <opt:equals value="value1">Body...</opt:equals>
  <opt:equals value="value2">Body...</opt:equals>
</opt:switch>

In this case, OPT simply checks the expression type. If it is a container, it matches the opt:contains tags, otherwise - opt:equals. Nested cross-mixing of opt:equals and opt:contains is also possible.

We get even better functionality, if we use the attribute form of opt:switch:

<ul opt:switch="$row">
 <li>Name {$row.name}</li>
 <li>Age: {$row.age}</li>
 <li opt:contains="city">City: {$row.city}</li>
 <li opt:contains="email">E-mail: {$row.email}</li>
 <li>Date: {$row.date}</li>
 <li opt:contains="other">Other: {$row.other}</li>
</ul>

And the finale that will help in the last example. Suppose that we would like to add the class="last" attribute to the last displayed list element. This could be done by the following code:

<ul opt:switch="$row">
 <li>Name {$row.name}</li>
 <li>Age: {$row.age}</li>
 <li opt:contains="city">City: {$row.city}</li>
 <li opt:contains="email">E-mail: {$row.email}</li>
 <li>Date: {$row.date}</li>
 <li opt:contains="other">Other: {$row.other}</li>
 <opt:append to="last">
   <opt:attribute name="str:class" value="str:last" />
 </opt:append>
</ul>

For everyone who claim that "PHP is the best template language": enjoy implementing similar functionality manually.

Futher changes in syntax

During the development process, there have occured some inconsistences in the naming style of the instructions. OPT 2.1 will attempt to fix them. Basically, all the tags named opt:someTag will be renamed to the form opt:some-tag. In sections, monsters like opt:sectionelse will be shortened to opt:else. opt:use will be renamed to opt:insert, and opt:on into opt:omit-tag.

Inflectors

This feature concerns the public library API and is related to finding the templates in the filesystem. I was talking to some programmers and the developers of CMS-es etc. They were often complaining about the current, hard-coded algorithm of mapping the template names to the files based on the streams. They said that it is very impractical, when it comes to the modular applications - it is very hard to manage the registration of dozens of quasi-streams, and operating them on the script- and template side. They would enjoy a system that would give them more flexibility. This is how the idea of inflectors was born. An inflector is a simple class that gets the "public template name", such as `db:user/edit.tpl` and returns the real filesystem path to it. The default OPT inflector is basically the reimplementation of the current template locating algorithm. It is just extended by an extra API for managing the streams. However, if the programmer needs some specific rules, he can write a custom inflector and deal with it on his own.

Backward compatibility

OPT 2.1 is intended to be used in new projects, however - if you stared writing something with OPT 2.0, you do not have to worry. Although the syntax changes can bother many programmers, there is no need to worry. OPT 2.1 will have a backward compatibility mode. Basically it activates the whole old and replaced syntax, with keeping the new features at the same time. You can use this mode to make use of the new features and give yourself some time to make a migration. What is more, many changes are just syntactic, not semantic. I plan to write a special tool that would convert the existing templates to the new syntax with one click.

Note that if you actually want to be up-to-date, the migration at some level will be necessary. The OPT 2.2 branch will be basically OPT 2.1 with the backward compatibility code removed.

Support

If you want to stay with OPT 2.0 in your current projects, the support for this branch will be continued long after the OPT 2.1 will be released. We will fix the encountered bugs and continue improving the stability of this branch. Actually, OPT 2.1 is going to be a transition branch to the next evolution level. The plans are as follows:

OPT 2.0 - works on PHP 5.2, the support is continued for at least one year after releasing OPT 2.1
OPT 2.1 - the new branch with new features and the full backward compatibility mode. The minimum requirement is PHP 5.3
OPT 2.2 - backward compatible with OPT 2.1, but without the 2.0 compatibility mode. Also, the very long support is planned.
OPT 3.0 - basically the same as OPT 2.2, but with the entire code migrated to PHP 6 and namespaces.

Note that OPT 2.0, 2.1 and 2.2 are also supposed to work on PHP 6 as soon as it will become more stable, because nowadays I even have serious problem with compiling the development snapshots.

Changes to the OPL core

As you can see, the new branches will require the improvements of the OPL core. The planned features are:

OPL 2.1 - new features: namespace support, command line interface tools, new debug console implementaton, improved error handling.
OPL 2.2 - removing legacy code for compatibility with OPL 2.0
OPL 3.0 - migration to PHP 6 and namespaces

Conclusion

OPL project is getting more and more mature. The new features will make it more suitable to deal with new problems and situations, where the template engine support is needed. At the same time, I think about how to introduce the new users in the project and teach them, how to use the offered tools. You can expect new tutorials, and personally, I think it would be nice to write a book after releasing OPF and OPT 2.2...

Open Power Template 2.0.0 is ready!

Zyx — Tue, 14 Jul 2009 09:14:00 +0200

Finally, after a 1.5 years of the development process, we managed to release the first stable version of Open Power Template. The road was very long but I hope that it was worth it. In this short entry, I'd like to make a small introduction to the project history for you, as it is quite interesting and could give some clue, how this all began.

Open Power Template is a quite old project, lasting for almost five years now. The development started in November 2004, as a small template engine for the open-source discussion board project called Open Power Board. The library was a creative variation of the Smarty engine, and I was the right person in the team to write it, as I was writing small template engines much earlier and had some experience in this area. Fortunately or not, the OpenPB project failed and my library was the only which survived the crash. After some huge code revisions, I released the first stable version, 1.0.0 in July 2006, three years ago. At the first sight, the template engine was quite similar to Smarty, especially because it used curly brackets, too. However, we could find there some ideas that were later improved and featured in the 2.0 branch. OPT 1.0.0 contained:

Sections
Components
Snippets (under different name)
I18n support

The 1.1 branch brought the pagination support and tree rendering instruction. The template engine did not understand the document structure, but the parser used some ideas taken from XML parsers. For example, it constructed a node tree for the content similar to DOM, and the nodes were instruction tags and the text content. Furthermore, there was a feature called XML Compatibility Mode which replaced the curly brackets with XML-like instruction delimiters. However, this was only a visual change.

Early in 2007, after some talks with the users, I started thinking about using a real XML parser in Open Power Template. On my blog, I published a note dated to April 19, where I suggested that a compiler with an XML parser could be written, as an extra compiler for the project. It did not mean that the works began then, as I was beginning my studies on AGH University in Cracow and had little time to deal with it, too. The first lines of code were written about seven months later, in November. On November 23, I wrote on my blog that the parser loaded the first XML document into memory, and a couple of days later it compiled its first (very simple) template.

OPT was a really productive project. Soon, I realized, that the old infrastructure from OpenPB used so far is crappy and unreliable which led to the formation of Invenzzia Group in January. You may ask where the name came from. We were thinking about it very long, and the further, the more stupid ideas were coming into our minds. As I was fascinated with linguistics and artifical (constructed) languages, we opened the dictionary of one of my languages, Ferrinti with the word: invenzzia - invention. By the way, it has nothing to do with Polish language, where the word for invention is wynalazek and inwencja means something completely different.

A couple of months later, I was becoming more and more annoyed with DocBook, especially because of the lack of any simple-in-use framework for generating user manuals and documentations from them. Because I did not want to deal with various XML parsers, incompatibility issues between Windows, Linux and everything else, I decided to write a completely new tool in PHP. This was the second project inspired by Open Power Template and it is known now as TypeFriendly. At the same time, we closed the 1.1 branch of OPT, to focus on the other projects. In July, the branch was actually rewritten from scratch, introducing a completely new API that had nothing to do with the earlier versions, as it was using many new features, such as autoloading, or better use of exceptions. Furthermore, I removed the explicit recursion from the compiler that produced very serious trouble due to the PHP stack limitations. Precisely after a year, on November 23, 2008, Invenzzia Group released the last 2.0-DEV version with actually all the planned features finished and began the beta-tests and the debugging process. After deep analysis, some parts of the code were rewritten once more (notably sections and data formats), providing a more stable and flexible source code, and many other features were widely tested both with unit tests and the real projects.

In March, I published the first revision of the Zend Framework port on our SVN. The community created a port for Kohana framework a couple of months earlier. I started also thinking about the tutorials. In the last few days, we managed to finish the user manual and the huge tutorial A photo gallery with Doctrine and Open Power Template, and we decided that we are ready to release the first stable version. And here we are!

Open Power Template 2.0-RC1

Zyx — Tue, 12 May 2009 13:06:00 +0200

Finally, the first Release Candidate of Open Power Template is available to download, after 4 months of testing and implementing the remaining features. The library seems to be completed and stable enough for both development and production environments. Of course, there is still a small possibility that a fatal bug will be found, but I do not think this will happen, as there are lots of unit tests present and the code has been checked with two real-world web applications.

Where to download?

Of course, from our website: Invenzzia. There are available the ordinary packages, and moreover, for the first time we publish the PHAR-s. PHAR means PHP ARchive - it is a package of executable PHP code that works much like JAR-s in Java. It is going to become a part of the official PHP release since 5.3.0 version. Unfortunately, our web system is not prepared to publish so many different files in one branch, so I publish the links right here till the problem will be solved:

The packages contain two PHAR-s: for OPL and OPT, license texts and the README. They do not include the documentation.

What now?

Well, Release Candidate does not mean the end. There is also the stable release, and lots of plans for the 2.1 branch. However, for some time the development process of OPT will slow down, as there are some other projects that need our urgent attention. First of all, we have all the resources for the programmers: tutorials, tips, case studies etc. Open Power Template brings a completely new philosophy of creating the presentation layer and there must be good articles available in order to show, how to make a good use of it. Moreover, there are next libraries on the way: Open Power Classes, Open Power Forms and the Zend Framework port.

Conclusion

At the end, we would like to thank all the current users of the library, especially those ones who have helped us with the development process. These projects are created for you.

OPT 2.0-beta2

Zyx — Sat, 21 Feb 2009 09:22:00 +0100

It's true, the new beta version of Open Power Template is available to download. It was planned to be released a week ago, but because of problems with SVN I didn't manage to do that (again, but hopefully for the last time). This is mostly a bugfix release, but there are some small improvements and changes in the component API, which was not documented then. The changes are not too big and were quite necessary to be introduced.

The changes in the component and block API:

setOptInstance() became setView and takes the current Opt_View object as an argument. Now the components and blocks can refer to the template variables.
createAttributes() became manageAttributes(), and moreover, the semantics has been changed. The new method takes an array of tag attribute values and returns the modified version. OPT translates it to the valid XML attribute list.

The tutorials can be found in the documentation. Another improvement are the get() and getGlobal() methods in Opt_View that allow to read the template variable values from the view object.

The things that need to be done now are:

To check the caching port interface.
To test several rarely-used compiler API features.
To check, whether all the features mentioned in the documentation are implemented and vice versa.
To fix more bugs :).

By the way, it is possible that in a few next days, there will be PHAR files with the beta2 available to download for use with PHP 5.3.

The library can be downloaded from our file repositories and the English documentation can be found here.

OPT 2.0.0-beta1 released

Zyx — Tue, 06 Jan 2009 22:45:00 +0100

Well, after many adventures with SVN and recently discovered problems with XML prologs and DTD I finally managed to release the first beta version of Open Power Template. From now, we are going to focus on removing as many bugs and problems as possible and we need here your help. Furthermore, we also need to complete the English manual, which is quite big now, but still does not explain some issues.

I hope the library will mature soon, because I want to start works on next libraries: Open Power Forms and Open Power Classes. Thankfully, our quite small community is quite active. Recently, Nowaker informed me he prepared a port for Kohana framework and he is going to release it, soon. As eXtreme will complete the new Invenzzia website, we will be able to put such add-ons in one place, as well as tutorials and sample projects.

The library can be found in our download sites. The packages contain English manual, which is also available on-line.

One year of hard work

Zyx — Thu, 20 Nov 2008 18:59:00 +0100

Somewhere in the middle of November 2007, I wrote the first (working!) lines of the Open Power Template 2.0.0 source code. After a year of hard work, I've already uploaded a revision that finally gives you all the invented features implemented. For that time, OPTv2 grew and changed its shape. The first development releases resembled OPT 1.1.x when it comes to API and some features. They were just ported to the new XML compiler. Nowadays, OPT is a totally different library. It is much more modularized and designed to make the integration with frameworks easier. There is still a lot to do. The incoming release will be the last one with the "-dev" prefix, and in the next few weeks, the beta versions will appear, then Release Candidates and at last, the first stable release.

I still get questions, why the hell I'm writing "yet another template engine if PHP is a template language itself". In my opinion, it is not PHP which is so good. The template engines are simply too bad, and unsually offer some control structures taken from PHP, variables, put them into different syntax and that's all. Why the hell someone would like to use such stuff if PHP gives him the same things plus object oriented programming to make the code more reusable. But take a look at another issue. Do you think that imperative languages, like PHP, are comfortable enough to be good languages to write, in fact, simple templates? In my opinion, the language that required to be taught, how to display a nested list and escape script data every time we want it is a mistake in this place. Such languages are created to express fast and advanced algorithms, where we need the full control over the process, not to take some data from the script and put them in a string. Yes, we can modularize them with loads of classes, interfaces etc. but is it still a comfortable way and a good use of OOP? In my opinion - no.

What give us new languages

When people noticed that writing in assembler is too complicated, they invented Fortran and Lisp, the first two high-level programming languages. They allowed to express the same algorithms and calculations in more human-readable form. Later, they invented C and C++, two general-purpose imperative languages. As the WWW appeared on the Internet, another group of people noticed that existing programming languages are too complicated to make dynamic websites. Would you like to deal with memory allocation while writing a CMS? The computers are nowadays faster and faster, and people noticed that sometimes it is much better to use simpler language designed for certain tasks at a cost of performance. In the template engines, the rule is similar. PHP in its current shape was never a good template language, so we try to create something different that simplify this process. And how to achieve this? We have to change the programming paradigm. As we design the new programming language, we can do almost everything with it and add features that are impossible in another language.

OPT template language

The basis of OPT template language is XML. Unlike PHP, OPT understands entire structure of the XHTML code in the templates and checks its syntax before we see something in the browser. All the language constructs are allowed to manipulate the code in the way that is impossible in PHP, using an interface similar to DOM. They can add new attributes, create tags, and the compiler prevents them from generating invalid output. Moreover, I wanted to get rid of programming in the templates, so I designed the control structures in a declarative style. An example of such declarative language is SQL, where you describe, what data you want to have and the database engine decides, how to find them. Here, you simply inform, what you want to see, and the template compiler provides you the implementation. Below, we have an example of displaying a hierarchical list of categories (a category tree):

xml
<opt:tree name="categories">
 	<opt:list><ul><opt:content /></ul></opt:list> <!-- 1 -->
  	<opt:node><li>{$categories.title} <opt:content /></li></opt:node> <!-- 2 -->
  	<opt:treeelse><p>Sorry, there are no categories in the system.</p></opt:treeelse> <!-- 3 -->
</opt:tree>

Do you see here any tree rendering algorithm? We have only elegant definitions, how a flat list looks like (1), how a node looks like (2) what what to show if there are no categories. We even know nothing about the data format. The template would not tell us whether the category tree must be provided as a big object or an array, and what are the details of its structure. As OPT provides a complete declarative programming model, similar solutions can be found almost everywhere in OPT. It really works. A couple of days ago I started writing a small website for my friend. Thanks to OPT, I managed to write all the presentation layer in 15 minutes. The fact that I did not have a working script yet did not concern me. I knew that OPT would do everything for me once I finally decide, where the script would return arrays, where objects, and what they would really be. I do not worry about code refactoring, because the templates remain untouched. Once I make a critical change, I simply recompile the templates and that's all. Try to achieve this in pure PHP or Smarty!

OPT language features:

True XML parser with configurable level of standard-compliance (from strict mode to quirks mode, where everything that is not directly parsed by OPT is a static text).
Variables are data format independent - the data structures are chosen during the compilation thanks to information from the PHP script.
Two complete models of handling modular templates: by including and by template inheritance (also dynamic).
Intuitive support for nested lists
Smart and semi-automatic data escaping.
Additional useful instructions to create cyclically changed values or to add separators between elements.
XML manipulation instructions.
Support for runtime instructions: blocks and components that are designed especially to deal with HTML forms.
Code reusability: write once, use everywhere.
Set of data manipulation and aggregation functions.
Support for i18n.
A complete expression language designed especially for use with XML. It includes the complete PHP object access support.
A basic set of programming structures: conditions and loops, if they are really necessary.

What about the API?

OPT source code design resembles Zend Framework conventions, when it comes to the class/method naming and file structure. It is fully objective - the templates are represented in the script as view objects, where we can assign the script results to and set the data format. Once the view is fed with the data, we create an output object and order to render the view. OPT provides two default views: HTTP and Return. The first one sends the results to the browser and maintains the HTTP headers. The second one returns the generated code back to the script. Of couse everything is extendable: you can write new outputs, define new data formats, write new functions, components and even instructions with the powerful compiler API. The management of all this stuff is also simple, because there is a plugin system.

The templates can be loaded and stored everywhere thanks to PHP streams. All you have to do is to write a new stream wrapper and register it in PHP. The advantage of this solution is that the wrapper can be used with all the file access functions in your script, even with require and include. The output results can be cached, but this time OPT does not provide its own cache handler, allowing the frameworks to do their job. In order to help the programmers manage the templates, the debug console is provided that shows information about the current configuration, executed and compiled templates and the resources used to perform those tasks. The errors are handled with the exceptions and the default error handler provides extra context information with additional explainations, how to fix the problem and where to find the solutions.

Below, you can see a sample use of OPT in the scripts:

php
try
{
	$view = new Opt_View('template_file.tpl');		
	$view->foo = value;
		
	$out = new Opt_Output_Http;
	$out->setContentType(Opt_Output_Http::XHTML, 'utf-8'); // Content negotiation available!
	$out->render($view);
}
catch(Opt_Exception $e)
{
	Opt_Error_Handler($e);
}

However, OPT is not a standalone script. It is the first library in the Open Power Libs series. They share a common core with the basic core that provides the necessary features, like autoloader, configuration, plugin architecture, debugging issues and PHAR support.

Project status

As I mentioned in the excerpt, all the planned features are currently completed, but not fully tested yet. The code seems to work in all the common use cases, but lots of deeper interactions between various elements are a big unknown. At this time we are going to start beta tests that will take some time. Meanwhile, the following tasks are waiting:

To complete the English documentation.
To write the complete unit tests and support for PHAR-s.
To write tutorials, FAQ-s and sample codes for OPT.
To provide the initial add-ons: Unicode function set for the templates, complex port to Zend Framework and probably Kohana. In the near future we are going to provide the support for Symfony, too.

Moreover, the next OPL projects are going to be opened. Currently, we have already started works on Open Power Classes, a set of small PHP classes to deal with various useful stuff. Our next big goal is Open Power Forms - a complete form handling solution for your script that shows the real power of Open Power Template.

How to help?

In the next few days, OPT 2.0.0-dev8 should appear. The source code for this version is already available in SVN, but I want to finish some API reference translations. Test it and send us you opinions. If you think you found a bug, use the bugtracker and let us know about it. Do not be afraid to ask - our community is still quite small, but the Invenzzia team is very helpful and does its best to help you deal with the coding problems.

To sum up

I know this post is full of propaganda stuff, but on this blog, it has not been announced so far and I think it is necessary to show, what this project really is and why we are writing it. Open Power Template is something more that just a template engine - it is probably the first presentation layer framework designed as a complex solution, and in addition - fast. I hope you'll enjoy it. Many people managed to enjoy OPT 1.1.x, and incoming 2.0.0 is simply much better.

Update: OPT 2.0.0-dev8 is available to download. The English manual can be found here.

OPT requirements

Zyx — Sun, 06 Apr 2008 10:28:00 +0200

I've found a new entry on the bugtracker recently. The author informed that he gets "Nesting level too deep - recursive dependency" error while running the development examples provided by the newest OPTv2 dev version. I looked at it deeper and found out that he lowered the default nesting level limit 4 times from 64 to just 16. Obviously, this is a quite drastic change and would affect not only OPT, but many other scripts as well. However, I think it is time to take a look at the OPT issues concerning the requirements, because several of them needs a deeper explaination.

The optimizations implemented in OPT, mostly concern the main parser class and the compiler. The first one is the part that is loaded every time the script wants to use templates, so it must provide huge quality standards. There are no sophisticated algorithms, so the speed improvements are:

To minimize the size of the code.
To minimize the amount of points that try to access the hard drive.
To use the fastest elements of the language.

The point is this class must be as light as possible; I would even say: almost transparent for the script.

The situation is quite different, when it comes to the compiler. It is loaded from time to time, so it can be big and splitted for more files (of course without exaggeration). The key role is played by the limits set in the PHP configuration, because in the pesimistic situations, the compiler runs out of resources to complete the task. Unfortunately, some of the new features cost a lot, and if the users wanted XML tag parsing, they must face the consequences: every such tag requires its own node object in the document tree and they need some preparation tasks to be done. My role is to minimize the time and memory necessary to do this.

The elements that can be exhausted during the compilation are:

The stack
The memory

The algorithms used in the compiler operate on a data structure called tree, and the use lots of recursion. It consumes the stack, because the number of currently executed methods and functions grows quite fast. In PHP, the default limit for the stack depth is 64 and if you are going to use OPT, the worst you can do is to lower it to ridiculous values like "16" that would affect not only OPT, but also many other scripts :). However, every recursion has an imperative version, constructed for example by queues. Maybe it does not consume less memory, but at least it does not depend on the limits. Such changes have been applied to many compiler methods so far, but some of them still require my attention, because they are not so trivial:

Processing the tree by the instruction processors
Expression compilation

In the first one the situation is clear: we get a node of some instruction and want to process it. We find its processor and redirect the node to it. It generates some PHP code and decides, what to do with its children. By default, it wants to process them, too. Here comes the recursion and the process starts from the beginning. There are some difficulties. The algorithm is split over many classes, and the programmers may create their own. What is more, some instructions need to start the subnode processing immediately, because their next decision depends on the code generated by the children. Obviously, this eliminates queues. An elegant replacement could be made, if we had an access to the low level programming.

Let's take a look, how it affects the compilation process. We have a huge template, with the tags nested up to the level 30. OPT with the compiler can take another 10 calls, and in the deepest place we have a complicated expression with brackets. To sum up, the library uses approx. 45 calls from 64 available, which is more than 70% of the limit! I know this case is quite pesimistic and in normal life such HTML documents are very rare, but it shows, how demanding the compilation can be.

If we can't remove the recursion without ugly tricks which would be never accepted by the instruction authors, how we can get around it? The idea is not to remove recursion completely, but we keep it in some places. I use here the fact that the immediate access to the subnode compilation results is necessary only for some instructions, and for example the HTML tags do not need it at all. If we get a random node from the tree, we have a big chance that it would not need a recursion here. So, why don't we provide a dual version of this algorithm? By default, it is executed with a queue, but for more sophisticated needs, some nodes can switch it to recursion. Now it can happen that a tree with a depth 30 is processed in one single loop, with no recursion! I predict that in normal situations, the algorithm will consume only 3, 4 spare "items" in the stack, which is fully acceptable.

The second issue concerns the memory usage. It's not so tragic here, but we can notice a significant rise when compared to the OPT 1.x compiler, even with using some patents from that version. The most memory is consumed by the tree. I've measured some single nodes recently and the results were from 2 kilobytes for a single node to 3 kilobytes for a node with one attribute. When the instructions start to add the PHP code to the buffers, the things are getting worse, but it happens only in some nodes in the whole template. We can safe some memory here. The objects use various arrays, which are initialized just in the beginning, for example:

php
abstract class optScannable extends optNode implements Iterator
{
	protected $subnodes = array();

In most of the nodes, the arrays are empty, so it it no sense for them to exist all the time. I fixed the code so that it creates an array only when it is necessary. From the declarations, the = array() parts were removed. The effect is quite interesting. On a single array, we safe 400 bytes. The new results are:

Single node with no attributes: 1400 b.
Single node with one attribute: 2350 b.

The additional memory may be safed by releasing the huge data that will not be used anymore, for example the template content.

What do we have to remember, while using the new OPT, when it comes to the requirement issues:

With the terribly low limits, the compiler will always crash. I thing 16 megabytes for the script (the default limit) is big enough both for the compiler and the script.
The best place to run the template parsing is the end of the script. We can delete some unncessary data earlier to free some memory.
The templates using the inheritance are more demanding, especially when overwriting snippets, because the compiler must keep also the old versions in the memory.
The quirks mode will consume less memory (simpler tree - no XML tags), but it is not implemented yet.
The compilation is run once for a long time. During the normal execution, we can forget about the limit issues etc.

I think this note shows, how difficult the library creation can be. The key is to know, what we can use in current place and that there is also the rest of the script which uses the same memory and has its own requirements.

OPT 2.0.0-dev5

Zyx — Fri, 21 Mar 2008 14:38:00 +0100

The new development version became available a bit later that I planned, but on the other hand, I managed to add much more new things. The DTD issue is solved, as well as XML prologs, CDATA support and code escaping. The expression compiler is done and the programmers may use now brand new component implementation. This is a significant step towards the first beta release and the end is closer than further. There are only a few instructions to code, the rest is almost completed and works. The compiler, programmers' API and most of the stuff also works. I think of making the first real tests on my home website and creating the Open Power Forms library compatible with the new OPT.

In the new OPT, I've redesigned the component design from scratch. It is much more flexible now and allows to make easily many interesting effects. The base rules are still the same, but the rest has changed. This is a sample code showing, how the component-based work with the forms is going to look like:

xml
<opt:snippet name="componentView">
  <com:div class="field" invalid="field error">
    <p>{$opt.component.title}: <opt:display /></p>
    <p class="desc" opt:if="$opt.component.description">{$opt.component.description}</p>
    <opt:onEvent name="error" message="info">
      <p class="error">{@info}</p>
    </opt:onEvent>
  </com:div>
</opt:snippet>

<div class="form">
<form method="post" action="skrypt.php">

<opt:input name="name" template="componentView">
   <opt:set name="title" str:value="Enter your name" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="surname" template="componentView">
   <opt:set name="title" str:value="Enter your surname" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="age" template="componentView">
   <opt:set name="title" str:value="Enter your age" />
   <opt:set name="description" str:value="Scope: 10-99 years" />
</opt:input>

<div class="foot"><input type="submit" value="Send" /></div>
</form>
</div>

The new implementation makes use of the opt:snippet instruction and template inheritance. If you had enough time, you might code the component objects, so that the field titles and descriptions will be unnecessary in the template. A component can handle events, displaying itself and its neighbourhood, managing parameters and tag attributes (we use the com namespace for it). OPT contains only the component mechanism, and the components itself must be written by the end user. The reason is very simple. In order to make a useful component, we should integrate it with the input validation classes, language system, error management etc. The demands are much bigger than OPT is able and is supposed to provide. However, there will be a ready-to-use component implementation: Open Power Forms library.

Unfortunately for the English manual is not started yet. Even the Polish users must use a partially-complete text, without formatting, because the new rendering system is not ready. However, we'll do everything to translate all the text together with some other materials (advices, FAQ, maybe even a tutorial?) into English, before the first final release is done. There is a team, I'll try also to ask some friends and we'll make a serious English support. Just be patient.

Download OPT-2.0.0-dev5

OPT 2.0.0-dev4

Zyx — Sun, 02 Mar 2008 11:28:00 +0100

There is a new Open Power Template development version available to download. The project begins to gel, as the main parser class is almost completed. Inspite of the output cache system mentioned earlier, I've implemented new instructions and completed the snippets. It means that all the template inheritance should work now. I hope the library will be usable by the end of this month. The real projects are really necessary to detect possible bugs and various shortcomings.

In snippets, we have a new possibility to insert the parent content using the opt:parent tag:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<opt:extend file="test_inherited_a.tpl">
	<opt:snippet name="header">
		<h1>Webmaster Of Puppets</h1>
		<p>Here is the parent content:</p>
		<opt:parent/>
		<p>End of the parent content</p>
	</opt:snippet>
</opt:extend>

In this case, if the test_inherited_a.tpl contains another snippet with the same name, its content will be put in the place of opt:parent. Such nesting works much deeper. For example, let's say that snippet also uses this tag. OPT will try to locate another parent and so on. It is even possible to put the default content of the opt:insert tag, which is used to place the snippets. The code contains also a simple code to detect the infinite recursion. If the parser finds it, it returns an exception and the programmer sees a nice, red error message :).

During the using of the new OPT, the new debug console should be also useful. It has been extended, compared to the similar option in OPT 1.1.x. The most significant change is splitting the list of templates to the smaller ones which show different information:

Executed templates
Currently compiled templates
Templates read from cache.
Inherited templates - it shows all the inheritance chains, as well, as the information whether they have been recompiled or not.

Moreover, the console displays all the warnings. They are no longer put onto the screen, but available only with the debug console or error messages, if we set the proper debug level.

The performance issue also looks very good. The total size of opt.class.php file is about 25 KB, compared to 32 KB in the OPT 1.1.x. Currently, its size influences the performance much, as PHP must load the parser and compile it to the internal bytecode. Of course, if we parse a lot of templates, it won't be so visible, however I'm quite pleased of the effect.

The package can be downloaded from here.

The main class almost finished

Zyx — Fri, 22 Feb 2008 08:59:00 +0100

The result of the latest works on the OPT source code is complete opt.class.php file which contains (as some of the readers may know) the main parser class. I must admit that this file was more demanding that the compiler. In the second case we can afford with quite a lot. Just an idea for the algorithm and prediction if and how it will influence the rest of the code. In the case of the main class, we must pay attention to the performance. It must contain everything that is run permanently and it must be fast and small. But the number of features is getting bigger and bigger.

The new development version will not be compatible with the previous ones when it comes to the programming interface. To implement the output cache system, I had to rebuild almost all of the output system. In the 1.x.x series, the concept of the output system did not exist. Either the result will be seen on the screen or put into a variable, it depended on the right method choice. There were some tricky connections which redirected everything to the right place, however it was not well designed and a bit messy. Well, such things simply must happen, if we start with keeping the similarities to Smarty, and I have to remind that in the very early times, it was one of the OPT goals. In OPTv2, there is a well-declared ioptOutput interface allowing the programmer to code the output processing on his own. It is also used in the implementation of output cache system. How - it is explained below.

So far, there have been two output systems: optNetOutput sending everything on the screen and checking, how many templates we are going to send in the XML mode. The second one was optReturnOutput, which returned the result. I added a possibility to decorate them with something else and I created optCachedOutput. Its only goal is to process the cache. The features are the same as in the previous versions, but the usage is a bit different and unfortunately, the initialization is more complicated:

$cache = new optCachedOutput('./cache/'); // this is the cache, here we configure it
$tpl -> register(OPT_OUTPUT, 'my_cache', $cache);
$tpl -> decorate(OPT_OUT_SCREEN, 'my_cache');
$tpl -> decorate(OPT_OUT_RETURN, 'my_cache');

if(!$cache -> isCached('template.tpl', 'some_id'))
{
  // data generation
}

$cache -> setCache(5, 'some_id'); // turn the cache on for 5 seconds
$tpl -> parse('template.tpl', 'my_cache');

I'll try to shorten it and make simpler the first lines of the code, because I don't like them, too. Moreover, I must check if I've made the decoration in the right way (according to the logic).

The support for plugins is not fully completed yet. The only ready part is placed in opt.class.php file. The rest is moved somewhere else, which should improve the performance. Like in OPT 1.1.3, there will be a possibility to set many plugin directories, but the parser will not require them to be writeable. There were thrown some files describing the content and allowing to include everything without scanning all the directory. As we know, the more includes, the less performance and some people can produce more plugins than all the files in my framework :). The new OPT will try to link all these files into one, mega-file containing everything we need. By the way, I'll have also a new algorithm for the OPT Toolset sequel, which will be used now for all the libraries, not only OPT.

The last issue concerns the available function set for the templates. Stormfly has recently written on his blog that OPT 1.1.x it was quite senseless and in fact, he was right. I took a look, what Smarty offers and it turned out that it has some nice and useful functions. The first list looks like that:

spacify - puts a string between every two neighbour characters in the text.
indent - makes an indentation in the text.
truncate - reduces the text to the specified length (allowing not to splitting the words)
strip - reduces all the white character groups to a single character
capitalize - equivalent to ucfirst()
upper - equivalent to strtoupper()
lower - equivalent to strtolower()
countWords - equivalent to str_word_count()
countChars - equivalent to strlen()
money - equivalent to money_format(), but the settings can be written in the OPT configuration.
number - equivalent to number_format(), but the settings can be written in the OPT configuration.
wordWrap - equivalent to word_wrap()
replace - equivalent to str_replace()
regexReplace - equivalent to preg_replace(), but the parameter order is changed.
date - equivalent to date() :)
firstof - returns the first non-empty element from the specified ones.
nl2br - equivalent to nl2br() :)
array - creating arrays on the template side
build - it was called "apply()" in the previous versions, but I still don't know, whether I keep this name or not.

The set contains no escaping functions, because the compiler is able to put them on his own in most of the places. There will be a possibility to escape something manually, but it won't be done through functions. Anyway, the point is the compiler must know, what is escaped. Otherwise it may add additional function call and this would produce a complete mess on the screen.

To sum up, I've written so much functionality that I haven't tested all of them yet. I don't know if something works, however I don't expect any revelations. Many of the algorithms are reimplementations of the same code from OPT 1.1.x. The next dev version will be available in the nearest days.

Some statistics for the end:

- opt.class.php - 24 KB and still growing. In OPT 1.1.x, this file was 8 KB bigger, but offered less functionality.
- opt.compiler.php - 65 KB and still growing.
- opt.instructions.php - 25 KB and still growing.

OPT 2.0.0-dev3

Zyx — Fri, 15 Feb 2008 13:26:00 +0100

According to the my announcement from the last week, the development versions are going to appear much faster than before. This is the first of such events. The third dev of OPT 2.0.0 is ready to download. This time, I've paid much attention to the attribute processing. They can do a lot of interesting things with the tags they've been added to. The earl implementation has been ready so far - there were even examples showing the "opt:section" working. But to satisfy both me and probably other programmers, I had to extend it.

Here are some examples showing, what can be done now. Let's start from something simple. In OPT, we use curly brackets to put expressions into a text, like:

text text text {$block} text text text

There is also opt:put instruction, which does the same for the first time we see it:

text text text <opt:put value="$block"/> text text text

However, it is an instruction, not curly brackets. It means we can make some nice things with it using attributes. Let's say we want to display something like "Value 1 / Value 2 / Value 3". We will use opt:put, a section and a separator:

<p><opt:put value="$section.value" opt:section="section" str:separator=" / "/></p>

As we see, such section affects also the tag which it was linked to and we can put the values of section elements in the attributes. The separator accepts expressions by default, but if we don't like writing separator="'something'" (double, and then a single quote), we can just switch the namespace to "str".

The implementation of reversed quotes is also ready. Compared to the previous versions, they change their function. Now it is the programmer who can give them a meaning and program them to do what he wants. It can be done by writing a simple function and giving its name to the parser. A sample use is an elegant implementation of language system a'la gettext:

<p>{`Some useful information...`}</p>
<p>{`Author:`} {$author}</p>

This is the last of the changes. As we have an XML compiler, we need some special instructions that allow to manage the tree directly, for example by adding an attribute with a variable name. OPT is going to provide two instructions for this task: opt:tag and opt:attribute. The second one has been already added to the newest dev. We can use it to read the attribute name from the block:

<div>
  <opt:attribute name="$some_name" value="$some_value"/>
  body...
</div>

This is not the end - connected with sections, we get a powerful tool to add a lot of new attributes:

<div>
  <opt:attribute name="$attr.name" value="$attr.value" opt:section="attr"/>
  body...
</div>

If we send the following table to this template: array(0 => array('name' => 'class', 'value' => 'dude'), array('name' => 'id', 'value' => 'lol')), the result will be:

<div class="dude" id="lol">
  body...
</div>

Unfortunately, the English manual is not started yet, but it will be ready before the first stable release.

Download: http://www.invenzzia.org/download/opt-2.0.0-dev3.tar.gz

New dev

Zyx — Sat, 09 Feb 2008 11:02:00 +0100

The website now has the "Download" page. In it, you can find a new development version of Open Power Template. It also contains a simple script "docgen" which we use for automated generation of the DocBook manuals. The new development versions should appear now much more often than before. In case of many new features, they might appear even every 2-3 days. The reason why this was impossible in the past was the time. The process was not automated and it takes a while to prepare such archive (for example - it takes a couple of hours to publish new OPT 1.x version on the Internet).

The pack contains everything I've implemented for last month:

Nearly completed sections.
Separators.
"For" and "Foreach" loops.
Mostly completed template inheritance.
HTTP header management.
Partially completed prolog parser.
Some fixes in the compiler.

The "/dev" directory contains some scripts that are used to test the new features. There are also unit tests: a complete set to test the expression parser and a part of the API test. There will be also a one to process the instructions, however the whitespace characters cause many problems, when it comes to prepare the files. If we try to compare the generated content with the expected result, we have to watch out on every space or line break. Otherwise the test fails, so you must be very patient to write a simple, stupid test. But I think this issue will be solved. To work, the tests require "phpUnit 3" package available in PEAR.

Template inheritance

Zyx — Sat, 02 Feb 2008 17:03:00 +0100

Unsually, the final HTML code of the website is composed of several smaller templates containing smaller pieces of it. There are many techniques of combining them. In PHP template engines, the most popular are two forms: manual execution of the templates from the script side and the include directive. Similar solution can be found in OPT 1.x, however in the new version, it should be forgotten - unless we use the quirks mode, the parser does not let us to call the parse() method more than once. In the beginning, I planned to replace it by extending inclusion with "sequences" - a groups of templates to include with their own variables and cache settings. However, one day I was asked on my blog by Coldpeer, whether OPT will support template inheritance. I googled for a while, I looked at it and thought it might be cool. That's how the implementation began.

As far as I know, template inheritance appeared for the first time in the Python framework, Django. The idea is taken from the object oriented programming, where we can extend classes by adding new methods, fields and overwriting the existing ones. Here it looks similar. We have a main template where we mark some characteristic places and fill them with a default content. Another template extends the main one and is able to overwrite that places with its own content. There can go the next template, and next...

I was quite surprised that a lot of required functionality already exists in OPT 1.x - the bind and insert instructions allowed to move whole template parts somewhere else (as well as their functionality, not only the result, like in Smarty's capture), even if the destination was in the other template. Unfortunately, there was no proper detection, whether one of the files was modified, so unsually the parser decided that the main part needs to be compiled, but the template with snippet definitions - does not. As a result, the places remained empty, although they should not. The development of such system for OPT 2.0 was quite complicated. You do not have to check just the template you've called, whether it is modified. You must also look at all the templates it inherits. Yesterday, I decided to solve this problem definietly and a couple of hours ago everything seemed to work properly:

The base template (test_inherited_a.tpl):

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en_US" xml:lang="en_US">
 <head>
  <title>Instruction test: snippet and insert</title>
 </head>
 <body>
  <opt:insert snippet="header">
   <h1>I'm a standard header</h1>
   <p>Foo bar joe</p>  
  </opt:insert>
  
  <hr/>
  
  <opt:insert snippet="content">
  	<p>Well, i'm also a standard content.</p>
  
  </opt:insert>
  
  <hr/>
  
  <opt:insert snippet="footer">
  	<p>And I'm a footer.</p>  
  </opt:insert>
  
  <p>&copy; Pasteright 2008 by LMAO, It seems to work!</p>
 </body>
</html>

The extending template (test_inherit_1.tpl):

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<opt:extend file="test_inherited_a.tpl">
	<opt:snippet name="header">
		<h1>Webmaster Of Puppets</h1>
	</opt:snippet>

	<opt:snippet name="content">
		<p>Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Phasellus ut tellus id nulla adipiscing eleifend. Sed dictum accumsan ante. Nullam at nisl vitae elit aliquet fringilla. Praesent egestas eros eget tellus. Praesent id odio a sapien rhoncus vehicula. Nunc fringilla, diam eget euismod tempor, tortor metus tincidunt sapien, eu cursus magna tellus at risus. Praesent non tellus eget magna facilisis pulvinar. Praesent libero mi, adipiscing a, pharetra eget, condimentum sodales, mi. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Donec ac elit. Duis iaculis tortor a metus. Aliquam id purus et eros faucibus fringilla. Praesent quis quam. In lectus urna, fringilla sit amet, iaculis eget, aliquet ac, quam. Donec vulputate dui sit amet lectus. Aenean tempor, orci at pretium ornare, tortor tortor venenatis ligula, eget blandit nisi risus eget dolor. Duis nunc neque, sodales porta, viverra non, tristique eu, sem. Curabitur magna neque, blandit ullamcorper, congue quis, tristique ut, felis.</p>
	</opt:snippet>

	<opt:snippet name="footer">
		<p>Bye!!!</p>
	</opt:snippet>
</opt:extend>

Some PHP code:

<?php

	define('OPT_DIR', '../lib/');
	require(OPT_DIR.'opt.class.php');

	try
	{
		$tpl = new optClass;
		$tpl -> sourceDir = './templates/';
		$tpl -> compileDir = './templates_c/';
		$tpl -> stripWhitespaces = false;
		$tpl -> printComments = false;
		$tpl -> setup();
		
		$tpl -> parse('test_inherit_1.tpl');
	}
	catch(optException $e)
	{
		optErrorHandler($e);
	}
?>

And the result can be admired below:

Some facts:

If we modify any of the templates in the inheritance chain, the parser will notice it and recompile proper files.
If we remove any of "opt:snippet" instructions, the default content of "opt:insert" will be displayed.
There is a possibility to overwrite an existing snippet by the more important template. You will be also able to call the parent content, however it is not implemented yet (some kind of "opt:parent" instruction).
It can be mixed with "opt:include" and "opt:sequence" (the sequence system briefly mentioned in the introduction).

What is more, it's quite possible that the template inheritance will work faster than traditional way, although it will consume more disk space. Simply, if we have an inheritance like this: "A extends B extends C", the compiler puts all their code in the file belonging to the "A" template. It reduces the amount of disk operations (the execution of one bigger template is faster than the same content in five files). On the other hand, by default OPT still checks the file modification times, but fortunately, the number of tests is also smaller. The exact results will be available, when some benchmarks are done. Before we end, I'll say that two days ago I made a small test (a simple list with 15 items). The result:

OPT 1.x: 1850 req/s
OPT 2.0: 2100 req/s

Well, it looks promising, but we have to remember that the new opt.class.php file still does not have half of the options, which obviously will increase its size. Moreover, some optimizations are not implemented yet. But in fact, you can assume that the new features should not destroy the performance, and there is a chance that it will be even better.

Deeper processing

Zyx — Thu, 31 Jan 2008 13:42:00 +0100

The last project I've created using Open Power Forms, shown some weaknesses of the current form implementation. There were some forms allowing a mass addition of many records in the following way: a table, in the columns the following fields: Name, Surname, PESEL number, etc. The numer of rows was 30, so the user is able to add 30 people to the database with one form. The adaptation of the map() method to my problem was not too hard; in fact it seemed that opfArrayContainer solved everything. But then the form renderer connected with Open Power Template went off.

My code looked like that:

php
$this -> map('names', new opfArrayContainer(
	new opfConstraint(MAP_TYPE, TYPE_STRING),
	new opfConstraint(MAP_LEN_GT, 2)
), OPF_OPTIONAL);
$this -> map('surnames', new opfArrayContainer(
	new opfConstraint(MAP_TYPE, TYPE_STRING),
	new opfConstraint(MAP_LEN_GT, 2)
), OPF_OPTIONAL);
$this -> map('pesels', new opfArrayContainer(
	new sConstraint(MAP_PESEL)
), OPF_OPTIONAL);

In other words, the script must have received for example the "names" field, which was an array of 30 elements with the names. If the user made a mistake while filling in the form, the form renderer did not have a clue, which component to display as invalid (in this case it should be colored red instead of green). Finally, I added a support for the IGNORE_OPF_ERRORS constant so that shut it up and just display everything without all this color rotation stuff.

Of course, it is not possible to keep this problem unsolved in the "public" releases and I decided to look for solution. It would be nice, if the improvement easily matched to the code, as well as was intuitive. It's not a problem to add something in 20 minut and a month later self-wishing a rapid death. The best solutions are simple and effective, so I started to think, how to describe a form with some definitions and transformations, so that it would be possible to construct every form and process it with OPF. My idea is to go somewhere deeper and remove distinguishing between forms and form elements. The reasons:

Both have a value (in the form case we define it as a set of subelement values).
Both have a state.
Both can be processed.

Now we can easily treat a form as an element which is capable of storing other fields. If we try to build our form described above, it will look like this: we create a form for a single person: name, surname, PESEL, etc. Next, we put it into a bigger form and order to loop up to 30 times. We can also achieve an effect of looping a single field, like it was possible with opfArrayContainer() so far. We just put in such loop not a form, but a single element.

Well, now the problem is, how to render it :). However, I don't think it will be complicated, especially if I look at the new Open Power Template.