Invenzzia... in English

TypeFriendly 0.1.0 released

Zyx — Wed, 23 Jul 2008 10:34:00 +0200

At last, the translation of the manual into English is complete and the source code has been already released as the first version of TypeFriendly, the documentation generator. In is available in Files on the main site. The user manual is provided in the source form and it works also as an example, so please do not worry that something is missing. All the necessary information to build the HTML version can be found in /info/README.txt file. All you need is to type one command in the system console. The on-line documentation will be available soon, as we are preparing new website. To get more information about TypeFriendly, take a look at the previous note.

TypeFriendly

Zyx — Wed, 02 Jul 2008 21:38:00 +0200

Together with eXtreme, we work hard to make the first release available as soon as possible and we are quite close now. This is a perfect opportunity to tell something more about this project, which should be appreciated by those programmers who write their own scripts and libraries and look for a smart tool to make a documentation. The idea to write TypeFriendly was born in my head after I failed to add some necessary things to the XSLT sheets for DocBook. The PHP manual shows, what could be done with it (syntax coloring, lots of different formats, enormous amount of tags), but don't become too happy too fast. To achieve similar effects, we need weeks of coding and we must also fight against the portability. In fact, I'd rather use those weeks to write something more useful and friendly.

So, TypeFriendly is a documentation writing tool, but it works differently than for example phpDocumentor which scans the source code of your project and makes a reference list of all classes and functions. Here, we write all the chapters and put them in the required order on our own, like in DocBook. However, in this place the similarities end, because TF already contains everything we need. Moreover, it was designed to be intuitive and easy to learn.

Features:

In TF, we write our manual in standard text files using the Markdown syntax. We can put some extra information with tags placed at the beginning of the file. They allow us to define a title, version information or "See also" section.
TF automatically sorts our files and connects them into bigger chapters, basing on their file names. By default, all the content is sorted alphabetically, but we can easily change it for all the files we need. We just create an additional file with "sorting hints", where we write the required order.
There is a built-in syntax highlighting option.
TF generates all the navigation and table of contents.
TF is designed to create a multilingual documentation. Apart from storing many versions of the same file and interface texts, it contains a simple tool to detect whether the "derived language versions" are up-to-date.
TF allows to generate the XHTML version very quickly, on a single page, as well as on many. We are also creating a special format for importing the manual to the database (to publish it online and for example, to add user comments).
TF packs your manual by default in the nice, green layout.

Although there haven't been any release so far, you can always connect to our new SVN and download the code from there. The link to the checkout commands is: http://svn.invenzzia.org/svn/typefriendly/trunk/ - we encourage you to test the script now. In the repository, there is a README file and a nearly finished Polish version of the TF manual. But don't worry. The first release will be available with the English version, too. It's much shorter than OPL manual, and nice to translate :).

Now, I think it's a good idea to show an example. The first one would be one of the files from OPT template engine manual that we develop, too:

Title: optNode class
ShortTitle: optNode
Status: abstract
Extends: library.optcodebuffer
ExtendedBy:
 - library.optscannable
 - library.optcharacterdata
 - library.optexpression

----
An abstract XML node class. It provides the basic support for the most important features, such as type and parent.

Class fields
------------
All the fields are protected.

 Name          | Type      | Description
---------------|-----------|---------------------------------------
 $type         | Integer   | Numerical type identifier
 $parent       | optNode   | Node parent

Available type identifiers:

1. `OPT_CDATA_NODE` - `optCharacterData`
2. `OPT_TEXT_NODE` - `optText`
3. `OPT_EXPRESSION_NODE` - `optExpression`
4. `OPT_ELEMENT_NODE` - `optElement`
5. `OPT_ROOT_NODE` - `optRoot`

At the beginning, we have a header with tags. There we define the exact title, and in this case also the dependencies between the PHP classes. Notice that we refer to the other classes by writing the identifier of the chapter, where they are described. Of course, there are additional tag versions, where we can write the class names directly, for example if we inherit from some PHP built-in class. There are more available tags. Let's say we want to make a "See also" section:

SeeAlso:
 - chapter.text1
 - chapter.text2
 - chapter.text3

Once the tag list finishes, we begin the exact content in the Markdown format. The author designed it basing on the formatting used in text files and e-mails (it can be said that he wrote a parser for a syntax that self-evolved :)). This means that the source text is really clean and easy to read. In fact, it is just a bit lesser comfortable than a parsed one. The problem is, if someone is accustomed to some wiki syntax, because MD not always works in this way (take a look at the table). But it is quite easy to switch. We use Markdown even on our website and I must say that it is a very convienient tool. I'm sure you'll find it in more Invenzzia products. There is only one big shortcoming - the only available output is XHTML. For now, you have to forget about LaTeX and the rest, but I can assure you that we plan to add it, too.

The most frustrating problem while writing a manual is navigation. The links "Previous", "Next", "Up" are added by TypeFriendly, we create the "See also" section by adding a tag, but what about clickable function and class names? I thought about it a bit and I invended a brilliant and simple idea. We have an additional text file, where we define, what text points to what. For example, we write there that `optClass::parse()` must always be a link to library.optclass.parse and that's all. Unfortunately, we haven't hacked Markdown in this way so far and the first release will not contain it yet.

TypeFriendly works in the operating system command line. We've tested it both on Linux and Windows and it works there without any problems. All we need is a PHP parser. Togethet with a simple syntax and built-in multilingual support, TF should encourage much more people to help in translations of your manuals, improving the base version and maybe even writing new outputs. Generating the XHTML version is really easy:

php ./typefriendly.php -l en -o xhtml ./path/to/manual/

Let's compare it to DocBook. The format itself is a bit longer, but relatively easy to learn. However, the conversion needs more tools. We must download the DTD, DocBook XSL Stylesheets and an XSTL parser. The choice is important, because fast and easy xsltproc will not highlight your syntax in the examples. On the other hand, the Windows users are f@#$ off if the author decides to write half of the processing tools in Bash. We don't have to look such examples too far. Just take a look at the PHP manual. Guess, what is it written in? :)

SVN is active!

Zyx — Mon, 30 Jun 2008 08:10:00 +0200

Unfortunately, the configuration of a version control system on our current server is not as simple as on Sourceforge.net. But now I found some time and ideas, how to make it work and finally... there it is, we have an SVN repository. It's available under the address http://svn.invenzzia.org/ - we still have to install a good browser. Today I'm going to upload the source code of the new OPL and TypeFriendly, a documentation generator almost ready to its first release. All the instructions, how to connect, will be published on the main site.

OPT requirements

Zyx — Sun, 06 Apr 2008 10:28:00 +0200

I've found a new entry on the bugtracker recently. The author informed that he gets "Nesting level too deep - recursive dependency" error while running the development examples provided by the newest OPTv2 dev version. I looked at it deeper and found out that he lowered the default nesting level limit 4 times from 64 to just 16. Obviously, this is a quite drastic change and would affect not only OPT, but many other scripts as well. However, I think it is time to take a look at the OPT issues concerning the requirements, because several of them needs a deeper explaination.

The optimizations implemented in OPT, mostly concern the main parser class and the compiler. The first one is the part that is loaded every time the script wants to use templates, so it must provide huge quality standards. There are no sophisticated algorithms, so the speed improvements are:

To minimize the size of the code.
To minimize the amount of points that try to access the hard drive.
To use the fastest elements of the language.

The point is this class must be as light as possible; I would even say: almost transparent for the script.

The situation is quite different, when it comes to the compiler. It is loaded from time to time, so it can be big and splitted for more files (of course without exaggeration). The key role is played by the limits set in the PHP configuration, because in the pesimistic situations, the compiler runs out of resources to complete the task. Unfortunately, some of the new features cost a lot, and if the users wanted XML tag parsing, they must face the consequences: every such tag requires its own node object in the document tree and they need some preparation tasks to be done. My role is to minimize the time and memory necessary to do this.

The elements that can be exhausted during the compilation are:

The stack
The memory

The algorithms used in the compiler operate on a data structure called tree, and the use lots of recursion. It consumes the stack, because the number of currently executed methods and functions grows quite fast. In PHP, the default limit for the stack depth is 64 and if you are going to use OPT, the worst you can do is to lower it to ridiculous values like "16" that would affect not only OPT, but also many other scripts :). However, every recursion has an imperative version, constructed for example by queues. Maybe it does not consume less memory, but at least it does not depend on the limits. Such changes have been applied to many compiler methods so far, but some of them still require my attention, because they are not so trivial:

Processing the tree by the instruction processors
Expression compilation

In the first one the situation is clear: we get a node of some instruction and want to process it. We find its processor and redirect the node to it. It generates some PHP code and decides, what to do with its children. By default, it wants to process them, too. Here comes the recursion and the process starts from the beginning. There are some difficulties. The algorithm is split over many classes, and the programmers may create their own. What is more, some instructions need to start the subnode processing immediately, because their next decision depends on the code generated by the children. Obviously, this eliminates queues. An elegant replacement could be made, if we had an access to the low level programming.

Let's take a look, how it affects the compilation process. We have a huge template, with the tags nested up to the level 30. OPT with the compiler can take another 10 calls, and in the deepest place we have a complicated expression with brackets. To sum up, the library uses approx. 45 calls from 64 available, which is more than 70% of the limit! I know this case is quite pesimistic and in normal life such HTML documents are very rare, but it shows, how demanding the compilation can be.

If we can't remove the recursion without ugly tricks which would be never accepted by the instruction authors, how we can get around it? The idea is not to remove recursion completely, but we keep it in some places. I use here the fact that the immediate access to the subnode compilation results is necessary only for some instructions, and for example the HTML tags do not need it at all. If we get a random node from the tree, we have a big chance that it would not need a recursion here. So, why don't we provide a dual version of this algorithm? By default, it is executed with a queue, but for more sophisticated needs, some nodes can switch it to recursion. Now it can happen that a tree with a depth 30 is processed in one single loop, with no recursion! I predict that in normal situations, the algorithm will consume only 3, 4 spare "items" in the stack, which is fully acceptable.

The second issue concerns the memory usage. It's not so tragic here, but we can notice a significant rise when compared to the OPT 1.x compiler, even with using some patents from that version. The most memory is consumed by the tree. I've measured some single nodes recently and the results were from 2 kilobytes for a single node to 3 kilobytes for a node with one attribute. When the instructions start to add the PHP code to the buffers, the things are getting worse, but it happens only in some nodes in the whole template. We can safe some memory here. The objects use various arrays, which are initialized just in the beginning, for example:

abstract class optScannable extends optNode implements Iterator
{
	protected $subnodes = array();

In most of the nodes, the arrays are empty, so it it no sense for them to exist all the time. I fixed the code so that it creates an array only when it is necessary. From the declarations, the = array() parts were removed. The effect is quite interesting. On a single array, we safe 400 bytes. The new results are:

Single node with no attributes: 1400 b.
Single node with one attribute: 2350 b.

The additional memory may be safed by releasing the huge data that will not be used anymore, for example the template content.

What do we have to remember, while using the new OPT, when it comes to the requirement issues:

With the terribly low limits, the compiler will always crash. I thing 16 megabytes for the script (the default limit) is big enough both for the compiler and the script.
The best place to run the template parsing is the end of the script. We can delete some unncessary data earlier to free some memory.
The templates using the inheritance are more demanding, especially when overwriting snippets, because the compiler must keep also the old versions in the memory.
The quirks mode will consume less memory (simpler tree - no XML tags), but it is not implemented yet.
The compilation is run once for a long time. During the normal execution, we can forget about the limit issues etc.

I think this note shows, how difficult the library creation can be. The key is to know, what we can use in current place and that there is also the rest of the script which uses the same memory and has its own requirements.

OPT 2.0.0-dev5

Zyx — Fri, 21 Mar 2008 14:38:00 +0100

The new development version became available a bit later that I planned, but on the other hand, I managed to add much more new things. The DTD issue is solved, as well as XML prologs, CDATA support and code escaping. The expression compiler is done and the programmers may use now brand new component implementation. This is a significant step towards the first beta release and the end is closer than further. There are only a few instructions to code, the rest is almost completed and works. The compiler, programmers' API and most of the stuff also works. I think of making the first real tests on my home website and creating the Open Power Forms library compatible with the new OPT.

In the new OPT, I've redesigned the component design from scratch. It is much more flexible now and allows to make easily many interesting effects. The base rules are still the same, but the rest has changed. This is a sample code showing, how the component-based work with the forms is going to look like:

<opt:snippet name="componentView">
  <com:div class="field" invalid="field error">
    <p>{$opt.component.title}: <opt:display /></p>
    <p class="desc" opt:if="$opt.component.description">{$opt.component.description}</p>
    <opt:onEvent name="error" message="info">
      <p class="error">{@info}</p>
    </opt:onEvent>
  </com:div>
</opt:snippet>

<div class="form">
<form method="post" action="skrypt.php">

<opt:input name="name" template="componentView">
   <opt:set name="title" str:value="Enter your name" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="surname" template="componentView">
   <opt:set name="title" str:value="Enter your surname" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="age" template="componentView">
   <opt:set name="title" str:value="Enter your age" />
   <opt:set name="description" str:value="Scope: 10-99 years" />
</opt:input>

<div class="foot"><input type="submit" value="Send" /></div>
</form>
</div>

The new implementation makes use of the opt:snippet instruction and template inheritance. If you had enough time, you might code the component objects, so that the field titles and descriptions will be unnecessary in the template. A component can handle events, displaying itself and its neighbourhood, managing parameters and tag attributes (we use the com namespace for it). OPT contains only the component mechanism, and the components itself must be written by the end user. The reason is very simple. In order to make a useful component, we should integrate it with the input validation classes, language system, error management etc. The demands are much bigger than OPT is able and is supposed to provide. However, there will be a ready-to-use component implementation: Open Power Forms library.

Unfortunately for the English manual is not started yet. Even the Polish users must use a partially-complete text, without formatting, because the new rendering system is not ready. However, we'll do everything to translate all the text together with some other materials (advices, FAQ, maybe even a tutorial?) into English, before the first final release is done. There is a team, I'll try also to ask some friends and we'll make a serious English support. Just be patient.

Download OPT-2.0.0-dev5

OPT 2.0.0-dev4

Zyx — Sun, 02 Mar 2008 11:28:00 +0100

There is a new Open Power Template development version available to download. The project begins to gel, as the main parser class is almost completed. Inspite of the output cache system mentioned earlier, I've implemented new instructions and completed the snippets. It means that all the template inheritance should work now. I hope the library will be usable by the end of this month. The real projects are really necessary to detect possible bugs and various shortcomings.

In snippets, we have a new possibility to insert the parent content using the opt:parent tag:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<opt:extend file="test_inherited_a.tpl">
	<opt:snippet name="header">
		<h1>Webmaster Of Puppets</h1>
		<p>Here is the parent content:</p>
		<opt:parent/>
		<p>End of the parent content</p>
	</opt:snippet>
</opt:extend>

In this case, if the test_inherited_a.tpl contains another snippet with the same name, its content will be put in the place of opt:parent. Such nesting works much deeper. For example, let's say that snippet also uses this tag. OPT will try to locate another parent and so on. It is even possible to put the default content of the opt:insert tag, which is used to place the snippets. The code contains also a simple code to detect the infinite recursion. If the parser finds it, it returns an exception and the programmer sees a nice, red error message :).

During the using of the new OPT, the new debug console should be also useful. It has been extended, compared to the similar option in OPT 1.1.x. The most significant change is splitting the list of templates to the smaller ones which show different information:

Executed templates
Currently compiled templates
Templates read from cache.
Inherited templates - it shows all the inheritance chains, as well, as the information whether they have been recompiled or not.

Moreover, the console displays all the warnings. They are no longer put onto the screen, but available only with the debug console or error messages, if we set the proper debug level.

The performance issue also looks very good. The total size of opt.class.php file is about 25 KB, compared to 32 KB in the OPT 1.1.x. Currently, its size influences the performance much, as PHP must load the parser and compile it to the internal bytecode. Of course, if we parse a lot of templates, it won't be so visible, however I'm quite pleased of the effect.

The package can be downloaded from here.

New Invenzzia.org

eXtreme — Tue, 26 Feb 2008 17:20:00 +0100

One month ago we had no time to prepare a good website for our group so Zyx made something temporary. Today I present a better and more complex solution. The website has been updated with texts and articles about our team. We've got still a lot to do with organization but our website and devblog are almost finished.

I installed and configured dotClear2 from scratch. Now, there is no problem with mixed Polish and English entries because I set up two independent miniblogs on one installation. User language detection is now integrated to the website's system and it is based on cookies and browser settings. I made also nicer and more friendly theme.

And why not Wordpress? I don't like it. There are no real and friendly templates (yes, I love sweet chaos with mixed PHP and HTML). So far, I had an unpleasant opportunity to create from scratch a template for Wordpress and it was a terrible experience. I have been working with dotClear2 earlier and it is very easy to configure everything and convert default theme to website's design. Moreover, it is a nice piece of OOP code in PHP5. ;-)

Team Invenzzia hasn't got the official logo yet. This one on website is only a temporary solution.

The main class almost finished

Zyx — Fri, 22 Feb 2008 08:59:00 +0100

The result of the latest works on the OPT source code is complete opt.class.php file which contains (as some of the readers may know) the main parser class. I must admit that this file was more demanding that the compiler. In the second case we can afford with quite a lot. Just an idea for the algorithm and prediction if and how it will influence the rest of the code. In the case of the main class, we must pay attention to the performance. It must contain everything that is run permanently and it must be fast and small. But the number of features is getting bigger and bigger.

The new development version will not be compatible with the previous ones when it comes to the programming interface. To implement the output cache system, I had to rebuild almost all of the output system. In the 1.x.x series, the concept of the output system did not exist. Either the result will be seen on the screen or put into a variable, it depended on the right method choice. There were some tricky connections which redirected everything to the right place, however it was not well designed and a bit messy. Well, such things simply must happen, if we start with keeping the similarities to Smarty, and I have to remind that in the very early times, it was one of the OPT goals. In OPTv2, there is a well-declared ioptOutput interface allowing the programmer to code the output processing on his own. It is also used in the implementation of output cache system. How - it is explained below.

So far, there have been two output systems: optNetOutput sending everything on the screen and checking, how many templates we are going to send in the XML mode. The second one was optReturnOutput, which returned the result. I added a possibility to decorate them with something else and I created optCachedOutput. Its only goal is to process the cache. The features are the same as in the previous versions, but the usage is a bit different and unfortunately, the initialization is more complicated:

$cache = new optCachedOutput('./cache/'); // this is the cache, here we configure it
$tpl -> register(OPT_OUTPUT, 'my_cache', $cache);
$tpl -> decorate(OPT_OUT_SCREEN, 'my_cache');
$tpl -> decorate(OPT_OUT_RETURN, 'my_cache');

if(!$cache -> isCached('template.tpl', 'some_id'))
{
  // data generation
}

$cache -> setCache(5, 'some_id'); // turn the cache on for 5 seconds
$tpl -> parse('template.tpl', 'my_cache');

I'll try to shorten it and make simpler the first lines of the code, because I don't like them, too. Moreover, I must check if I've made the decoration in the right way (according to the logic).

The support for plugins is not fully completed yet. The only ready part is placed in opt.class.php file. The rest is moved somewhere else, which should improve the performance. Like in OPT 1.1.3, there will be a possibility to set many plugin directories, but the parser will not require them to be writeable. There were thrown some files describing the content and allowing to include everything without scanning all the directory. As we know, the more includes, the less performance and some people can produce more plugins than all the files in my framework :). The new OPT will try to link all these files into one, mega-file containing everything we need. By the way, I'll have also a new algorithm for the OPT Toolset sequel, which will be used now for all the libraries, not only OPT.

The last issue concerns the available function set for the templates. Stormfly has recently written on his blog that OPT 1.1.x it was quite senseless and in fact, he was right. I took a look, what Smarty offers and it turned out that it has some nice and useful functions. The first list looks like that:

spacify - puts a string between every two neighbour characters in the text.
indent - makes an indentation in the text.
truncate - reduces the text to the specified length (allowing not to splitting the words)
strip - reduces all the white character groups to a single character
capitalize - equivalent to ucfirst()
upper - equivalent to strtoupper()
lower - equivalent to strtolower()
countWords - equivalent to str_word_count()
countChars - equivalent to strlen()
money - equivalent to money_format(), but the settings can be written in the OPT configuration.
number - equivalent to number_format(), but the settings can be written in the OPT configuration.
wordWrap - equivalent to word_wrap()
replace - equivalent to str_replace()
regexReplace - equivalent to preg_replace(), but the parameter order is changed.
date - equivalent to date() :)
firstof - returns the first non-empty element from the specified ones.
nl2br - equivalent to nl2br() :)
array - creating arrays on the template side
build - it was called "apply()" in the previous versions, but I still don't know, whether I keep this name or not.

The set contains no escaping functions, because the compiler is able to put them on his own in most of the places. There will be a possibility to escape something manually, but it won't be done through functions. Anyway, the point is the compiler must know, what is escaped. Otherwise it may add additional function call and this would produce a complete mess on the screen.

To sum up, I've written so much functionality that I haven't tested all of them yet. I don't know if something works, however I don't expect any revelations. Many of the algorithms are reimplementations of the same code from OPT 1.1.x. The next dev version will be available in the nearest days.

Some statistics for the end:

- opt.class.php - 24 KB and still growing. In OPT 1.1.x, this file was 8 KB bigger, but offered less functionality.
- opt.compiler.php - 65 KB and still growing.
- opt.instructions.php - 25 KB and still growing.

OPT 2.0.0-dev3

Zyx — Fri, 15 Feb 2008 13:26:00 +0100

According to the my announcement from the last week, the development versions are going to appear much faster than before. This is the first of such events. The third dev of OPT 2.0.0 is ready to download. This time, I've paid much attention to the attribute processing. They can do a lot of interesting things with the tags they've been added to. The earl implementation has been ready so far - there were even examples showing the "opt:section" working. But to satisfy both me and probably other programmers, I had to extend it.

Here are some examples showing, what can be done now. Let's start from something simple. In OPT, we use curly brackets to put expressions into a text, like:

text text text {$block} text text text

There is also opt:put instruction, which does the same for the first time we see it:

text text text <opt:put value="$block"/> text text text

However, it is an instruction, not curly brackets. It means we can make some nice things with it using attributes. Let's say we want to display something like "Value 1 / Value 2 / Value 3". We will use opt:put, a section and a separator:

<p><opt:put value="$section.value" opt:section="section" str:separator=" / "/></p>

As we see, such section affects also the tag which it was linked to and we can put the values of section elements in the attributes. The separator accepts expressions by default, but if we don't like writing separator="'something'" (double, and then a single quote), we can just switch the namespace to "str".

The implementation of reversed quotes is also ready. Compared to the previous versions, they change their function. Now it is the programmer who can give them a meaning and program them to do what he wants. It can be done by writing a simple function and giving its name to the parser. A sample use is an elegant implementation of language system a'la gettext:

<p>{`Some useful information...`}</p>
<p>{`Author:`} {$author}</p>

This is the last of the changes. As we have an XML compiler, we need some special instructions that allow to manage the tree directly, for example by adding an attribute with a variable name. OPT is going to provide two instructions for this task: opt:tag and opt:attribute. The second one has been already added to the newest dev. We can use it to read the attribute name from the block:

<div>
  <opt:attribute name="$some_name" value="$some_value"/>
  body...
</div>

This is not the end - connected with sections, we get a powerful tool to add a lot of new attributes:

<div>
  <opt:attribute name="$attr.name" value="$attr.value" opt:section="attr"/>
  body...
</div>

If we send the following table to this template: array(0 => array('name' => 'class', 'value' => 'dude'), array('name' => 'id', 'value' => 'lol')), the result will be:

<div class="dude" id="lol">
  body...
</div>

Unfortunately, the English manual is not started yet, but it will be ready before the first stable release.

Download: http://www.invenzzia.org/download/opt-2.0.0-dev3.tar.gz

New dev

Zyx — Sat, 09 Feb 2008 11:02:00 +0100

The website now has the "Download" page. In it, you can find a new development version of Open Power Template. It also contains a simple script "docgen" which we use for automated generation of the DocBook manuals. The new development versions should appear now much more often than before. In case of many new features, they might appear even every 2-3 days. The reason why this was impossible in the past was the time. The process was not automated and it takes a while to prepare such archive (for example - it takes a couple of hours to publish new OPT 1.x version on the Internet).

The pack contains everything I've implemented for last month:

Nearly completed sections.
Separators.
"For" and "Foreach" loops.
Mostly completed template inheritance.
HTTP header management.
Partially completed prolog parser.
Some fixes in the compiler.

The "/dev" directory contains some scripts that are used to test the new features. There are also unit tests: a complete set to test the expression parser and a part of the API test. There will be also a one to process the instructions, however the whitespace characters cause many problems, when it comes to prepare the files. If we try to compare the generated content with the expected result, we have to watch out on every space or line break. Otherwise the test fails, so you must be very patient to write a simple, stupid test. But I think this issue will be solved. To work, the tests require "phpUnit 3" package available in PEAR.

Licensing

Zyx — Thu, 07 Feb 2008 10:19:00 +0100

From time to time, I return to the library licensing issue. Undoubtedly, the right choice of a license is quite significant for the project success or failure. Simple, permissive licenses are very easy to follow and use, but in my opinion, they are quite dangerous for the project itself and can lead to some abuses. On the other hand, we have a GNU LGPL license, which has been used by Open Power Template so far. It provides the correct protection of the code, however the linking with non-GPL software is quite complex and produces many questions. Let's take a look: we write the libraries for all the website programmers and authors of complete scripts, whether they are open-source or not.

I decided to check, how much place the "simple" explaination of LGPL conditions to the end user would take, especially if he is about to make a commercial use of his software. I added the new appendix to the documentation and began to write. The result was a two-screen-long list of explainations, why and how the website creators can use it, when the source code must be published etc. Of course, it is unacceptable. I don't like too complex constraints and my intention is not to force the others to follow them unless there is a significant reason.

The reason why I chose LGPL, was to protect the library against changing it into a proprietary software. Such story happened to Wine project, distributed under the X11 (MIT) license. After a while, a proprietary versions appeared, whose creators did not wish to share their source code. Although one 150-KB library is not too valuable, I prefer to be calm and be sure, especially if the other project are going to appear. Unfortunately, there is no license that would suit to our needs:

- Full right to link with open-source and proprietary software under the condition of retain/reproduce copyright information, and retain the license conditions in case of source code.
- Requirement to redistribute the code itself (intented to link with non-author's software) with the source code, and as a consequence, retaining the license.

I think such license would be the best protector of end-user laws. By forbidding the redistribution of the library itself only in a binary form and/or witch changed license, he is sure that every OPT package found in the Internet can be used in the same way, as the original one. If he encounters a closed-source version, it means that it is adapted to this exact project and it is not indented to link with other scripts.

The best candidate is one of the modified BSD licenses without the advertising clause. Unfortunately, I'll have to add the statement mentioned above on my own in English, and the legal language is not so easy. It will take some time and probably will require some help from the West. That's why the development versions of OPT will be still available under GNU LGPLv3 and the final release will be released under the terms of much easier license.

Except that, I would also like to provide a paid versions of the libraries, inspite of using a liberal license. They would not be expensive - the price of $10-20 should be acceptable to everyone interested. The income would be used to keep the domains, servers and advertising, because it requires some money. The paid version would be the same, as the free one, when it comes to the source code, but it would include some add-ons. The ideas are:

- Additional articles and tutorials.
- The right to include them into your documentation.
- Sample projects with well-documented source code.
- Earlier access to the new releases.
- Additional releases in case of finding any non-critical bug.
- The right to provide detailed information about your project on our website.

Unfortunately for the international user, for some time it will be available only for Poland. The international service will require some extra preparations. However, I think the idea is good and everyone should be happy. We have the money to keep our website and resources alive, the customers, because they have some extra content and rights and the rest of the users too - in order to convince them, we will try to prepare new tutorials, FAQ-s etc.

Template inheritance

Zyx — Sat, 02 Feb 2008 17:03:00 +0100

Unsually, the final HTML code of the website is composed of several smaller templates containing smaller pieces of it. There are many techniques of combining them. In PHP template engines, the most popular are two forms: manual execution of the templates from the script side and the include directive. Similar solution can be found in OPT 1.x, however in the new version, it should be forgotten - unless we use the quirks mode, the parser does not let us to call the parse() method more than once. In the beginning, I planned to replace it by extending inclusion with "sequences" - a groups of templates to include with their own variables and cache settings. However, one day I was asked on my blog by Coldpeer, whether OPT will support template inheritance. I googled for a while, I looked at it and thought it might be cool. That's how the implementation began.

As far as I know, template inheritance appeared for the first time in the Python framework, Django. The idea is taken from the object oriented programming, where we can extend classes by adding new methods, fields and overwriting the existing ones. Here it looks similar. We have a main template where we mark some characteristic places and fill them with a default content. Another template extends the main one and is able to overwrite that places with its own content. There can go the next template, and next...

I was quite surprised that a lot of required functionality already exists in OPT 1.x - the bind and insert instructions allowed to move whole template parts somewhere else (as well as their functionality, not only the result, like in Smarty's capture), even if the destination was in the other template. Unfortunately, there was no proper detection, whether one of the files was modified, so unsually the parser decided that the main part needs to be compiled, but the template with snippet definitions - does not. As a result, the places remained empty, although they should not. The development of such system for OPT 2.0 was quite complicated. You do not have to check just the template you've called, whether it is modified. You must also look at all the templates it inherits. Yesterday, I decided to solve this problem definietly and a couple of hours ago everything seemed to work properly:

The base template (test_inherited_a.tpl):

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en_US" xml:lang="en_US">
 <head>
  <title>Instruction test: snippet and insert</title>
 </head>
 <body>
  <opt:insert snippet="header">
   <h1>I'm a standard header</h1>
   <p>Foo bar joe</p>  
  </opt:insert>
  
  <hr/>
  
  <opt:insert snippet="content">
  	<p>Well, i'm also a standard content.</p>
  
  </opt:insert>
  
  <hr/>
  
  <opt:insert snippet="footer">
  	<p>And I'm a footer.</p>  
  </opt:insert>
  
  <p>&copy; Pasteright 2008 by LMAO, It seems to work!</p>
 </body>
</html>

The extending template (test_inherit_1.tpl):

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<opt:extend file="test_inherited_a.tpl">
	<opt:snippet name="header">
		<h1>Webmaster Of Puppets</h1>
	</opt:snippet>

	<opt:snippet name="content">
		<p>Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Phasellus ut tellus id nulla adipiscing eleifend. Sed dictum accumsan ante. Nullam at nisl vitae elit aliquet fringilla. Praesent egestas eros eget tellus. Praesent id odio a sapien rhoncus vehicula. Nunc fringilla, diam eget euismod tempor, tortor metus tincidunt sapien, eu cursus magna tellus at risus. Praesent non tellus eget magna facilisis pulvinar. Praesent libero mi, adipiscing a, pharetra eget, condimentum sodales, mi. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Donec ac elit. Duis iaculis tortor a metus. Aliquam id purus et eros faucibus fringilla. Praesent quis quam. In lectus urna, fringilla sit amet, iaculis eget, aliquet ac, quam. Donec vulputate dui sit amet lectus. Aenean tempor, orci at pretium ornare, tortor tortor venenatis ligula, eget blandit nisi risus eget dolor. Duis nunc neque, sodales porta, viverra non, tristique eu, sem. Curabitur magna neque, blandit ullamcorper, congue quis, tristique ut, felis.</p>
	</opt:snippet>

	<opt:snippet name="footer">
		<p>Bye!!!</p>
	</opt:snippet>
</opt:extend>

Some PHP code:

<?php

	define('OPT_DIR', '../lib/');
	require(OPT_DIR.'opt.class.php');

	try
	{
		$tpl = new optClass;
		$tpl -> sourceDir = './templates/';
		$tpl -> compileDir = './templates_c/';
		$tpl -> stripWhitespaces = false;
		$tpl -> printComments = false;
		$tpl -> setup();
		
		$tpl -> parse('test_inherit_1.tpl');
	}
	catch(optException $e)
	{
		optErrorHandler($e);
	}
?>

And the result can be admired below:

Some facts:

If we modify any of the templates in the inheritance chain, the parser will notice it and recompile proper files.
If we remove any of "opt:snippet" instructions, the default content of "opt:insert" will be displayed.
There is a possibility to overwrite an existing snippet by the more important template. You will be also able to call the parent content, however it is not implemented yet (some kind of "opt:parent" instruction).
It can be mixed with "opt:include" and "opt:sequence" (the sequence system briefly mentioned in the introduction).

What is more, it's quite possible that the template inheritance will work faster than traditional way, although it will consume more disk space. Simply, if we have an inheritance like this: "A extends B extends C", the compiler puts all their code in the file belonging to the "A" template. It reduces the amount of disk operations (the execution of one bigger template is faster than the same content in five files). On the other hand, by default OPT still checks the file modification times, but fortunately, the number of tests is also smaller. The exact results will be available, when some benchmarks are done. Before we end, I'll say that two days ago I made a small test (a simple list with 15 items). The result:

OPT 1.x: 1850 req/s
OPT 2.0: 2100 req/s

Well, it looks promising, but we have to remember that the new opt.class.php file still does not have half of the options, which obviously will increase its size. Moreover, some optimizations are not implemented yet. But in fact, you can assume that the new features should not destroy the performance, and there is a chance that it will be even better.

Deeper processing

Zyx — Thu, 31 Jan 2008 13:42:00 +0100

The last project I've created using Open Power Forms, shown some weaknesses of the current form implementation. There were some forms allowing a mass addition of many records in the following way: a table, in the columns the following fields: Name, Surname, PESEL number, etc. The numer of rows was 30, so the user is able to add 30 people to the database with one form. The adaptation of the map() method to my problem was not too hard; in fact it seemed that opfArrayContainer solved everything. But then the form renderer connected with Open Power Template went off.

My code looked like that:

$this -> map('names', new opfArrayContainer(
	new opfConstraint(MAP_TYPE, TYPE_STRING),
	new opfConstraint(MAP_LEN_GT, 2)
), OPF_OPTIONAL);
$this -> map('surnames', new opfArrayContainer(
	new opfConstraint(MAP_TYPE, TYPE_STRING),
	new opfConstraint(MAP_LEN_GT, 2)
), OPF_OPTIONAL);
$this -> map('pesels', new opfArrayContainer(
	new sConstraint(MAP_PESEL)
), OPF_OPTIONAL);

In other words, the script must have received for example the "names" field, which was an array of 30 elements with the names. If the user made a mistake while filling in the form, the form renderer did not have a clue, which component to display as invalid (in this case it should be colored red instead of green). Finally, I added a support for the IGNORE_OPF_ERRORS constant so that shut it up and just display everything without all this color rotation stuff.

Of course, it is not possible to keep this problem unsolved in the "public" releases and I decided to look for solution. It would be nice, if the improvement easily matched to the code, as well as was intuitive. It's not a problem to add something in 20 minut and a month later self-wishing a rapid death. The best solutions are simple and effective, so I started to think, how to describe a form with some definitions and transformations, so that it would be possible to construct every form and process it with OPF. My idea is to go somewhere deeper and remove distinguishing between forms and form elements. The reasons:

Both have a value (in the form case we define it as a set of subelement values).
Both have a state.
Both can be processed.

Now we can easily treat a form as an element which is capable of storing other fields. If we try to build our form described above, it will look like this: we create a form for a single person: name, surname, PESEL, etc. Next, we put it into a bigger form and order to loop up to 30 times. We can also achieve an effect of looping a single field, like it was possible with opfArrayContainer() so far. We just put in such loop not a form, but a single element.

Well, now the problem is, how to render it :). However, I don't think it will be complicated, especially if I look at the new Open Power Template.

Let's go!

Zyx — Wed, 30 Jan 2008 12:29:00 +0100

Welcome to the devblog of the new programming group Invenzzia. The reason that I formed them are my open-source projects which need a good organization and infrastructure. To begin with, the group will take care of my advanced template engine that exists for over three years so far - Open Power Template. It will become the first part of a bigger series "Open Power Libraries". Currently, we have already begun setting up everything and our website is run part by part. I'll do everything to make sure it won't last too long.

As a typical English-language user you haven't probably heard about Open Power Board project. It was a Polish initiative started in the end of 2004 as a result of commercializing Invision Power Board. The goal was to create an open-source discussion board by the Polish PHP community. However, the project was closed after a couple of weeks, because almost everyone said they have no time to develop it. In fact, the only people who were still active were me with Open Power Template and Tomasz "Slump" Szczupliński as a manager. I tried to reactivate the project in 2005, but there was the same story: everyone except Slump and me went away. I gave up and decided to create more open-source libraries, using the infrastructure left by OpenPB. I managed to make the first stable release of OPT and started Open Power Forms, a library for data validation and form processing integrated with OPT. The code is still marked as beta, because I did not have anyone to finish the JavaScript part, although the PHP-side solutions proved their effectiveness many times.

Of course the situation could not last forever. Two months ago I decided to close OpenPB definietly and throw away almost everything that had any connections with that project. The idea of a new programming group came to my mind and I asked some people whether they would like to participate in it. The team is formed by:

Jacek "eXtreme" Jędrzejewski - PHP/JS programmer and XHTML/CSS hacker. Personally, he's my brother.
Radosław "Radzio" Piekarz - PHP/JS programmer. Personally, we study together.
Tomasz "Slump" Szczupliński - our manager mentioned above.
Tomasz "Zyx" Jędrzejewski - or in other words - me. I code libraries and everything that speeds up other software.

In the beginning, we are going to finish Open Power Template 2, as well as Open Power Forms. We have some more ideas, but they will be coded later. Anyway, we'll try to provide a great set of open-source tools for PHP that present a completely new point of view to many issues.