Invenzzia... in English - development

TypeFriendly 0.1.0 released

Zyx — Wed, 23 Jul 2008 10:34:00 +0200

At last, the translation of the manual into English is complete and the source code has been already released as the first version of TypeFriendly, the documentation generator. In is available in Files on the main site. The user manual is provided in the source form and it works also as an example, so please do not worry that something is missing. All the necessary information to build the HTML version can be found in /info/README.txt file. All you need is to type one command in the system console. The on-line documentation will be available soon, as we are preparing new website. To get more information about TypeFriendly, take a look at the previous note.

TypeFriendly

Zyx — Wed, 02 Jul 2008 21:38:00 +0200

Together with eXtreme, we work hard to make the first release available as soon as possible and we are quite close now. This is a perfect opportunity to tell something more about this project, which should be appreciated by those programmers who write their own scripts and libraries and look for a smart tool to make a documentation. The idea to write TypeFriendly was born in my head after I failed to add some necessary things to the XSLT sheets for DocBook. The PHP manual shows, what could be done with it (syntax coloring, lots of different formats, enormous amount of tags), but don't become too happy too fast. To achieve similar effects, we need weeks of coding and we must also fight against the portability. In fact, I'd rather use those weeks to write something more useful and friendly.

So, TypeFriendly is a documentation writing tool, but it works differently than for example phpDocumentor which scans the source code of your project and makes a reference list of all classes and functions. Here, we write all the chapters and put them in the required order on our own, like in DocBook. However, in this place the similarities end, because TF already contains everything we need. Moreover, it was designed to be intuitive and easy to learn.

Features:

In TF, we write our manual in standard text files using the Markdown syntax. We can put some extra information with tags placed at the beginning of the file. They allow us to define a title, version information or "See also" section.
TF automatically sorts our files and connects them into bigger chapters, basing on their file names. By default, all the content is sorted alphabetically, but we can easily change it for all the files we need. We just create an additional file with "sorting hints", where we write the required order.
There is a built-in syntax highlighting option.
TF generates all the navigation and table of contents.
TF is designed to create a multilingual documentation. Apart from storing many versions of the same file and interface texts, it contains a simple tool to detect whether the "derived language versions" are up-to-date.
TF allows to generate the XHTML version very quickly, on a single page, as well as on many. We are also creating a special format for importing the manual to the database (to publish it online and for example, to add user comments).
TF packs your manual by default in the nice, green layout.

Although there haven't been any release so far, you can always connect to our new SVN and download the code from there. The link to the checkout commands is: http://svn.invenzzia.org/svn/typefriendly/trunk/ - we encourage you to test the script now. In the repository, there is a README file and a nearly finished Polish version of the TF manual. But don't worry. The first release will be available with the English version, too. It's much shorter than OPL manual, and nice to translate :).

Now, I think it's a good idea to show an example. The first one would be one of the files from OPT template engine manual that we develop, too:

Title: optNode class
ShortTitle: optNode
Status: abstract
Extends: library.optcodebuffer
ExtendedBy:
 - library.optscannable
 - library.optcharacterdata
 - library.optexpression

----
An abstract XML node class. It provides the basic support for the most important features, such as type and parent.

Class fields
------------
All the fields are protected.

 Name          | Type      | Description
---------------|-----------|---------------------------------------
 $type         | Integer   | Numerical type identifier
 $parent       | optNode   | Node parent

Available type identifiers:

1. `OPT_CDATA_NODE` - `optCharacterData`
2. `OPT_TEXT_NODE` - `optText`
3. `OPT_EXPRESSION_NODE` - `optExpression`
4. `OPT_ELEMENT_NODE` - `optElement`
5. `OPT_ROOT_NODE` - `optRoot`

At the beginning, we have a header with tags. There we define the exact title, and in this case also the dependencies between the PHP classes. Notice that we refer to the other classes by writing the identifier of the chapter, where they are described. Of course, there are additional tag versions, where we can write the class names directly, for example if we inherit from some PHP built-in class. There are more available tags. Let's say we want to make a "See also" section:

SeeAlso:
 - chapter.text1
 - chapter.text2
 - chapter.text3

Once the tag list finishes, we begin the exact content in the Markdown format. The author designed it basing on the formatting used in text files and e-mails (it can be said that he wrote a parser for a syntax that self-evolved :)). This means that the source text is really clean and easy to read. In fact, it is just a bit lesser comfortable than a parsed one. The problem is, if someone is accustomed to some wiki syntax, because MD not always works in this way (take a look at the table). But it is quite easy to switch. We use Markdown even on our website and I must say that it is a very convienient tool. I'm sure you'll find it in more Invenzzia products. There is only one big shortcoming - the only available output is XHTML. For now, you have to forget about LaTeX and the rest, but I can assure you that we plan to add it, too.

The most frustrating problem while writing a manual is navigation. The links "Previous", "Next", "Up" are added by TypeFriendly, we create the "See also" section by adding a tag, but what about clickable function and class names? I thought about it a bit and I invended a brilliant and simple idea. We have an additional text file, where we define, what text points to what. For example, we write there that `optClass::parse()` must always be a link to library.optclass.parse and that's all. Unfortunately, we haven't hacked Markdown in this way so far and the first release will not contain it yet.

TypeFriendly works in the operating system command line. We've tested it both on Linux and Windows and it works there without any problems. All we need is a PHP parser. Togethet with a simple syntax and built-in multilingual support, TF should encourage much more people to help in translations of your manuals, improving the base version and maybe even writing new outputs. Generating the XHTML version is really easy:

php ./typefriendly.php -l en -o xhtml ./path/to/manual/

Let's compare it to DocBook. The format itself is a bit longer, but relatively easy to learn. However, the conversion needs more tools. We must download the DTD, DocBook XSL Stylesheets and an XSTL parser. The choice is important, because fast and easy xsltproc will not highlight your syntax in the examples. On the other hand, the Windows users are f@#$ off if the author decides to write half of the processing tools in Bash. We don't have to look such examples too far. Just take a look at the PHP manual. Guess, what is it written in? :)

SVN is active!

Zyx — Mon, 30 Jun 2008 08:10:00 +0200

Unfortunately, the configuration of a version control system on our current server is not as simple as on Sourceforge.net. But now I found some time and ideas, how to make it work and finally... there it is, we have an SVN repository. It's available under the address http://svn.invenzzia.org/ - we still have to install a good browser. Today I'm going to upload the source code of the new OPL and TypeFriendly, a documentation generator almost ready to its first release. All the instructions, how to connect, will be published on the main site.

OPT requirements

Zyx — Sun, 06 Apr 2008 10:28:00 +0200

I've found a new entry on the bugtracker recently. The author informed that he gets "Nesting level too deep - recursive dependency" error while running the development examples provided by the newest OPTv2 dev version. I looked at it deeper and found out that he lowered the default nesting level limit 4 times from 64 to just 16. Obviously, this is a quite drastic change and would affect not only OPT, but many other scripts as well. However, I think it is time to take a look at the OPT issues concerning the requirements, because several of them needs a deeper explaination.

The optimizations implemented in OPT, mostly concern the main parser class and the compiler. The first one is the part that is loaded every time the script wants to use templates, so it must provide huge quality standards. There are no sophisticated algorithms, so the speed improvements are:

To minimize the size of the code.
To minimize the amount of points that try to access the hard drive.
To use the fastest elements of the language.

The point is this class must be as light as possible; I would even say: almost transparent for the script.

The situation is quite different, when it comes to the compiler. It is loaded from time to time, so it can be big and splitted for more files (of course without exaggeration). The key role is played by the limits set in the PHP configuration, because in the pesimistic situations, the compiler runs out of resources to complete the task. Unfortunately, some of the new features cost a lot, and if the users wanted XML tag parsing, they must face the consequences: every such tag requires its own node object in the document tree and they need some preparation tasks to be done. My role is to minimize the time and memory necessary to do this.

The elements that can be exhausted during the compilation are:

The stack
The memory

The algorithms used in the compiler operate on a data structure called tree, and the use lots of recursion. It consumes the stack, because the number of currently executed methods and functions grows quite fast. In PHP, the default limit for the stack depth is 64 and if you are going to use OPT, the worst you can do is to lower it to ridiculous values like "16" that would affect not only OPT, but also many other scripts :). However, every recursion has an imperative version, constructed for example by queues. Maybe it does not consume less memory, but at least it does not depend on the limits. Such changes have been applied to many compiler methods so far, but some of them still require my attention, because they are not so trivial:

Processing the tree by the instruction processors
Expression compilation

In the first one the situation is clear: we get a node of some instruction and want to process it. We find its processor and redirect the node to it. It generates some PHP code and decides, what to do with its children. By default, it wants to process them, too. Here comes the recursion and the process starts from the beginning. There are some difficulties. The algorithm is split over many classes, and the programmers may create their own. What is more, some instructions need to start the subnode processing immediately, because their next decision depends on the code generated by the children. Obviously, this eliminates queues. An elegant replacement could be made, if we had an access to the low level programming.

Let's take a look, how it affects the compilation process. We have a huge template, with the tags nested up to the level 30. OPT with the compiler can take another 10 calls, and in the deepest place we have a complicated expression with brackets. To sum up, the library uses approx. 45 calls from 64 available, which is more than 70% of the limit! I know this case is quite pesimistic and in normal life such HTML documents are very rare, but it shows, how demanding the compilation can be.

If we can't remove the recursion without ugly tricks which would be never accepted by the instruction authors, how we can get around it? The idea is not to remove recursion completely, but we keep it in some places. I use here the fact that the immediate access to the subnode compilation results is necessary only for some instructions, and for example the HTML tags do not need it at all. If we get a random node from the tree, we have a big chance that it would not need a recursion here. So, why don't we provide a dual version of this algorithm? By default, it is executed with a queue, but for more sophisticated needs, some nodes can switch it to recursion. Now it can happen that a tree with a depth 30 is processed in one single loop, with no recursion! I predict that in normal situations, the algorithm will consume only 3, 4 spare "items" in the stack, which is fully acceptable.

The second issue concerns the memory usage. It's not so tragic here, but we can notice a significant rise when compared to the OPT 1.x compiler, even with using some patents from that version. The most memory is consumed by the tree. I've measured some single nodes recently and the results were from 2 kilobytes for a single node to 3 kilobytes for a node with one attribute. When the instructions start to add the PHP code to the buffers, the things are getting worse, but it happens only in some nodes in the whole template. We can safe some memory here. The objects use various arrays, which are initialized just in the beginning, for example:

abstract class optScannable extends optNode implements Iterator
{
	protected $subnodes = array();

In most of the nodes, the arrays are empty, so it it no sense for them to exist all the time. I fixed the code so that it creates an array only when it is necessary. From the declarations, the = array() parts were removed. The effect is quite interesting. On a single array, we safe 400 bytes. The new results are:

Single node with no attributes: 1400 b.
Single node with one attribute: 2350 b.

The additional memory may be safed by releasing the huge data that will not be used anymore, for example the template content.

What do we have to remember, while using the new OPT, when it comes to the requirement issues:

With the terribly low limits, the compiler will always crash. I thing 16 megabytes for the script (the default limit) is big enough both for the compiler and the script.
The best place to run the template parsing is the end of the script. We can delete some unncessary data earlier to free some memory.
The templates using the inheritance are more demanding, especially when overwriting snippets, because the compiler must keep also the old versions in the memory.
The quirks mode will consume less memory (simpler tree - no XML tags), but it is not implemented yet.
The compilation is run once for a long time. During the normal execution, we can forget about the limit issues etc.

I think this note shows, how difficult the library creation can be. The key is to know, what we can use in current place and that there is also the rest of the script which uses the same memory and has its own requirements.

OPT 2.0.0-dev5

Zyx — Fri, 21 Mar 2008 14:38:00 +0100

The new development version became available a bit later that I planned, but on the other hand, I managed to add much more new things. The DTD issue is solved, as well as XML prologs, CDATA support and code escaping. The expression compiler is done and the programmers may use now brand new component implementation. This is a significant step towards the first beta release and the end is closer than further. There are only a few instructions to code, the rest is almost completed and works. The compiler, programmers' API and most of the stuff also works. I think of making the first real tests on my home website and creating the Open Power Forms library compatible with the new OPT.

In the new OPT, I've redesigned the component design from scratch. It is much more flexible now and allows to make easily many interesting effects. The base rules are still the same, but the rest has changed. This is a sample code showing, how the component-based work with the forms is going to look like:

<opt:snippet name="componentView">
  <com:div class="field" invalid="field error">
    <p>{$opt.component.title}: <opt:display /></p>
    <p class="desc" opt:if="$opt.component.description">{$opt.component.description}</p>
    <opt:onEvent name="error" message="info">
      <p class="error">{@info}</p>
    </opt:onEvent>
  </com:div>
</opt:snippet>

<div class="form">
<form method="post" action="skrypt.php">

<opt:input name="name" template="componentView">
   <opt:set name="title" str:value="Enter your name" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="surname" template="componentView">
   <opt:set name="title" str:value="Enter your surname" />
   <opt:set name="description" str:value="At least 3 letters" />
</opt:input>

<opt:input name="age" template="componentView">
   <opt:set name="title" str:value="Enter your age" />
   <opt:set name="description" str:value="Scope: 10-99 years" />
</opt:input>

<div class="foot"><input type="submit" value="Send" /></div>
</form>
</div>

The new implementation makes use of the opt:snippet instruction and template inheritance. If you had enough time, you might code the component objects, so that the field titles and descriptions will be unnecessary in the template. A component can handle events, displaying itself and its neighbourhood, managing parameters and tag attributes (we use the com namespace for it). OPT contains only the component mechanism, and the components itself must be written by the end user. The reason is very simple. In order to make a useful component, we should integrate it with the input validation classes, language system, error management etc. The demands are much bigger than OPT is able and is supposed to provide. However, there will be a ready-to-use component implementation: Open Power Forms library.

Unfortunately for the English manual is not started yet. Even the Polish users must use a partially-complete text, without formatting, because the new rendering system is not ready. However, we'll do everything to translate all the text together with some other materials (advices, FAQ, maybe even a tutorial?) into English, before the first final release is done. There is a team, I'll try also to ask some friends and we'll make a serious English support. Just be patient.

Download OPT-2.0.0-dev5