Not Blocks nor Files but Words
Paper presented at the EuroFORML conference 1989
A programming environment for Forth is described, which is based on a database and a browser instead of blocks or files. The collection of words, which constitute the program, is hierarchically structured into modules and groups. The programmer acts on the program through a browser interface, which shows modules, groups, words and definitions in separate windows. The database structure allows useful actions on the program like a hypertext facility, easy factoring of a program's structure, reloading of single words, glossaries and selective documentation.
Creating a program is equivalent to building a machine. The machine is constructed in two major steps: factoring of the functionality into components (modules) and development of the components. In Forth the components are described by words.The traditional way to program in Forth, with blocks or files, relies solely on text editors. From a software engineering point of view, however, the editor is only a sub function of a productive programming environment.This paper describes an alternative.
Blocks originated in the days of scarce resources and are a clever way to organize data like the text of a Forth program. With good block editors the development of Forth programs can be fast and efficient. The program can be structured into block ranges, and parts can be loaded separately. Blocks are used in productive ways. It is easy to implement a facility in order to look up the definition of a compiled word (view field added to headers, e.g. F83). It takes a lot of experience, however, to reach this level of high productivity. The problem is to handle changes in the program. This amounts to shifting words, together with the comments of the words (which may be on a different (shadow) block).
The free formatting of a linear text file possible with text editors has converted many Forth programmers to the use of files. Building the program as a collection of files automatically imposes a modular structure and makes the programs readable. Comments can be kept with the words (where they belong) and the pages can be neatly laid out for documentation. Like a range of blocks, parts of a text file can be loaded with cut and paste operations. If programs become large, however, the words are not easy to find anymore. A view facility is cumbersome to construct for files, unless the file is organized by line numbers.
Both blocks and files limit the creativity of the programmer. A Forth program consists of words. Programming means defining, changing, inserting, deleting and shifting words. Using blocks and files, we treat words as chunks of text on which we operate. The text view limits our perception of the program. It is difficult to see the complete picture if we constantly look at all the details. We need the paper documentation to understand the program and we must take care to keep updating the documentation. The methods used for direct access to the text of words (e.g. VIEW) rely on the dictionary structure of Forth. This implies that the words are compiled, and that means a bottom-up approach to programming. Especially in the starting phase of a project, it would be useful to construct words, bottom-up and top-down, without worrying about their compile-ability for the moment and still to make use of fancy word finding features. I believe that a better approach is possible and would like to show you my way of programming in Forth. This method uses a database as the central repository of a program and a browser as the interface to the programmer.
Before I describe the database let us look at the structure of a program. Most Forth programmers divide their programs into modules, a module being a range of blocks or one of several files. I have found one more subdivision helpful, the 'group'. A typical program of, let's say, 1000 words thus divides into the easily manageable sizes of about 10 modules, each consisting of about 10 groups, each with about 10 words.
The database reflects the modular structure of the program with three types of records: module, group and word records. These records build up a database, which is kept in memory during work and written to a file for backup. All records are doubly linked. This makes the tasks of building, deleting, inserting and shifting records simple to implement. All fields but the name field contain 16 bit pointers, limiting the size of the record space to 64 KB (with room for approximately 2000 words). A fixed amount of space (16 bytes) is reserved for the name field in order to make renaming of entities easy. Thus names are limited to 15 characters (plus a count byte).The text of the words is located in a separate file. Only the text of the current word is read into memory for editing. Thus a program is represented by two files, a structure file and a text file.
previous module (2 bytes)
next module (2)
first group (2)
last group (2)
name of module (16)
previous group (2)
next group (2)
first word (2)
last word (2)
name of group (16)
previous word (2)
next word (2)
code address (2)
code length (2)
The text field points to the location of the text in the file. Text starts on 16 byte boundaries, thus a text file has a maximum size of 1 MB.The alpha field contains a (one way) link between all words in alphabetical order of the names. This order is built and maintained with each editing operation on a word. The minor task of keeping the words in alphabetical order is richly repaid by the ease of implementing hypertext and other features.The code address points to the code location (cfa) of a (compiled) word. The code length is also stored.
This division of a program into modules, groups and words is especially useful if we can easily visualize it. A browser is the adequate means. Figure 1 shows the programmers view of the program. In separate windows we see the modules, the groups of the active module, the words of the active group and the text (comment and definition) of the active word. By scrolling through each window the complete program can be studied. One of the MODULES, GROUPS or WORDS windows is chosen with the left/right keys, and the active item in the window is selected with the up/down keys. Other actions are related to the function keys (actions changing with the window).The browser always shows the current state of the program. Changes are immediately effective.
Figure 1: Browser view of a program. The underlined item of a window has been selected. The window with the boxed item is currently active. The lowest line displays the current actions of the function keys.
The organization of a Forth program as a database with a browser interface has some nice properties and facilitates the implementation of useful features:
- The structure of the program is clearly visible. In all windows scrolling shows the contents. The dependent windows are updated.
- Changes to a program are immediately effective. Programming becomes sculpturing.
- Words, groups and modules can be moved around freely. A method of deleting and inserting is used. Deleted entities are transferred to a delete stack, one each for modules, groups and words. Thus several entities can be deleted together and inserted in a new place. A group or module is moved with all included parts.
- The program can be loaded as a whole, as a single module, a group or just a word.
- Each colon-word can be reloaded after redefinition without having to reload higher words. If space permits, the new code replaces the old (otherwise it is appended at the end of the code space).
- The text of a word used in a higher word can be quickly found: the word at the cursor location becomes the current word, i.e. the browser switches to the text, group and module of this word. This 'hypertext' facility works as many levels down as desired and back again.
- Another quick find facility is used outside the WORD window. After pressing the space bar and entering one ore more characters, the first word with the given initial string is shown. Thus only the minimal distinctive number of characters must be entered for the search.
- A glossary of all words can be printed at any time.(The last three features use the alpha links to advantage).
- The words are listed on paper as modules or groups with the words in their present order, the comments adjacent to (above) the words definition. The program is thus documented like a text file.
- The compiler has become simpler since it does not have to create headers (these are already part of the database).
- The generated code is header less. It is compiled as a separate entity (not on top of the development system).
- The code address is shown alongside each word (zero when not compiled). This gives a feeling for what is happening in the code space. And the code generated for the current word can be displayed in the browser (useful for debugging).
The structure presented is not very different from original Forth. We may think of the database as an extended dictionary structure. Module and group records correspond to vocabularies (in representation but not in search order; all words are globally accessible). And the name and link fields and eventual view field of a dictionary header have been expanded with more information about a word.There is one significant difference: The dictionary is tightly linked with the code aspect of a program. The headers exist only when the program is loaded. The database point of view is independent of the code aspect. The links exist in the text and all browsing facilities are available without any code being loaded. This is very useful in the initial (defining) state of a program.
This new programming environment for Forth (named 'Holon') has by now been applied to the development of one real project. The basic concept seems to be sound since so many useful features easily derive from it. Holon was an exercise in factoring the functionality of Forth. The additional layer of complexity added to the Forth system makes the life of the Forth programmer considerably less complex. Moreover the database overhead is balanced by several possible simplifications of the original system. I welcome comments or questions regarding Holon.
Holon has been realized in HS-Forth (Harvard Softworks, P.O.Box 69, Springboro, Ohio 45066), profiting from its segmentation of code space and window functions, among others fine utilities. Innumerable publications of the Forth community have contributed ideas to the project, which on a long and winding road finally evolved to the present system. I am deeply grateful to all who shared their ideas and helped me to better understand the meaning of computers. Charles Moore has originated a way which is really worth following and which still is far from explored.
I owe the concept of Holons to Arthur Koestler (The Ghost in the Machine, also in Janus: A Summing Up). A Holon expresses the idea, that something can at the same time be a whole and a part of another whole. This is typical of organisms. It is also a characteristic of Forth words (and of machines in general).
Copyright © 1999 by Wolf Wejgaard