Building Persistent Web Applications with Coyote
- A Tutorial -

How Terminal Applications Work

Program Flow

The MultiValue environment was originally designed for developing application software that would interact with users through serial, green-screen, terminals. This programming envionment would allow developers to write applications in an exended BASIC programming language. BASIC programs would run top-down and could be further modularized through the use of execution levels, internal and external subroutine calls, as well as a number of program flow branching and looping constructs.

Variable Declaration and Scoping

MultiValue BASIC application traditionally provide for two types of variables. Local variables and common variables. Local variables are storage locations that can contain either simple or dimensioned array storage. Array storage variables must be explicitely declared while simple variables are declared by the system automatically. For either simple or dimensioned array storage, you can store nearly any type of data in an mvBASIC variable including:

mvBASIC local variables are only visible within the scope of the current application source module. If you application calls external subroutines, then those subroutines have their own private set of local variables. mvBASIC does not provide mechanisms to further scope variables. You cannot build declared structures or have variables that are local to one routine within a source item.

mvBASIC common variables are similar to local variables except that you must explicitly declare them prior to use. These variables operate just like local variables except that they are global in scope. Because of the nature of the run-time environment, you must be careful to declare common variables identically in all modules that use common variables. For this reason, most applications will declare common variables in an included source item instead of directly within the main-line program source item.

Some MultiValue implementations also support a third type of variable scoping called "named common". Named common allows you to declare common variables that are private to a particular name. One problem associated with standard common declarations is that you must be careful to declare exactly the same variables in all source items. This can be inconvenient for some application designs. Named common allows you to declare blocks of variables that may only be used by certain routines as global variables and ignored by other routines.

Communicating with the User

Terminal applications communicate with users through a terminal data stream. The application has commands that allow output to be generated to the user (PRINT and CRT) and allow input to be collected from the user (INPUT). Because all processing is sequential and flow-of-control oriented, the user can interact with an application on a character by character basic with the application literally processing every keystroke that is pressed. Of course, system functions including line-editing input routines are available to enable applications to function without requiring low-level keyboard handlers implemented in mvBASIC (some mvBASIC applications implement low-level keyboard support routines anyway).

How Web Applications Work

Web applications differ from terminal applications in a number of aspects. These have to do with the differences in architechure, communications channels, and client-server aspects of a browser-based application as opposed to a terminal-based application. Here are some of the traditional differences between web applications and terminal applications:

Communications Channels

Terminal applications run with a permenant communications channel that remains open between the client (terminal) and the server (host computer). This communicatons channel may be an actual serial RS-232 connection, or may be a network stream connection implemented over top of various networking protocols including IPX and TCP/IP. A standard communcations protocol TELNET exists for TCP/IP network connections.

Browser applications run over network connections only using a protocol called HTTP (HyperText Transfer Protocol). HTTP is a lightweight RPC (Remote Proceedure Call) protocol that can also be used as a lightweight file transfer protocol (HTTP has nothing to do with the FTP file transfer protocol). One of the more powerful aspects of the HTTP protocol is that clients (browsers) and servers (web servers) do not maintain permenant network connections with each other. This is particularily good for the servers because it allows for much better hardware utilization. Consider a web server that needs to support 50,000 clients. Because HTTP only opens TCP/IP network connections as needed, these 50,000 clients can be supported with a much smaller number of active TCP/IP network connections. HTTP also provides powerful services that allow intermediate computer systems called proxy servers to sit between the client and server systems. This architechure further extends communcations efficiency and can actually reduce communcations bandwidth needs by a factor of about 10:1 in some cases.

Communications Granularity

When a terminal user presses a key, the host server receives the information immediately. There is no intermediate processing that occurs. Most mvBASIC applications also run in "full-duplex" mode, so it is actually the host system's responsibility to echo each character that is typed back to the user's terminal.

Web applications behave differently. With a web application, the browser only communcates with the web server when the user takes some action that requires the retrieval of a new web page. In a data-entry environment, this is akin to block-mode data-entry terminals that were popular 20+ years ago and used with mainframe style computers. While this page-at-a-time granularity is looked upon by some as a limitation of web applications, it is also this page-at-a-time granularity that makes the web so efficient and useful when operating over potentially bad communcations circuits such as the public internet.

Applications Flow

When a terminal application is running the application always has a current execution location (program counter) as well as an execution return stack. This top-down flow makes it easy to envision what the application is doing at any point in time.

Web applications are typically "event driven". This means that the application is written as a collection of seperate processing routines that each handle specific requests from the browser. There is no enforced flow to the application and the user can theoreticaly take arbitrary paths through the application. The problem with event-driven applications is that it can be very difficult to design event handlers that will correctly respond to all possible scenios of application sequence. In addition, many applications including many accounting and businesss type functions, do not lend themselves well to ad-hoc user-controlled flow so the events must enforce a specific sequence of events anyway.

Application Persistence

With terminal applications, persistence is inherent in the run-time environment. Each terminal applications is running on a specific and non-shared MultiValue "port". This allows the application to store as much application data as desired in a collection of local, common, and named common variables.

Web applications typically execute with very little state stored on the server. The browser can store some data in form input fields and cookies, but it becomes the application's responsibility to save it's own state, usually by reading and writing records from database files. While it is possible for a web application to create any level of persistence desired, it is usually the applcation developer's responsibility to maintain persistence as a part of the application itself.

How Coyote Melds the Terminal and Web Application Models.

The Coyote web server and it's included development tools allow you to implement web application in a variety of programming styles. While these includes easy-to-use event-driven methodologies, one of tbe most exciting aspects of Coyote lies in the ability to run terminal-style, fully persistent, top-down applications written in mvBASIC that use a web browser to communicate with the end-user. This top-down programming model preserves all of the benefits of a browser-based client-server programming model while still allowing the application developer to write terminal-style top-down fully persistent code.

Part 1 - Web Application Modules

When you write an mvBASIC application that will run on the web, you store the application in standard modules stored in MultiValue database items. This is exactly the same environment that you are familiar with when writing terminal applications. Because the run-time environment is a little different, program source modules have slightly differnent rules for how they are setup.

Terminal Application Modules

Web Application Modules

There are also some additional limitations imposed by web application modules. Primarily this has to do with including all modules in a single application "object" file. Terminal applications have the concept of cataloged external subrouines. Web applications do not require that external subroutines be cataloged, but instead there is a requirement for external subroutine "object" to be stored in the same database file.

Part 2 - Variable Scoping

Terminal applications have two types of variables:

Web applications have three types of variables:

In a terminal application, you do not need to define local varibles because the system will do this for you the first time a variable name is used. Common variables must be defined with a COMMON statement.

Web applications also allow you to just use local variables. These variables are called transient variables. The reason that these variables are called transient is becuase they will lose their values whenever the application yields control to the user by displaying a web page.

If you wish to use variables that do not lose their values when a web page is called, then you must declare these variables as persistent. This declaration is accomplished by using the _PVAR statement at the top fo the program. Once defined as a persistent variable, you can freely use these variables to store application data that will not be lost when a user web page is displayed.

Common variables are always persistent with web applications and are defined with the _CVAR statement. For the same reason as COMMON statements should be declared in an INCLUDE item, _CVAR statements should be declared in an _INCLUDE item.

Part 3 - Program Flow

Coyote web applications support all of the same program flow constructs as do terminal applications. Some of the syntaxes have been changed to allow web processing to more explicitly control the applicaton flow:

  Terminal Applications Web Applications
External Subroutine Calls CALL SUB.NAME(x,y,z) _CALL "SUB.NAME" x,y,z
External Indirect Subroutine Calls CALL @SUB.NAME(x,y,z) _CALL SUB.NAME x,y,z
Internal Subroutine Calls GOSUB label _GOSUB label
Subroutine Returns RETURN _RETURN

The primary reason for these syntax changes is that you are allowed (with limitations) to call existing compiled BASIC application subroutines with standard BASIC syntax provided that these subroutines follow several simple rules:

Part 4 - Interacting with the User

The main area where Coyote web applications and terminal applications differ is in interacting with the end-user. Most terminal applications use PRINT or CRT statements to display information on the users terminal using some sort of control characters that define a terminal's behaviour, operating mode, or "emulation". These interactions are serialized and can occur at any time spacing. Input is usually processed by using the mvBASIC INPUT statement. Again, input takes place sequentially and the application processes input one field at a time.

Web applications inherently operate page-at-a-time instead of field-at-a-time. This means that a web application will build an entire web page and transmit it to the user. This page is created in a markup language called HTML (HyperText Markup Language) which allows for display formatting as well as data entry fields through the use of HTML "forms". Web applications interact with the user by invoking HTML "template" pages using the _PAGE statement from mvBASIC. These _PAGE command is best viewed as the biggest PRINT/INPUT statement that you can imagine. With a single statement, used in conjunction with an HTML template, you can format and display hundreds of fields on data on the end-user's browser as well as collect hundreds of fields of input textboxes, checkboxes, select boxes, etc. It is the display/entry capability of Coyote that brings the true power of the browser data entry/display model to the MultiValue environemnt.

Building a Coyote Web Application

The steps to building a web application with Coyote are:

  1. Create database files to store the application's source and object items.
  2. Setup the Coyote web server so that the application's object file is reachable.
  3. Design a rough application page flow structure
  4. Build rough HTML template pages for each page in the application
  5. Write mvBASIC source code that executes the application's business logic
  6. Compile the application
  7. Test the application with a browser
  8. Debug the application with a terminal session
  9. Repeat steps 3-8 above ...

Step 1 - Creating application database files

A web application is stored in two database files.

Source code - The source code is stored in a standard hashed database file. In our example here, this file is called APP.SRC

Object code - When the application is "compiled" it's web-executable code will be stored in a seperate data file. In our example here, this file is called APP.OBJ. It is important that an entire application be compiled to a single object file as external subroutine calls between object files are not supported.

Step 2 - Setting up Coyote to have access to the application object file

In order to access the application that you are writing from a browser, you need to configure Coyote to have access to the application's object file as web content. This is done with an HTTP DIR line in the Coyote CONFIG item:

HTTP DIR=*:80 /APP/ FILE APP.OBJ

This configuration will allow you to access the application with a URL of:

http://www.yourdomain.com/app/main.htm

Step 3 - Designing a rough application flow

This step is identical to building a terminal application. You need to decide how the application will be presented to the end-user. Here is a simple management application that allows users to request database reports that will be displayed on the browser. The application flow might be:

1. Validate the user

Web applications do not log on the system in the standard sense. As such, if you wish to restrict an application to a particular user, you will need to ask the user for an account name and a password. You can use this information to authenticate that the user is allowed to access the application. You can also subsequently use this information to decide which data the user has access to.

2. Present the user with a main menu

Most web applications work just like terminal applications. The user logs on and is then presented with a menu. You can give every user the same menu, create a seperate menu for each user, or if you want to get fancy, dynamically generate the main menu based on the user priviledges of the current user.

3. Based on the menu selection, branch to subroutines for each function

Web applications do support EXECUTE, but not in terms of generating a user HTML interface. If your application EXECUTEs a TCL command, the command must complete without asking for input. This is ideal for running reports and other queries. This structure prohibits using EXECUTE for application menu flow. The most common way to implement menus is to have the menu routine call an external subroutine for each menu selection.

4. Write each subroutine as a stand-alone module

Once the menu processor calls the actual logic subroutine, you can implement the specific function at hand. This may involve doing additional data entry, presenting sub-menus, or other functions specific to your application. By making each operation a seperate external subroutine you can re-use modules thoughout the application.

Step 4 - Build rough HTML templates

This application has several HTML pages:

Validating the end-user

Welcome to Coyote Widget Processing ...

Enter your account name
Enter your password

Displaying the main menu

Welcome |ACCT.DATA<2>| ... Please select ...

Display today's orders
Display today's shipments
Log-off

Display the results of each report in a general "report" page

|ACCT.DATA<2>| ... Here is your report ... |TIMEDATE()|

|R.TITLE|

|PL_AMTOCRLF(RPT)|

Continue

These HTML templates are designed to both accept input from the user as well as display results to the user. You will notice two characteristics of these templates:

  1. There are insertion points that allow you to display mvBASIC variables and expressions as a part of an HTML page.
  2. There are HTML form input controls that gather input from the user.

Insertion points can contain:

  1. Simple mvBASIC variables such as |R.TITLE|.
  2. mvBASIC dynamic array expressions such as |ACCT.DATA<2>|.
  3. mvBASIC expressions, functions, and calculations such as |TIMEDATE()|
  4. Coyote "functions" such as |PL_AMTOCRLF(RPT)|

    The PL_AMTOCRLF(...) function is a Coyote mvBASIC extension that converts mv Attribute Mark characters into carriage return - line feed two character sequences.

HTML form entry fields can be:

  1. Text boxes
  2. Text password boxes
  3. Radio Buttons
  4. Check Boxes
  5. Select Menus
  6. Submit Buttons
  7. Clickable Images

Each of these fields has their own unique "behaviour" which is defined in more detail elsewhere in the Coyote documentation. The important thing to remember is that communicating with an HTML input control is as easy as naming the control after the mvBASIC variable who's data it manipulates. For example, the first text control in screen 1 above is defined as:

<input type="text" name="ACCT">

This text input control will manipulate an mvBASIC variable named ACCT. It is also allowable to declare input controls that manipulate dynamic array elements as:

<input type="text" name="D<3,7>">

In actuality, this is an illegal HTML control name. HTML does not like field names that contain a "<" character. As such, you would actually specify this control in the HTML "quoted" format of:

<input type="text" name="D&lt;3,7&gt;">

Fortunately, Coyote will process this HTML quoting for you. Also, most HTML editors allow you to specify the field name as "D<3,7>" and will store it using the &lt; and &gt; quoting for you.

Step 5 - Writing mvBASIC code to tie things together.

This application probably will consist of one main module which validates the user and presents the main menu and two external subroutines modules which run each report. Here are sample code for this application

The main module

      _PROGRAM APP.OBJ MAIN
*
*
*
*
      _FILE USER.FILE USERS
*
      _PVAR ACCT,PSWD,ACCT.DATA
*
      ACCT = ''
      PSWD = ''
      LOOP
          _PAGE C:/WEBPAGES/LOGIN.HTM
          READ ACCT.DATA FROM USER.FILE , ACCT.NAME ELSE ACCT.DATA = '*'
      WHILE ACCT.DATA<1> <> '*' AND ACCT.DATA<1> <> PSWD DO REPEAT
*
      LOOP
         _PAGE C:/WEBPAGES/MENU.HTM
         BEGIN CASE
            CASE B.ORDER
               _CALL "ORDER.REPORT"
            BASE B.SHIPMENT
               _CALL "SHIPMENT.REPORT"
            CASE B.OFF
               R.ERR = '302 MOVED TEMPORARILY'
               PL_ADD_HDR 'Location: http://www.homepage.com'
               RETURN
         END CASE
      WHILE 1 DO REPEAT

This routine is used to control the main flow of the application. Here are some details about the program:

_PROGRAM ... - This line defines the type of program module that is contained herein. The first parameter is the file that the "compiled" web application will be stored in. The second parameter is the name of the URL that will be used to access this program (without the .HTM extension).

_FILE ... - This statement is used to define an open file that this module will have access to. Because of the nature of variable declarations, file variables must be handled specially. This example will open the database file "USERS" into the mvBASIC variable "USER.FILE".

_PVAR ... - This statement will define the variables ACCT, PSWD, and ACCT.DATA as persistant local variables. A persistant variable will retain it's value across web page displays.

The first executable part of the program deals with asking the user for an account name and password. The LOGIN.HTM page template includes two text input fields named ACCT and PSWD. Because text input fields support beginning values, it is important to start these variables as null (otherwise, you will get a 'variable not assigned zero used' run-time error).

_PAGE ... - The HTML page itself is called with the Coyote statement _PAGE followed by the page name. In this case, the HTML page itself is not stored in a database file, but is stored on the c: drive of the underlying operating system (this is probably a Windows 9x/NT host). Other syntaxes are available to access HTML pages from unix directories for unix hosts and through PicLan DOS Services Gateway systems for native hosts.

After the user submits the first page, you can read the user record from the USERS file and verify that the correct password was entered. If it is not, this application just keeps asking for the account name and password again and again (probably not the best design, but this is a sample application after all).

Once the user has supplied a valid account name and password, they are presented with their main menu. The main menu HTML template page contains submit buttons named B.ORDER, B.SHIPMENT, and B.OFF. Submit button operate by allowing the user to press a single button. When the mvBASIC application continues, the variable associated with the one button that was pressed will be set to a value of '1' and all other button variables will be set to a value of '0'. This allows menus to directly use 'BEGIN CASE, CASE, ..., END CASE' program logic to branch to different code depending on which button's variable is set to '1'.

_CALL ... - Each menu selection (except logging off) is handled by calling an external web subroutine using the _CALL statement. In this example, each module is called directly and there are no parameters.

Logging of a web application may look a little strange. The reason is that web applications are never really logged on. This example uses an HTTP "redirect" to send the user's browser to another URL when they press the logoff button. After the browser leaves, the application is effectively dead and unreachable.

The Order Report Subroutine

      _SUBROUTINE APP.OBJ ORDER.REPORT
*
*
*
*

      S = 'SORT ORDER.FILE'
      S = S : ' WITH DATE = "' : OCONV(DATE(),'D4/') : '"'
      S = S : ' BY INV.NO'
      S = S : ' CUST.NO'
      S = S : ' AMT'
*
      EXECUTE S CAPTURING RPT
*

      R.TITLE = 'Order report'
      _PAGE C:/WEBPAGES/REPORT.HTM
*
      _RETURN

This subroutine is very simple.

_SUBROUTINE ... - This line declares this module as a web subroutine. It's object will be stored in the file APP.OBJ and it will be called ORDER.REPORT.

The first part of this subroutine builds an ACCESS statement in the variable S. S does not need to be declared as persistent because it is only signifigant for a short period of time during the subroutine's execution and does not need to survive a web page call.

After the ACCESS report statement is built in the variable S, it is executed capturing the result in the variable RPT.

_PAGE ... - The report that has just been captured can then be presented to the end-user call transmitting the HTML page REPORT.HTM. REPORT.HTM contains an insertion point:

|PL_AMTOCRLF(RPT)|

This insertion point will take the variable RPT (this is the target of the capturing clause) and convert attribute marks into CRLF pairs. Also, because the insertion point is not tagges as "RAW(...)" data that is inserted into the page will be HTML "quoted" (characters like '<' will be replaced with '&lt;').

_RETURN - After the user receives their report, they can return to the main menu.

The Shipment Report Subroutine

      _SUBROUTINE APP.OBJ SHIPMENT.REPORT
*
*
*
*

      S = 'SORT SHIP.FILE'
      S = S : ' WITH DATE = "' : OCONV(DATE(),'D4/') : '"'
      S = S : ' BY INV.NO'
      S = S : ' CUST.NO'
      S = S : ' AMT'
*
      EXECUTE S CAPTURING RPT
*
      R.TITLE = 'Shipment report'
      _PAGE C:/WEBPAGES/REPORT.HTM
*
      _RETURN

The shipment subroutine is identical to the ORDER.REPORT subroutine except that the ACCESS statement is different.

Admitedly, these are very simple examples. They do show how top-down applications are structured. This is also a good programming style using small application modules as external subroutines.

Step 6 - Compiling the application

The applicatoin is compiled by using the PLZ command:

PLZ APP.SRC MAIN
PLZ APP.SRC ORDER.REPORT
PLZ APP.SRC SHIPMENT.REPORT

You can also compile multiple modules by using active select lists or by specifying '*' to compile all of the modules in the source file.

Step 7 - Testing the application from a browser

You can access the appliation from a browser by using the URL:

http://www.yourdomain.com/app/main.htm

When you access the application from a browser for the first time, it's mvBASIC code will actually be compiled (The PLZ step only generates the target source code. The Coyote web server compiles to real mvBASIC object code the first time each 'page' is accessed.) If there are compile error, you will see the capture of the compiler output on your browser.

Step 8 - Debugging the application

The easiest way to debug the applicaton, assuming that it at least compiles, it to:

  1. Use the Coyote PL_PRINT command to print display messages to the PLIP-MONITOR.
  2. Run the Coyote Web Server in the forground (instead of on a phantom port) which gives you full use of the mvBASIC debugger.

Either way, you application is just an mvBASIC program (with a somewhat different end-user interface). All of the mvBASIC rules apply including setting variables and insuring that the application goes where you wish it.

Step 9 - Refining the application

After the shell application is running, you will probably need to refine it. This may including adding graphics to the HTML pages (yes, Coyote does support this), collecting data as parameters for reports, building error pages, Logging activity, etc. With each step you will need to re-compile the application modules that you change with the PLZ command and then use the browser to test your changes.

Summary

These are the steps involved in designing a simple browser-based application with Coyote. The steps are very similar to writing a terminal application. With these simple tools, you can build very sophisticated application that perform reporting, data-entry, and nearly any other function that terminal applications can perform using a purely browser-based interface.

Many web applicatons go beyond the simple user interface described here. With Coyote, you can dynamically generate any and every aspect of the web pages that the user will be presented. If you wish to build an HTML table with all of the current open order and let the user directly click on an order to query, Coyote allows you to do this quickly and efficiently with tools to dynamically build input controls and tables which are then inserted into HTML page templates. With Coyote, you are only limited by the layout features of the browser and by your imagination.