Next Generation Wasabi Platform

“Pipes”

This is a view of an operating system running the platform, there are 3 types of processes: pKernel (unique) implements the system, packages (multiple) extend the system and both of these live within the OS as sibblings of legacy OS processes. User applications are not visible in such a view, because they are made up by links between objects and are run within several process-spaces at once. The XmlShell contains a list of all pipes chains, the list of all first pipe objects is an analogue of a list of user application entry points.

1. pKernel

pKernel (sometimes called “the core”) is the platform server, it is the single system process providing the basic services for the rest of the packages, components and applications.

1a. Service Management

The first task of pKernel is to provide a way for all running packages to expose themselves to each other. This is accomplished via the service manager, which keeps track of all instantiated services, their parent factories, and later routes the requests for object destruction accordingly. Services are provided by service factories as pKernel components, packages, or package components (see “2. packages”).

1b. XML Shell Script

An interpreted XML grammar embeds class/data/call serialization, set operations, control structures, variables, event mapping, external scripting languages (ie: perl, tcl, lua) and so forth, and makes up the shell scripting language that unifies communication between all pipes in the system.

The xml shell has knowledge of all instantiated pipelines and connects them to one another upon requests, it also creates new pipelines out of xml script, and destroys them when they are no longer needed. All pipable objects communicate with and can be controlled by the shell. For instance, an object may receive xml shell commands in its standard input that is an order to call a particular method of the object, with particular parameters. These commands are implemented by the shell using interface discovery on the object. In addition, the shell may install shell callbacks into an object, such that triggering an event on it will execute a particular piece of xml script. High level languages such as perl or python may be embedded in the xml, and thus become directly usable to build simple yet powerful pipable objects.

1c. I/O, Security & Network Pipe Layers

All input/output pipe operations in the system are routed thru the pKernel. All objects inputs come from the pKernel, and all objects outputs go to it, this ensures that all pipe objects can be used transparently across a network, and may equally transparently be encrypted and decrypted on the fly (one exception to this rule is mentioned later in this text, see section “2c. Components”)

1d. Call Server

See “2b. Call Server”.

1e. RPC Server

Finally, pKernel implements a remote procedure call server, whose architecture is similar to the multi-thread call server described in section “2b. Call Server”. This server allows remote machines to issue calls within the platform as if the object was local, and (in addition to what the package call server does) allows local processes to issue calls to remote machine as if the caller was a remote object as well. The same bridge may also be established between remote and local packages. Network restrictions apply on all communications and obviously default to denying all RPC.

2. Packages

To create objects, factories are exposed to the system via packages. A package is a process that runs on the underlying OS and exposes its factories to the pKernel. One package may also use several process entries at once (ie: multiple threads on linux). Typically, the C++ code for a package, including factory exposure, dispatching of classes and i/o management via the pKernel, will be generated by XPP.

2a. Factories

The role of each factory is to provide new functionality to the pipe platform by allowing the creation of existing or new classes of objects. The factory usually waits on a thread-wait object, then wakes up when it is asked to process creation/destruction requests. The factory returns cross-process pointers to the calling applications, which are GUIDs and which are mapped to their corresponding process-space pointers on the package's side. Factories may provide both high level pipe objects (for the platform) or low level objects (such as pKernel objects [ie: services] or third party (non-pipe) objects).

2b. Call Server

Each package must allow the pKernel to call into the package's code. This call will be accomplished by generating cross-process dispatchable classes via XPP for each object class created. Variables payload transfer is achieved via shared memory, and calls are synchronized using thread-wait and mutex objects. The role of the call server is to maintain a package's call-ready status by dynamically creating and destroying threads if needed, or to block, awaiting thread availability.

A call server may specify the minimum and maximum number of threads to run. A typical single-thread package will behave like this :

This single thread model ensures that objects within the package are only running code when no other process is issuing an inter-process call into them. Although the model still allows multiple processes (and therefore, multiple threads) to call the package's objects, a single method is ran at a time, which keeps the single-threaded code safe. Note the embedding of the underlying OS' message loop within our own, so that messaged native OS calls may be processed synchronously.

Multi-threaded packages have slightly more complicated call servers :

The multi-thread model creates “call threads” on the fly: external processes may call the package's objects while previous calls have not yet returned. Each call thread waits on a signal from the main thread to process a call, then pushes the return payload back on the shared memory communication channel and unlocks the calling (possibly extra-process) thread, then goes back to waiting for a call to execute. A thread garbage collection mechanism destroys threads which have not been used since a specific amount of time (possibly, a global pKernel setting limits the number of threads running throughout the platform). Note that there is no reference to the underlying OS' message loop here, because it is executed in the main process' thread, concurrently with the server thread (native OS messages are processed asynchronously here).

The callback server architecture is symmetrical to the call server architecture: each new pipe class that is written and that goes thru XPP generates C++ code to handle both call directions (method/callback). Since each direction of communication requires a client (running in the caller's process) and a server (running in the callee's process), this is a total of four generated classes for each pipe object (counting only IPC-related classes). Each server class is handled by its own package's' process' call server; each call uses generated client code for one of the two classes. The package will use the generated client code for the callback class to trigger callbacks on the remote process, and use the generated server code for the main class to listen for method calls coming from other processes. The remote process will use the generated client code for the main class to call methods on the object (in the package's process space), and use the generated server code for the callback class in order to listen for callbacks coming back from packages.

2c. Components

Components are “in-process packages”. They have the same function as packages (exposing factories, instantiating services, pipe objects, etc.), but they run within the process space of a host application (in the native OS sense): either a package or the pKernel. This allows third parties to implement low level extentions (ie: non-piped) to specific packages (or to the pKernel itself). In addition, since the pKernel automatically optimizes a connection between two objects running in the same process by bypassing a costly part of i/o management (function call parameter payload transfer via shared memory), components enable low-latency applications (ie: media playing) to execute their piped chains without going thru i/o management, boosting their performances significantly. For this reason, a clever organization of related objects into packages and components will help optimize performances. Possibly, the power user may want to redo this organization, and for instance, put everything in his kernel, join two particular packages into one, or remove everything he does not use from another, etc.

3. Pipes

Pipe objects are the basic high level objects on the platform, they are created by pipe object factories (PipeObject family) and automatically inherit the i/o features implemented by the pKernel. Pipe objects are the building blocks of high level applications (xml+script). In terms of code, a pipe object may be composed of yet more pipe objects linked via scripting, or may be programmed natively in C++, using XPP to generate ancestors that automatically expose the class to the pKernel (as well as the component or package infrastructure code).

3a. Pipelines

A pipeline is a chain of connected sources and sinks. Each pipe object in the pipeline may implement several different callback source and sinks.

Standard input is received on a standard onData on the root Pipe callback interface. Multiple data format are supported, objects may request automatic transcoding from the system for those formats it does not handle (obviously, such a request may fail). Standard output is sent via custom printf/fwrite-like member on the object implemented by the Pipe ancestor.

3b. Hooks

Most connections will be established by the shell rather than the objects themselves, but the shell has no knowledge of the interfaces to connect. This is not an issue since all callbacks are multiplexed into a single function call and later dispatched, this allows third party objects to connect/disconnect as well as monitor/hook any event on any object regardless of wether they know of its interface or not (ie: passthru/forward/log objects).

4. Applications

4a. Functional Wiring

An application is just a particular wiring of a chunk of objects. Using the xml shell and ultimately, the graphical IDE, one describes the pipes to create and connect when the application starts, the application events to provide to the scripts and objects, etc. For all intent and purpose, the functional wiring *is* the application, just a “high level” application. Note however that a functional wiring does not define a visible interface on the screen, only the framework in which the skinner will give it one.

A very simple http server might look something a little bit like this in the IDE, one day, if we work hard:

4b. User Interface

The user interface for an application is defined using a freeform skinning system that piggybacks on the shell xml syntax. The role of the skin is to attach itself to the functional wiring, to add events and UI widgets, and to hook them up so as to implement user interaction. Generic UI widgets are handled via “themes”, sets of high level object implemented using other high level objects: when a skinner uses generic UI widgets, he asks the system to implement objects that have been skinned by another skinner (the “themer” ?), and that will be consistant throughout the platform.