This article was originally posted on omniscentral.com January, 2007. I’ve reposted it here, as omniscentral.com has recently closed their doors.
I was recently taken in by a company who is working on expanding their well established Omnis 7 based desktop application line into the realm of the Internet. We have been working on developing add on and replacement web applications to these desktop applications. Our technology choices for moving into web based development focuses on PHP as a server side language, and content being delivered through the Apache web server.
For the short term, our desktop based Omnis 7 applications needs to co-exist with our web products. Also, this web application suite needs to build on already established data structures in the Omnis data file format. In addition, we needed a way to be able to re-use pieces of business logic established in Omnis should we opt not to re-write them on our web platform.
This created some interesting challenges for us. Migrating our data storage from the Omnis data file format to a full fledged SQL backend was not a short term option. That would have required a considerable commitment to recoding our desktop applications. Also, we are moving forward with MySQL which Omnis 7 doesn’t support natively. We did consider a wholesale move to Omnis Studio to solve this and other problems, but this again was too large a commitment for our desktop apps. We needed a way to gain read/write access to the Omnis data file format, as well as call Omnis procedures. And for good measure, we needed to do this in a fast, flexible manner, that wasn’t tied to a particular operating system or to a particular singe machine (IE: networked solution).
Phase Old
The initial solution to this problem was solved by building out a TCP/IP server in Omnis 7. Using DirectLine Technologies’ APPCOM external, we exposed this server to our web applications. Our web applications would then communicate to the server issuing remote procedure calls using a tab delimited format for sending and receiving data. Calls coming in would be mapped to a procedure and parameters would be scraped out of the tabs. Procedures would then bundle up a result in a tab delimited format, and that would then be sent back to the web app for processing. We rode this solution for some time, but reached the limit of what we could expect from this solution.
The biggest problem was how we communicate the data from one language to another and back. Tab delimited format does not lend itself to expressing data other than simple strings and numbers. Booleans, arrays, dates, and faults are difficult to express in this format, and resulted in tab delimited soup. This proved to be a large implementation and maintenance problem resulting in a fragile system. We did look at replacing tab delimited with XML, but with no XML tools in Omnis 7, this was not possible.
Also, because of this data representation issue, there was no easy way to gain access to an Omnis data file. We toyed with the idea of building out an SQL wrapper procedure that would take an SQL call and transmit the result back to the caller. This would have proved to be a herculean task. Building out tab delimited structures to represent variable SQL result sets, as well as handling issues with interpreting this mess on the client end was more a hassle than it was worth.
Phase New
Enter Remote Studio. Remote Studio, or RStudio, is “…an applet technology, which provides an equivalent of remote procedure call to an Omnis server…” * RStudio came bundled with Omnis Studio 4.0, and promised to simplify inter-language remote procedure calls (RPCs). It provides a language and platform agnostic mechanism for communication to Omnis, map remote procedure calls coded in Omnis to RStudio requests, and handle “seamless” data type conversations between the client and server. Nice.
RStudio works within a client server model. For our deployment, it requires four components to make this all work:
- A client library that handles building a RPC call, including call parameters, dispatching, and then handling the result.
- A web server CGI script or Apache module that routes communications between the RStudio client and the Omnis server.
- An Omnis server (Studio has a built in server, or the Omnis Web Client Server runtime) which handles mapping incoming requests to a particular Omnis library and remote task class, and handles converting the request parameters to native Omnis data types.
- An Omnis Remote Task that implements the dispatching logic that directs the request to an Omnis method, and back.
Unfortunately, RStudio appears to be a feature not well evangelized by Raining Data. Documentation and use cases are limited to a small handful of pages in Programming Omnis. Also, adoption of this technology by the Omnis community is minimal. Getting good information on how to properly use RStudio was a struggle. More problematic, the PHP RStudio implementation provided by Raining Data proved to be riddled with many issues.
Phase New, Round One
With Apache setup with the CGI, and Omnis Studio running with the server listening on port 5112, I got my first test implementation together. Within PHP, I built my RPC call with the Omnis supplied Rstudio library as follows:
mOmnisLibrary = 'BRIDGE'; // Omnis remote task to call $rs->mOmnisClass = 'connector'; // Port Omnis server is listening on $rs->mOmnisServer = '5112'; // URL / IP of web server $rs->mServerURL = '127.0.0.1'; // Path to Omnis web server gateway $rs->mServerScript = '/cgi-bin/nph-omniscgi.exe'; // Build RPC parameters $string = 'this is a test string'; $var = new RSvar(); $var->setChar($string, strlen($string)); $params = array( $var ); // Call 'testRPC' remote procedure $result = $rs->execute('testRPC', $params); // handle result echo $result->getChar(); ?>Within Omnis, I loaded up the following remote task class BRIDGE.controller:
##### Method '$construct' #####
No. Parameter Type Subtype Init.Val/Calc Description
1 pParams Row Remote call parameters
No. Method text
1 Calculate iCanClose as kTrue ;; Allow auto closing of instance
2
3 ; Check that the connection is comming from localhost, otherwise can it
4 If isnull($clientaddress)
5 Quit method $raiseError(8,'Cannot connect from IP',pParams)
6 Else If $clientaddress<>'127.0.0.1'
7 Quit method $raiseError(8,'Cannot connect from IP',pParams)
8 End If
9
10 Quit method $processmsg(pParams) ;; remote procedure dispatcher
##### Method '$event' #####
No. Parameter Type Subtype Init.Val/Calc Description
1 pParams Row Remote call paramaters
No. Method text
1 On evPost
2 Quit method $processmsg(pParams) ;; remote procedure dispatcher
##### Method '$processmsg' #####
No. Parameter Type Subtype Init.Val/Calc Description
1 pParams Row
1 ; do check to see that method exits
2 If $cinst.$methods.$makelist($ref().$name).$search($ref.C1=pParams.RSname)<>0
3 Do method [pParams.RSname] (pParams) Returns pParams
4 Else
3 Do method $raiseError (16,'Procedure does not exist',pParams) Returns pParams
9 End If
10
11 Quit method pParams
##### Method '$canclose' #####
No. Method text
1 Quit method iCanClose
##### Method '$raiseError' #####
No. Parameter Type Subtype Init.Val/Calc Description
1 iNumber Short integer (0 to 255) 0 Error number
2 cMessage Character 100000000 'No message defined' Error String
3 pParams Row
No. Local Variable Type Subtype Init.Val/Calc Description
1 %lResult Number floating dp
2 rError Row
No. Method text
1 Signal error {iNumber,cMessage}
2
3 ; Define error message and return it as the RT result
4 Do rError.$cols.$add('error',kCharacter)
5 Do rError.$cols.$add('number',kNumber)
6 Do rError.$cols.$add('message',kCharacter)
7 Do rError.$assigncols('__ERROR__',iNumber,cMessage)
8 Do pParams.$cols.RSrval.$coltype.$assign(kRow)
9 Calculate pParams.RSrval as rError
10 Quit method pParams
##### Method 'testRPC' #####
No. Parameter Type Subtype Init.Val/Calc Description
1 pParams Row
No. Method text
1 Calculate pParams.RSrval as pParams.RSparm1
2
3 Quit method pParams
The remote task is initialized by RStudio when the request is received by the Omnis server. The Omnis server passes a row variable pParams to the constructor of the object. pParams contains the name of the RPC name in pParams.RSname. All RPC parameters are contained in pParams.RSparm1, pParams.RSparm2, etc… All RPC parameters found in pParams are converted to native Omnis data types when they are received by the called remote task.
The meat of the class is within the $processmsg method which handles dispatching the request to the actual called method. A check is made to ensure that the method actually exists. Should it not exist, or if any other terminal error occurs that needs to be communicated back to the client, the raiseError method can be called to build out an error message formatted in a unique way that the client can then trap.
The actually RPC called is implemented as method testRPC. testRPC does a few things here. It grabs the passed parameter from the client as found in pParams.RSparm1 and then assigns it to pParams.RSrval. pParams.RSrval should contain the result that you wish to communicate back to the client. Before Omnis server transmits the result back to the client, RStudio will encode the value found in pParams.RSrval in such a way that it can be decoded by the RStudio client so it can be mapped to native data types on the client side.
Phase New, Round Two
This all work pretty well but performance tests showed that the CGI middleware was a significant bottleneck. CGI is not especially performant as the CGI script needs to be started up for each request. The Apache Omnis module, mod_omnis.so, would certainly be a more performant option as the module is loaded on startup of the web server, not on each request. FastCGI could be another option, but I am unsure if the supplied Omnis CGI can be run in a FastCGI environment. I didn’t try either of these options for a variety of reasons, but most importantly I felt I could squeeze the most performance by removing this middle tier entirely.
Raining Data, unfortunately, does not disclose the inner workings of how this middleware works. However, through careful inspection of the client library, some reverse engineering of the data communicated between Omnis server and the middleware, and insightful comments from the Omnis community, I was able to circumvent the middleware. And it was well worth it. I saw roughly a three fold performance increase of the number of requests I could execute per minute through continuous requests. All this for a few lines of code changed in the Omnis PHP RStudio library
OMNIS_ORFC is the key to doing direct communication to the Omnis server. Within OMNIS_ORFC, a header is communicated that indicates the size of the body. The body itself is made up of URL encoded parameters. Talking about how OMNIS_ORFC works is beyond the scope of what I want to discuss here. But for those interested in this, perhaps for other language implementations of RStudio, the attached PHP code is pretty well documented.
Phase New, Round Three
Moving the middleware out proved to be the least of my problems. The shipped Omnis PHP RStudio library had several small, but annoying bugs and error handling issues here and there that took some time to track down. Thanks to open code, I was able to work through these issues quite quickly.
I then soon discovered that if I tried to return a list, let’s say 158 rows / 9 cols, from Omnis, the processes of decoding the response in the Omnis PHP RStudio client was incredibly slow. My tests showed that requests that involved decoding this response took upwards to 20 seconds round trip. Also, seemingly related, large string result sets would take a considerable amount of time to decode. Finally, the inverse also held true. If I tried to send a 150 X 10 array or a large string from the PHP RStudio client to Omnis, it would take unacceptable lengths of time.
After weeks of frustrating back and forth with RD tech support, and careful study of the Omnis PHP implementation, I decided to rewrite the entire PHP RStudio implementation. I was convinced that I could resolve this performance issue, and it was not a limitation of the implementation language as I was told. There were several other issues that were also bugging me:
- Making the API easier to use so that it would handle the conversion from PHP data types to OMNIS_ORFC and back again more seamlessly. The API just demanded too much from a lazy programmer like me.
- PHP coding standards.
- Upgrading the code to use PHP5 vs. PHP4 language features.
Well, after several iterations of coding, fixing, and trying to figure out how RStudio works under the hood, I got to a point where my custom implementation sufficiently resolved all performance, API, and coding issues. To boot, I was able to resolve a few other bugs that were inherited from the Omnis PHP implementation (namely a nasty bug that resulted in arbitrary data to become corrupted in Omnis after decode).
To execute an RStudio call, my API went from this:
mOmnisLibrary = 'BRIDGE'; // Omnis remote task to call $rs->mOmnisClass = 'connector'; // Port Omnis server is listening on $rs->mOmnisServer = '5112'; // URL / IP of web server $rs->mServerURL = '127.0.0.1'; // Path to Omnis web server gateway $rs->mServerScript = '/cgi-bin/nph-omniscgi.exe'; // Build RPC parameters $string = 'this is a test string'; $var = new RSvar(); $var->setChar($string, strlen($string)); $params = array( $var ); // Call 'testRPC' remote procedure $result = $rs->execute('testRPC', $params); // handle result echo $result->getChar(); ?>to simply this:
execute('testRPC', $params); ?>Performance wise – and I’m pretty happy about this – my 158 rows / 9 cols list went from 20 seconds round trip, to 0.2 +/- seconds.
But, their are some drawbacks to my Implementation. It for one breaks the RStudio API in several areas. Most importantly, I removed the entire variable object, as I found this to be too heavy and requiring too much work from the end user. Instead I implemented “magic” type checking at the language level to handle converting between PHP data types to Omnis data types. Also, should this not be sufficient, the end use can offer “type hints” to control how data is converted between languages.
Also, this implementation does not support encoding PHP arrays that are more than 2 dimensions, or decoding Omnis lists that are more than 2 dimensions.
For my purpose, these limitations are more than acceptable.
Phase New, Round Next
With this new PHP RStudio implementation, we now have a more solid foundation for PHP / Omnis interoperability.
I was able to solve the read / write Omnis data file format problem in what I consider a elegant fashion. Using the PHP Zend Framework database abstraction library (think Omnis DAM), I was able to build out an Omnis database adapter using RStudio. This offers close to all the functionality found in Omnis SQL – plus emulates some “must have” missing features like the SQL AS specifier – without needing to tie your Omnis data file exclusively to Omnis code.
Also, should we not opt to build business logic out in PHP, and leverage logic originally implemented in Omnis 7, we could migrate pieces of this code to Omnis Studio to be accessed via RStudio. Granted this is not ideal, as a conversion process would need to be undertaken. However, this is much preferable than converting our entire legacy applications from Omnis 7 to Omnis Studio which would be a much more involved process.
Attached is a copy of my PHP RStudio library for your perusal and use. Should you have any comments / suggestions, I’d be delighted to hear them.
In the future, I can see a few feature enhancements that would be valuable for our work here. RStudio can also transport “raw” data from the client to server without being put through the data mapping process. This could be a lighter way to transfer simple data. Also, I would like to see bi-directional RStudio support built out allowing data to be pushed from Omnis to PHP (or other language implementations), much like how Omnis to Omnis RStudio communication works. Lastly, being a PHP programmer first, I’m not entirely happy with my RStudio remote task implementation. This should make parameter and result mapping to methods more seamless.
Checkout our Remote Studio PHP Client on bitbucket.