Breaking the Log Jam

In our last release we added a feature so that individual endpoints can be excluded from logging. Still, the number of log files that get generated can still be daunting and difficult to sift through when specific issues need to be investigated. Never fear...the new HTTP Logs utility is here! Just launch the NDW_HTTP_LOGS form, confirm the log directory, and wonder why we never created this sooner!

Paving the Road to Integration

Up until now, the Resource Manager feature of the HTTP Framework Setup form maintained exclusive control over resource endpoint definitions (read: tightly coupled code design). Recent projects necessitated a way for 3rd party utilities to tap into this same functionality. Rather than copy blocks of code, we decoupled the resource endpoint logic and relocated it into its own service module: HTTP_Resource_Manager_Services. This will be documented soon but, unless you have a need to manage endpoints outside of the HTTP Framework Setup form, you probably don't need to know about this.


Ping Me Up Sometime

A new Ping API has been added. Many systems need a way to monitor whether or not the API server is still running. The Ping API was created specifically for this purpose. It returns the current date and time in GMT format.

To Log, or Not to Log, that is the Question

Logging is great until you need to sort through thousands of logs that exist only to monitor the API server. Each resource endpoint now has the ability to be excluded from logging. The new Ping API is excluded by default.

Home URL on the Range

The HTTP Framework Setup provides a way to enter the expected Home URL for your APIs. This is used for logging and for producing hypermedia links in your API responses. However, this might not be useful for sites that support multiple URLs or IP addresses. Therefore, the Home URL will now be derived from the incoming request.

Time Off for Good Behavior

The new Web Accounts authentication system works really well at tracking the number of invalid password attempts. In fact, it works a little too well. It turns out that once an account has incurred an invalid password attempt, this incursion is remembered by the system even after the account successfully provides a valid password. This eventually causes an account to immediately become disabled after only one invalid password attempt. The system will now reset the number of invalid password attempts whenever a valid password is provided. The number of invalid passwords tracked by the overall system, however, is still accumulated until manually reset.


A Good Thing just gets Better

We hate to sound repetitive, but this is again a minor release that ties up more loose ends.


A Good Thing just gets Better

This release mostly ties up a few loose ends that were introduced in v4.0.0.


API Automation

Our next major release to the SRP HTTP Framework introduces some major improvements to the way resources and APIs are created. The HTTP Framework Setup form has been heavily extended to support new features. The most significant of these is the Resource Manager, a visual tool that allows developers to define endpoints and automatically generate API templates with the click of a few buttons!

Authentication in a Box

While the SRP HTTP Framework has included support for HTTP Basic Authentication since its first release and it has made it easy to embrace other methods of authentication, it had never ship with an out-of-the box authentication management tool. We have rectified this by including a Web Accounts manager. Users can now be created with expiring passwords. The SRP HTTP Framework authentication service has been significantly updated to provide support for Web Accounts while making it even easier to support other methods as well. Also, unsuccessful authentication requests are now tracked and the system can be configured to automatically go into containment mode based on a predetermined number of invalid passwords.

APIs are the new Services

As a way of embracing the awesomeness and convenience of our new enhanced BASIC+ syntax, we've added a new statement: GoToAPI. This effectively eliminates maintaining those strings of allowed methods and services in addition to the long and complicated Case statements to route each method and endpoint combination to an appropriate GoSub label. Seeing is believing...so check it out!

GoToAPI else
   // The specific resource endpoint doesn't have a API handler yet.
   HTTP_Services('SetResponseStatus', 204, 'This is a valid endpoint but a web API handler has not yet been created.')

Return Response OR ''

// Endpoint Handlers

API contacts.POST
   HTTP_Resource_Services('PostDatabaseItem', 'CONTACTS', FullEndpointURL)
end api

API contacts.ID.PATCH
   KeyID   = EndpointSegment
   HTTP_Resource_Services('PatchDatabaseItem', 'CONTACTS', FullEndpointURL, KeyID)
end api

API contacts.ID.PUT
   KeyID   = EndpointSegment
   HTTP_Resource_Services('PutDatabaseItem', 'CONTACTS', FullEndpointURL, KeyID)
end api

API contacts.ID.DELETE
   KeyID   = EndpointSegment
   HTTP_Resource_Services('DeleteDatabaseItem', 'CONTACTS', KeyID)
end api

Due to the new syntax and the desire to maintain backwards compatibility with the legacy service code, the old format of HTTP_{Resource}_Services is now replaced with {Resource}_API.

Resource Building Your Way

The HTTP_Resource_Services routine has always been a convenient way to quickly convert OpenInsight data into rich JSON while following good RESTful practices such as including self-referencing and collection URIs. This is accomplished through a family of high-level services (e.g., GetDatabaseItem, PostDatabaseItem, DeleteDatabaseItem). However, there are times when a little more low-level effort is required. To help with this, several new services have been added to be a middle ground between these high-level services and direct calls to the SRP_JSON function.


Protecting Column Data

In addition to a couple of minor bug fixes, this release enhances POST, PUT, and PATCH APIs that rely upon HTTP_Resource_Services. Developers can now pass in an @FM delimited list of column names that are allowed to be updated. Hence, even if the client submits a payload containing every known column name in the dictionary, these services will now cross-check each column name against the approved list and by-pass any columns that are not mentioned. If this allowed column names list is empty, then these APIs will attempt to update every column referenced in the payload (i.e., default behavior).


Keeping in Step

Another minor release to allow the Entry Point Service entry to be formatted like the other service routines in the HTTP Framework Setup form. This prompt normally expects "HTTP" and "SERVICES" to be omitted, but the more recent Aborted Service and Debugger Setting entries expect fully qualified names. With this update, entry_point or HTTP_ENTRY_POINT_SERVICES will work equally well.


Missing Metadata

What!? A new version already? Yes, but nothing to get too excited over. We realized that even though most of our routines are written using a service-oriented approach, we were not shipping the metadata that the SRP Editor can use to help automate the building of calls into our services. This oversight is now rectified.


In version 3.0.1, virtually every major component has been overhauled to make RESTful web services even easier to create and debug. Just read on...

HTTP Framework Setup

Introduced in v3.0.0, the HTTP Framework Setup form (NDW_HTTP_FRAMEWORK_SETUP) form includes more options:

  • Enable Logging - No longer is it necessary to rename your log folder to turn off logging. Now you can simply toggle this on or off!
  • Log Errors Only - Are you getting lost in a sea of log files? Perhaps you only need to find those pesky error responses. Easy peasy! Just check this box and only 4xx and 5xx HTTP status code responses will be sent to your folder.
  • Aborted Service - Miss having an INET_ABORTED equivalent feature? You're in luck. A template for handling aborted requests now ships with v3.0.1 called HTTP_ABORTED_SERVICE. Feel free to modify this or copy it into your own custom aborted service handler.
  • Debugger Setting - Tired of running back and forth to the Database Manager to change the Debugger Setting? Are you concerned that modifying this setting will affect your desktop users? If you answered yes to either question we have you covered! Now you can specify your own Debugger Setting that will only affect your web services. Of course, what good is having an Intercept option without a Debugger Intercept service? Glad you asked because we have that covered too. The HTTP_DEBUGGER_SERVICE routine also ships with v3.0.1. Like HTTP_ABORTED_SERVICE, you are free to modify this routine or make a copy to use.

Log like a Boss

Logging is a critical tool in the developer's belt when troubleshooting and profiling web services. In the last release we updated the Response log to include useful metadata. In 3.0.1, we go full throttle with the logging enhancements.

  • Request Logs - Once upon a time the Request logs were nothing but raw text dumps of the Request argument that the OECGI passed into HTTP_MCP. We are now proud to showcase our new and improved Request log format:

    Request Log
    Request Argument
    <01> HTTPQueryString          : 
    <02> HTTPPathInfo             : contacts
    <03> HTTPContentType          : 
    <04> HTTPContentLength        : 0
    <05> HTTPGatewayInterface     : CGI/1.1
    <06> HTTPHTTPS                : off
    <07> HTTPAccept               : */*
    <08> HTTPCookie               : 
    <09> HTTPFrom                 : 
    <10> HTTPReferer              : 
    <11> HTTPUserAgent            : PostmanRuntime/6.1.6
    <12> HTTPTranslated           : C:\MyWebsite\www\contacts
    <13> HTTPRemoteAddr           :
    <14> HTTPRemoteHost           : 
    <15> HTTPRemoteIdent          : 
    <16> HTTPRemoteUser           : 
    <17> HTTPRequestMethod        : GET
    <18> HTTPScriptName           : /cgi-bin/oecgi4.exe
    <19> HTTPServerName           : www.contacts.com
    <20> HTTPServerPort           : 80
    <21> HTTPServerProtocol       : HTTP/1.1
    <22> HTTPServerSoftware       : Abyss/ AbyssLib/
    <23> HTTPServerURL            : 
    <24> HTTPNoURLDecode          : 
    <25> HTTPResponseIsBinary     : 
    <26> HTTPRegistrySettings     + 
         RegistryInfo             : SOFTWARE\RevSoft\OECGI4
         EngineName               : 
         ServerURL                : localhost
         ServerPort               : 8088
         ApplicationName          : FRAMEWORKS
         UserName                 : FRAMEWORKS
         StartupFlags             : 1
         ShutdownFlags            : 1
         FilePath                 : 
         FilePathMapped           : 
         FileMode                 : 1
         SysDownPage              : 
         OILocation               : 
    <27> HTTPOECGIVersion         : VERSION:OECGI4
    <28> HTTPGetString            : 
    <29> HTTPPostString           : 
    <30> HTTPAdditionalValues     + 
         Media-Type               : 
         Accept-Encoding          : gzip, deflate
         Accept-Charset           : 
         Accept-Language          : 
         Authorization            : 
  • Response Logs - Turns out more metadata is a good thing so we went ahead and updated the Response log with the current HTTP Framework version, Authorization information, and Query Params used:

    Response Log
    HTTP Framework : v3.0.1 - 07/11/2017 12:58PM
    Time to Execute: 00h 00m 00s 063ms
    Request Method : GET
    Request URL    : http://www.contacts.com/api/contacts
    Authorization  : None
    Query Params   : company=benton
    Status: 200 OK
    Access-Control-Allow-Origin: *
    Content-Type: application/hal+json
    Content-Length: 507
       "_embedded" : {
          "item" : [
                "_links" : {
                   "self" : {
                      "href" : "http://www.contacts.com/api/contacts/1"
                "company" : "Benton, John B Jr",
                "email" : "jbutt@gmail.com",
                "first_name" : "James",
                "last_name" : "Butt"
       "_links" : {
          "self" : {
             "href" : "http://www.contacts.com/api/contacts"
  • Aborted Logs - We added a new log type for aborted web services to make it easy to identify the offending code:

    Aborted Log
    ProcErr Argument
    ENG0010:  HTTP_CONTACTS_SERVICES, line 315.  Variable has not been assigned a value. 
  • Debugger Logs - For those who want to utilize a Debugger Intercept option for more indepth troubleshooting in their web applications, we also provide a JSON formatted log file for the basic intercept content:

    Debugger Log
       "CallDepth" : 5,
       "CallStack" : [
             "LineNo" : 315,
             "ProcName" : "HTTP_CONTACTS_SERVICES"
             "LineNo" : 155,
             "ProcName" : "HTTP_SERVICES"
             "LineNo" : 81,
             "ProcName" : "HTTP_ENTRY_POINT_SERVICES"
             "LineNo" : 155,
             "ProcName" : "HTTP_SERVICES"
             "LineNo" : 135,
             "ProcName" : "HTTP_MCP"
       "Curr_Program" : "HTTP_CONTACTS_SERVICES",
       "LineNo" : 315,
       "SPStatCode" : "ENG0010:  HTTP_CONTACTS_SERVICES, line 315.  Variable has not been assigned a value.",
       "SPStatus" : 1

Log File Names

What's in a name? Actually, quite a bit. We've even updated the names assigned to our log files for more diagnostic muscle:

The format of each file name is YYYY-MM-DD_HH-MM-SS_PID_LogType.log. The PID (Process ID) can be quite useful when multiple requests are processed simultaneously. Each OpenEngine process will have its own unique PID. Since each HTTP Request and associated Response will be handled by the same OpenEngine, it's quite easy now to pair up the relevant log files. The PID also makes it easy to see when one OpenEngine process may have been terminated due to a runtime error. For instance, in the above screen shot the Debugger log file introduces a new OpenEngine (PID 43028) because the original OpenEngine (44448) used by the previous HTTP Request terminated.

Making Contacts

No, we are not adding social networking to the product but we are offering a full featured Contacts web service. A sample HTTP_CONTACTS_SERVICES has shipped with the product for some time, but it was mostly there as example code and was never connected to a database table. No more! We are now including a sample CONTACTS database table (tied to the GLOBAL database). Just attach and then run the Contacts web service out of the box. We have also updated HTTP_CONTACTS_SERVICES to support all major CRUD activities, including searching via query parameters. We've added a lot of comments to this routine to get both novice and experienced web developers productive even quicker. Use this routine as a template for your own web services or copy bits and pieces to augment exist services.

Ah CRUD...

Speaking of create, read, update, and delete functionality, we enhanced the HTTP_RESOURCE_SERVICES utility to fully support proper POST, PUT, and PATCH operations against database table resources. POST will create new database rows when the server needs to generate the next Key ID. PUT creates (or updates) database rows when the client specifies the Key ID. Finally, PATCH updates only those specific columns that are passed into the web service (OECGI4 v4.0.0.3 or higher required).


Version 3.0.0 introduces new features to make it easier to configure setup values and to accept special URLs without requiring authentication.


This form can be used to configure all setup parameters without the need to edit the SRP_HTTP_FRAMEWORK_SETUP record directly.

Non-Authenticated URLs

A new setup parameter has been created to store one or more URLs that should pass through the authentication logic. The HTTP_Authentication_Services module has been updated to verify if the URL requires authentication. If not, the current URL request is marked as authenticated so it can continue to be processed by the end point API.

This feature is helpful when OAuth-type requests need to be supported. OAuth relies upon communication between trusted servers, away from client-side web and mobile applications. This is part of the security mechanism of OAuth. Generally, these URLs cannot be configured to support server-side authentication so these need to be allowed through without authentication.

HTTP_MCP and Logging

The logging feature of HTTP_MCP has been updated to provide more useful information in the response log. In addition to the basic response information (e.g., status code, response headers, and the body), metadata for the URL request itself is added so that it is easier to match the request to the response. Additionally, the time to execute the request from start to finish is also included as a way of helping developers identify potential bottlenecks. Here is an example of an updated response log:

Response Log
Time to Execute: 00h 00m 00s 875ms
Request Method : POST
Request URL    : https://api.srpcs.com/srpteam/users/don
Status: 201 Created
Access-Control-Allow-Origin: *
Content-Type: application/hal+json
Content-Length: 129
   "URL" : "https://api.srpcs.com/srpteam/users/don",
   "method" : "POST",
   "phrase" : "Created",
   "status" : 201
  • No labels