Data-driven API Creation and Processing in Node.js

Data-driven web services API processing greatly simplifies supporting, maintaining, and documenting many different APIs.


Earlier this year, I took some time off from my own projects to write a web services server for my friend’s popular photobooth software.  Until then, the photobooth software was a heavyweight client application.  However, he wanted to add more sharing and social functions, such as sending photobooth pictures to users via texting and e-mail.  Also, some recent changes in Tumblr’s API made uploading pictures directly from the client rather burdensome.   For these and other reasons, he wanted to create a web server to handle all of the social and sharing integration.  At the same time, I wanted to dive into node.js for my own app,, so it seemed like a worthwhile side project.

The Problem: So Many APIs to Connect Together

It seems that virtually every conceivable function is available via a REST API.  The primary purpose of the web services server is to act as “glue” for all of these various services.  For this project, the initial set of APIs included:

  • Cloud table storage for the server’s own data
  • Cloud blob storage to temporarily store pictures
  • Text messaging (SMS and MMS) service to send and receive text messages and photobooth pictures
  • E-mail service to request and receive photobooth pictures
  • Tumblr upload integration
  • OAuth Callback API
  • A photobooth client API to enable photobooth applications to access these web services

Some of these interfaces are outgoing-only, but others, such as the e-mail and SMS interfaces, are two-way and may receive calls from the cloud at anytime.  Additionally, we needed the ability to quickly add other social and sharing services to the server, support API versioning, and a server administration and testing interface.  Because these APIs are exposed to the web, user authentication, URL validation, and API parameter validation are extremely important.

Data-driven API Processing Eliminates the Tedium of Supporting Different APIs

Rather than trying to implement each of these APIs separately, I decided to build a data-driven, self-documenting, modular API processing system.  Each API is defined by a Javascript object that defines that API’s specific function calls, associated URLs, and parameters. This API specification object is used by the server to process all incoming web requests and generate a testing interface.

By controlling the API processing pipeline using the API specification object, changes or modifications to an API specification object are automatically implemented by the server. Additionally, new API functions and even entirely new APIs can be easily added just by passing their API specification objects to the API processing pipeline. Despite being code, the API specification objects are easy to read and understand.

API Specification Object

For example, this is an abridged API specification:

module.exports.PhotocacheAPIList = [
								name: "uploadFile",
								type: "file",
								inputtype: "file",
								optional: true								
								name: "tempFilePath",
								type: "path",
								hidden: true,
								optional: false								
				, {
									name: "cachecode",
									type: "alpha-numeric",

This Javascript array includes objects representing each API function. Each API function’s specification object includes the function name, version, calling URL, the callback function used to process the API call, and any function parameters. For each API function’s parameters, this object specifies the parameter names, the parameters’ data types, and whether the parameters are required or optional.

Processing API Requests Using the API Specification Object

The web services server processes all incoming requests as follows:


The server initializes the API processing pipeline with the API specification objects for the supported API. The API processing pipeline then uses these API specification objects to evaluate each incoming API request.

For example, this function checks the incoming API request’s URL against the URLs defined by the API specification:

validateApiUrl: function (req, res, apiSpecArray){
			var choppedUrl = req.path.slice(1); //node provides url with a leading "/"
			var APISearch = apiSpecArray.filter(function (val) {
				return (val.url == choppedUrl);
			if (APISearch.length != 1){
				 if(console_logging <= 2) console.log("API Not found");
				return undefined;
				 if(console_logging <= 2) console.log("API found: " + APISearch[0].name);
				return APISearch[0];

This function simply searches through all of the url fields of the API specification objects to see if any API functions’ URLs match the API request’s URL. If so, the function returns the API specification object for the matching API function. This object describes the API function and its parameters.

If the API request’s URL is matched with an API function, then the API processing pipeline validates the request’s parameters against the API specification. This is done in two parts: first, the API request is checked to make sure all of the required parameters are included; and second, the API request’s parameters are validated and sanitized based on the parameter types defined by the API specification.

Checking for required parameters is done by cycling through the specification of API function parameters to identify a list of required parameters, and then checking to see if the request includes every parameter in this list.

var reqParams = spec.params.filter(function (val) {
			return (val.optional == false);
for (var p in reqParams){
				var param = params.filter(function (val){ return ( == reqParams[p].name);});
					if(param[0].value != ""){
						if(console_logging <= 1) console.log("Req parameter found " + reqParams[p].name);
				catch (e){
					if(console_logging <= 2) console.log("Parameter Missing ", reqParams[p].name);
					return false;

If all of the required parameters are present, then all of the API request’s parameters (including any optional or extraneous parameters) are validated and sanitized.

//If all req params found, validate all params included in request
			for (p in params){
				if(selectedParamSpec = spec.params.filter(function (val){ return ( == params[p].name);})){
					if(console_logging <= 1) console.log("Parameter matched " + params[p].name);
						if(params[p].value != ""){  //No need to process empty optional parameters, empty req parameters caught above.
						//Validate parameter type and length here.
						//Copy matching params to new object -- this has the effect of ignoring any bogus params.
							//Default behavior is to bounce request if any invalid params are received.							
							validatedParams = undefined;
			return validatedParams;

Parameters are validated using a standard validation module – node validator. However, this validation code is wrapped in another function to allow for complex and user-defined parameter types.

validateApiParamTypes: function (pValue,pType){
				case "url":
				case "alpha-numeric":
				case "alpha-numeric6":
						check(pValue.value,"Not a 6 digit alphanumeric").isAlphanumeric().len(6,6);
					} catch (e) {
							 if(console_logging <= 2) console.log(e.message);
							return undefined; 
				case "int":

In addition to common data types such as “url” and “int”, custom data types are supported such as a 6 character alphanumeric code.

Finally, if the incoming API request passes all of these validation steps, then the API request is forwarded to the API callback function defined by the API specification object for processing.

if(validatedParams = this.validateApiParams(postedParams,selectedAPISpec)){
	//run API callback function
	selectedApiObject[selectedAPISpec.callback](req, res, validatedParams);


Other Uses for the API Specification Object

In addition to controlling the API processing pipeline, these API specification objects can be used for other purposes, such as for documentation and testing. For example, I also use it to generate a web-based testing interface. The set of API specification objects is passed to a Jade web template, which then generates a web testing page including a set of web forms for the API functions. Each web form is configured with the URL and parameters of one of the API functions. The resulting web page includes web forms for all of the API functions.

Here is the node.js call to create an API test web page:

app.get('/api_test', auth, function(req, res){
	apiGateway.showAPITestPage(req, res,'api_test.jade', APISpecArray);

And this is the jade template for creating web forms based on the API specification objects:

div(style='border: 1px solid #888; border-radius: 3px;')
            - each item in APISpecArray
              != "<div id=\"" + + "_form-container\" class=\"API-form-container\" style=\"display: block;\">"
                            != "<form id=\"" + + "_form\" enctype=\""+(item.enctype!=undefined ? item.enctype : "application/x-www-form-urlencoded")+"\" action=\"" + item.url +"\" method=\"post\">"
                - each param in item.params
                  - if (param.hidden != true)
                    !="<label for=\"" + + + "\"> Property " + + (param.optional == true ? " (Optional)":" (Required)") + "</label>"
                    !="<input type=\""+(param.inputtype=="file" ? "file" : "text")+"\" id=\"" + + + "\" name=\"" + +"\">"                    
              button(id ="#{}").APITestSubmit  Test #{}
              != "</form>" 
              != "</div>"

Comments are closed.

Copyright 2018 · RSS Feed · Log in

Business Theme by Organic Themes · WordPress Hosting

Organic Themes