REST API

Re-thinking STARLIMS architecture

Re-thinking STARLIMS architecture

Michel Roberge

There is something about STARLIMS that has been bugging me for a long time. Don’t get me wrong – I think it is a great platform. I just question the wellness of XFD in 2024, and the selection of Sencha for the HTML part of it.

But an even more critical point: I question the principle of using the same server for the “backend” and the “frontend”. Really, the current architecture of STARLIMS (in a simplified way) is something like this:

Sure, you can add load balancers, multiple servers, batch processors… But ultimately, the Server’s role is both backend and Web Rendering, without really following Server-Side-Rendering (SSR) pattern. It hosts / provides the code to render from backend and let client do rendering. So, in fact, it is Client-Side-Rendering (CSR) with most of the SSR drawbacks.

This got me thinking. What if we really decoupled the front end from the backend? And what if we made this using real micro services? You know, something like this:

Let me explain the layers.

React.js

React does not need presentation. The infamous open-source platform behind Facebook. Very fast and easy, huge community… Even all the AI chatbot will generate good React components if you ask nicely! For security, it’s like any other platform; it’s as secure as you make it. And if you pair it with Node.js, then it’s very easy, which brings me to the next component…

Node.js

Another one in no need of presentation. JavaScript on a backend? Nice! And there, on one end, you handle the session & security (with React) and communicate with STARLIMS through the out of the box REST API. Node can be just a proxy to STARLIMS (it is the case currently) but should also be leveraged to extend the REST APIs. It is a lot easier to implement new APIs and connect to STARLIMS (or anything else for that matter!) and speed up the process. Plus, you easily get cool stuff like WebSockets if you want, and you can cache some reference data in Redis to go even faster!…

Redis

Fast / lightweight / free cache (well, it was when I started). I currently use it only for sessions; since REST API is stateless in STARLIMS, I manage the sessions in Node.js, and store them in Redis, which allows me to spin multiple Node.js instances (load balancing?) and share sessions across. If you don’t need to spin multiple proxy, you don’t need this. But heh, it’s cooler with it, no?

I was thinking (I haven’t done anything about this yet) to have a cron job running in Node.js to pull reference data from STARLIMS (like test plans, tests, analytes, specifications, methods, etc) periodically and update Redis cache. Some of that data could be used in the UI (React.js) instead of hitting STARLIMS. But now, with the updated Redis license, I don’t know. I think it is fine in these circumstances, but I would need to verify.

… BUT WHY?

Because I can! – Michel R.

Well, just because. I was learning these technologies, had this idea, and I just decided to test the theory. So, I tried. And it looks like it works! There are multiple theoretical advantages to this approach:

  1. Performance: Very fast (and potentially responsive) UI.
  2. Technology: New technology availability (websockets, data in movement, streaming, etc.).
  3. Integration: API first paradigm, Node.js can make it really easy to integrate with any technology!
  4. Source control: 100% Git for UI code, opening all git concepts (push, pull requests, merge, releases, packages, etc.).
  5. Optimization: Reduce resource consumption from STARLIMS web servers.
  6. Scalability: High scalability through containerization and micro-services.
  7. Pattern: Separation of concerns. Each component does what its best at.
  8. Hiring – there is a higher availability of React.js and Node.js developers than STARLIMS developers!

Here’s some screenshots of what it can look like:

As you can see, at this stage, it is very limited. But it does work, and I like a couple of ideas / features I thought of, like the F1 for Help, the keyboard shortcuts support, and more importantly, the speed… It is snappy. In fact, the speed is limited to what the STARLIMS REST API can provide when getting data, but otherwise, everything else is way, way faster than what I’m used to.

How does it work, really?

This is magic! – Michel R.

Magic! … No, really, I somewhat “cheated”. I implemented a Generic API in the STARLIMS REST API. This endpoint supports both ExecFunction and RunDS, as well as impersonation. Considering that the REST API of STARLIMS is quite secure (it uses anti-tampering patterns, you can ask them to explain that to you if you want) and reliable, I created a generic endpoint. It receives a payload containing the script (or datasource) to run, with the parameters, and it returns the original data in JSON format.

Therefore, in React, you would write code very similar to lims.CallServer(scriptName, parameters) in XFD/Sencha.

Me being paranoid, I added a “whitelisting” feature to my generic API, so you can whitelist which scripts to allow running through the API. Being lazy, I added another script that does exactly the same, without the whitelisting, just so I wouldn’t have to whitelist everything; but hey, if you want that level of control… Why not?

Conclusion

My non-scientific observations are that this works quite well. The interface is snappy (a lot faster than even Sencha), and developing new views is somewhat easier than both technologies as well.

Tip: you can just ask an AI to generate a view in React using, let’s say, bootstrap 5 classNames, and perhaps placeholders to call your api endpoints, et voilà! you have something 90% ready.

Or you learn React and Vite and you build something yourself, your own components, and create yourself your own STARLIMS runtime (kind-of).

This whole experiment was quite fun, and I learned a ton. I think there might actually be something to do with it. I invite you to take a look at the repositories, which I decided to create a public version of for anyone to use and contribute under MIT with commercial restrictions license:

You need to have both projects to get this working. I recommend you check both README to begin with.

Right now, I am parking this project, but if you would like to learn more, want to evaluate this but need guidance, or are interested in actually using this in production, feel free to drop me an email at [email protected]! Who knows what happens next?

Here’s the missing API helper

Michel Roberge

Well, all was good under the sun, until a reader pointed out that I had omitted a very important piece. I was expecting STARLIMS developers to know how to manage; but it is not so. Re-reading, I realized that indeed, one might need directions.

I’m talking about the API_Helper_Custom.RestApiCustomBase class.

This class is not really needed, you can instead inherit of the RestApi.RestApiBase class.

But having our own custom base is good! It allows us to implement common functionalities that all your services may need. In this example, I’ll provide an impersonation method, very useful if you wish to have a single integration user, but actually know who the user should be impersonating.

:CLASS RestApiCustomBase;
:INHERIT RestApi.RestApiBase;

:DECLARE APIEmail;
:DECLARE LangId;
:DECLARE UserName;

/* do stuff here that applies to all custom API's;

:PROCEDURE Constructor;

    :DECLARE sUser;
    Me:LangId := "ENG";
    Me:UserName := GetUserData();

    Me:APIEmail := Request:Headers:Get("SL-API-Email");

    sUser := LSearch("select USRNAM from USERS where EMAIL = ? and STATUS = ?", "", "DATABASE", { Me:APIEmail, 'Active' });             

    :IF ( !Empty(sUser) ) .and. ( sUser <> GetUserData() );
        Me:Impersonate(sUser);
    :ENDIF;

    Me:LangId := LSearch("select LANGID from USERS where USRNAM = ?", "ENG", "DATABASE", { MYUSERNAME });

:ENDPROC;

/* Allow system to impersonate a user so transactions are corrected against the correct user;
:PROCEDURE Impersonate;
    :PARAMETERS sUser;
    :IF !IsDefined("MYUSERNAME");
        :PUBLIC MYUSERNAME;
    :ENDIF;
    MYUSERNAME := sUser;
    SetUserData(MYUSERNAME);        
:ENDPROC;

As you can see, this is pretty simple. Once you have this REST API ready, inherit this class, and you should be fine to have a working API. In the above example, the code expect a header SL-API-Email that will contain the email of the user to impersonate. If it is not provided, then the user to whom the key belong is the current user.

Hope this helps those who didn’t yet figure it out!

Open up STARLIMS with its REST API!

Open up STARLIMS with its REST API!

Michel Roberge

Alright folks, I was recently involved in other LIMS integrations and one pattern that is very much alike is a “click this functionality to enable the equivalent API” approach. Basically, by module, you decide what can be exposed or not. And then, by role or by user (or lab, of all of that), you grant consuming rights.

It got me thinking “heh, STARLIMS used to do that with the generic.asmx web service”. RunAction and RunActionDirect anyone?

So, that’s just what I did, for fun, but also thinking that if I’d go around re-writing routing and scripts for every single functionalities, that would be a total waste of time. Now, don’t get me wrong! Like everything, I think that it depends.

You can (and should) expose only bits and pieces you need to expose, unless your plan is to use STARLIMS as a backend mostly and integrate most (if not all) of the features to external systems (those of you who you want a React or Angular front end, that’s you!).

So, if you’re in the later group, take into consideration security. You will want to set the RestApi_DevMode setting to false STARLIMS’ web.config file. This is to ensure that all communication is hashed using MD5 and not tampered with. Then, of course, you’ll check https and all these things. This is out of scope, but still worthy of note.

Once that’s done, you need 2 pieces.

  1. You need to define a route. Personnally, I used the route /v1/generic/action . If you don’t know how to do that, I wrote and article on the topic.
  2. You need a script to do all of this! Here’s the simplified code: 
:PROCEDURE POST;
	:PARAMETERS payload;
	:IF payload:IsProperty("action") .and. !Empty(payload:action);
		:DECLARE response;
		response := CreateUdObject();
		response:StatusCode := 200;
		response:Response := CreateUdObject();
		response:Response:data := "";
		:IF payload:IsProperty("parameters") .and. !Empty(payload:parameters);
			response:Response:data := ExecFunction(payload:action, payload:parameters);
		:ELSE;
			response:Response:data := ExecFunction(payload:action);
		:ENDIF;
		:RETURN response;
	:ELSE;
		:DECLARE response;
		response := CreateUdObject();
		response:StatusCode := 400;
		response:Response := CreateUdObject();
		response:Response:message := "invalid action/parameters";
		:RETURN response;
	:ENDIF;
:ENDPROC;

In my case, I went a little bit more fancy by adding an impersonation mechanism so the code would “run as”. You could add some authorization on which scripts can be ran, by who, when, etc. Just do it at the beginning, and the return a 403 forbidden response if execution is denied.

Yeah, I know, this is not rocket science, and this is not necessarily the most secure approach. In fact, this really opens up your STARLIMS instance to ANY integration from a third party software… But, as I mentioned in the beginning, maybe that’s what you want?

NB: I used DALL·E 2 (openai.com) to generate the image on the front page using “STARLIMS logo ripped open”. I had to try!

STARLIMS REST API & POSTMAN – Production Mode

Michel Roberge

Alright folks! If you’ve been playing with the new STARLIMS REST API and tried production mode, perhaps you’ve run into all kind of problems providing the correct SL-API-Signature header. You may wonder “but how do I generate this?” – even following STARLIMS’s c# example may yield unexpected 401 results.

At least, it did for me.

I was able to figure it out by looking at the code that reconstructs the signature on STARLIMS side, and here’s a snippet of code that works in POSTMAN as a pre-request code:

// required for the hash part. You don't need to install anything, it is included in POSTMAN
var CryptoJS = require("crypto-js");

// get data required for API signature
const dateNow = new Date().toISOString();
// thhis is the API secret found in STARLIMS key management
const privateKey = pm.environment.get('SL-API-secret');
// this is the API access key found in STARLIMS key management
const accessKey = pm.environment.get('SL-API-Auth');
// in my case, I have a {{url}} variable, but this should be the full URL to your API endpoint
const url = pm.environment.get('url') + request.url.substring(8);
const method = request.method;
// I am not using api methods, but if you are, this should be set
const apiMethod = "";

var body = "";
if (pm.request.body.raw){
    body = pm.request.body.raw;
}

// this is  the reconstruction part - the text used for signature
const signatureBase = `${url}\n${method}\n${accessKey}\n${apiMethod}\n${dateNow}\n${body}`;

// encrype signature
var data = CryptoJS.enc.Utf8.parse(signatureBase);
const hash = CryptoJS.HmacSHA256(data, privateKey);
const encodedHash = encodeURIComponent(CryptoJS.enc.Base64.stringify(hash));

// set global variables used in header
pm.globals.set("SL-API-Timestamp", dateNow);
pm.globals.set("SL-API-Signature", encodedHash);

One point of interest – if it still is not working, and if you can’t figure out why, an undocumented STARLIMS feature is to add this application setting in the web.config to view more info:

<add key="RestApi_LogLevel" value="Debug" />

I hope this helps you use the new REST API provided by STARLIMS!

REST API Cloud

STARLIMS REST API – Add and Route your own endpoints

Michel Roberge

With the version 12 technology platform, STARLIMS offers a new REST API engine. It is really great – until you want to enhance it and add your own endpoints. That’s where it gets … complicated. Well – not so much – if you know where to start. Nothing here is hidden information, it is all written in the technology release documentation; just not easily applied.

If you read the doc, you’ve read something like this:

Routing maps incoming HTTP API requests to their implementation. If you are a Core Product team, you must implement routing in pre-defined Server Script API_Helper.RestApiRouter; if you are a Professional Services or Customer team, you must implement routing in pre-defined Server Script API_Helper_Custom.RestApiRouter (which you need to create, if it doesn’t exist).

STARLIMS Technology Platform Documentation
09-016-00-02 REV AB

That section is accessible using the /building_rest_api.html Url of the platform documentation.

It is really good, and it works, and everything listed is appropriate. I would only add 2 points for your sanity.

1- handle your routes in a different way than what STARLIMS suggest. Their example is very simple, but you’ll want to have something scalable / reusable. I went with a single function and nested hashtables. By default, the custom routing needs a Route method. To “store” the route, I’ll also add a private getRoutes method. In the future, we’ll only add entries in the getRoutes, which will simplify our life.

:PROCEDURE getRoutes;

	/* 
		structure is:
		
		hashTable of version
			hashTable of of service
				hashTable of entity
	;
	:DECLARE hApiVersions;
	
	/* all route definition should be in lowercase;
	
	/* store API Verions at 1st htable level;
	hApiVersions := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v2"] := LimsNetConnect("", "System.Collections.Hashtable");
	
	/* store each service within the proper version;
	hApiVersions["v1"]["examples"] := LimsNetConnect("", "System.Collections.Hashtable");
	/* then store each endpoint per entity;
	hApiVersions["v1"]["examples"]["simple"] := "API_Examples_v1.Simple";

	/* store each service within the proper version;
	hApiVersions["v1"]["system"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"]["system"]["status"] := "API_CustomSystem_v1.status";
	
	/* process-locks endpoints;
	hApiVersions["v1"]["process-locks"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"]["process-locks"]["process"] := "API_ProcessLocks_v1.Process";
	
	/* user-management endpoints;
	hApiVersions["v1"]["user-management"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"]["user-management"]["user-session"] := "API_UserManagement_v1.UserSession";
	
	hApiVersions["v1"]["sqs"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"]["sqs"]["message-queue"] := "API_SQS_v1.message";

	hApiVersions["v1"]["load"] := LimsNetConnect("", "System.Collections.Hashtable");
	hApiVersions["v1"]["load"]["encrypt"] := "API_Load_v1.encrypt";
	hApiVersions["v1"]["load"]["origrec"] := "API_Load_v1.origrec";

	:RETURN hApiVersions;
:ENDPROC;


:PROCEDURE Route;
	:PARAMETERS routingInfo;
	
	/* 	routingInfo
			.Version : string - e.g. "v1"
			.Service : string - e.g. "folderlogin"
			.Entity : string - e.g. "sample";
	
	:DECLARE hRoutesDef, sVersion, sService, sEntity;
	
	hRoutesDef := Me:getRoutes();
	
	/* remove case route;
	sVersion := Lower(routingInfo:Version);
	sService := Lower(routingInfo:Service);
	sEntity := Lower(routingInfo:Entity);
	
	:IF !Empty(hRoutesDef[sVersion]);
		:IF !Empty(hRoutesDef[sVersion][sService]);
			:RETURN hRoutesDef[sVersion][sService][sEntity];
		:ENDIF;
	:ENDIF;
	
	:RETURN "";
	
:ENDPROC;

When you need to add new routes, all you do is add new lines to the getRoutes method, the logic in the Route method is static and shouldn’t change. Then, you create the corresponding categories and scripts to actually run your logic, and you’re set.

Of course, you can build your own mechanism – it is by no mean the best one; but I do find it to be easier to manage than STARLIMS’ suggestion.

Now, I know: you might be tempted to write a generic data-driven routing. I was tempted to do it. In the end, it is a balance between convenience and security. If you let it be data-driven, you loose control on what can be routed. Someone may modify the route to, let’s say, get result, to instead return all user information, and you wouldn’t know. If it’s in the code, then you’ll know. So – although it is not as convenient, don’t get your routes handled by the database. It would also add extra load on the database. So – no good reasons other than convenience, really.

2- properly document your APIs. Heck, document your APIs before you implement them! I recommend https://swagger.io/ to generate some .yaml files. Trust me: whoever will be consuming your API will thank you!

All in all, I think the STARLIMS REST API really brings the system to an all new level. Theoretically, one could build a full UI stack using React or Angular and just consume the API to run the system on a new front end.

Or one could expose data endpoints for pipelines to maintain a data mart.

Or anything. At this point, your creativity is the limiting factor. Do you have great ideas for use cases?