copy/sync system configurations and user accounts between IRIS instances to simplify user and configurations management.

IRIS configurations and user accounts contain various data elements that need to be tracked, and many people struggle to copy or sync those system configurations and user accounts between IRIS instances. So how can this process be simplified?

In software engineering, CI/CD or CICD is the set of combined practices of continuous integration (CI) and (more often) continuous delivery or (less often) continuous deployment (CD). Can CI/CD eliminate all our struggles?

I work in a team which develops and deploys IRIS clusters. We run IRIS in containers on Red Hat OpenShift container platform.

Please do not stop reading if you are not currently using Kubernetes. You are likely to encounter similar challenges to those my team and I faced even if you are not utilizing Kubernetes or running IRIS in containers.

We have decided to separate code from configuration and place them in different GitHub repos. Every time a commit is made in a code repo, a pipeline run gets triggered. As a result, a new image from the files in the code repo is built.

We define configuration as code to be utilized in a GitOps way by adding YAML files and other configuration artifacts to a deployment GitHub repo. GitOps is a software development framework that enables organizations to continuously deliver software applications while efficiently managing IT infrastructure (and more) using Git as a single source of truth. One of the benefits of GitOps is the ability to easily roll back. All you need is to revert to prior status in Git.

DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle.[1]

I read on Wikipedia that Continuous Delivery is “When teams produce software in short cycles with high speed and frequency so that reliable software can be released at any time, and with a simple and repeatable deployment process when deciding to deploy.”

 At the same time, Wikipedia defined Continuous Deployment as “When new software functionality is rolled out completely automatically.”

We have decided to store YAML files in a deployment GitHub repo.

iris-cpf (line 19 above) refers to a ConfigMap which contains files used for the CPF Merge.

There are various available CD pipeline tools that can push deploy a configuration as code instead of having to apply files manually.

My team, for instance, uses Argo CD. It is aGitOps tool that gets deployed in clusters as a Kubernetes extension. It is special because it has visibility in the cluster. Its user interface shows the Application status in a browser because Argo CD is a Kubernetes extension.

Unlike external CD tools that only enable push-based deployments, Argo CD can pull updated (something as) code from Git repositories and deploy it directly to Kubernetes resources.

Pull deployment tools like Argo CD compare the actual state of our Kubernetes cluster with the desired one described in our deployment repo.

Argo CD watches both our deployment repo and our Kubernetes clusters. Our deployment repo is the single source of truth. If something changes in the GitHub repo, Argo CD will update the cluster to match the desired state defined in the repo.

Argo CD agent syncs GitHub repo and Kubernetes clusters. If we manually apply a change, it will get wiped out by Argo CD when it syncs the deployed application to the desired state defined in Git.

 For deploying different configurations to different clusters, we use Kustomize. We have a base configuration defined in the deployment repo. We also have overlays defined in the deployment GitHub repo to configure different system default settings and different images for various environments like development, SQA, stage, and production.

In the line 417, we determine that our environment-specific System Default Settings are stored in SDS_ENV.xml.

 What about those situations where Kubernetes is not used? I have created many applications which are available in Open Exchange. I have learned how to define IRIS configuration in GitHub repo using Installer class and then build an image and run a container.

Yet, what about cases when the configuration or user accounts require some modifications after an application has already been deployed? Things become complicated when persistent volumes are involved. It happens because we do not want to lose the data that exists on our persistent volumes.

What are all those things that are stored on persistent volumes? IRIS configuration is kept in mgr directory of the data directory. The CPF Merge feature should enable us to modify any configuration settings in iris.cpf.

My team has added a lot of code to %ZSTART routines that are executed whenever an IRIS compute or data instance is started. One concern we have is what we refer to as the zero-sized CPF error. We frequently encounter a situation when an IRIS instance crashes with a zero-sized CPF file. Unfortunately, we have not yet discovered the root cause of that issue. We suspect that CPF Merge actions and numerous Routine, Global, and Package Mappings getting added and deleted inside a %ZSTART routine contribute to the zero-sized CPF error.

We have written code to delete all System Default Settings and import them from a Kubernetes ConfigMap that gets mounted in the IRIS container as a volume. As a matter of fact, we have two sets of System Default Settings: a base one that is imported in all environments and an environment-specific one that varies from one environment to another.

We have decided to import users from an XML file. Occasionally, we ran into problems while we executed this code directly from %ZSTART routine. We moved this code into a scheduled task based on recommendation by InterSystems. Apparently, we had uncovered a bug that could wipe out security global under certain circumstances. Anyway, for some reason that I cannot recall right now, this problem is no longer an issue when the User accounts import gets executed by a task that is scheduled to run on demand from %ZSTART routine. It must have been a timing issue. The scheduled task runs later than the %ZSTART routine.

We have created a custom password validation routine to enforce password rules.

What should we do when we need a new web application? How about CSP Merge in addition to CPF Merge?

I believe Web Applications are stored in %SYS IRIS.dat file. I consider trying to define web applications inside a file that can be mounted using a ConfigMap. We could add code to %ZSTART routine or add another scheduled task to look for the file and create any web applications that do not yet exist in IRIS.

Webgateway containers deployed using InterSystems Kubernetes Operator have a persistent data volume which holds CSP.conf and CSP.ini. However, we have not yet implemented an automated way to update those files as necessary when adding a new web application.

Lorenzo Scalese created config-api and config-copy apps available in Open Exchange. He suggests using the IRS-Config-API library in your application installer module.

IRIS-Config-API can export IRIS configuration to JSON document in one place and import the IRIS configuration from JSON document in another environment.

Lorenzo created the iris-config-copy tool to export configuration from one InterSystems IRIS instance and import it into another one. If we install iris-config-copy on both the source and the target instance, the target instance uses REST to get the config from the source instance.

We need to create a web application on the source instance to enable the target instance to retrieve the IRIS configuration from the source instance.

Iris-config-copy can export IRIS configuration of local instance or remote instance.

There are methods to import specific configuration files. We can import Security, globals containing SQL Connections, CPF configuration data, or Tasks.