New features in 7.1

Machine learning and AI in CloverDX

We are introducing a set of four new components which will help you work with machine learning models – AITextClassifier, AITokenClassifier, AIZeroShotClassifier and AIAnonymizer.

The first three components allow you to run various data classification machine learning models locally on your own hardware – on CloverDX Server as well as in Designer. Each runs a different type of machine learning model to help you implement different kinds of use cases.

To use these components, you can either download machine learning models from our CloverDX Marketplace and install them on your CloverDX Server / Designer or you can even use your own models after converting them to TorchScript format.

When using these components on your Server, you can use a new AI / ML module in Server console to see what models are available in your environment. You can install new ones in the Libraries module by grabbing them from our Marketplace or by providing your own libraries with AI models.

The last component – AIAnonymizer – is a combination of token classifier (same type of model you can run in AITokenClassifier component) and data masking algorithm that uses the model output to mask data flagged by the model. This can be very effectively used for e.g. personal data redaction.

Note that these components are all considered Incubation and their interface may change in future versions of CloverDX.

OpenAIClient component

The new OpenAIClient component allows you to use large language models published by OpenAI in your graphs.

The component connects to OpenAI API and allows you to send your data together with your custom prompt to the API. It will then process the response from the LLM giving you full control over the flow of the “conversation” with the API. This can be used to implement various strategies for response processing and error handling (e.g., asking LLM to regenerate response if the response does not match the expected format and more).

This component is currently considered an Incubation feature.

New AI image on DockerHub

To make it easier for you to work with accelerated machine learning models we are publishing a new Docker image on DockerHub – CloverDX AI image.

This image is preconfigured to allow you to run your machine learning inference via CloverDX components with GPU acceleration providing for significantly higher performance compared to CPU-only execution.

The image can be deployed on any machine where all software and hardware requirements are met. The machine requires NVIDIA GPU with properly configured drivers, CUDA, container toolkit and more.

The easiest way to use this image is to run it on AWS in AWS Deep Learning Base GPU AMI (Ubuntu 24.04) which runs on AWS EC2 G4dn or AWS EC2 G5 instances.

Machine learning models in CloverDX libraries and Marketplace

To use the new AI components, you must install machine learning models in your CloverDX. To make this process as simple as possible, we’ve extended CloverDX libraries to also allow you to store and publish machine learning models.

Several new libraries with machine learning models are available in CloverDX Marketplace.

These libraries can all be installed via Libraries module in CloverDX Server. Once installed, the models are automatically enabled on the Server. You can then view their properties in the new AI / ML module.

To use the models you’ve installed, you can then simply select which model to use in your AI component in Designer.

Sharing Data Manager across multiple environments

Since the introduction of the Data Manager in CloverDX 7.0 you can easily create and manage data sets within a single CloverDX environment.

With CloverDX 7.1 we are introducing the ability for multiple CloverDX Servers (or clusters) to share a single Data Manager instance. This allows you to centralize all your data sets in a single location and use them across multiple CloverDX environments.

This approach offers multiple benefits like easier data governance as well as more predictable performance since you can pool all your interactive users of CloverDX Business Tools on a single machine and leave your automation running on different instances. This can then ensure that automated processes that may adhere to strict SLAs are not influenced by users working in Wrangler or in Data Manager.

Working with reference data in Wrangler

You can now easily work with reference data sets in your Wrangler jobs – whether you need to access your reference data as lookup or insert data into your reference data set.

All reference data sets now appear in the Data Catalog. Each data set can be accessed with two different connectors – one source and one target.

Reference data set sources allow you to use reference data sets whenever you need to read from it – typically when the data set is used as a lookup table in Wrangler job. This way you can very easily use shared reference data instead of having to define your own lookups via files uploaded to your Wrangler workspace.

Similarly, if you’d like to update your reference data set, you can use the target to do this. The target will add or update the records in the data set. Interactive users will then have to approve the changes in the Data Manager. This approach guarantees that the changes are not applied silently and that your data owners still retain full control over what is published in reference data sets.

Additional improvements

Descriptions for Listeners and Schedulers

We’ve added the ability to add descriptions to your Event Listeners and Schedulers. This will help you navigate more complex environments where you may have dozens or even hundreds of Listeners or Scheduler entries. Search in both modules searches within the description field so you can find what you are looking for faster.

At the same time, we are also providing more information about when the given entity was created or updated and who made the change.

Public REST API for Data Manager

We’ve introduced a read-only version of the Data Manager’s public API. IN this version, you can use the API to query information about your transactional and reference data sets as well as query the data in the data sets.

More complete API with the ability to write or modify data sets will be coming in CloverDX 7.2.

Platform support

DB2 no longer supported as Server system database

Starting with this release, IBM DB2 is no longer supported as the database that is used by the Server as its system / backend database. If you are using DB2, you will have to migrate to a different database – PostgreSQL, MySQL, Microsoft SQL Server, or Oracle. Consult our System requirements section in the documentation for more details.

Please note that this does not affect the ability to connect to DB2 databases from your graphs – this remains without any change.