New features in 6.3

Data anonymization

In this release we added an ability to anonymize sensitive data to ensure that it can be safely used, for example, when sharing your data with 3rd parties or when test data is needed. CloverDX now supports:

• A set of anonymization steps for Wrangler users
• A more powerful and complex rule-based anonymizer for data engineers using Designer and Server.

For Wrangler users, we are introducing a new step category – Anonymization. These are the steps that help with basic anonymization data:

Data engineers creating their processes in Designer can now leverage a new CloverDX Marketplace library – AnonymizationLib.

This library allows you to create shared configuration with anonymization rules and simply anonymize each data set by passing it through the Anonymization component. Anonymization rules are automatically applied based on the field names, so you don’t have to do that every time you need to anonymize a similar set of data.

Excel source in Wrangler

We’ve introduced a new built-in data source in Wrangler – a Microsoft Excel data source allowing you to work with Excel files directly, significantly simplifying the data extraction process.

Just like the CSV source, the Excel source will analyze the layout of the spreadsheet file to determine the columns in the data. The data source allows you to work with your files right away – just drop an Excel file to Wrangler and start working with your data.

Append to your output files in Wrangler

We’ve extended both built-in data targets – Excel and CSV – to allow you to append to a previously written output file. This will allow you to, for example, run a report repeatedly and append new data to existing output file, so that you do not have to merge the files manually later.

New Wrangler steps

We are continuing to expand the functionality that allows you to manipulate your data in CloverDX Wrangler. In this release we added several new steps:

• Anonymization steps: Add noise to date and Add noise to number.
• Merge columns step that allows you to combine multiple columns into one.

New column info tooltip

We improved the tooltip shown when hovering over the column header. It now shows more details about the data quality in the column and allows you to easily copy technical column name to clipboard to help you write formulas faster:

Visual update of CloverDX Server Console

We've finished unifying the visual theme of the Server Console with the modern look & feel introduced with CloverDX 6.0. This is only a cosmetic update with minor improvements and does not require re-training.

CloverDX on Java 17

We are continuing updates of CloverDX stack and with CloverDX 6.3 we’ve upgraded CloverDX Designer to use Java 17. The Designer will now install Eclipse Temurin JDK 17 by default.

The recommended deployment stack for CloverDX is therefore Apache Tomcat 9.x with Eclipse Temurin JDK 17 and CloverDX Designer with Temurin JDK 17.

Java 17 is an LTS (Long Term Support) version with Eclipse Foundation providing support at least until October 2027 compared to Java 11 with support ending in October 2024.

If you with to use CloverDX with Java 11, you can do so on the Server. CloverDX Designer will no longer run with Java 11, but it can be set to Java 11 compatibility mode if required to work with CloverDX Server deployed on Java 11.

Smaller updates

New CTL functions for data anonymization

We’ve added several variants of CTL function addNoise to help you mimic the behavior of the anonymization steps from Wrangler in your CTL code.

List archive content in ListFiles components

ListFiles component can now look inside archive files (zip, gzip, tar, tgz, tar.gzip) and list files that are compressed in the archive. The component supports full set of functionality with regards to archives on other protocols (e.g., you can list content of zip archive in an s3 bucket without having to download the archive first) as well as nested archives (zip within zip etc.).

Dynamically populate Data App enumeration parameters on App start

We’ve added the ability to dynamically populate Data App enumeration parameters with items returned by other Data Services. Previously this was only possible by using JavaScript linked to the Data App which is more complex and error prone.

User management improvements

We’ve updated the design of the user and group management modules in CloverDX Server Console. The user interface has been improved to work better with larger amounts of users and groups and to be more consistent and easier to use.

Execution properties for Wrangler workspaces

We’ve introduced a concept of Execution properties to Wrangler workspaces. You can now easily configure that your Wrangler jobs should have debugging enabled (so that you can see data on edges in Job Inspector) and you can also change the Log level.

Both settings will be useful mainly to CloverDX Server administrators and connector developers who will be able to easily see what is going on within Wrangler job in more detail compared to previous CloverDX versions.

Better integration of Data Catalog into source and target settings

Data Catalog now opens inside the data source and data target settings when you wish to select a different source or target instead of the current one. This helps you preserve the context of the job you are working on instead of switching out to a separate Data Catalog page.

HL7Reader no longer in Incubation

We’ve graduated the HL7Reader component from Incubation category in Designer. It can now be found in Readers category in the Component Palette.

At the same time, we’ve fixed composite naming when parsing HL7 files. This change may be incompatible with existing jobs that rely on previously produced composite names. Please review your jobs to ensure that they still work as expected.