Customer Portal

In CloverDX 6.3 we are introducing anonymization functionality to help our users deal with sensitive data in their jobs. We are also adding an ability to work with Excel files directly in Wrangler and many other improvements across the board.

The new data anonymization functionality will allow users to anonymize sensitive data in their jobs regardless of how they use CloverDX. For Wrangler users, we are introducing new anonymization steps while the engineers using Designer can take advantage of the new Anonymization library as well as new built-in CTL functions to build more complex anonymization processes.

We are also introducing a new Excel data source for Wrangler. This source will allow you to simply drop an Excel file to Wrangler and start working with your data right away. At the same time, our built-in data targets for CSV and Excel are now able to append to existing files, making reconciliation reports and continuous data update scenarios much easier to accomplish directly in CloverDX Wrangler.

We’ve updated CloverDX Designer to be compatible with Java 17. With this upgrade, we recommend you use Java 17 for all CloverDX products for best performance and security.

About this release: This is a production bugfix. We recommend upgrading to this version from all previous versions. Please check Installation/Upgrade section.

New features in 6.3

Data anonymization

In this release we added an ability to anonymize sensitive data to ensure that it can be safely used, for example, when sharing your data with 3rd parties or when test data is needed. CloverDX now supports:

  • A set of anonymization steps for Wrangler users
  • A more powerful and complex rule-based anonymizer for data engineers using Designer and Server.

For Wrangler users, we are introducing a new step category – Anonymization. These are the steps that help with basic anonymization data:

Anonymization steps in Wrangler

Data engineers creating their processes in Designer can now leverage a new CloverDX Marketplace library – AnonymizationLib.

Usage of AnonymizationLib in CloverDX graph.

This library allows you to create shared configuration with anonymization rules and simply anonymize each data set by passing it through the Anonymization component. Anonymization rules are automatically applied based on the field names, so you don’t have to do that every time you need to anonymize a similar set of data.

Excel source in Wrangler

We’ve introduced a new built-in data source in Wrangler – a Microsoft Excel data source allowing you to work with Excel files directly, significantly simplifying the data extraction process.

Just like the CSV source, the Excel source will analyze the layout of the spreadsheet file to determine the columns in the data. The data source allows you to work with your files right away – just drop an Excel file to Wrangler and start working with your data.

Append to your output files in Wrangler

We’ve extended both built-in data targets – Excel and CSV – to allow you to append to a previously written output file. This will allow you to, for example, run a report repeatedly and append new data to existing output file, so that you do not have to merge the files manually later.

New Wrangler steps

We are continuing to expand the functionality that allows you to manipulate your data in CloverDX Wrangler. In this release we added several new steps:

New column info tooltip

We improved the tooltip shown when hovering over the column header. It now shows more details about the data quality in the column and allows you to easily copy technical column name to clipboard to help you write formulas faster:

New column header tooltip with Data Quality bar.

Visual update of CloverDX Server Console

We've finished unifying the visual theme of the Server Console with the modern look & feel introduced with CloverDX 6.0. This is only a cosmetic update with minor improvements and does not require re-training.

New design of CloverDX Server Console.

CloverDX on Java 17

We are continuing updates of CloverDX stack and with CloverDX 6.3 we’ve upgraded CloverDX Designer to use Java 17. The Designer will now install Eclipse Temurin JDK 17 by default.

The recommended deployment stack for CloverDX is therefore Apache Tomcat 9.x with Eclipse Temurin JDK 17 and CloverDX Designer with Temurin JDK 17.

Java 17 is an LTS (Long Term Support) version with Eclipse Foundation providing support at least until October 2027 compared to Java 11 with support ending in October 2024.

If you with to use CloverDX with Java 11, you can do so on the Server. CloverDX Designer will no longer run with Java 11, but it can be set to Java 11 compatibility mode if required to work with CloverDX Server deployed on Java 11.

Smaller updates

New CTL functions for data anonymization

We’ve added several variants of CTL function addNoise to help you mimic the behavior of the anonymization steps from Wrangler in your CTL code.

List archive content in ListFiles components

ListFiles component can now look inside archive files (zip, gzip, tar, tgz, tar.gzip) and list files that are compressed in the archive. The component supports full set of functionality with regards to archives on other protocols (e.g., you can list content of zip archive in an s3 bucket without having to download the archive first) as well as nested archives (zip within zip etc.).


Dynamically populate Data App enumeration parameters on App start

We’ve added the ability to dynamically populate Data App enumeration parameters with items returned by other Data Services. Previously this was only possible by using JavaScript linked to the Data App which is more complex and error prone.


User management improvements

We’ve updated the design of the user and group management modules in CloverDX Server Console. The user interface has been improved to work better with larger amounts of users and groups and to be more consistent and easier to use.

Execution properties for Wrangler workspaces

We’ve introduced a concept of Execution properties to Wrangler workspaces. You can now easily configure that your Wrangler jobs should have debugging enabled (so that you can see data on edges in Job Inspector) and you can also change the Log level.

Both settings will be useful mainly to CloverDX Server administrators and connector developers who will be able to easily see what is going on within Wrangler job in more detail compared to previous CloverDX versions.

Better integration of Data Catalog into source and target settings

Data Catalog now opens inside the data source and data target settings when you wish to select a different source or target instead of the current one. This helps you preserve the context of the job you are working on instead of switching out to a separate Data Catalog page.

HL7Reader no longer in Incubation

We’ve graduated the HL7Reader component from Incubation category in Designer. It can now be found in Readers category in the Component Palette.

At the same time, we’ve fixed composite naming when parsing HL7 files. This change may be incompatible with existing jobs that rely on previously produced composite names. Please review your jobs to ensure that they still work as expected.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.9 to 5.0). If you're on our maintenance program, we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.9 to 5.0) and we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Excel Data Source Improvements

Excel source - Improve detection of formula columns
CLO-28300 Improvement
Handle more custom excel number formats
CLO-28280 Improvement

Wrangler UI Improvements

Tooltip for "Use in a new job"
CLO-27177 Improvement
Keep scroll position in list of transformation steps
CLO-26667 Improvement
Wrangler - rows read/written - format improvement
CLO-28239 Improvement

Fixes

XMLExtract is very slow when the graph runs in Server project
CLO-28525 Fix
Missing hover state for import d&d panel
CLO-28295 Fix
Drag'n'drop import of XML broken on Safari
CLO-28292 Fix
Support enumeration parameter with labels in server GUI
CLO-25355 Fix
Server - Sandbox zip - "Upload in progress" window appearance
CLO-28248 Fix
Excel source - negative zeros
CLO-28507 Fix
Wrangler login page - fix styling of error/info messages
CLO-26259 Fix
January 23, 2024 Documentation Download CloverDX 6.3 6.3.1

Reading Excel in Wrangler

Excel Data Source for Wrangler
CLO-27591 Improvement

Anonymization

New CTL functions for anonymization
CLO-27931 Improvement
New step: Add noise to number
CLO-27864 Improvement
New step: Add noise to date
CLO-27912 Improvement
Move Mask text step to Anonymization category
CLO-27913 Improvement

Wrangler Improvements

New step: Merge columns step
CLO-26290 Improvement
Allow user to create new target if the existing target is invalid
CLO-26302 Improvement
Edit action for My Jobs table
CLO-27001 Improvement
Deleting of sources/targets when deleting job
CLO-27040 Improvement
Unify Source and Target detail tabs in catalog
CLO-27175 Improvement
Job editor - No source selected screen
CLO-27178 Improvement
Copy column name for formulas
CLO-27265 Improvement
Add new item to the Navigation diagram in job
CLO-27323 Improvement
Hints / examples for steps with regex property
CLO-27439 Improvement
Make the error icon in Step panel interactive
CLO-27528 Improvement
Append for CSV data target
CLO-25220 Improvement
Append for Excel data target
CLO-27585 Improvement
Add server identification to note in Wrangler mapping subgraph
CLO-27602 Improvement
Integrate Data Catalog into source / target selection in job
CLO-27655 Improvement
Make vertical content scrollable
CLO-27657 Improvement
Cleanup mapping graphs
CLO-27754 Improvement
Move Delete, Rename and Duplicate steps to the top of the Context menu
CLO-27662 Improvement
Check metadata when appending in wrangler
CLO-27833 Improvement
Perform check config when user changes column formatting
CLO-28014 Improvement
Target should indicate that it is configured to append
CLO-28163 Improvement

Listing Archives

Add support for listing archives to ListFiles
CLO-24624 Improvement

Server UI Redesign

Redesign Server UI
CLO-27961 Improvement

User and Groups Management

Server user/group filter is case-sensitive
CLO-27104 Improvement
Mark problematic users
CLO-27107 Improvement
Improvements in user/group management
CLO-27108 Improvement
Enhance component for group/user assignment
CLO-27112 Improvement

Java 17

Designer on Java 17
CLO-25145 Improvement
Update java of cloud marketplace offers to 17
CLO-26691 Improvement
Update Server RPM packages to Java 17
CLO-27694 Improvement

Miscellaneous

Optimize CTL switch statement
CLO-18154 Improvement
Azure marketplace - use Standard IP SKU
CLO-25492 Improvement
Health check - add info about the last run
CLO-27194 Improvement
Replace com.jcraft:jsch by com.github.mwiede:jsch
CLO-27294 Improvement
Populate DataApp enum parameter
CLO-27589 Improvement
Move HL7Reader component out of incubation
CLO-27609 Improvement
Execution properties for Wrangler workspaces: debug and logging
CLO-27914 Improvement
Show information about the non-production environment
CLO-27939 Improvement
Delete usage statistics telemetry
CLO-27940 Improvement
Move "Add new monitor" button in Ops Dashboard to the toolbar
CLO-27597 Improvement
Fix http encoding in OAuth2 authorization
CLO-27369 Improvement
Libraries: Add indication that there is an issue with libraries
CLO-26435 Improvement

Fixes

getFieldProperties doesn't return all values properly
CLO-7580 Fix
JMS Listener fails while reading message properties
CLO-20284 Fix
JMS Listener batch mode does not work with RabbitMQ
CLO-20287 Fix
Log level - setting on certain jobs get overriden by sandbox or root configuration
CLO-27138 Fix
Positions of scrollbar error indicators are not recomputed
CLO-27301 Fix
Connecting edge to some components logs error
CLO-27558 Fix
Incorrect naming of parsed HL7 composites
CLO-27608 Fix
The same target is not used for the duplicated job as for the original
CLO-27711 Fix
Designer+Server installer - uninstallation can wipe all shortcuts from current user's start menu
CLO-28020 Fix
Cannot delete source/target from detail header
CLO-28097 Fix
Azure CLI authentication failing with Azure Key Vault (secret manager) in Server v6.2.1
CLO-28177 Fix

Security

Limit http methods accepted by CloverDX Server APIs
CLO-26701 Security
Upgrade PrimeFaces to v13.0.1
CLO-27437 Security
CVE FIX - Upgrade postgresql to 42.6.0
CLO-25039 Security
CVE FIX - Upgrade woodstox to 6.4.0
CLO-25601 Security
CVE FIX - Upgrade kafka-clients to 3.5.1
CLO-27471 Security
CVE FIX - Upgrade snappy-java to 1.1.10.5
CLO-27473 Security
CVE FIX - Upgrade embedded Tomcat to v9.0.82
CLO-27716 Security

Compatibility

getFieldProperties doesn't return all values properly
CLO-7580 Compatibility
Designer on Java 17
CLO-25145 Compatibility
Limit http methods accepted by CloverDX Server APIs
CLO-26701 Compatibility
Replace com.jcraft:jsch by com.github.mwiede:jsch
CLO-27294 Compatibility
Incorrect naming of parsed HL7 composites
CLO-27608 Compatibility
December 12, 2023 Documentation Download CloverDX 6.3 6.3.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)