Customer Portal

CloverDX 6.5 is bringing numerous “quality of life” improvements across all corners of our platform. We’re also introducing an incubation version of a major new feature that will be coming in full force in the upcoming 6.6 and later releases – the Data Manager. You will be able to support business processes that involve direct manual intervention into data coming through your data pipelines, as well as use the Data Manager to store and manage your shared static (reference) data (lookup tables).

We are introducing a new more powerful mapping in the Aggregate component. The new mapping allows you to configure conditional aggregations reducing the need for coding and allowing you to build your jobs much faster.

We made several improvements to CloverDX Wrangler. These include the ability to automatically perform data type conversions in your output mapping, ability to preview data sent to your data target and improvements in the Delete columns step.

We are also introducing a new CloverDX Home app which simplifies the navigation in CloverDX platform.

We’ve updated the visual design of Data Apps. New Data Apps are easier to navigate and use and align better with the overall visual design of CloverDX.

With this release we improved the auto-configuration of CloverDX to make it easier to build cluster of CloverDX Servers. This feature can be effectively used to build CloverDX instances in Kubernetes or in other infrastructure as code platforms. We’ve published two new examples which show you how to run a single CloverDX Server instance of a cluster of multiple CloverDX Servers in Kubernetes.

And as with every release, we made many smaller changes that will help you when using CloverDX to manage your data. This includes new CTL functions, better Component Palette in Designer, LookupTableReaderWriter improvements and more.

About this release: This is a production release. We recommend upgrading to this version from all previous versions. Please check Installation/Upgrade section.

New features in 6.5

More powerful Aggregate component

Our improved Aggregation mapping allows you to use Aggregator to conditionally aggregate your data. This will allow you to use Aggregator component in situations that would have required usage of Denormalizer component in previous versions of CloverDX. This in turn reduces the need for coding and allows even less experienced users to create more complex aggregations without overcomplicating their graphs.

The updated component also allows you to create metadata as you go along so that you no longer need metadata defined before you start working on your aggregation. This helps you build your mappings more effectively without having to go back and forth between multiple different dialogs.

Wrangler improvements

Automatic type conversions in mapping

Mapping to target in your Wrangler jobs will now automatically apply conversions between some data types. Automatic conversions are only applied in cases where the conversion is safe and no data is lost. This includes conversions of various data types to string or conversion from integer to decimal.

Preview of data sent to target

You can now click on the data target "step" in your Wrangler jobs in the Step list panel to see the data exactly as it is sent to the target – including the mapping if any was present.

This allows you to better see and understand what kind of data your target is getting, which is especially nice when automatic data conversions are applied as described above.

Delete columns step improvements

Delete columns step has an additional parameter – Mode of operation – which allows you to select whether to keep or remove the selected columns. The “keep selected” mode that was added in CloverDX 6.5 is especially useful in cases where you have large number of input fields and only need to retain small number of them – instead of selecting each of the many fields to remove, you can select the fields you wish to keep.

Improved platform navigation

With the addition of the Data Manager (see below), CloverDX now includes four apps that help non-technical users to work with their data. To help you navigate the platform, we are introducing CloverDX Home – a new page that allows you to see which apps you can use. The new Home page is available via https://your.server.com/clover/home.

To help you navigate in various apps, we added a new App menu which allows you to quickly jump to other apps in CloverDX platform.

Visual Refresh of Data Apps

We’ve improved the design and usability of Data Apps. Data Apps now have more modern look that fits better into CloverDX platform.

Kubernetes deployments

We are introducing new examples that show how you can run CloverDX Server or a cluster of CloverDX Server instances in Kubernetes. The deployment examples have been configured with reasonable defaults and can be used directly or as templates for your own deployments.

To get the examples, please visit our GitHub repository.

Incubation: Data Manager

We are introducing a preview of CloverDX Data Manager – a new part of CloverDX platform that helps users to collaboratively improve data quality in CloverDX platform.

Data Manager implements a workflow that allows technical users to submit data for review to domain experts. Domain experts can use Data Manager web app to review, edit and approve changes in the data. Once that is done, the approved records can be picked up by CloverDX graphs and further processed before being sent to downstream systems.

This collaborative workflow allows each user to contribute in their area of expertise – whether it is technical users who build the “backend” processes to ingest data and feed it to Data Manager or non-technical users who know the data very well and can manually fix the issues that cannot be resolved automatically.

Data Manager is an Incubation feature. If you’d like to try it, please contact your CloverDX account manager or [email protected].

Smaller changes

New Component Palette in Designer

We improved Component Palette in CloverDX Designer. New Palette works much better with a larger number of components and categories. This is especially helpful for users who have a lot of libraries deployed in their CloverDX Server.

LookupTableReaderWriter component propagates metadata

When working with lookups, the LookupTableReaderWriter component will now automatically propagate metadata corresponding to the lookup table on its input and output ports. This will help you build your jobs more effectively and reduce issues coming from misaligned metadata between your graphs and lookups.

Improved export/import of sandbox configuration

The export of sandbox configuration will now export name of the sandbox as well similarly like is done for “global” configuration export. This will allow the sandbox-specific configuration to be imported on another Server easily as it will allow the receiving Server to create a copy of the sandbox right aways rather than requiring the sandbox to be present before importing the configuration.

Simplified cluster configuration

We’ve improved the configuration management for CloverDX Cluster. It is no longer necessary to provide a complete jGroups configuration on each node. Nodes in the cluster will apply reasonable default settings and share them across all nodes.

This is especially useful for initial_hosts property which will now be configured automatically by the Server allowing you to easily automate the deployment since it is no longer necessary to know the host names in advance.

It is still possible to fine-tune the configuration if needed – for example to change ports or other settings.

New CTL functions

We’ve added four new CTL functions in CloverDX 6.5:

  • getDayOfWeek
  • validateCreditCard
  • validateEmail
  • validatePhoneNumber

The last three functions replicate the functionality of Wrangler validation steps allowing you to get the same powerful data validation in your CTL code without having to build a Wrangler job.

New REST API to query Server users

New REST API endpoint /users will allow you to list user accounts configured on the Server.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.9 to 5.0). If you're on our maintenance program, we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.9 to 5.0) and we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Wrangler Improvements

Empty screen for step list
CLO-25808 Improvement
Data target preview
CLO-26986 Improvement
Excel target custom number formats
CLO-28524 Improvement
Delete columns step: add ability to "keep selected columns"
CLO-28916 Improvement
Automatic type conversions in output mapping in Wrangler
CLO-28918 Improvement

Kubernetes

New Kubernetes examples
CLO-29451 Improvement

Aggregate Component

Aggregate component improvements - functionality and GUI
CLO-29283 Improvement

API Improvements

Improve server configuration import/export APIs for sandbox configuration
CLO-28915 Improvement
REST API for users
CLO-29355 Improvement

Miscellaneous

Simplify jgroups configuration
CLO-29427 Improvement
SAML does not redirect to the original URL
CLO-29292 Improvement
New CTL functions
CLO-29281 Improvement
Improve format for edge memory size logs
CLO-29250 Improvement
Better location for runtime log file
CLO-29107 Improvement
Improve usability of the graph editor palette
CLO-29096 Improvement
toJson() function for JSONReader
CLO-29000 Improvement
Add access to other applications from the Server
CLO-28731 Improvement
CloverDX Server Deployment via Azure Marketplace with Custom Subnet Causes Tomcat Startup Failure
CLO-28390 Improvement
DBExecute: Propagate SQL statement to the error port
CLO-26891 Improvement
Single/multiple field(s) parameter editor types are missing variant options in the required type drop-down
CLO-25160 Improvement
Showing relevant information in monitoring dials
CLO-15639 Improvement

Security

Show the content cached by secret managers
CLO-25328 Security
Upgrade postgresql jdbc driver to 42.7.3
CLO-29007 Security
CVE FIX - upgrade spring-web to 5.3.34
CLO-29276 Security
CVE FIX - Upgrade Tomcat to v9.0.88
CLO-29145 Security
CVE FIX - upgrade cxf-core
CLO-29022 Security
CVE FIX - xmlsec
CLO-28952 Security
CVE FIX - commons-fileupload
CLO-28951 Security
CVE FIX - Upgrade ion-java
CLO-28937 Security
CVE FIX - Upgrade jakarta.el
CLO-28936 Security
CVE FIX - spring-web
CLO-28815 Security
CVE FIX - upgrade Apache commons-compress
CLO-28780 Security
CVE FIX - nimbus-jose-jwt
CLO-28741 Security

Fixes

Add check for maximum target name length
CLO-27744 Fix
Problem with libraries on server with proxy
CLO-28419 Fix
Using sftp URL in Output file URL property of HTTPConnector saves the file in SFTP but throws an error
CLO-28886 Fix
jGroups properties should be trimmed
CLO-28889 Fix
REST API support package download inconsistent behavior on non-cluster Server instances
CLO-28957 Fix
CORS Filter Does Not Work Properly for Data Services on HTTPS Connectors
CLO-29117 Fix
CTL editor linter stuck / false positive
CLO-29138 Fix
User group permission upgrade - better message when not possible to upgrade
CLO-29173 Fix
DB schema dialog - filter bugs
CLO-29342 Fix
Components password in plaintext in Execution History/Job Inspector
CLO-29353 Fix
The user group cannot be updated because it does not exist
CLO-29489 Fix

Compatibility

LookupTableReaderWriter should propagate metadata
CLO-6660 Compatibility
Remove Server JMX MBean API
CLO-18673 Compatibility
Drop RPM packages for Linux
CLO-28867 Compatibility
Unified interface for business users
CLO-28699 Compatibility
HTTPConnector - use sandbox url instead of local url
CLO-29142 Compatibility
Change HTTPConnector timeout default value
CLO-29217 Compatibility
June 25, 2024 Documentation Download CloverDX 6.5 6.5.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)