CloverDX 5.13

Go to Downloads (5.13.1) Latest update January 11, 2022 Documentation

In CloverDX 5.13 we are improving overall security of your deployments by integrating native support for Secret managers.  This integration allows seamless use of managed passwords in CloverDX jobs using standard parameter injection which allows secrets to be used in any context – authenticating database connections, APIs, file protocols and more. This release ships with connectivity to Azure Key Vault.

Job Inspector is also receiving significant update. It allows you to view detailed information about various items from job Outline – connections, lookup tables, sequences, and dictionary items. Other improvements include ability to view data for partitioned graphs on a cluster and many quality-of-life improvements such as syntax highlighting in code view and more.

Besides new features, CloverDX 5.13 also focuses on platform and security updates. We upgraded MySQL backend database support to support MySQL version 8.0. We are also introducing new application deployment stack built around  Open Liberty and IBM SDK 8 (including support for OpenJ9 VM).

Release also includes security fix for Log4Shell vulnerability of log4j Java library.

About this release: This is a production bugfix. We recommend upgrading to this version from all previous versions. Please check Installation/Upgrade section.

New features in 5.13

Secret manager framework

Keeping secrets secret is generally a difficult task. That’s why software like Azure Key Vault, AWS Secrets Manager or HashiCorp Vault are growing in popularity for both small and large enterprises. Having centralized password management makes it easy to automatically rotate access credentials for whole sets of APIs and systems or have the ability to revoke passwords from a single place provisioning new credentials at the same time.

CloverDX Server can now provision passwords from supported Secret managers directly into executed jobs using extended parameter notation - ${secret:<manager-name>/<secret id>} (for example, ${secret:prod/dwh-pass}). Such notation can be used anywhere where parameters can be used including directly in Connections, CTL code, Component properties, etc. 

In this release we launch with support for Azure Key Vault. Additional secret managers to follow in next release.

Job Inspector updates

This release marks major update of Job Inspector features. Job Inspector now allows you to view details of all objects from job Outline. You can display properties of your database connections, lookup tables, sequences as well as dictionary.

Server administrators can further restrict Data Inspector permissions only to specified group of users and prevent them from seeing edge data.

 

Operations

There were several improvements related to CloverDX Server maintenance. Besides new stack support discussed in separate chapter, CloverDX 5.13 comes with support of MySQL 8.0 as its backend database, ability to mask sensitive information injected from environment in Server UI and Support package. Support package can now also contain server health information (CPU/Memory utilization, Job queue status, etc.) along with advanced SAML logging and backend database communication log (both subject to additional logging configuration).

The largest change for DevOps is the ability to run CloverDX jobs on suspended server by means of manual override. This override may be provided either by a dialogue (when running from Designer or Server UI) or by special header when running job from REST API. Manual override will help you for situations where server automation features need to be suspended but still some operations have to run – for example, to simplify cluster migrations, software upgrades and code deployment.

Open Liberty platform stack

Along with RedHat, VMWare (Pivotal) stacks, CloverDX now supports Open Liberty application server on IBM SDK 8 or OpenJDK 1.8.

 

 

Smaller updates

Apache Avro graduation
CTL functions of Apache Avro message format were graduated from incubation. The functions work just like they did before, so if you use them in your solutions, you do not need to make any changes to your code.

CloverDX Designer usability
Many reported usability and development improvements were implemented. Few highlights:

  • You will now be able to get design-time hints for various runtime parameters like RUN_ID and others which are injected into jobs when they run.
  • DataIntersection component now only requires one connected port allowing you easily ignore data sets that you do not need without requiring explicit Trash component.
  • Warning and error decorations in CloverDX Designer Navigator view can now be disabled.

CloverDX Server performance improvements
Event listeners can now run parallel jobs, instead of running them in sequence for different triggers. This can improve performance if you have many listeners of the same type.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.9 to 5.0). If you're on our maintenance program, we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.9 to 5.0) and we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Security Fixes

Upgrade Log4j to 2.17.1
CLO-22512 Security
January 11, 2022 Documentation Download CloverDX 5.13 5.13.1

Secret Managers

Secret Managers: Azure Key Vault
CLO-21985 Improvement

Job Inspector Improvements

Add a permission for reading edge debug data in the Job Inspector
CLO-21942 Improvement
Outline for Job Inspector
CLO-21453 Improvement
Add more outline elements to Job Inspector
CLO-21938 Improvement
Support Data Partitioning in Data Inspector
CLO-21937 Improvement
Add job name / runId to browser tab (title)
CLO-21805 Improvement
Syntax highlight for CTL
CLO-21213 Improvement
Return RUN ID back into parameter table
CLO-21499 Improvement
Job inspector displays only the latest version of the graph
CLO-21992 Improvement
Reorder input parameters in Job Inspector
CLO-21902 Improvement
Select new execution when restarting a job from Execution History
CLO-21881 Improvement

Deployment Stacks

Add support for Open Liberty stack for Server deployment
CLO-21943 Improvement
Support MySQL 8 as a system database
CLO-21203 Improvement
Introduce support macOS 12 (Monterey)
CLO-21514 Improvement
Test Designer on Windows 11
CLO-21513 Improvement

Miscellaneous

Inform user when connection will stop working
CLO-21790 Improvement
Remove useless file encoding parameter
CLO-21767 Improvement
Add health monitor, saml and hibernate logs to support package
CLO-21601 Improvement
Remove incubation annotation for Avro CTL functions
CLO-21077 Improvement
Missing parameter value warnings in Designer are too aggressive
CLO-20996 Improvement
Allow manual execution on suspended nodes
CLO-20411 Improvement
Display error message when trying to link (shared) metadata with invalid file URL
CLO-18452 Improvement
Add option to not initialize EmailSender connection at the beginning of a phase
CLO-13967 Improvement
Remove syntax error warning dialog from transform editor
CLO-21788 Improvement
Remember size and position of common dialogs
CLO-21518 Improvement
SMTP Timeout in Milliseconds Only
CLO-20332 Improvement
Rename SalesforceWaveWriter to SalesforceEinsteinWriter
CLO-12715 Improvement
JOB_FILE, SANDBOX_CODE and SANDBOX_ROOT should be resolved for graph development
CLO-10410 Improvement
Merge forces ports 0 and 1 connected
CLO-21906 Improvement
Confusing Error Message when Creating Metadata Field Name Beginning with a Number
CLO-20778 Improvement
Dashboard/Resource tabs - optional landing pages
CLO-20627 Improvement

Security Fixes

Fix JQuery CVEs (upgrade PrimeFaces to v10)
CLO-21983 Security
Fix xmlsec CVE
CLO-22013 Security
Fix jsoup CVE
CLO-21963 Security
Update embedded Tomcat to 9.0.54
CLO-22109 Security
Mask KeyStore and TrustStore Password Values in System Info
CLO-21446 Security
Upgrade Log4j to 2.15.0
CLO-22450 Security

Fixes

Cannot login to Server with SAML Auth and reverse Proxy
CLO-21800 Fix
Broken layout of the CTL code in Job Inspector
CLO-22306 Fix
Session token expiration breaks Server projects with SSO
CLO-22129 Fix
JSONExtract NULL issue
CLO-22116 Fix
Job queue does not work correctly in containers
CLO-21980 Fix
Unable to send test email from the gui (websphere)
CLO-21974 Fix
Remove incubation annotation for Parquet components
CLO-21964 Fix
Special chars in parquet column names
CLO-21926 Fix
KafkaCommit fails when records take longer than 5 minutes
CLO-21872 Fix
Job Inspector - error message is shown in a single line
CLO-21419 Fix
macOS Server App: ExecuteScript fails with "No such file or directory"
CLO-21348 Fix
StackOverflowError when opening parameters file in CloverDX Parameters Editor
CLO-20918 Fix
Trigger health state resets after import, monitor state is incorrect
CLO-20220 Fix
DB Lookup Table with externalized connection is not properly found in Validator
CLO-19298 Fix
DataIntersection: Join Key dialog doesn't refresh key order
CLO-7652 Fix
Export graph to filesystem overwriting existing file throws an error
CLO-21659 Fix
Externalizing connection loses custom JDBC properties
CLO-21603 Fix
Job Inspector - text of long note overrides components
CLO-21491 Fix
Multiple event listeners listening on the same event run graphs sequentially
CLO-15271 Fix
When the "File URL" attribute is sent through the input port (source processing type) it doesn't work with the escaped URL
CLO-21487 Fix
Secure parameter value shows in tooltip
CLO-21466 Fix
Changing context of where the CloverDX Server runs to root context (/clover -> /) prevents Worker from starting
CLO-20317 Fix

Compatibility

Drop support for Tomcat 8.5
CLO-21865 Compatibility
Drop support for Pivotal tc Server 3.x
CLO-21858 Compatibility
December 14, 2021 Documentation Download CloverDX 5.13 5.13.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)