CloverDX 5.7

Go to Downloads (5.7.0) Latest update July 21, 2020 Documentation

CloverDX 5.7 brings a job queue to better automatically manage load on the Server. The job queue protects the Server from CPU and memory overloading. Additionaly, new user's experience of Server is improved by a guided tour of the Server Console, with the help of a new Data Warehouse example. The installation and statup of the Server is more robust with the help of database update confirmation, better behavior in cluster and better progress reporting. The variant CTL type is integrated deeper into CloverDX.

New features in 5.7

Job Queue

The new job queue handles requests for job execution, and determines when to actually start each job. It manages the overall load on the Server to prevent CPU or memory overloading. This helps the Server especially in case of high production load and in cases of spikes of data processing.

In case of high CPU load or heap memory consumption, jobs are enqueued and wait to be started later. The job queue automatically starts the jobs later when the overall load decreases.

The Server Console monitoring and logging capabilities were extended to provide information about enqueued jobs.  APIs and components were also extended. The job queue manages executions on each cluster node, and the cluster load balancer is aware of job queues of each node to better spread the load.

Job queue in Monitoring

Server Evaluation Improvements

Use experience and learning curve of new users of Server is simplified by a new guided tour of the Server Console. The tour takes the users trough major parts of the Server and explains the main concepts.

To support the guided tour there is a new example. The Data Warehouse example shows a more complex scenario that uses a wider range of Server's functionality - jobs, schedules, listeners, data services etc.

CloverDX Server guided tour

Server Installation Robustness

We have improved the installation and upgrade process of Server to prevent some common issues and simplify the whole process.

User approval before changing database schema

Upgrades to a new Server version can require updates to the Server's database schema. We have modified this process to required user's approval in the Server Console. This helps to guide the users to create backups of the Server's database and to understand that this process might take some time.

For automation purposes, it is possible to disable this manual approval.

Parallel start of cluster nodes when upgrading

When upgrading CloverDX Cluster to a new version, it is now possible to start all nodes in parallel and only one of them will perform changes to the Server's database. So now it's not necessary to manually start the nodes in sequence to prevent database issues.

Reporting of progress during startup

The Server Console shows more detailed progress information during startup of the Server. You can see the where in the startup process the Server currently is, and you can see the Server's logs in the browser.

Facilitate SAML debugging

If single sign-on using SAML is enabled, it can be complex to analyze issues in the login process - communication problems, configuration issues etc. We've improved logging of SAML integration and we provide a way to enable more detailed logging.

CloverDX Server in cloud marketplaces

The AWS and Azure Marketplace offerings with CloverDX Server were improved:

  • it is possible to enable centralized collection of Server logs in a cloud logging service 
  • security was improved by enabling disk encryption by default and implementing recommended security updates
  • it is possible to enable JMX monitoring

Variant improvements

The CTL type "variant" was more integrated into the rest of the CloverDX stack:

  • variant can be used in dictionary
  • ability to use variant in additional container functions, foreach statement and addtional utility functions

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.9 to 5.0). If you're on our maintenance program, we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.9 to 5.0) and we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Job Queue

Job Queue
CLO-19409 Improvement
Submit time should be provided by jobflow components
CLO-19112 Improvement
Performance log should contain size of Job Queue
CLO-18952 Improvement
Simple HTTP API operation "graph_status" provides new Submit time instead of Started time
CLO-18765 Improvement Compatibility
Updated cluster load balancing algorithm in compliance with Job Queue
CLO-18714 Improvement
Add 'Submit time' to RunRecord
CLO-18698 Improvement
CloverDX Server suspend operation should allow to finish enqueued jobs
CLO-19266 Improvement

Server Evaluation

Welcome page for evaluators at Server UI
CLO-19217 Improvement
Distribute data warehouse example (DWHExample) with server
CLO-19161 Improvement

Server Installation

Require user approval before applying system database patches
CLO-19125 Improvement Compatibility
Prevent corruption of system database during parallel start of multiple cluster nodes
CLO-19026 Improvement
Reporting of progress during server startup
CLO-19024 Improvement
Facilitate SAML debugging
CLO-18910 Improvement
Persist suspended state of Server and sandboxes
CLO-19031 Improvement

Cloud Marketplaces

Enable VM disk encryption
CLO-18958 Improvement
Centralized log collection in Cloud
CLO-18776 Improvement
CloudFormation stack should be finished when CloverDX Server is ready
CLO-18760 Improvement
Enable JMX for monitoring in Cloud offerings
CLO-18544 Improvement

Variant

Add variant dictionary entry type to engine
CLO-18286 Improvement Compatibility
Variant record2map and findAllValues functions
CLO-18504 Improvement
Variant foreach and in statements
CLO-18503 Improvement
Dynamic field access functions for variant
CLO-18360 Improvement
Dictionary type Variant
CLO-18286 Improvement
Additional container functions for variant
CLO-18050 Improvement
Allow automatic upcasting in CTL cast function
CLO-18916 Improvement

Additional Improvements

Ability to specify a certificate using keystore name and/or certificate alias
CLO-19009 Improvement
Server Console should allow to specify Execution Properties for Data Service jobs
CLO-19337 Improvement
Update bundled PostgreSQL JDBC driver
CLO-18860 Improvement
Support PostgreSQL 11.7 as server DB
CLO-18833 Improvement
Streaming directory listing for local files
CLO-18845 Improvement
Inherit user.timezone and locale properties from Core to Worker
CLO-18794 Improvement
Upgrade POI to 4.1.2 version
CLO-18682 Improvement Compatibility
Easier reading of optional form parameters in Data Service
CLO-18413 Improvement
GarbageManJob should not check remote job execution - performance issue
CLO-18331 Improvement
Option to enable CORS filter in CloverDX Server
CLO-18172 Improvement
DefaultReadOnly property in Worker JNDI
CLO-17808 Improvement
Return the number of affected rows from DBExecute
CLO-13531 Improvement
Worker has redundant property worker.ssl.enabled on command line
CLO-17590 Improvement

Fixes

Spreadsheet components high memory usage with in-memory mode
CLO-18639
Server on tcServer doesn't start because of scanning of spring-web*.jar
CLO-19145
Extract metadata from Hive DB fails with 'Method not supported' error
CLO-18973
JSONExtract Unable to Create Schema from Valid JSON
CLO-18753
Spring framework 5.1.2.RELEASE Security Vulnerability
CLO-18736
Nothing happens after attempting to rename externalized metadata
CLO-18414
NPE in DBExecute when trying to assign multiple IN OUT params
CLO-18236
Each data service invocation causes unneeded round trips to DB
CLO-18128
mail.smtp.starttls.enable parameter isn't working for Clover Server e-mail settings
CLO-9369
CTL: fix parsing of floating point numbers
CLO-18918 Compatibility
Broken "See Execution Properties" link in file properties in Sandboxes module on Server
CLO-18809
Change MySQL run record datetime precision to support milliseconds
CLO-17495
Accept certificate dialog does not work with certificate chains
CLO-15269
JDBC connections are opened sequentially
CLO-13377
July 21, 2020 Documentation Download CloverDX 5.7 5.7.0
  • ETL Developers (This item is most useful for developers; it either brings new functionality for transforming data or brings optimizations)
  • Administrators (This item is an improvement or feature that will help setup, install, administer and manage the application)
  • Support (This item helps staff supporting the production operation to identify potential problems or avoid such)