Customer Portal

Releases

CloverETL 4.9

Go to Downloads (4.9.1) Latest update August 14, 2018

CloverETL 4.9 is all about Server robustness. It brings a major architectural change under the hood of CloverETL Server, aimed at increased fault tolerance and stability in mission critical deployments.

With 4.9, we are splitting the Server's internal architecture into two units. Don't worry though—for users and admins the change is almost invisible, most of the new stuff happens under the hood. First, we have a "worker" runtime module that executes transformation graphs and jobflows. The rest of the functionality of the Server remains in the "core" module (Server Web Console, user management, scheduler, data services, etc.).

By isolating job execution into a separate process of the host operating system, we are making the Server resilient against potential issues in individual jobs and we allow users more convenient control over resources the worker gets (e.g. user-friendly memory settings).

This new architecture is backwards compatible with previous CloverETL versions and does not require additional modules or licenses.

New features in 4.9

Increased Fault Tolerance in Server (Server Worker)

Server's internal architecture has been split into two separate modules, each running in a separate OS process. First, a "worker" runtime, executes transformation graphs and jobflows and the second module, "a core" manages all other functions of the Server (Server Console, user management, scheduler, data services, etc.).

By isolating job execution into a separate process of the host operating system, we are making the Server resilient against potential issues in individual jobs and we allow users more convenient control over resources allocated to the worker (e.g. user-friendly memory settings).

This under-the-hood change is designed to have minimal impact on users. No changes to your projects and or package installation or licensing are required.

You will notice new "Worker" panel under Monitoring section and new configuration options under Configuration/Setup. See the updated installation instructions for more details.

This new architecture is backwards compatible with previous CloverETL versions and does not require additional modules and licenses.

Filtering in Scheduling module view

For users who manage large numbers of schedules in the Server, we're introducing filtering and paging of the Scheduling module, allowing for quick access to individual scheduled jobs.

Use of Java "direct memory" disabled by default

In the past, internal buffers in the Runtime used so called Java "direct memory". This brought us marginal performance improvements but proved to cause trouble in many different Java setups.

Direct memory (USE_DIRECT_MEMORY) is now disabled by default. However, you can switch it back on if you wish to. Refer to Engine Configuration documentation for details.

Compatibility Notes

Please pay close attention to Compatibility Notes when upgrading to 4.9 from previous production versions or Milestone releases.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • This is a milestone release. We recommend you install this version on a test Server first and use it as a preview of upcoming features and for development. Milestone releases are not supported for production use.

     

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.6 to 4.7). If you're on our maintenance program, we automatically renew the keys for you. Just go to the download area again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • New in 4.9: There is no special configuration required. We dropped the requirement to set Server URL in Milestone 1—As of Milestone 2 this is no longer needed.
  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.6 to 4.7) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Features & Improvements

JMS Event listener—Added option to define Durable subscriber name 		Feature CLO-13996
Renamed Worker JMS properties to better match Tomcat's resource properties 		Compatibility Feature CLO-13875

Fixes

Fixed StackOverflowError in a subgraph in a jobflow 		Fix CLO-14060
Fixed passing "byte" data through Dictionary when using Worker 		Fix CLO-14011
Several minor updates to SSL configuration in Worker. See the linked ticket for details. 		Fix CLO-13982

Compatibility

Disabled Groovy servlet that allowed calling Groovy scripts from a sandbox via HTTP API 		Compatibility CLO-14115
Renamed Worker JMS properties to better match Tomcat's resource properties 		Compatibility Feature CLO-13875
August 14, 2018 Download CloverETL 4.9 4.9.1

Launch Services

Launch Services are now deprecated. All users are advised to switch to Data Services instead. However, existing users with some Launch Services configured in their environments will be able to continue using them. We will advise long ahead of time when we plan to fully phase out Launch Services. 		Compatibility CLO-13391 4.9.0

Server Security

Added a new permission allowing to set API-only users (who can use Server APIs but can't log into the Server Console) 		New Feature CLO-12704 4.9.0

Runtime

Changed isDecimal() and str2decimal() and isNumber()/str2number() behavior to be consistent in handling numbers written in exponential notation. 		Compatibility Improvement CLO-8756 4.9.0
Introduced new isDecimal functions with format and locale parameters 		New feature CLO-13311 4.9.0
JNDI and SSL passwords in configuration are now encrypted using the external secure-cfg-tool, no longer as standard secure parameters 		Compatibility Improvement CLO-13846 CLO-13852 4.9.0
HTTPS protocols (-Dhttps.protocols=TLSv1.2,TLSv1.1,TLSv1) are automatically set on Worker, enabling Salesforce to work with Java 7 		Improvement CLO-13807 4.9.0

Fixes

Fixed: Retrieving execution history via API with Data Service run records doesn't work 		Fix CLO-13751 4.9.0
Fixed ParallelReader creating duplicate records when reading small files in Cluster environment 		Fix CLO-13541 4.9.0
Fixed Parameter Editor occasionally freezing Designer on MacOS 		Fix CLO-13460 4.9.0
Fixed CreateFiles failing to create files on SFTP (AWS) 		Fix CLO-13275 4.9.0
Fixed Server crashing when browsing large execution history (>100,000 entries) 		Fix CLO-13009 4.9.0
Passwords handled as secure parameters are now properly hidden in Detail tab of Execution History 		Fix CLO-12872 4.9.0
Fixed manual schedule execution producing bogus run records 		Fix CLO-12524 4.9.0

Compatibility

Updated the bundled PostgreSQL driver version to 42.2.2 		Compatibility Improvement CLO-13924 4.9.0
Upgraded internal JGroups library to version 3.6.15.Final 		Compatibility CLO-13375 4.9.0
Launch Services are now deprecated. All users are advised to switch to Data Services instead. However, existing users with some Launch Services configured in their environments will be able to continue using them. We will advise long ahead of time when we plan to fully phase out Launch Services. 		Compatibility CLO-13391 4.9.0
Changed isDecimal() and str2decimal() and isNumber()/str2number() behavior to be consistent in handling numbers written in exponential notation. 		Compatibility Improvement CLO-8756 4.9.0
JNDI and SSL passwords in configuration are now encrypted using the external secure-cfg-tool, no longer as standard secure parameters 		Compatibility Improvement CLO-13846 CLO-13852 4.9.0
JNDI resources need separate configuration for worker 		Compatibility CLO-12796 4.9.0-M1
ProfilerProbe cannot persist results in Worker 		Compatibility CLO-12796 4.9.0-M1
Dropped support for Apache Tomcat 8 		Compatibility CLO-12632 4.9.0-M1

Server Robustness - Worker

Configuring the "server.http.url" property during installation/upgrade is not needed anymore. 		Improvement CLO-13022 CLO-12837 4.9.0-M2
Performance charts now include worker stats 		Improvement CLO-11462 4.9.0-M2
Cluster has now better overview of worker status of each node 		Improvement CLO-11464 4.9.0-M2
Server is now split between two modules: a worker that handles execution of jobs and a "core" that takes care of all the other functions of the Server 		Major New Feature 4.9.0-M1

Memory Improvements

Direct memory disabled by default 		Improvement CLO-12734 4.9.0-M2

Tomcat support

CloverETL Server now supports Apache Tomcat 8.5 & Pivotal tcServer 3.2.9 		Improvement CLO-12634 CLO-12632 4.9.0-M2
Dropped support for Apache Tomcat 8 		Compatibility CLO-12632 4.9.0-M2

Performance

Increased performance when using global secure parameters 		Improvement CLO-12691 CLO-12870 4.9.0-M2
Increased performance when using global secure parameters 		Improvement CLO-11441 4.9.0-M1

Scheduling

You can now filter the Scheduling tab in Server + better performance for large numbers of schedules. 		New feature CLO-11163 4.9.0-M1

Server Robustness - Worker (changes from Milestone 2)

Server Core and Worker now internally communicate over HTTP, increasing reliability over Milestone versions which used (less reliable) JMX messages. 		Improvement CLO-13032 4.9.0
Server Core and Worker HTTP messaging can be secured with SSL 		New feature CLO-13192 4.9.0
Worker JNDI—support all parameters of Apache DBCP2 		Improvement CLO-13798 4.9.0
Worker classpath parsing supports all separators on all platforms (semicolon and colon on all platforms) 		Improvement CLO-13565 4.9.0
Renamed workerManager.* properties to worker.* properties 		Improvement CLO-13437 4.9.0
Renamed worker.portPool to worker.portRange 		Improvement CLO-13310 4.9.0
Renamed worker.additional.classpath to worker.classpath 		Improvement CLO-13398 4.9.0

Known Issues

CTL debugging does not work with Worker 		Known Issue CLO-12778 4.9.0
JMS event listener cannot pass non-serializable objects to worker 		Known Issue CLO-12552 4.9.0
July 23, 2018 Download CloverETL 4.9 4.9.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)