CloverDX 5.10

Go to Downloads (5.10.1) Latest update May 11, 2021 Documentation

In this release we focused on connectivity improvements of CloverDX data platform. We added native connectivity to Snowflake, Parquet and implemented Kafka Message Listener to empower Kafka capabilities introduced in previous release. Further development of REST APIs included migration of Simple HTTP APIs capability into more up-to-date and consumable format, allowing Server to be more efficiently embedded in broader application landscape. Simple HTTP API will stay in the product but is now considered to be in maintenance mode and will be deprecated in some future release.

Improvements for Operations Dashboard aim for quick identification and subsequent fixes for business-critical process failures using visuals aids.

In continued effort to improve security and harden the software, we fixed 9 critical security issues found. These fixes will be backported to all currently supported versions.

Supporting new formats, known for being engineered to support structured data, 5.10 marks a steppingstone for future product focus on broader native use of hierarchical data.

New features in 5.10

Native Snowflake connections

CloverDX is now a Snowflake technology partner, which allowed us to bundle official driver for family of existing database components and implement efficient, native connectivity to Snowflake. New component SnowflakeBulkWriter was created to support highly efficient inserts (you can find this component in Writers category in Designer palette). On average, new SnowflakeBulkWriter is 8x faster for same data volume compared to standard DatabaseWriter’s INSERT statement.

Apache Parquet format support

To support data analytic use cases in Hadoop ecosystem, CloverDX 5.10 ships with two new components, ParquetReader and ParquetWriter. These components start in Incubation category where they will stay for approximately 2 releases. These components contain basic set of features which are going to be extended over incubation period.

ParquetReader: with performance in mind, developers can choose which columns to read – improving speed by minimizing unnecessary I/O operations. Works with file’s schema document, supporting metadata propagation.

ParquetWriter: using configurable mapping, serializes data streams into a Parquet file to any data target supported by other file-based components (including direct writes to HDFS and S3).

Alert level in Operations Dashboards

Latest improvement to the Dashboard, originally introduced in CloverDX 5.8.0, allows users to assign a severity level to each monitored resource. Higher level will manifest with more visible highlight and will be shown before alerts with lower configured severity.

Alert level helps support to identify and resolve critical issues in a timely manner.

We also made it easier to manage monitors on the Dashboard since it is now possible to add triggers to monitors directly from Alerts and Notification tab for each trigger and Data Service.

Kafka Message Listener

With latest release, collection of event listeners grown for Kafka Message Listener, capable detecting new arrivals into Kafka message queues. It allows to define both, specific topic(s) and topics by use of regular expressions to match them by common pattern.

Configuration which used to detect changes is subsequently passed to triggered task to do the processing.

Server management REST APIs

We created a new REST API to help us address shortcomings of existing Simple HTTP API. Starting with this release, we ported most used methods from Simple HTTP API. New API is easier to use and more powerful and will be further improved in future releases.

All methods come with complete Swagger documentation with test interface and examples. It is accessible from the Server’s login page via Server REST API Catalog link.

Simple HTTP API is still supported and will not be removed anytime soon. It is now however in maintenance mode and will not be actively developed.

 

Product features removed

  • Launch Services: Launch Services were hidden to most CloverDX Server users since they were replaced by Data Services and deprecated in CloverETL 4.9. Note that upgrading to CloverDX 5.10 from previous versions will remove all your Launch Service configuration from the database but will not affect any graphs, jobflows or any other logic in your sandboxes. We strongly recommend any Launch Services in use to be reimplemented as Data Services prior upgrade and set up HTTP proxy to adjust for new URLs when required. Make sure database is backed up before upgrade.
  • RSE projects: RSE projects were used to connect CloverDX Designer to the Server. They were deprecated since CloverETL 4.2 and were replaced by “sync projects”. Starting with CloverDX 5.10, it will not longer be possible to open RSE projects or create new ones. To access your data in existing RSE project, simply create new Server Project from Designer and point it to appropriate sandbox. You will not lose any project data.
  • XLSDataReader and XLSDataWriter: we removed very old and deprecated components. Both were replaced by more powerful and faster components SpreadsheetDataReader and SpreadhseetDataWriter in CloverETL 3.0.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.9 to 5.0). If you're on our maintenance program, we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.9 to 5.0) and we automatically renew the keys for you. Just go to the License Keys again and copy/paste the license key from there.

 

Release Compatibility/Upgrade notes, Features & Fixes Published Download

Fixes

Backport - FileEventListeners stop to perform file system check
CLO-20955 Fix
Cannot install plugins into Designer
CLO-20951 Fix
Error when Installing IBM MDM Plugin in Designer
CLO-20956 Fix
Quickbase Import CSV Fails to Load Pound Symbol
CLO-20926 Fix
OpenSSH config file is being ignored in release 5.10
CLO-21023 Fix
CloudFormation templates for AWS offer contain incorrect supported instance types
CLO-20994 Fix
Kafka listener does not work in cluster
CLO-20958 Fix
Parquet Schema dialog - NPE when Parquet Decimal is mapped on a not-decimal field
CLO-20954 Fix
Unable to insert user password with length greater than 32
CLO-21042 Fix
May 11, 2021 Documentation Download CloverDX 5.10 5.10.1

Snowflake

Add SnowflakeBulkWriter component
CLO-20261 Improvement
Snowflake JDBC Specific
CLO-20261 Improvement

Parquet Components

ParquetWriter component
CLO-20614 Improvement
ParquetReader component
CLO-20613 Improvement

Kafka

Kafka event listener
CLO-20471 Improvement
Support headers in KafkaWriter
CLO-20462 Improvement
KafkaCommit should check phase number
CLO-20324 Improvement
KafkaReader - Assigned topic partition does not work for multiple topics
CLO-20291 Improvement
KafkaCommit component should provide build-in metadata for input port
CLO-20288 Improvement
Allow to validate Kafka connection in edit dialog
CLO-20066 Improvement
Upgrade Kafka library in clover from 2.6.0 to 2.7.0
CLO-20601

REST API

REST API - sandbox create and delete
CLO-20104 Improvement
REST API - suspend & resume of sandbox
CLO-20103 Improvement
REST API - status and suspend/resume of server
CLO-20102 Improvement
REST API - configuration import & export
CLO-20100 Improvement
REST API - new execution (run graph)
CLO-20099 Improvement
REST API - list of executions
CLO-20097 Improvement

Operations Dashboard

Alert level in monitoring
CLO-19983 Improvement
Permissions for Operations Dashboard
CLO-20348 Improvement
Show warnings in the list of dashboards
CLO-20300 Improvement

Platform Updates & Security

Update JSF to version 2.2.16
CLO-20786 Security
Remove the XSS vulnerability from the error page
CLO-20814 Security
Update jsch from 0.1.54 to 0.1.55
CLO-20378 Security
Update embedded tomcat to 9.0.41 and 8.5.61
CLO-20360 Security
Update quartz to 2.3.2
CLO-20359 Security
Update jackson-databind to 2.12.1
CLO-20356 Security
Update dom4j to 2.1.3
CLO-20355 Security
Update derby to 10.14.2.0
CLO-20354 Security
Update Eclipse to 2020-06
CLO-20499
Update JDBC driver for MS SQL Server
CLO-5344

Improvements

Synchronization on component thread pool
CLO-20586 Improvement
Enable file operations outside of Jobflows
CLO-20446 Improvement
WebServiceClient SSL checkconfig
CLO-20108 Improvement
Change layout of the Login page on the Server
CLO-20477 Improvement
JobQueue log is not accurate enough for further analysis
CLO-20473 Improvement
Track thread allocations at runtime
CLO-20469 Improvement
Detect unsupported RSE projects and help the user
CLO-20362 Improvement
SequenceChecker: remove unnecessary serialization
CLO-6309 Improvement
Merge: remove unnecessary serialization
CLO-3325 Improvement
Meaning of Incubation category should be explained directly in Designer
CLO-20322 Improvement
Simple HTTP API sandbox_create with createDirs=false fails with error
CLO-19154 Improvement
Change default order of authentization options for SFTP connection
CLO-18831 Improvement
Add option to disable Kerberos authentication on SFTP connections
CLO-15906 Improvement
Add syntax highlighting to DBExecute
CLO-20584 Improvement

Fixes

Executions API does not return 403 Forbidden, but 200 / 400 / 404
CLO-20576 Fix
Suspend happens twice
CLO-20545 Fix
Cannot open CTL in tab
CLO-20513 Fix
REST API - sandboxes with space in sandboxCode
CLO-20500 Fix
DataServices - upload file in multipart - files stay in tmp dir
CLO-20447 Fix
KafkaCommit with no input port must be higher phase than KafkaReader
CLO-20413 Fix
Azure data disk mounted incorrectly
CLO-20386 Fix
KafkaCommit can fail on partitions rebalance
CLO-20337 Fix
Charset dialog does not provide "no value" item
CLO-20314 Fix
Hanging SFTP write cannot be aborted
CLO-20223 Fix
Invalid connection attempts to MongoDB on localhost
CLO-19952 Fix
Data Apps fail with HTTP 500 if path to branding zip contains trailing space
CLO-19438 Fix
JSONReader: there is no warning shown when field names contain underscores
CLO-18929 Fix
OracleDataWriter Inevitable Warning
CLO-13025 Fix
NPE when closing graph before checkConfig finishes
CLO-20590 Fix
NPE in DB connection dialog
CLO-20292 Fix
Charset selector doesn't pre-select selected charset
CLO-20207 Fix

Compatibility

Remove support for Launch Services in CloverDX 5.10
CLO-20307 Compatibility
Remove deprecated components: XLSDataReader and XLSDataWriter
CLO-20309 Compatibility
Remove XLS methods from ServerFacade
CLO-20785 Compatibility
Drop support for RSE projects in CloverDX Designer
CLO-20308 Compatibility
March 30, 2021 Documentation Download CloverDX 5.10 5.10.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)