New features in 5.3

Dynamic metadata driven by a SQL Query

With this feature, you can configure CloverDX to dynamically generate metadata from a predefined SQL query. This enables you to create reusable, data-agnostic transformations and subgraphs that automatically adapt to changing structures in your database.

Dynamic metadata has been available to skilled developers (in the underlying graph XML code) for a long time. With 5.3 we’ve added a user interface to make the feature easily accessible to all users. Metadata defined in this new UI is backwards compatible — if you had used dynamic metadata in the past, you won’t need to make any changes.

Since CloverDX 5.3, SQL query metadata can be easily created and edited directly in the Designer. We’ve also added additional configuration options so that you’re not limited just to the default metadata properties (for example, you can set delimiters now).

Metadata propagation from CustomJava components

Here’s another feature geared towards reusability of code. Custom components implemented in Java (using CustomJavaReader, CustomJavaWriter, etc.) can now define how they propagate (or create their own) metadata on input and output ports.

This way you can better define the component’s interface and behavior. Development in Designer is also easier as your custom components behave more predictably. You have full control over the propagation in the component’s code — you can reuse existing metadata, take metadata from other ports or create your own.

Improved CloverDX Cluster load balancer

Optimizing the utilization of system resources and scaling of data integration workloads has been another key area we’ve focused on.

In 5.3 we’ve modified the Cluster load balancing algorithm to spread the workload much more evenly across available nodes. This leads to a better utilization of available resources, increased throughput, shorter processing times per job, and reduces the likelihood of losing nodes due to system overload. Additionally we’ve specifically focused on preventing nodes from crashing due to low system memory. 

Our extensive internal testing has shown significantly better results over previous versions, especially when processing large numbers of jobs that start at the same time. We’ve been able to reach up to 40% faster data processing.

CloverDX Server console changes

On the Server Monitoring screen, we’ve revamped the performance charts for better readability (and they’re prettier too).

Color coding of different Server instances is another improvement for better and safer management of multiple environments (dev, test, UAT, prod). You can now add a little colored tag to your Server instance to indicate the environment you’re currently working with. No more accidental mistakes on prod!

Additionally, we’ve made minor improvements to the Server UI; improved the lifecycle of info messages displayed in the Server Console and introduced better control over auto refresh in the Execution History.

Docker

With CloverDX 5.3 we’ve added a Docker file as another option to get and deploy the Server (previous options are still available without change). The Docker file contains all the necessary settings and configuration to easily run production-grade CloverDX Server in a container, greatly shortening the time it takes to configure a production Server. 

This allows to create a modern data processing architecture easily using CloverDX. Just deploy containerized Server and mount additional volumes. No more configuration, just focus on code and run your data jobs anywhere where Docker lives.

For more information and details how to use containerised CloverDX, see our github repository at https://github.com/cloverdx/cloverdx-server-docker

Other changes

• Support for Pivotal tcServer 3.2.14

• Java in CloverDX Designer bundle has been updated to Oracle JDK 8 b202