New features in 5.11

Job Inspector

Job Inspector is a new, integral part of the Server Console that will allow you to view your graphs and other types of jobs like jobflows, subgraphs or even data services directly in server environment without having to open CloverDX Designer.

Job Inspector will help you to quickly review overall data flow, metadata on all edges, job logs, component configuration and more. This makes it ideally suited for issue investigation on production. It does not allow any job editing so even users with limited privileges can use it to view jobs.

When job fails, Job Inspector provides means to efficiently investigate why an error occurred. For running jobs, Job Inspector provides real-time monitoring of job progress.

CSV output preview in Data Apps

Data Apps have become one of our core features as many users now use them to run variety of ad-hoc processes in simple and easy user interface.

CloverDX 5.11 extends Data App interface with more powerful data viewer, allowing users to work with results directly in browser. CSV data will now be displayed in rich table view with headers as defined in output metadata. You can sort, search and browse through your data.

Variant and semi-structured data

With our introduction of variant data type in CloverDX 5.6 we allowed engineers to build processes more effectively by native support of semi-structured formats - JSON and BSON.

In this release, we are improving this functionality by adding support for variant fields in metadata. You can now transport variant data throughout your transformations, use it in components without need for conversion to and from variant.

JSONExtract and JSONWriter now natively support variant so you can easily read or write whole (sub)sections of JSON documents into a single metadata field.

Formulas and hyperlinks in Microsoft Excel files

SpreadsheetDataReader and SpredsheetDataWriter were extended for reading and writing of formulas and hyperlinks. With the improved components you can read and write more complex and more interactive spreadsheets.

We also improved Map by name feature which is commonly used when reading files where headers can appear out of order. Automated mapping is more robust and predictable, thanks to normalization of header data and improved field matching algorithm. It will help you match your columns in wider range of cases regardless of letter case, extraneous whitespaces and use of special characters.

Azure Blob Storage connectivity

To support growing number of CloverDX deployments in Azure cloud, we are introducing new remote protocol - Azure Blob Storage. Azure Blob protocol can be used in all Readers, Writers and File Operations components. This allows you to work with any file type or file format (like csv, json, etc.) or work with files directly (to copy, move, delete, etc.). To access data in Azure Blob, use URL with az-blob:// prefix.

Apache Avro format support (Incubation)

CloverDX 5.11 adds one more data format to help you work with complex, semi-structured data – Apache Avro. We are introducing three new functions that allow you to work with Avro data directly in your transformations:

• variant parseAvro(byte avroData, string avroSchema)

• byte writeAvro(variant value, string avroSchema)

• string getAvroSchema(variant data)

These functions allow you to convert any Avro data to, and from variant. You can then use variant to implement business logic to transform Avro data as needed.

Main use case for these functions is to help you work with messaging use cases that use Avro as the data format. For example, you can now easily connect to Kafka topic, pull Avro messages from it, map incoming data to your internal format and load into your target system.

These functions are now considered to be in Incubation. We will be working on further upgrades to extend these functions. Even though they are in Incubation, they are fully supported and you can safely use them in your production code as needed.

Server REST API

Latest additions to CloverDX Server REST API are methods which allow you to enable or disable automation features - Schedulers, Event Listeners and Data Services.

These API endpoints will help you build more complex automated deployments for your DevOps or DataOps processes.

Platform support

To help you run CloverDX in demanding and business critical applications, CloverDX 5.11 adds support for two deployment stacks – combinations of operating system, application server and Java JDK distribution – all with commercial support by their vendors.

To ensure the stability of the deployment, these stacks are supported as single configurations and should not be mixed with open-source alternatives or between each other.

Red Hat stack
This stack is built on top of Red Hat technology:

• Red Hat Enterprise Linux 8
• Red Hat OpenJDK 11
• Red Hat JBoss Web Server 5.4

VMware (Pivotal) stack
VMware stack uses VMware (formerly Pivotal) application server and JDK:

• VMware tc Server 4.1
• VMware-supported OpenJDK 11 (BellSoft Liberica JDK 11)

We test VMware stack on CentOS and on Ubuntu Linux. We are still supporting existing tc Server 3.1 deployments even though tc Server 3.1 is no longer officially supported.

Compatibility notice

JBoss EAP 7.1 support

CloverDX 5.11 is the last CloverDX release that will support JBoss EAP 7.1 as target application server. CloverDX releases starting with CloverDX 5.12 (planned in September 2021) will only support JBoss EAP 7.2 deployments. We advise customers running CloverDX on JBoss EAP 7.1 to upgrade to JBoss EAP 7.2.