New features in 5.10

Native Snowflake connections

CloverDX is now a Snowflake technology partner, which allowed us to bundle official driver for family of existing database components and implement efficient, native connectivity to Snowflake. New component SnowflakeBulkWriter was created to support highly efficient inserts (you can find this component in Writers category in Designer palette). On average, new SnowflakeBulkWriter is 8x faster for same data volume compared to standard DatabaseWriter’s INSERT statement.

Apache Parquet format support

To support data analytic use cases in Hadoop ecosystem, CloverDX 5.10 ships with two new components, ParquetReader and ParquetWriter. These components start in Incubation category where they will stay for approximately 2 releases. These components contain basic set of features which are going to be extended over incubation period.

ParquetReader: with performance in mind, developers can choose which columns to read – improving speed by minimizing unnecessary I/O operations. Works with file’s schema document, supporting metadata propagation.

ParquetWriter: using configurable mapping, serializes data streams into a Parquet file to any data target supported by other file-based components (including direct writes to HDFS and S3).

Alert level in Operations Dashboards

Latest improvement to the Dashboard, originally introduced in CloverDX 5.8.0, allows users to assign a severity level to each monitored resource. Higher level will manifest with more visible highlight and will be shown before alerts with lower configured severity.

Alert level helps support to identify and resolve critical issues in a timely manner.

We also made it easier to manage monitors on the Dashboard since it is now possible to add triggers to monitors directly from Alerts and Notification tab for each trigger and Data Service.

Kafka Message Listener

With latest release, collection of event listeners grown for Kafka Message Listener, capable detecting new arrivals into Kafka message queues. It allows to define both, specific topic(s) and topics by use of regular expressions to match them by common pattern.

Configuration which used to detect changes is subsequently passed to triggered task to do the processing.

Server management REST APIs

We created a new REST API to help us address shortcomings of existing Simple HTTP API. Starting with this release, we ported most used methods from Simple HTTP API. New API is easier to use and more powerful and will be further improved in future releases.

All methods come with complete Swagger documentation with test interface and examples. It is accessible from the Server’s login page via Server REST API Catalog link.

Simple HTTP API is still supported and will not be removed anytime soon. It is now however in maintenance mode and will not be actively developed.

Product features removed

• Launch Services: Launch Services were hidden to most CloverDX Server users since they were replaced by Data Services and deprecated in CloverETL 4.9. Note that upgrading to CloverDX 5.10 from previous versions will remove all your Launch Service configuration from the database but will not affect any graphs, jobflows or any other logic in your sandboxes. We strongly recommend any Launch Services in use to be reimplemented as Data Services prior upgrade and set up HTTP proxy to adjust for new URLs when required. Make sure database is backed up before upgrade.
• RSE projects: RSE projects were used to connect CloverDX Designer to the Server. They were deprecated since CloverETL 4.2 and were replaced by “sync projects”. Starting with CloverDX 5.10, it will not longer be possible to open RSE projects or create new ones. To access your data in existing RSE project, simply create new Server Project from Designer and point it to appropriate sandbox. You will not lose any project data.
• XLSDataReader and XLSDataWriter: we removed very old and deprecated components. Both were replaced by more powerful and faster components SpreadsheetDataReader and SpreadhseetDataWriter in CloverETL 3.0.