CloverETL 4.4 expands Salesforce connectivity with new SalesforceReader and Writer components that bring support for complex SOQL queries (function calls, joins, subqueries) and binary attachments.
In this update, we also continued working on Server features—adding support for Remote File Listeners (the ability to watch FTP, SFTP, Samba, and S3 folders) and introducing Task Failure Listeners, an additional automation option for monitoring and recovery of Event listeners.
Don’t forget to pay attention to the Compatibility section. We’re changing some defaults that may potentially affect many core components. However, you shouldn’t see any negative impact, as the changes we’re making are aimed towards more predictable, and thus error-proof, behavior.
We've expanded our Salesforce connector family with SOAP-based SalesforceReader/Writer components. The SOAP API gives you support for complex SOQL queries—aggregation and multi-currency functions, joins, aliases, reading deleted records (for capturing incremental changes) and subqueries—as well as reading and uploading of attachments.
With these components you also save your API limits—each query or write takes only a single API call (per 2,000 records) and returns results faster compared to SalesforceBulkReader/Writer, which work best for large batches.
However, these new components are limited to low data volumes, typically up to thousands of records. That makes them great for lookups or ad hoc queries, but for large data sets, SalesforceBulkReader/Writer would be a much better option.
CORPORATE SERVER AND CLUSTER ONLY
You can now watch "hot" remote folders—FTP, SFTP, Amazon S3 and Samba shares. Previously, only local folders were supported.
Also, you can now avoid jobs triggering on unfinished files (large files might appear in a folder before they finish copying/uploading). Do so by setting a polling interval that blocks the trigger until the file size has stabilized.
CORPORATE SERVER AND CLUSTER ONLY
Task Failure listeners allow you to configure reactions to failures of other listeners (e.g. a File Event Listener watching a "hot" folder starts failing because the folder becomes unavailable).
The reaction can be anything a normal listener can do, from sending emails (default) to running cleanup jobs, scripts, etc.
We've integrated these directly into Listeners modules (look for "Create Notification" button).
You'll find the Amazon Redshift driver in the DB connection configuration dialog out-of-the-box now. Amazon recommends using this driver instead of the Postgres driver.
ParallelReader component now supports native parallel reading from Amazon S3.
Right-click a value to quickly turn it into a filter in Data Inspector.
You can instruct XMLWriter to omit <?xml ... ?> declaration to write out fragments which you can later easily append or merge together.
By default the declaration is generated (set to "false"), maintaining backwards compatibility.
Please read carefully:
We've changed the default encoding for many components to UTF-8 (from ISO-8859-1).
This might affect your existing transformations (e.g. failing or records that are were okay before but are now invalid in UTF-8 definition).
We recommend reviewing the Compatibility section for this release. We're introducing some changes to defaults that could affect your production data (only if you are relying on some strangely behaved defaults).
To help you install or upgrade to this version, we've prepared a simple checklist:
Before You Upgrade
|Release||Compatibility/Upgrade notes, Features & Fixes||Published||Download|
|February 13, 2017||Download CloverETL 4.4 4.4.1|
Salesforce CRM Connector
Task Failure Listeners
Remote File Event Listeners
File Event Listeners
Amazon Web Services
Differences/Fixes since the Milestone Releases
|December 14, 2016||Download CloverETL 4.4 4.4.0|