EasyMorph has already been offering some task automation capabilities (e.g. parameters and loops, running external programs) that are missing in less sophisticated data preparation tools. We've been witnessing large data analytics vendors entering the data preparation market after realizing that good data analysis requires good data preparation. However, we at EasyMorph understand that good data preparation almost always requires good automation. Manual data preparation is not enough anymore because it increasingly takes too much time. More and more internal and external systems get involved into data analysis every day. More and more public data APIs opened. More and more files are received, processed, sent out, or uploaded. Traditional batch scripting doesn't help much because it's not data-aware. Too frequently data-related workflows depend on data itself. For instance, before processing a file extract it may be necessary to obtain the report date from the extract contents, because it's not available in the file name or timestamp. Or, data quality has to be verified before a file is copied, or uploaded into a database automatically. Sometimes data quality verification can require querying multiple data sources. Such logic can be non-trivial to arrange with scripting.
With almost 100 transformations (as of ver.3.7), EasyMorph is already well positioned for the new reality as it combines comprehensive data transformation features with certain elements of automation. However, we're doing a step further and doubling down on the automation capabilities in EasyMorph. The goal is to provide a convenient mechanism for automating routine data-related tasks, even those that might not be classified as data transformation at all, e.g. automated rule-based alerts and file transfers.
Some of the new features are already available in version 3.7 released recently, but more is coming. Here is what to expect:
- Conditional IF/THEN/ELSE workflows (already available in 3.7)
- Deep PowerShell integration (already available in 3.7) that opens the vast .NET ecosystem as well as Windows automation for EasyMorph users. With it you can, for instance, trigger VBA macros in spreadsheets, or fetch email attachments from your Outlook folder right from EasyMorph projects.
- In version 3.8 we will be lifting the existing restrictions on transformation order. Currently an import transformation must always come first in a table. This restriction will be removed and any transformation will be possible to insert before or after any transformation. Tables will be able to start with a workflow transformation such as Download File or Run Program, and then have an import transformation.
- In 3.9 EasyMorph Scheduler will evolve into EasyMorph Launcher -- a handy utility residing in the system tray with a list of frequently used tasks that either run on schedule or are triggered manually. You will be able to launch in two clicks daily routines, such as transferring files between computers, fetching updates from Amazon AWS, uploading data and refreshing Tableau Server workbooks or Qlik Server dashboards, triggering EasyMorph Server tasks, sending out emails and even text messages (SMS, for those in Europe). Think of it as high-level replacement for batch scripts.
- Triggering various actions through public APIs in external and internal systems. For instance, extract refreshing in Tableau Server (coming in v3.7.1), reloading apps in Qlik Sense Server (coming in v3.7.2), various actions with Amazon AWS, Azure, Google Apps, Slack and other cloud applications.
- Easier iterations (loops) that won't require creating another project (v4.0).
- Event-based task triggering (e.g on file change), automatic email retrieval and attachment processing in EasyMorph Server.
- Error handling.
- Transformations renamed to actions.
All these changes will be gradually rolled out during 2018 with some features coming sooner, and some later. As always, we're keeping ears open for your suggestions.
Of course, expanding into data-aware task automation doesn't mean that the "traditional" data transformation part in EasyMorph will be neglected. There are many new ETL-related features on the 2018 roadmap but that would be a topic of another post.
Discuss this article on EasyMorph Community