May 6, 2018

What's new in version 3.8


Version 3.8 has been released and is available for downloading on our web-site. Here is an overview of the most notable changes:

Transformations are now called actions

From the beginning, EasyMorph has had a number of transformations such as “Run Program” or “Export to Excel” that actually didn’t transform anything. As its automation capabilities keep expanding, new transformations “that don’t transform anything” keep adding, making the term “transformation” increasingly confusing. Therefore, starting from this release all transformations now called actions.

For some time, both terms action and transformation will be used interchangeably, as we make relevant edits in the documentation, tutorial and other places. Eventually, the term “transformation” will only be used occasionally for actions that actually transform data.

Arbitrary order of actions

Up until the previous release, all actions in EasyMorph were divided into two categories: start actions, and regular actions. A start action must always come first in a table, while regular actions could only come after a start action, or another regular action. With the growth of automation use cases for EasyMorph, such separation introduced unnecessary limitations. For instance, importing a file is frequently done after copying it from another location. In such cases, importing had to be done in a separate project called via the “Call” action. This was inconvenient and cumbersome.

Starting from this release, this division is removed. From now on, any action can be first in a table, and any action can be inserted after any action. As the number of available actions in EasyMorph has become rather big (90+ actions as of this release), they have been separated into 5 categories, to simplify navigation:
  • Import – actions that import data (e.g. “Import from Excel”).
  • Create – actions that generate a table (e.g. “List of files”).
  • Transform – actions that modify input dataset (e.g. “Filter by expression”).
  • Export – actions that export data
  • Workflow – actions that affect internal workflows (e.g. “Iterate”), or interact with external systems and data (e.g. “Download file” or “Run program”).
If an action doesn’t expect any input data (typically actions from categories “Import” and “Create”), then the result of previous action is silently ignored without a warning or indication.
The equalization of actions has caused a few changes in the user interface of EasyMorph of which the most significant are two:
  1. The button “Insert/create table” used for selecting a start action and creating a table, has been removed. The “Add new action” button has moved to 1st position in the “Main” toolbar.
  2. The catalog of available actions is now two-tiered (it was one-tiered). Actions are grouped by categories and, optionally, by subcategories.

Column profiler

To simplify detection of data quality issues, the “Filter/Profile” dialog (invoked by double-clicking a column header) now implements a column profiler.

The profiler automatically calculates various counts. Each count has according “Filter” button to keep only rows with counted values.

The profiler also builds a histogram with distribution of numeric values in the column. The histogram allows filtering a range of values visually.

(click to zoom)

Notice that the profiler dialog window is floating meaning that you can click different columns in different tables without closing the profiler window.

Error handling

The "Call" action now has a new mode that allows capturing errors in the called project ("Iterate" has had this ability for a while):
The new mode makes it possible to arrange an error handling workflow in case of failure of the called project. For instance, send an email notification if the called project failed. The ability to handle project errors was frequently requested by EasyMorph users.

MS Exchange email connector

The generic SMTP email connector and the ability to send email notifications introduced in v3.7.1 sparked a lot of interest among EasyMorph users. However, in many cases sending emails through SMTP in MS Exchange was not possible due to Exchange settings or network connectivity issues.

In this release we're adding a native email connector for MS Exchange. Now, sending emails for organizations that use MS Exchange is not a problem.

New Plus licensing

The Plus license now always includes following features :
  • Unlimited actions
  • The “Input” action
  • The “Split delimited file” action
It also has become more expensive. See the new pricing at http://easymorph.com/buy.html.

Support for HTTPS in EasyMorph Server

EasyMorph Server now allows installing SSL certificates and has HTTPS-only mode. To simplify procurement and setup, EasyMorph Server license now includes 1 SSL certificate from a major certificate authority. 

Also, from now on version numbers for the Server are synchronized with desktop EasyMorph. Thus the latest Server version is also 3.8.

What's next

In a few weeks we will be releasing minor version 3.8.1 with a few new actions as well as data quality suggestions introduced in the Column Profiler. It will also have new error cards that will simplify debugging call/iteration errors.

In version 3.9 planned for release in early July we will continue improving automation capabilities in EasyMorph with the addition of two major features: transferring files via SFTP/SCP and receiving emails with attachments. This would make EasyMorph Server a very versatile tool for data collection and publishing.

Read also:
Release notes v3.8

Apr 19, 2018

Press-release: EasyMorph Announces Record Business Expansion and Adds 100th Customer

TORONTO, April 19, 2018 — EasyMorph Inc., a provider of self-service data preparation software for non-technical business users, announced record growth in its first quarter ending March 31, 2018. The company has achieved its highest revenue since its inception in 2015 and added its one-hundredth customer, a 300% increase in its customer base within the past year.

EasyMorph launched in 2015 with a vision to empower non-technical business users to easily merge, filter and manipulate data from multiple platforms and data sources without having to code or do complex programming. To date, there are over 15,000 downloads of the software with users in 20 different countries.

"We have added our one-hundredth customer in record time and I see it as a confirmation of EasyMorph's vision," said Dmitry Gudkov, Founder of EasyMorph Inc. "Reducing dependency on enterprise IT departments lets our business users solve data-related problems autonomously without delays and bureaucratic hassle."

EasyMorph's agile data preparation self-service solution helps non-technical business users to:
  • Extract and aggregate data from multiple platforms and data sources.
  • Slice and dice data, group information, calculate new metrics and more without having to code or do complex programming.
  • Automate routine file processing. 
  • Minimize their reliance on IT departments.
EasyMorph's powerful data preparation and automation capabilities can be used with any business intelligence or data visualization tool including Tableau dashboards, Qlik apps and Excel spreadsheets.

Interested parties can click here to download a free version of EasyMorph.

About EasyMorph

EasyMorph, Inc. provides self-service, agile data preparation at your fingertips. Business users can easily access, combine, prep, and manipulate data from multiple platforms and data sources with no coding required. EasyMorph is optimized for non-technical users to independently conquer and automate tedious data-related tasks quickly and efficiently.

Jan 22, 2018

What's new in version 3.7.1

With version 3.7.1 we continue expanding the automation capabilities in EasyMorph together with making data transformation more convenient. Key novelties include:

Sending emails

Sending email notifications is a cornerstone automation capability. There is a number of scenarios that have become possible due the introduction of the "Send email" action in EasyMorph:
  • Notification about failed data quality checks. A workflow can send an email before it gets halted.
  • Rule-based notifications. A Server task can be scheduled to check regularly certain conditions (e.g. existence of a file, consistency of data, or timeliness of new submissions). If a condition is not met, the task sends an according email notification.
  • Per-user data publishing. Sometimes, it's necessary to deliver each user a customized copy of a report or data extract. Sending an personalized email with an attachment could do the job.
  • Log monitoring. A scheduled project or task can parse logs and notify about possible errors or unwanted events.
The "Send email" action is highly configurable and allows specifying recipients, subject and attachment path using parameters. Parameters can also be dynamically inserted into the message body. The message text can be HTML formatted (only in Plus/Pro editions).


We're also working on making EasyMorph capable of receiving emails and saving attachments.

Triggering Tableau Server extract refreshes

This release introduced the "Tableau Server Command" action which employs the Tableau Server REST API to perform various operations (commands) on a remote Tableau Server. In version 3.7.1 the only command available is extract refreshing for a data source or a workbook. This command is available for Tableau Server version 10.3 and up.

With the expansion of the REST API new commands will become available in the action.


Easier creation of new parameters

Now it's easier to start using a parameter instead of a transformation property such as file name. For this, select <Add new parameter> in the list of parameters in the property, and a new parameter will be created automatically. The name and value of the new parameter will be defined after the transformation property it for which it was created.

For instance, when changing folder path to a parameter in the "List of files" transformation, the new parameter will automatically named "Folder path" and its value will be the folder path previously  in the transformation.


Other changes

  • The new "Metadata" transformation offers a convenient way to obtain a list of columns of a table, as well as the total number of rows and columns in it.
  • The "File Command" transformation now has a new mode for cloning existing files.
  • The "Convert Data Types" transformation has a new mode for automatically converting any data type in selected columns to text.
  • New connector "Email Server" in the Connector Manager.
  • The Start screen has a refreshed look with more choices to start a new workflow.
  • A new two-column selector is now displayed when selecting a project table, parameter, or connector. The selector allows searching and filtering that simplifies dealing with large projects.
  • Transformations Call, Iterate, and Iterate table now display parameter annotations.

Read also Release Notes v3.7.1.pdf

PS. Notice that our posts start using term action interchangeably with transformation. Eventually, in order to minimize confusion, we will reduce the use of term transformation only to the actions that actually transform data in EasyMorph.

Dec 25, 2017

EasyMorph doubles down on data-aware task automation



EasyMorph has already been offering some task automation capabilities (e.g. parameters and loops, running external programs) that are missing in less sophisticated data preparation tools. We've been witnessing large data analytics vendors entering the data preparation market after realizing that good data analysis requires good data preparation. However, we at EasyMorph understand that good data preparation almost always requires good automation. Manual data preparation is not enough anymore because it increasingly takes too much time. More and more internal and external systems get involved into data analysis every day. More and more public data APIs opened. More and more files are received, processed, sent out, or uploaded. Traditional batch scripting doesn't help much because it's not data-aware. Too frequently data-related workflows depend on data itself. For instance, before processing a file extract it may be necessary to obtain the report date from the extract contents, because it's not available in the file name or timestamp. Or, data quality has to be verified before a file is copied, or uploaded into a database automatically. Sometimes data quality verification can require querying multiple data sources. Such logic can be non-trivial to arrange with scripting.

With almost 100 transformations (as of ver.3.7), EasyMorph is already well positioned for the new reality as it combines comprehensive data transformation features with certain elements of automation. However, we're doing a step further and doubling down on the automation capabilities in EasyMorph. The goal is to provide a convenient mechanism for automating routine data-related tasks, even those that might not be classified as data transformation at all, e.g. automated rule-based alerts and file transfers.

Some of the new features are already available in version 3.7 released recently, but more is coming. Here is what to expect:
  • Conditional IF/THEN/ELSE workflows (already available in 3.7)
  • Deep PowerShell integration (already available in 3.7) that opens the vast .NET ecosystem as well as Windows automation for EasyMorph users. With it you can, for instance, trigger VBA macros in spreadsheets, or fetch email attachments from your Outlook folder right from EasyMorph projects.
  • In version 3.8 we will be lifting the existing restrictions on transformation order. Currently an import transformation must always come first in a table. This restriction will be removed and any transformation will be possible to insert before or after any transformation. Tables will be able to start with a workflow transformation such as Download File or Run Program, and then have an import transformation.
  • In 3.9 EasyMorph Scheduler will evolve into EasyMorph Launcher -- a handy utility residing in the system tray with a list of frequently used tasks that either run on schedule or are triggered manually. You will be able to launch in two clicks daily routines, such as transferring files between computers, fetching updates from Amazon AWS, uploading data and refreshing Tableau Server workbooks or Qlik Server dashboards, triggering EasyMorph Server tasks, sending out emails and even text messages (SMS, for those in Europe). Think of it as high-level replacement for batch scripts.
  • Triggering various actions through public APIs in external and internal systems. For instance, extract refreshing in Tableau Server (coming in v3.7.1), reloading apps in Qlik Sense Server (coming in v3.7.2), various actions with Amazon AWS, Azure, Google Apps, Slack and other cloud applications.
  • Easier iterations (loops) that won't require creating another project (v4.0).
  • Event-based task triggering (e.g on file change), automatic email retrieval and attachment processing in EasyMorph Server.
  • Error handling.
  • Transformations renamed to actions.

All these changes will be gradually rolled out during 2018 with some features coming sooner, and some later. As always, we're keeping ears open for your suggestions.

Of course, expanding into data-aware task automation doesn't mean that the "traditional" data transformation part in EasyMorph will be neglected. There are many new ETL-related features on the 2018 roadmap but that would be a topic of another post.

Discuss this article on EasyMorph Community

Dec 18, 2017

What's new in EasyMorph Server 1.3

Spaces

Previously all users could access all Server tasks as well as files in the public folder. Starting from this release EasyMorph Server introduces spaces in order to offer more granularity and control in managing access to tasks and files.

Each space is independent from other spaces, and can have its own:
  • tasks
  • public web-folder
  • connector repository

Spaces are managed in a new tab called “Spaces” visible only to the Server administrator.

Server spaces

Spaces allow limiting access to tasks using one of the 4 modes:
  • Disabled – tasks are entirely disabled and not available in this particular space
  • Lock task – a user can only trigger tasks, but not create or edit them
  • Lock project – a user can edit task schedule, task parameters, but not create tasks, or change the project in a task.
  • Full access – no restrictions, the only mode where new tasks can be created.

A combination of task access mode with one of the file access modes introduced earlier allows flexibly configuring spaces for different types of users and use cases. See the table below for a few examples:

Examples of spaces with different roles

Password-protected spaces

A space can be password-protected. In this case, a user should provide correct password in order to access tasks and/or files.

Performing operations through the API with password-protected spaces also requires providing a password. The command-line client and EasyMorph Server .NET SDK have been updated accordingly.

Remote administration

By default, the Server settings and space configuration is only allowed from localhost. However, it is now possible to enable remote administration. In this case, it is possible to configure the Server from a remote computer, after providing a valid administrator password.

Start/stop batch scripts

It is now possible to execute a batch script when the Server service is starting or stopping. Modify onstart.bat or onstop.bat located in C:\Program Files\EasyMorph Server\systemscripts accordingly.

This capability can be used, for instance, to map additional network drives on server start, or to send an email notification when the Server service is shutting down or rebooting.