Aug 30, 2018

What's new in version 3.9


EasyMorph version 3.9 is now available for downloading on our website. The new release adds  powerful new automation capabilities to EasyMorph:

Fetching emails with attachments

Our goal is to make the everyday work of a data analyst as automated as possible so that tedious activities take less time, and more time can be spent on interesting and high-value work. Receiving data files by email and further processing them is a daily and monotonous routine for many people, but now it can be automated with EasyMorph. The "Fetch email" action introduced in v3.9 automatically retrieves email messages with attachments from Exchange as well as from mail servers that support IMAP (most modern mail services do).

The "Fetch email" action (tabs "Pre-filter" and "Attachments").

Using this action, it is possible to automate the process of receiving data files from email attachments and then transforming them with further uploading into a database, or publishing them to a Tableau Server, or transferring to another machine through SFTP (more on this later).

With a bit of creativity, more advanced scenarios can be accomplished, for instance:

  • Sending an automated reminder if someone didn't send an expected data file in time.
  • Automatically verifying incoming data in email attachments and then sending back a request to resubmit the data if a data quality checks fails.
  • Using a 3rd party service to collect information through web-forms. The service sends form submissions by email which are automatically retrieved by EasyMorph, parsed, and uploaded into a database.
  • Performing a particular action (e.g. run program, or execute PowerShell script) when an email with specific keywords in the subject is received.

SFTP/SCP file transfer

It is now possible to automatically send to and receive files from a remote computer via SFTP (FTP over SSH) and SCP. Key pair authentication is supported as well as use of a proxy.
The "File transfer" action.

Note that sending files by regular (non-secure) FTP is not supported.

Data collection/publishing

With this release, EasyMorph (Desktop and Server) has become a versatile tool for automated data collection and publishing. It can now be done in multiple ways which cover the vast majority of cases in a corporate environment:
  • Receive/send emails with attachments (Desktop and Server).
  • Transfer files to/from a remote computer via SFTP/SCP (Desktop and Server).
  • Upload/download files via a web-browser (Server only).
  • Transfer files between Desktop and Server over secure HTTPS connection using the "EasyMorph Server Command" action.
  • Transfer files between any computer and Server using Server CLI or EasyMorph .NET SDK

Native data format

This release introduces for the first time, the ability to save and load EasyMorph datasets into files in the native EasyMorph data format (.dset). The native data file format stores data in a form that is very close to the internal data representation in the EasyMorph’s in-memory engine. Therefore, saving a dataset into a file and reading a dataset from a file doesn’t have performance overhead issues such as data conversion or compression/decompression, and therefore is very fast.

The native format compresses data. The compression ratio on par with Tableau Data Extracts (.tde) and Qlik QVDs (.qvd). The difference is typically within 10% deviation compared to one or another format depending on data type.

Later we will add the ability to publish datasets to and load from the Server in one click. This will make the Server a convenient place for collaboration -- storing and sharing even large datasets with millions of rows.

Data Preparation University

This newsletter is the last one in its current form. Starting with out next newsletter, we're introducing a new format, our "Data Preparation University" where most of its content will be dedicated to practical advice on data preparation for beginners and experienced users. We will continue to include short overviews of new releases, however the main description of what's new will be in the release notes that accompany every new version.

Links:
Download EasyMorph
Release Notes v3.9 (pdf)


May 6, 2018

What's new in version 3.8


Version 3.8 has been released and is available for downloading on our web-site. Here is an overview of the most notable changes:

Transformations are now called actions

From the beginning, EasyMorph has had a number of transformations such as “Run Program” or “Export to Excel” that actually didn’t transform anything. As its automation capabilities keep expanding, new transformations “that don’t transform anything” keep adding, making the term “transformation” increasingly confusing. Therefore, starting from this release all transformations now called actions.

For some time, both terms action and transformation will be used interchangeably, as we make relevant edits in the documentation, tutorial and other places. Eventually, the term “transformation” will only be used occasionally for actions that actually transform data.

Arbitrary order of actions

Up until the previous release, all actions in EasyMorph were divided into two categories: start actions, and regular actions. A start action must always come first in a table, while regular actions could only come after a start action, or another regular action. With the growth of automation use cases for EasyMorph, such separation introduced unnecessary limitations. For instance, importing a file is frequently done after copying it from another location. In such cases, importing had to be done in a separate project called via the “Call” action. This was inconvenient and cumbersome.

Starting from this release, this division is removed. From now on, any action can be first in a table, and any action can be inserted after any action. As the number of available actions in EasyMorph has become rather big (90+ actions as of this release), they have been separated into 5 categories, to simplify navigation:
  • Import – actions that import data (e.g. “Import from Excel”).
  • Create – actions that generate a table (e.g. “List of files”).
  • Transform – actions that modify input dataset (e.g. “Filter by expression”).
  • Export – actions that export data
  • Workflow – actions that affect internal workflows (e.g. “Iterate”), or interact with external systems and data (e.g. “Download file” or “Run program”).
If an action doesn’t expect any input data (typically actions from categories “Import” and “Create”), then the result of previous action is silently ignored without a warning or indication.
The equalization of actions has caused a few changes in the user interface of EasyMorph of which the most significant are two:
  1. The button “Insert/create table” used for selecting a start action and creating a table, has been removed. The “Add new action” button has moved to 1st position in the “Main” toolbar.
  2. The catalog of available actions is now two-tiered (it was one-tiered). Actions are grouped by categories and, optionally, by subcategories.

Column profiler

To simplify detection of data quality issues, the “Filter/Profile” dialog (invoked by double-clicking a column header) now implements a column profiler.

The profiler automatically calculates various counts. Each count has according “Filter” button to keep only rows with counted values.

The profiler also builds a histogram with distribution of numeric values in the column. The histogram allows filtering a range of values visually.

(click to zoom)

Notice that the profiler dialog window is floating meaning that you can click different columns in different tables without closing the profiler window.

Error handling

The "Call" action now has a new mode that allows capturing errors in the called project ("Iterate" has had this ability for a while):
The new mode makes it possible to arrange an error handling workflow in case of failure of the called project. For instance, send an email notification if the called project failed. The ability to handle project errors was frequently requested by EasyMorph users.

MS Exchange email connector

The generic SMTP email connector and the ability to send email notifications introduced in v3.7.1 sparked a lot of interest among EasyMorph users. However, in many cases sending emails through SMTP in MS Exchange was not possible due to Exchange settings or network connectivity issues.

In this release we're adding a native email connector for MS Exchange. Now, sending emails for organizations that use MS Exchange is not a problem.

New Plus licensing

The Plus license now always includes following features :
  • Unlimited actions
  • The “Input” action
  • The “Split delimited file” action
It also has become more expensive. See the new pricing at http://easymorph.com/buy.html.

Support for HTTPS in EasyMorph Server

EasyMorph Server now allows installing SSL certificates and has HTTPS-only mode. To simplify procurement and setup, EasyMorph Server license now includes 1 SSL certificate from a major certificate authority. 

Also, from now on version numbers for the Server are synchronized with desktop EasyMorph. Thus the latest Server version is also 3.8.

What's next

In a few weeks we will be releasing minor version 3.8.1 with a few new actions as well as data quality suggestions introduced in the Column Profiler. It will also have new error cards that will simplify debugging call/iteration errors.

In version 3.9 planned for release in early July we will continue improving automation capabilities in EasyMorph with the addition of two major features: transferring files via SFTP/SCP and receiving emails with attachments. This would make EasyMorph Server a very versatile tool for data collection and publishing.

Read also:
Release notes v3.8

Apr 19, 2018

Press-release: EasyMorph Announces Record Business Expansion and Adds 100th Customer

TORONTO, April 19, 2018 — EasyMorph Inc., a provider of self-service data preparation software for non-technical business users, announced record growth in its first quarter ending March 31, 2018. The company has achieved its highest revenue since its inception in 2015 and added its one-hundredth customer, a 300% increase in its customer base within the past year.

EasyMorph launched in 2015 with a vision to empower non-technical business users to easily merge, filter and manipulate data from multiple platforms and data sources without having to code or do complex programming. To date, there are over 15,000 downloads of the software with users in 20 different countries.

"We have added our one-hundredth customer in record time and I see it as a confirmation of EasyMorph's vision," said Dmitry Gudkov, Founder of EasyMorph Inc. "Reducing dependency on enterprise IT departments lets our business users solve data-related problems autonomously without delays and bureaucratic hassle."

EasyMorph's agile data preparation self-service solution helps non-technical business users to:
  • Extract and aggregate data from multiple platforms and data sources.
  • Slice and dice data, group information, calculate new metrics and more without having to code or do complex programming.
  • Automate routine file processing. 
  • Minimize their reliance on IT departments.
EasyMorph's powerful data preparation and automation capabilities can be used with any business intelligence or data visualization tool including Tableau dashboards, Qlik apps and Excel spreadsheets.

Interested parties can click here to download a free version of EasyMorph.

About EasyMorph

EasyMorph, Inc. provides self-service, agile data preparation at your fingertips. Business users can easily access, combine, prep, and manipulate data from multiple platforms and data sources with no coding required. EasyMorph is optimized for non-technical users to independently conquer and automate tedious data-related tasks quickly and efficiently.

Jan 22, 2018

What's new in version 3.7.1

With version 3.7.1 we continue expanding the automation capabilities in EasyMorph together with making data transformation more convenient. Key novelties include:

Sending emails

Sending email notifications is a cornerstone automation capability. There is a number of scenarios that have become possible due the introduction of the "Send email" action in EasyMorph:
  • Notification about failed data quality checks. A workflow can send an email before it gets halted.
  • Rule-based notifications. A Server task can be scheduled to check regularly certain conditions (e.g. existence of a file, consistency of data, or timeliness of new submissions). If a condition is not met, the task sends an according email notification.
  • Per-user data publishing. Sometimes, it's necessary to deliver each user a customized copy of a report or data extract. Sending an personalized email with an attachment could do the job.
  • Log monitoring. A scheduled project or task can parse logs and notify about possible errors or unwanted events.
The "Send email" action is highly configurable and allows specifying recipients, subject and attachment path using parameters. Parameters can also be dynamically inserted into the message body. The message text can be HTML formatted (only in Plus/Pro editions).


We're also working on making EasyMorph capable of receiving emails and saving attachments.

Triggering Tableau Server extract refreshes

This release introduced the "Tableau Server Command" action which employs the Tableau Server REST API to perform various operations (commands) on a remote Tableau Server. In version 3.7.1 the only command available is extract refreshing for a data source or a workbook. This command is available for Tableau Server version 10.3 and up.

With the expansion of the REST API new commands will become available in the action.


Easier creation of new parameters

Now it's easier to start using a parameter instead of a transformation property such as file name. For this, select <Add new parameter> in the list of parameters in the property, and a new parameter will be created automatically. The name and value of the new parameter will be defined after the transformation property it for which it was created.

For instance, when changing folder path to a parameter in the "List of files" transformation, the new parameter will automatically named "Folder path" and its value will be the folder path previously  in the transformation.


Other changes

  • The new "Metadata" transformation offers a convenient way to obtain a list of columns of a table, as well as the total number of rows and columns in it.
  • The "File Command" transformation now has a new mode for cloning existing files.
  • The "Convert Data Types" transformation has a new mode for automatically converting any data type in selected columns to text.
  • New connector "Email Server" in the Connector Manager.
  • The Start screen has a refreshed look with more choices to start a new workflow.
  • A new two-column selector is now displayed when selecting a project table, parameter, or connector. The selector allows searching and filtering that simplifies dealing with large projects.
  • Transformations Call, Iterate, and Iterate table now display parameter annotations.

Read also Release Notes v3.7.1.pdf

PS. Notice that our posts start using term action interchangeably with transformation. Eventually, in order to minimize confusion, we will reduce the use of term transformation only to the actions that actually transform data in EasyMorph.

Dec 25, 2017

EasyMorph doubles down on data-aware task automation



EasyMorph has already been offering some task automation capabilities (e.g. parameters and loops, running external programs) that are missing in less sophisticated data preparation tools. We've been witnessing large data analytics vendors entering the data preparation market after realizing that good data analysis requires good data preparation. However, we at EasyMorph understand that good data preparation almost always requires good automation. Manual data preparation is not enough anymore because it increasingly takes too much time. More and more internal and external systems get involved into data analysis every day. More and more public data APIs opened. More and more files are received, processed, sent out, or uploaded. Traditional batch scripting doesn't help much because it's not data-aware. Too frequently data-related workflows depend on data itself. For instance, before processing a file extract it may be necessary to obtain the report date from the extract contents, because it's not available in the file name or timestamp. Or, data quality has to be verified before a file is copied, or uploaded into a database automatically. Sometimes data quality verification can require querying multiple data sources. Such logic can be non-trivial to arrange with scripting.

With almost 100 transformations (as of ver.3.7), EasyMorph is already well positioned for the new reality as it combines comprehensive data transformation features with certain elements of automation. However, we're doing a step further and doubling down on the automation capabilities in EasyMorph. The goal is to provide a convenient mechanism for automating routine data-related tasks, even those that might not be classified as data transformation at all, e.g. automated rule-based alerts and file transfers.

Some of the new features are already available in version 3.7 released recently, but more is coming. Here is what to expect:
  • Conditional IF/THEN/ELSE workflows (already available in 3.7)
  • Deep PowerShell integration (already available in 3.7) that opens the vast .NET ecosystem as well as Windows automation for EasyMorph users. With it you can, for instance, trigger VBA macros in spreadsheets, or fetch email attachments from your Outlook folder right from EasyMorph projects.
  • In version 3.8 we will be lifting the existing restrictions on transformation order. Currently an import transformation must always come first in a table. This restriction will be removed and any transformation will be possible to insert before or after any transformation. Tables will be able to start with a workflow transformation such as Download File or Run Program, and then have an import transformation.
  • In 3.9 EasyMorph Scheduler will evolve into EasyMorph Launcher -- a handy utility residing in the system tray with a list of frequently used tasks that either run on schedule or are triggered manually. You will be able to launch in two clicks daily routines, such as transferring files between computers, fetching updates from Amazon AWS, uploading data and refreshing Tableau Server workbooks or Qlik Server dashboards, triggering EasyMorph Server tasks, sending out emails and even text messages (SMS, for those in Europe). Think of it as high-level replacement for batch scripts.
  • Triggering various actions through public APIs in external and internal systems. For instance, extract refreshing in Tableau Server (coming in v3.7.1), reloading apps in Qlik Sense Server (coming in v3.7.2), various actions with Amazon AWS, Azure, Google Apps, Slack and other cloud applications.
  • Easier iterations (loops) that won't require creating another project (v4.0).
  • Event-based task triggering (e.g on file change), automatic email retrieval and attachment processing in EasyMorph Server.
  • Error handling.
  • Transformations renamed to actions.

All these changes will be gradually rolled out during 2018 with some features coming sooner, and some later. As always, we're keeping ears open for your suggestions.

Of course, expanding into data-aware task automation doesn't mean that the "traditional" data transformation part in EasyMorph will be neglected. There are many new ETL-related features on the 2018 roadmap but that would be a topic of another post.

Discuss this article on EasyMorph Community