Nov 2, 2018

4 reasons to start hosting datasets on EasyMorph Server

We've just released EasyMorph v3.9.1 (both Desktop and Server). One of its major new features is the easy publishing of datasets from Desktop to Server and retrieving from Server back to Desktop. While it may look as a minor feature at first glance, there are 4 good reasons why you may want to start hosting your data on EasyMorph Server:

Easy publishing from Desktop to Server

When you keep datasets on Server instead of local drive, there is no need to use calculated parameters for paths to files in EasyMorph projects. Datasets will be loaded from the same location on Server no matter if your project is executed on Desktop or on Server. Therefore you can simply publish a project from Desktop to Server without any additional configuration of file locations.

It works well even for large datasets. Because the dataset format (.dset) is highly compressed and close to the internal representation of data in EasyMorph, loading a big dataset from Server over a network can be much faster than loading an equivalent CSV file stored locally.


Collaboration on datasets

Sending a interesting dataset to your colleague is now very easy: right-click the action which result you want to share, choose the "Send to Server" command, pick a folder on Server, and press "Publish". Then tell your colleague the folder location and the dataset name. The colleague right-clicks the Desktop workspace (the grey background behind tables), chooses the "Receive from Server" command, and receives the dataset in seconds, even if it contains millions of rows.


Project portability

When datasets are hosted on Server, there is no need to send source files together with project if a user wants to send the project to another user. You can send only the project itself, and when the other user opens it, the source datasets will be loaded automatically from Server and the project will work with exactly the same data, as if you sent it together with source data files.


Better security

Using Server for hosting source data instead of local files also works better from a security standpoint. Source data files have limited protection when you share them via a network folder or email message. However, when datasets are hosted on Server no one else will be able to access them unless they have been given access to the Server space that hosts the datasets.

Aug 30, 2018

What's new in version 3.9


EasyMorph version 3.9 is now available for downloading on our website. The new release adds  powerful new automation capabilities to EasyMorph:

Fetching emails with attachments

Our goal is to make the everyday work of a data analyst as automated as possible so that tedious activities take less time, and more time can be spent on interesting and high-value work. Receiving data files by email and further processing them is a daily and monotonous routine for many people, but now it can be automated with EasyMorph. The "Fetch email" action introduced in v3.9 automatically retrieves email messages with attachments from Exchange as well as from mail servers that support IMAP (most modern mail services do).

The "Fetch email" action (tabs "Pre-filter" and "Attachments").

Using this action, it is possible to automate the process of receiving data files from email attachments and then transforming them with further uploading into a database, or publishing them to a Tableau Server, or transferring to another machine through SFTP (more on this later).

With a bit of creativity, more advanced scenarios can be accomplished, for instance:

  • Sending an automated reminder if someone didn't send an expected data file in time.
  • Automatically verifying incoming data in email attachments and then sending back a request to resubmit the data if a data quality checks fails.
  • Using a 3rd party service to collect information through web-forms. The service sends form submissions by email which are automatically retrieved by EasyMorph, parsed, and uploaded into a database.
  • Performing a particular action (e.g. run program, or execute PowerShell script) when an email with specific keywords in the subject is received.

SFTP/SCP file transfer

It is now possible to automatically send to and receive files from a remote computer via SFTP (FTP over SSH) and SCP. Key pair authentication is supported as well as use of a proxy.
The "File transfer" action.

Note that sending files by regular (non-secure) FTP is not supported.

Data collection/publishing

With this release, EasyMorph (Desktop and Server) has become a versatile tool for automated data collection and publishing. It can now be done in multiple ways which cover the vast majority of cases in a corporate environment:
  • Receive/send emails with attachments (Desktop and Server).
  • Transfer files to/from a remote computer via SFTP/SCP (Desktop and Server).
  • Upload/download files via a web-browser (Server only).
  • Transfer files between Desktop and Server over secure HTTPS connection using the "EasyMorph Server Command" action.
  • Transfer files between any computer and Server using Server CLI or EasyMorph .NET SDK

Native data format

This release introduces for the first time, the ability to save and load EasyMorph datasets into files in the native EasyMorph data format (.dset). The native data file format stores data in a form that is very close to the internal data representation in the EasyMorph’s in-memory engine. Therefore, saving a dataset into a file and reading a dataset from a file doesn’t have performance overhead issues such as data conversion or compression/decompression, and therefore is very fast.

The native format compresses data. The compression ratio on par with Tableau Data Extracts (.tde) and Qlik QVDs (.qvd). The difference is typically within 10% deviation compared to one or another format depending on data type.

Later we will add the ability to publish datasets to and load from the Server in one click. This will make the Server a convenient place for collaboration -- storing and sharing even large datasets with millions of rows.

Data Preparation University

This newsletter is the last one in its current form. Starting with out next newsletter, we're introducing a new format, our "Data Preparation University" where most of its content will be dedicated to practical advice on data preparation for beginners and experienced users. We will continue to include short overviews of new releases, however the main description of what's new will be in the release notes that accompany every new version.

Links:
Download EasyMorph
Release Notes v3.9 (pdf)


May 6, 2018

What's new in version 3.8


Version 3.8 has been released and is available for downloading on our web-site. Here is an overview of the most notable changes:

Transformations are now called actions

From the beginning, EasyMorph has had a number of transformations such as “Run Program” or “Export to Excel” that actually didn’t transform anything. As its automation capabilities keep expanding, new transformations “that don’t transform anything” keep adding, making the term “transformation” increasingly confusing. Therefore, starting from this release all transformations now called actions.

For some time, both terms action and transformation will be used interchangeably, as we make relevant edits in the documentation, tutorial and other places. Eventually, the term “transformation” will only be used occasionally for actions that actually transform data.

Arbitrary order of actions

Up until the previous release, all actions in EasyMorph were divided into two categories: start actions, and regular actions. A start action must always come first in a table, while regular actions could only come after a start action, or another regular action. With the growth of automation use cases for EasyMorph, such separation introduced unnecessary limitations. For instance, importing a file is frequently done after copying it from another location. In such cases, importing had to be done in a separate project called via the “Call” action. This was inconvenient and cumbersome.

Starting from this release, this division is removed. From now on, any action can be first in a table, and any action can be inserted after any action. As the number of available actions in EasyMorph has become rather big (90+ actions as of this release), they have been separated into 5 categories, to simplify navigation:
  • Import – actions that import data (e.g. “Import from Excel”).
  • Create – actions that generate a table (e.g. “List of files”).
  • Transform – actions that modify input dataset (e.g. “Filter by expression”).
  • Export – actions that export data
  • Workflow – actions that affect internal workflows (e.g. “Iterate”), or interact with external systems and data (e.g. “Download file” or “Run program”).
If an action doesn’t expect any input data (typically actions from categories “Import” and “Create”), then the result of previous action is silently ignored without a warning or indication.
The equalization of actions has caused a few changes in the user interface of EasyMorph of which the most significant are two:
  1. The button “Insert/create table” used for selecting a start action and creating a table, has been removed. The “Add new action” button has moved to 1st position in the “Main” toolbar.
  2. The catalog of available actions is now two-tiered (it was one-tiered). Actions are grouped by categories and, optionally, by subcategories.

Column profiler

To simplify detection of data quality issues, the “Filter/Profile” dialog (invoked by double-clicking a column header) now implements a column profiler.

The profiler automatically calculates various counts. Each count has according “Filter” button to keep only rows with counted values.

The profiler also builds a histogram with distribution of numeric values in the column. The histogram allows filtering a range of values visually.

(click to zoom)

Notice that the profiler dialog window is floating meaning that you can click different columns in different tables without closing the profiler window.

Error handling

The "Call" action now has a new mode that allows capturing errors in the called project ("Iterate" has had this ability for a while):
The new mode makes it possible to arrange an error handling workflow in case of failure of the called project. For instance, send an email notification if the called project failed. The ability to handle project errors was frequently requested by EasyMorph users.

MS Exchange email connector

The generic SMTP email connector and the ability to send email notifications introduced in v3.7.1 sparked a lot of interest among EasyMorph users. However, in many cases sending emails through SMTP in MS Exchange was not possible due to Exchange settings or network connectivity issues.

In this release we're adding a native email connector for MS Exchange. Now, sending emails for organizations that use MS Exchange is not a problem.

New Plus licensing

The Plus license now always includes following features :
  • Unlimited actions
  • The “Input” action
  • The “Split delimited file” action
It also has become more expensive. See the new pricing at http://easymorph.com/buy.html.

Support for HTTPS in EasyMorph Server

EasyMorph Server now allows installing SSL certificates and has HTTPS-only mode. To simplify procurement and setup, EasyMorph Server license now includes 1 SSL certificate from a major certificate authority. 

Also, from now on version numbers for the Server are synchronized with desktop EasyMorph. Thus the latest Server version is also 3.8.

What's next

In a few weeks we will be releasing minor version 3.8.1 with a few new actions as well as data quality suggestions introduced in the Column Profiler. It will also have new error cards that will simplify debugging call/iteration errors.

In version 3.9 planned for release in early July we will continue improving automation capabilities in EasyMorph with the addition of two major features: transferring files via SFTP/SCP and receiving emails with attachments. This would make EasyMorph Server a very versatile tool for data collection and publishing.

Read also:
Release notes v3.8

Apr 19, 2018

Press-release: EasyMorph Announces Record Business Expansion and Adds 100th Customer

TORONTO, April 19, 2018 — EasyMorph Inc., a provider of self-service data preparation software for non-technical business users, announced record growth in its first quarter ending March 31, 2018. The company has achieved its highest revenue since its inception in 2015 and added its one-hundredth customer, a 300% increase in its customer base within the past year.

EasyMorph launched in 2015 with a vision to empower non-technical business users to easily merge, filter and manipulate data from multiple platforms and data sources without having to code or do complex programming. To date, there are over 15,000 downloads of the software with users in 20 different countries.

"We have added our one-hundredth customer in record time and I see it as a confirmation of EasyMorph's vision," said Dmitry Gudkov, Founder of EasyMorph Inc. "Reducing dependency on enterprise IT departments lets our business users solve data-related problems autonomously without delays and bureaucratic hassle."

EasyMorph's agile data preparation self-service solution helps non-technical business users to:
  • Extract and aggregate data from multiple platforms and data sources.
  • Slice and dice data, group information, calculate new metrics and more without having to code or do complex programming.
  • Automate routine file processing. 
  • Minimize their reliance on IT departments.
EasyMorph's powerful data preparation and automation capabilities can be used with any business intelligence or data visualization tool including Tableau dashboards, Qlik apps and Excel spreadsheets.

Interested parties can click here to download a free version of EasyMorph.

About EasyMorph

EasyMorph, Inc. provides self-service, agile data preparation at your fingertips. Business users can easily access, combine, prep, and manipulate data from multiple platforms and data sources with no coding required. EasyMorph is optimized for non-technical users to independently conquer and automate tedious data-related tasks quickly and efficiently.

Jan 22, 2018

What's new in version 3.7.1

With version 3.7.1 we continue expanding the automation capabilities in EasyMorph together with making data transformation more convenient. Key novelties include:

Sending emails

Sending email notifications is a cornerstone automation capability. There is a number of scenarios that have become possible due the introduction of the "Send email" action in EasyMorph:
  • Notification about failed data quality checks. A workflow can send an email before it gets halted.
  • Rule-based notifications. A Server task can be scheduled to check regularly certain conditions (e.g. existence of a file, consistency of data, or timeliness of new submissions). If a condition is not met, the task sends an according email notification.
  • Per-user data publishing. Sometimes, it's necessary to deliver each user a customized copy of a report or data extract. Sending an personalized email with an attachment could do the job.
  • Log monitoring. A scheduled project or task can parse logs and notify about possible errors or unwanted events.
The "Send email" action is highly configurable and allows specifying recipients, subject and attachment path using parameters. Parameters can also be dynamically inserted into the message body. The message text can be HTML formatted (only in Plus/Pro editions).


We're also working on making EasyMorph capable of receiving emails and saving attachments.

Triggering Tableau Server extract refreshes

This release introduced the "Tableau Server Command" action which employs the Tableau Server REST API to perform various operations (commands) on a remote Tableau Server. In version 3.7.1 the only command available is extract refreshing for a data source or a workbook. This command is available for Tableau Server version 10.3 and up.

With the expansion of the REST API new commands will become available in the action.


Easier creation of new parameters

Now it's easier to start using a parameter instead of a transformation property such as file name. For this, select <Add new parameter> in the list of parameters in the property, and a new parameter will be created automatically. The name and value of the new parameter will be defined after the transformation property it for which it was created.

For instance, when changing folder path to a parameter in the "List of files" transformation, the new parameter will automatically named "Folder path" and its value will be the folder path previously  in the transformation.


Other changes

  • The new "Metadata" transformation offers a convenient way to obtain a list of columns of a table, as well as the total number of rows and columns in it.
  • The "File Command" transformation now has a new mode for cloning existing files.
  • The "Convert Data Types" transformation has a new mode for automatically converting any data type in selected columns to text.
  • New connector "Email Server" in the Connector Manager.
  • The Start screen has a refreshed look with more choices to start a new workflow.
  • A new two-column selector is now displayed when selecting a project table, parameter, or connector. The selector allows searching and filtering that simplifies dealing with large projects.
  • Transformations Call, Iterate, and Iterate table now display parameter annotations.

Read also Release Notes v3.7.1.pdf

PS. Notice that our posts start using term action interchangeably with transformation. Eventually, in order to minimize confusion, we will reduce the use of term transformation only to the actions that actually transform data in EasyMorph.