tag:blogger.com,1999:blog-54274672327436434102024-02-07T21:55:42.247-05:00EasyMorph.BlogDmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comBlogger43125tag:blogger.com,1999:blog-5427467232743643410.post-58731687422709037642019-04-20T22:09:00.002-04:002019-04-20T22:11:04.185-04:00The blog is moving to the Community forumOur publishing was historically split between this blog (which appeared first), the <a href="https://community.easymorph.com/">EasyMorph Community forum</a> (which was created later), and our newsletter. This was confusing for our users, some of which didn't know about the Community while some didn't realize that there is a blog.<br />
<div>
<br /></div>
<div>
To simplify our communication, we're consolidating all our publishing activity on the Community forum. We will keep sending out the newsletter too, which will contain the best "How to" articles from the Community, and news about new features and updates (when applicable).</div>
<div>
<br /></div>
<div>
This blog will no longer be updated.</div>
<div>
<br /></div>
<div>
The kind of articles we used to publish on this blog are now published in <a href="https://community.easymorph.com/c/articles">the Articles category on the forum</a>.</div>
<div>
<br /></div>
<div>
Release notes are available on the <a href="https://easymorph.com/download.html">download page</a>, and cover both Desktop and Server.</div>
<div>
<br />
For RSS readers, subscribe to our RSS newsfeed: <a href="https://easymorph.com/feed.rss">https://easymorph.com/feed.rss</a>.</div>
<div>
<br /></div>
Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-66471599066107911622018-11-02T16:49:00.000-04:002018-11-02T16:49:48.686-04:004 reasons to start hosting datasets on EasyMorph ServerWe've just released EasyMorph v3.9.1 (both <a href="https://easymorph.com/download.html">Desktop</a> and <a href="https://easymorph.com/download-server.html">Server</a>). One of its major new features is the easy publishing of datasets from Desktop to Server and retrieving from Server back to Desktop. While it may look as a minor feature at first glance, there are 4 good reasons why you may want to start hosting your data on EasyMorph Server:<br />
<br />
<h3>
Easy publishing from Desktop to Server</h3>
When you keep datasets on Server instead of local drive, there is no need to use calculated parameters for paths to files in EasyMorph projects. Datasets will be loaded from the same location on Server no matter if your project is executed on Desktop or on Server. Therefore you can simply publish a project from Desktop to Server without any additional configuration of file locations.<br />
<br />
It works well even for large datasets. Because the dataset format (.dset) is highly compressed and close to the internal representation of data in EasyMorph, loading a big dataset from Server over a network can be much faster than loading an equivalent CSV file stored locally.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeDSKpqmCTP-fikKbrSkPZj5Kn7pW2Z-hocXD0GWAueG5sqoXzGiuMlVsIxVnLKq8cBlszNpvlM3ot5YrzUMDsbhPrV_aT4tifNIM1mE5nyA0rPQefE4j0gAV-cpv_tKmzAEbAJQ05234/s1600/easy-publishing.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="411" data-original-width="621" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeDSKpqmCTP-fikKbrSkPZj5Kn7pW2Z-hocXD0GWAueG5sqoXzGiuMlVsIxVnLKq8cBlszNpvlM3ot5YrzUMDsbhPrV_aT4tifNIM1mE5nyA0rPQefE4j0gAV-cpv_tKmzAEbAJQ05234/s1600/easy-publishing.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<h3>
Collaboration on datasets</h3>
Sending a interesting dataset to your colleague is now very easy: right-click the action which result you want to share, choose the "Send to Server" command, pick a folder on Server, and press "Publish". Then tell your colleague the folder location and the dataset name. The colleague right-clicks the Desktop workspace (the grey background behind tables), chooses the "Receive from Server" command, and receives the dataset in seconds, even if it contains millions of rows.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid1lmzUEsz_xhpdcau8Gie7fP10xJt5ED1DqTRnQ3mcPKVmSEfFyhZugwz_e2wJVxbfAuZ7VoWeNudiacAqNsE5TCB9kFc-Ra4Zz_4MHo1Hbdd4oRtRf1r-N1gJm7L7RFavQZaa33ltuc/s1600/sharing-dataset.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="379" data-original-width="403" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid1lmzUEsz_xhpdcau8Gie7fP10xJt5ED1DqTRnQ3mcPKVmSEfFyhZugwz_e2wJVxbfAuZ7VoWeNudiacAqNsE5TCB9kFc-Ra4Zz_4MHo1Hbdd4oRtRf1r-N1gJm7L7RFavQZaa33ltuc/s1600/sharing-dataset.png" /></a></div>
<br />
<h3>
Project portability</h3>
<div>
When datasets are hosted on Server, there is no need to send source files together with project if a user wants to send the project to another user. You can send only the project itself, and when the other user opens it, the source datasets will be loaded automatically from Server and the project will work with exactly the same data, as if you sent it together with source data files.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgng9Zbxdr_x4fwIMJAErUPyAKFT0FOlsA5DwphGmmXuEgHId0EcuFWLMHkidpxum2jazMQaIc6ulJGXEEGp5S_LE4ijLlW7ck_mkhOzksHff61mt_JtR6u05P3w9DBBabYrtuQOAVBGqU/s1600/sharing-project.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="381" data-original-width="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgng9Zbxdr_x4fwIMJAErUPyAKFT0FOlsA5DwphGmmXuEgHId0EcuFWLMHkidpxum2jazMQaIc6ulJGXEEGp5S_LE4ijLlW7ck_mkhOzksHff61mt_JtR6u05P3w9DBBabYrtuQOAVBGqU/s1600/sharing-project.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<h3>
Better security</h3>
<div>
Using Server for hosting source data instead of local files also works better from a security standpoint. Source data files have limited protection when you share them via a network folder or email message. However, when datasets are hosted on Server no one else will be able to access them unless they have been given access to the Server space that hosts the datasets.</div>
<div>
<br /></div>
Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-75113105694024127772018-08-30T20:30:00.000-04:002018-09-04T11:55:10.095-04:00What's new in version 3.9<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAbuZHUARIfJ1lL5UFp_HNsInTD8GVcAuPCV7mdQCKW-hwvce_pXS5ZpjY2i__gh2ws3BNVrMi_ZYnKcGVXWJKypCWH9kdloqQhhLZPIUsp60Xom-qC_iihT6_OuhIi3IVhtCfGETstS8/s1600/released.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="192" data-original-width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAbuZHUARIfJ1lL5UFp_HNsInTD8GVcAuPCV7mdQCKW-hwvce_pXS5ZpjY2i__gh2ws3BNVrMi_ZYnKcGVXWJKypCWH9kdloqQhhLZPIUsp60Xom-qC_iihT6_OuhIi3IVhtCfGETstS8/s1600/released.png" /></a></div>
<br />
EasyMorph version 3.9 is now available for downloading on our <a href="https://easymorph.com/download.html">website</a>. The new release adds powerful new automation capabilities to EasyMorph:<br />
<br />
<h3>
Fetching emails with attachments</h3>
Our goal is to make the everyday work of a data analyst as automated as possible so that tedious activities take less time, and more time can be spent on interesting and high-value work. Receiving data files by email and further processing them is a daily and monotonous routine for many people, but now it can be automated with EasyMorph. The "Fetch email" action introduced in v3.9 automatically retrieves email messages with attachments from Exchange as well as from mail servers that support IMAP (most modern mail services do).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTD0JIV_oWN39L6REbIAPIzKUtnU4e_jAP-ZGN3Mc1UZXtwiDESWpttRY6TlX3m9nkgwYzIAhC4X4FKsN2SYLjeA6b3etEWH-2j3ZnP_yjqeY9WGInpDDLcmQDjDVSqQccY_TXpyDBTPw/s1600/fetch+email.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="514" data-original-width="700" height="465" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTD0JIV_oWN39L6REbIAPIzKUtnU4e_jAP-ZGN3Mc1UZXtwiDESWpttRY6TlX3m9nkgwYzIAhC4X4FKsN2SYLjeA6b3etEWH-2j3ZnP_yjqeY9WGInpDDLcmQDjDVSqQccY_TXpyDBTPw/s640/fetch+email.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The "Fetch email" action (tabs "Pre-filter" and "Attachments").</td></tr>
</tbody></table>
<br />
Using this action, it is possible to automate the process of receiving data files from email attachments and then transforming them with further uploading into a database, or publishing them to a Tableau Server, or transferring to another machine through SFTP (more on this later).<br />
<br />
With a bit of creativity, more advanced scenarios can be accomplished, for instance:<br />
<br />
<ul>
<li>Sending an automated reminder if someone didn't send an expected data file in time.</li>
<li>Automatically verifying incoming data in email attachments and then sending back a request to resubmit the data if a data quality checks fails.</li>
<li>Using a 3rd party service to collect information through web-forms. The service sends form submissions by email which are automatically retrieved by EasyMorph, parsed, and uploaded into a database.</li>
<li>Performing a particular action (e.g. run program, or execute PowerShell script) when an email with specific keywords in the subject is received.</li>
</ul>
<div>
<br /></div>
<h3>
SFTP/SCP file transfer</h3>
<div>
It is now possible to automatically send to and receive files from a remote computer via SFTP (FTP over SSH) and SCP. Key pair authentication is supported as well as use of a proxy.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbwvMv3MNchSqY7cor5971GDuu6zDidLz9OD20BFY2qRr_YQRjlLBTjrmuyO5Xd3cYvIzeucgF-xvwIDS3tp8VvdnYyRiiDd2YnX4dG3OX-Ri9kzxrE6Z8KT6vgM_zvItnwKYemHkO5w/s1600/file-transfer.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="425" data-original-width="348" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbwvMv3MNchSqY7cor5971GDuu6zDidLz9OD20BFY2qRr_YQRjlLBTjrmuyO5Xd3cYvIzeucgF-xvwIDS3tp8VvdnYyRiiDd2YnX4dG3OX-Ri9kzxrE6Z8KT6vgM_zvItnwKYemHkO5w/s400/file-transfer.png" width="327" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The "File transfer" action.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Note that sending files by regular (non-secure) FTP is not supported.</div>
<div>
<br /></div>
<h3>
Data collection/publishing</h3>
<div>
With this release, EasyMorph (Desktop and Server) has become a versatile tool for automated data collection and publishing. It can now be done in multiple ways which cover the vast majority of cases in a corporate environment:</div>
<div>
<ul>
<li>Receive/send emails with attachments (Desktop and Server).</li>
<li>Transfer files to/from a remote computer via SFTP/SCP (Desktop and Server).</li>
<li>Upload/download files via a web-browser (Server only).</li>
<li>Transfer files between Desktop and Server over secure HTTPS connection using the "EasyMorph Server Command" action.</li>
<li>Transfer files between any computer and Server using <a href="https://github.com/easymorph/server-cmd">Server CLI</a> or <a href="https://github.com/easymorph/server-sdk">EasyMorph .NET SDK</a></li>
</ul>
<div>
<br />
<h3>
Native data format</h3>
This release introduces for the first time, the ability to save and load EasyMorph datasets into files in the native EasyMorph data format (.dset). The native data file format stores data in a form that is very close to the internal data representation in the EasyMorph’s in-memory engine. Therefore, saving a dataset into a file and reading a dataset from a file doesn’t have performance overhead issues such as data conversion or compression/decompression, and therefore is very fast.<br />
<br />
The native format compresses data. The compression ratio on par with Tableau Data Extracts (.tde) and Qlik QVDs (.qvd). The difference is typically within 10% deviation compared to one or another format depending on data type.<br />
<br />
Later we will add the ability to publish datasets to and load from the Server in one click. This will make the Server a convenient place for collaboration -- storing and sharing even large datasets with millions of rows.<br />
<br />
<h3>
Data Preparation University</h3>
This newsletter is the last one in its current form. Starting with out next newsletter, we're introducing a new format, our "Data Preparation University" where most of its content will be dedicated to practical advice on data preparation for beginners and experienced users. We will continue to include short overviews of new releases, however the main description of what's new will be in the release notes that accompany every new version.<br />
<br />
Links:<br />
<a href="https://easymorph.com/download.html">Download EasyMorph</a><br />
<a href="https://easymorph.com/download/release-notes/Release_notes_v3.9.pdf">Release Notes v3.9 (pdf)</a><br />
<br /></div>
</div>
<div>
<br /></div>
Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-40384993701519407732018-05-06T11:42:00.000-04:002018-05-06T15:38:12.649-04:00What's new in version 3.8<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQfblSLTed1wprP1KMuMoV78Ch3yqUl86Um0HkmkPUm3jtd07i-JISwnM46mQJh8vfQAlYO2GE5ax7mNSc-4z5Xbzk9GhmwcHKEL2M5MF6XkX9hH2Z-TrksDlWmUv8sX04eB5nJ5QanQg/s1600/released.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="192" data-original-width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQfblSLTed1wprP1KMuMoV78Ch3yqUl86Um0HkmkPUm3jtd07i-JISwnM46mQJh8vfQAlYO2GE5ax7mNSc-4z5Xbzk9GhmwcHKEL2M5MF6XkX9hH2Z-TrksDlWmUv8sX04eB5nJ5QanQg/s1600/released.png" /></a></div>
<br />
Version 3.8 has been released and is available for downloading on our <a href="https://easymorph.com/download.html">web-site</a>. Here is an overview of the most notable changes:<br />
<br />
<h4>
Transformations are now called actions</h4>
From the beginning, EasyMorph has had a number of transformations such as “Run Program” or “Export to Excel” that actually didn’t transform anything. As its automation capabilities keep expanding, new transformations “that don’t transform anything” keep adding, making the term “transformation” increasingly confusing. Therefore, starting from this release all transformations now called <i>actions</i>.<br />
<br />
For some time, both terms action and transformation will be used interchangeably, as we make relevant edits in the documentation, tutorial and other places. Eventually, the term “transformation” will only be used occasionally for actions that actually transform data.<br />
<div>
<br /></div>
<div>
<h4>
Arbitrary order of actions</h4>
<div>
Up until the previous release, all actions in EasyMorph were divided into two categories: start actions, and regular actions. A start action must always come first in a table, while regular actions could only come after a start action, or another regular action. With the growth of automation use cases for EasyMorph, such separation introduced unnecessary limitations. For instance, importing a file is frequently done after copying it from another location. In such cases, importing had to be done in a separate project called via the “Call” action. This was inconvenient and cumbersome.</div>
<div>
<br /></div>
<div>
Starting from this release, this division is removed. From now on, any action can be first in a table, and any action can be inserted after any action. As the number of available actions in EasyMorph has become rather big (90+ actions as of this release), they have been separated into 5 categories, to simplify navigation:</div>
<div>
<ul>
<li>Import – actions that import data (e.g. “Import from Excel”).</li>
<li>Create – actions that generate a table (e.g. “List of files”).</li>
<li>Transform – actions that modify input dataset (e.g. “Filter by expression”).</li>
<li>Export – actions that export data</li>
<li>Workflow – actions that affect internal workflows (e.g. “Iterate”), or interact with external systems and data (e.g. “Download file” or “Run program”).</li>
</ul>
</div>
<div>
If an action doesn’t expect any input data (typically actions from categories “Import” and “Create”), then the result of previous action is silently ignored without a warning or indication.</div>
<div>
The equalization of actions has caused a few changes in the user interface of EasyMorph of which the most significant are two:</div>
<div>
<ol>
<li>The button “Insert/create table” used for selecting a start action and creating a table, has been removed. The “Add new action” button has moved to 1st position in the “Main” toolbar.</li>
<li>The catalog of available actions is now two-tiered (it was one-tiered). Actions are grouped by categories and, optionally, by subcategories.</li>
</ol>
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD7S6CeUm1D4W_8HcvGHcmRbAYyFpGf2mzbELObq7h3HYqiVx8XMq0QQhC98xdQTVQM2xdRHaZ0cGBlJr0qT2wT3zVjGTT3fx70S9Gigdfh96qKcLiIPCGBrKG7Xzla8URZ4cRUSanFu0/s1600/2018-05-06+09_45_04-Release_notes_v3.8.docm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="399" data-original-width="353" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD7S6CeUm1D4W_8HcvGHcmRbAYyFpGf2mzbELObq7h3HYqiVx8XMq0QQhC98xdQTVQM2xdRHaZ0cGBlJr0qT2wT3zVjGTT3fx70S9Gigdfh96qKcLiIPCGBrKG7Xzla8URZ4cRUSanFu0/s1600/2018-05-06+09_45_04-Release_notes_v3.8.docm.png" /></a></div>
<div>
<br /></div>
<h4>
Column profiler</h4>
<div>
<div>
To simplify detection of data quality issues, the “Filter/Profile” dialog (invoked by double-clicking a column header) now implements a column profiler.</div>
<div>
<br /></div>
<div>
The profiler automatically calculates various counts. Each count has according “Filter” button to keep only rows with counted values.</div>
<div>
<br /></div>
<div>
The profiler also builds a histogram with distribution of numeric values in the column. The histogram allows filtering a range of values visually.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUjSMXvAO9XIls93on9YOtTZqeZhfgjUjm3Vf4rTgmtVVWdMbOzZQSTjnG7Ib4KHGF9s9Mjpjcko3HRjC8Gpe8lTh8jUL_kPgrm-QIunuf1SmT2ujksiAsyh7a3nYNh46SEFvd8qPrrGk/s1600/column-profiler.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="513" data-original-width="688" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUjSMXvAO9XIls93on9YOtTZqeZhfgjUjm3Vf4rTgmtVVWdMbOzZQSTjnG7Ib4KHGF9s9Mjpjcko3HRjC8Gpe8lTh8jUL_kPgrm-QIunuf1SmT2ujksiAsyh7a3nYNh46SEFvd8qPrrGk/s400/column-profiler.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Notice that the profiler dialog window is floating meaning that you can click different columns in different tables without closing the profiler window.</div>
</div>
<div>
<br /></div>
<h4>
Error handling</h4>
<div>
The "Call" action now has a new mode that allows capturing errors in the called project ("Iterate" has had this ability for a while):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasg5clEmVA7zcbLHzpryPLR6e8o7f5GhY07hLYiJAyqZsazPqk53MzsvwKuZIH_-W7XmDJodXfxu72IXeIx2mYCoKgHvTW3YiFhJW_QuurIemVAECvjwLjRHFUwgYg8pzzOmnDYMJ51E/s1600/call-new-mode.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="428" data-original-width="345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasg5clEmVA7zcbLHzpryPLR6e8o7f5GhY07hLYiJAyqZsazPqk53MzsvwKuZIH_-W7XmDJodXfxu72IXeIx2mYCoKgHvTW3YiFhJW_QuurIemVAECvjwLjRHFUwgYg8pzzOmnDYMJ51E/s1600/call-new-mode.png" /></a></div>
<div>
The new mode makes it possible to arrange an error handling workflow in case of failure of the called project. For instance, send an email notification if the called project failed. The ability to handle project errors was frequently requested by EasyMorph users.</div>
<div>
<br /></div>
<h4>
MS Exchange email connector</h4>
<div>
The generic SMTP email connector and the ability to send email notifications introduced in v3.7.1 sparked a lot of interest among EasyMorph users. However, in many cases sending emails through SMTP in MS Exchange was not possible due to Exchange settings or network connectivity issues.</div>
<div>
<br /></div>
<div>
In this release we're adding a native email connector for MS Exchange. Now, sending emails for organizations that use MS Exchange is not a problem.</div>
<div>
<br /></div>
<div>
<h4>
New Plus licensing</h4>
<div>
The Plus license now always includes following features :</div>
<div>
<ul>
<li>Unlimited actions</li>
<li>The “Input” action</li>
<li>The “Split delimited file” action</li>
</ul>
</div>
<div>
It also has become more expensive. See the new pricing at <a href="http://easymorph.com/buy.html">http://easymorph.com/buy.html</a>.</div>
</div>
<div>
<br /></div>
<h4>
Support for HTTPS in EasyMorph Server</h4>
<div>
EasyMorph Server now allows installing SSL certificates and has HTTPS-only mode. To simplify procurement and setup, EasyMorph Server license now includes 1 SSL certificate from a major certificate authority. </div>
<div>
<br /></div>
<div>
Also, from now on version numbers for the Server are synchronized with desktop EasyMorph. Thus the latest Server version is also 3.8.</div>
<div>
<br /></div>
<h4>
What's next</h4>
<div>
In a few weeks we will be releasing minor version 3.8.1 with a few new actions as well as <a href="https://twitter.com/easy_morph/status/986661965782241280">data quality suggestions</a> introduced in the Column Profiler. It will also have new error cards that will simplify debugging call/iteration errors.</div>
<div>
<br /></div>
<div>
In version 3.9 planned for release in early July we will continue improving automation capabilities in EasyMorph with the addition of two major features: transferring files via SFTP/SCP and receiving emails with attachments. This would make EasyMorph Server a very versatile tool for data collection and publishing.<br />
<br />
Read also:<br />
<a href="https://easymorph.com/download/release-notes/Release_notes_v3.8.pdf">Release notes v3.8</a></div>
Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-38774516709856975382018-04-19T07:45:00.000-04:002018-04-20T07:46:55.364-04:00Press-release: EasyMorph Announces Record Business Expansion and Adds 100th CustomerTORONTO, April 19, 2018 — <a href="https://easymorph.com/">EasyMorph Inc</a>., a provider of self-service data preparation software for non-technical business users, announced record growth in its first quarter ending March 31, 2018. The company has achieved its highest revenue since its inception in 2015 and added its one-hundredth customer, a 300% increase in its customer base within the past year.<br />
<br />
EasyMorph launched in 2015 with a vision to empower non-technical business users to easily merge, filter and manipulate data from multiple platforms and data sources without having to code or do complex programming. To date, there are over 15,000 downloads of the software with users in 20 different countries.<br />
<br />
"We have added our one-hundredth customer in record time and I see it as a confirmation of EasyMorph's vision," said Dmitry Gudkov, Founder of EasyMorph Inc. "Reducing dependency on enterprise IT departments lets our business users solve data-related problems autonomously without delays and bureaucratic hassle."<br />
<br />
EasyMorph's agile data preparation self-service solution helps non-technical business users to:<br />
<ul><li>Extract and aggregate data from multiple platforms and data sources.</li>
<li>Slice and dice data, group information, calculate new metrics and more without having to code or do complex programming.</li>
<li>Automate routine file processing. </li>
<li>Minimize their reliance on IT departments.</li>
</ul>EasyMorph's powerful data preparation and automation capabilities can be used with any business intelligence or data visualization tool including Tableau dashboards, Qlik apps and Excel spreadsheets.<br />
<br />
Interested parties can <a href="http://easymorph.com/download.html">click here</a> to download a free version of EasyMorph.<br />
<br />
<h3>About EasyMorph</h3><a href="https://easymorph.com/">EasyMorph, Inc.</a> provides self-service, agile data preparation at your fingertips. Business users can easily access, combine, prep, and manipulate data from multiple platforms and data sources with no coding required. EasyMorph is optimized for non-technical users to independently conquer and automate tedious data-related tasks quickly and efficiently.Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-91123171240391841882018-01-22T01:23:00.000-05:002018-01-22T10:22:28.780-05:00What's new in version 3.7.1With version 3.7.1 we continue expanding the automation capabilities in <a href="http://easymorph.com/">EasyMorph</a> together with making data transformation more convenient. Key novelties include:<br />
<br />
<h3>
Sending emails</h3>
Sending email notifications is a cornerstone automation capability. There is a number of scenarios that have become possible due the introduction of the "Send email" action in EasyMorph:<br />
<ul>
<li>Notification about failed data quality checks. A workflow can send an email before it gets halted.</li>
<li>Rule-based notifications. A <a href="http://easymorph.com/server.html">Server</a> task can be scheduled to check regularly certain conditions (e.g. existence of a file, consistency of data, or timeliness of new submissions). If a condition is not met, the task sends an according email notification.</li>
<li>Per-user data publishing. Sometimes, it's necessary to deliver each user a customized copy of a report or data extract. Sending an personalized email with an attachment could do the job.</li>
<li>Log monitoring. A scheduled project or task can parse logs and notify about possible errors or unwanted events.</li>
</ul>
The "Send email" action is highly configurable and allows specifying recipients, subject and attachment path using parameters. Parameters can also be dynamically inserted into the message body. The message text can be HTML formatted (only in <a href="http://easymorph.com/buy.html">Plus/Pro editions</a>).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIJ1u3h1ejCQELd0f_CLrtkiqJoyE6w4swN3gsgHPdTN6YkEwDEhI8_yQpDowVPym42SFbpRRPLX0v8d46UmCouKyS9oiV3Raf1IZGr6twGEyA4obbVydlFsS6tnWjONrBgz-fN-lKGQA/s1600/send-email.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIJ1u3h1ejCQELd0f_CLrtkiqJoyE6w4swN3gsgHPdTN6YkEwDEhI8_yQpDowVPym42SFbpRRPLX0v8d46UmCouKyS9oiV3Raf1IZGr6twGEyA4obbVydlFsS6tnWjONrBgz-fN-lKGQA/s1600/send-email.png" /></a></div>
<br />
We're also working on making EasyMorph capable of <i>receiving</i> emails and saving attachments.<br />
<br />
<h3>
Triggering Tableau Server extract refreshes</h3>
This release introduced the "Tableau Server Command" action which employs the <a href="https://onlinehelp.tableau.com/current/api/rest_api/en-us/help.htm">Tableau Server REST API</a> to perform various operations (commands) on a remote Tableau Server. In version 3.7.1 the only command available is extract refreshing for a data source or a workbook. This command is available for Tableau Server version 10.3 and up.<br />
<br />
With the expansion of the REST API new commands will become available in the action.<br />
<br />
<br />
<h3>
Easier creation of new parameters</h3>
Now it's easier to start using a parameter instead of a transformation property such as file name. For this, select <Add new parameter> in the list of parameters in the property, and a new parameter will be created automatically. The name and value of the new parameter will be defined after the transformation property it for which it was created.<br />
<br />
For instance, when changing folder path to a parameter in the "List of files" transformation, the new parameter will automatically named "Folder path" and its value will be the folder path previously in the transformation.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9QCQmD_BUlkBr14paXiQN3TdFoXwhQlr2bRw7kzsRJxs232wuX7aZGN56Gz7EPRZ_AFm-U31zYMzyQmbbtX4lEU71lzcGa_rrYADMIMXTOQJPtrVOH28vgCAAO2c_I3HyQMRWUQ82QKk/s1600/new+param.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="243" data-original-width="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9QCQmD_BUlkBr14paXiQN3TdFoXwhQlr2bRw7kzsRJxs232wuX7aZGN56Gz7EPRZ_AFm-U31zYMzyQmbbtX4lEU71lzcGa_rrYADMIMXTOQJPtrVOH28vgCAAO2c_I3HyQMRWUQ82QKk/s1600/new+param.png" /></a></div>
<br />
<h3>
Other changes</h3>
<ul>
<li>The new "Metadata" transformation offers a convenient way to obtain a list of columns of a table, as well as the total number of rows and columns in it.</li>
<li>The "File Command" transformation now has a new mode for cloning existing files.</li>
<li>The "Convert Data Types" transformation has a new mode for automatically converting any data type in selected columns to text.</li>
<li>New connector "Email Server" in the Connector Manager. </li>
<li>The Start screen has a refreshed look with more choices to start a new workflow.</li>
<li>A new two-column selector is now displayed when selecting a project table, parameter, or connector. The selector allows searching and filtering that simplifies dealing with large projects.</li>
<li>Transformations Call, Iterate, and Iterate table now display parameter annotations.</li>
</ul>
<br />
Read also <a href="http://easymorph.com/download/release-notes/Release_notes_v3.7.1.pdf">Release Notes v3.7.1.pdf</a><br />
<br />
PS. Notice that our posts start using term <i>action</i> interchangeably with <i>transformation</i>. Eventually, in order to minimize confusion, we will reduce the use of term <i>transformation</i> only to the actions that actually transform data in EasyMorph.Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-75992106590963473132017-12-25T14:36:00.000-05:002018-01-16T20:52:37.668-05:00EasyMorph doubles down on data-aware task automation<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3DP0ithaYYlqRgFC9IpHgEGph5rgkQsNP9_8PXVaJHzOedrsZzQ7F9qrReDVNKouxVHORLjyn9hs64_GcGRWzJiYuCOSoslkGaD0qsG4jkFebY9EVqC21zNWMh4H9FpfIzo_0e_ZU98Y/s1600/task-automation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="399" data-original-width="600" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3DP0ithaYYlqRgFC9IpHgEGph5rgkQsNP9_8PXVaJHzOedrsZzQ7F9qrReDVNKouxVHORLjyn9hs64_GcGRWzJiYuCOSoslkGaD0qsG4jkFebY9EVqC21zNWMh4H9FpfIzo_0e_ZU98Y/s400/task-automation.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<a href="http://easymorph.com/">EasyMorph</a> has already been offering some task automation capabilities (e.g. parameters and loops, running external programs) that are missing in less sophisticated data preparation tools. We've been witnessing large data analytics vendors <a href="http://bi-review.blogspot.ca/2017/10/tableau-maestro-vs-3rd-party-data-prep.html">entering the data preparation market</a> after realizing that <i>good data analysis requires good data preparation</i>. However, we at EasyMorph understand that good data preparation almost always requires good <i>automation</i>. Manual data preparation is not enough anymore because it increasingly takes too much time. More and more internal and external systems get involved into data analysis every day. More and more public data APIs opened. More and more files are received, processed, sent out, or uploaded. Traditional batch scripting doesn't help much because it's not <i>data-aware</i>. Too frequently data-related workflows depend on data itself. For instance, before processing a file extract it may be necessary to obtain the report date from the extract contents, because it's not available in the file name or timestamp. Or, data quality has to be verified before a file is copied, or uploaded into a database automatically. Sometimes data quality verification can require querying multiple data sources. Such logic can be non-trivial to arrange with scripting.<br />
<br />
With almost 100 transformations (as of ver.3.7), EasyMorph is already well positioned for the new reality as it combines comprehensive data transformation features with certain elements of automation. However, we're doing a step further and doubling down on the automation capabilities in EasyMorph. The goal is to provide a convenient mechanism for automating routine data-related tasks, even those that might not be classified as data transformation at all, e.g. automated rule-based alerts and file transfers.<br />
<br />
Some of the new features are already available in version 3.7 released recently, but more is coming. Here is what to expect:<br />
<ul>
<li><a href="http://blog.easymorph.com/2017/12/conditional-workflows-in-easymorph.html">Conditional IF/THEN/ELSE workflows</a> (already available in 3.7)</li>
<li>Deep <a href="http://blog.easymorph.com/2017/12/running-powershell-commands-in-easymorph.html">PowerShell integration</a> (already available in 3.7) that opens the vast .NET ecosystem as well as Windows automation for EasyMorph users. With it you can, for instance, trigger VBA macros in spreadsheets, or fetch email attachments from your Outlook folder right from EasyMorph projects.</li>
<li>In version 3.8 we will be lifting the existing restrictions on transformation order. Currently an import transformation must always come first in a table. This restriction will be removed and any transformation will be possible to insert before or after any transformation. Tables will be able to start with a workflow transformation such as Download File or Run Program, and then have an import transformation.</li>
<li>In 3.9 <a href="http://easymorph.com/scheduler.html">EasyMorph Scheduler</a> will evolve into EasyMorph Launcher -- a handy utility residing in the system tray with a list of frequently used tasks that either run on schedule or are triggered manually. You will be able to launch in two clicks daily routines, such as transferring files between computers, fetching updates from Amazon AWS, uploading data and refreshing Tableau Server workbooks or Qlik Server dashboards, triggering EasyMorph Server tasks, sending out emails and even text messages (SMS, for those in Europe). Think of it as high-level replacement for batch scripts.</li>
<li>Triggering various actions through public APIs in external and internal systems. For instance, extract refreshing in Tableau Server (coming in v3.7.1), reloading apps in Qlik Sense Server (coming in v3.7.2), various actions with Amazon AWS, Azure, Google Apps, Slack and other cloud applications.</li>
<li>Easier iterations (loops) that won't require creating another project (v4.0). </li>
<li>Event-based task triggering (e.g on file change), automatic email retrieval and attachment processing in <a href="http://easymorph.com/server.html">EasyMorph Server.</a></li>
<li>Error handling.</li>
<li>Transformations renamed to <i>actions</i>.</li>
</ul>
<br />
All these changes will be gradually rolled out during 2018 with some features coming sooner, and some later. As always, we're keeping ears open for your <a href="https://community.easymorph.com/c/feature-requests">suggestions</a>.<br />
<br />
Of course, expanding into data-aware task automation doesn't mean that the "traditional" data transformation part in EasyMorph will be neglected. There are many new ETL-related features on the 2018 roadmap but that would be a topic of another post.<br />
<br />
<a href="https://community.easymorph.com/t/blog-easymorph-doubles-down-task-automation/381">Discuss this article on EasyMorph Community</a>Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-55526411402239755072017-12-18T11:20:00.000-05:002017-12-18T11:21:25.126-05:00What's new in EasyMorph Server 1.3<h4>
Spaces</h4>
Previously all users could access all Server tasks as well as files in the public folder. Starting from this release <a href="http://easymorph.com/server.html">EasyMorph Server</a> introduces <i>spaces </i>in order to offer more granularity and control in managing access to tasks and files.<br />
<br />
Each space is independent from other spaces, and can have its own:<br />
<ul>
<li>tasks </li>
<li>public web-folder</li>
<li>connector repository </li>
</ul>
<br />
Spaces are managed in a new tab called “Spaces” visible only to the Server administrator.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPKC0kXkHbfsbztEHcegEMdNW7oyPtsiFPQNrBVLzJwTn7lSSXm4FUparQSPAZyPic1esPPTzHHxvKkf3M3wibzRYaJ6WcuMgFGBSGg1BNVrBvHqxDQudvF0gPOqV3z4ya2cpO6FHnoJk/s1600/server-spaces.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="532" data-original-width="746" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPKC0kXkHbfsbztEHcegEMdNW7oyPtsiFPQNrBVLzJwTn7lSSXm4FUparQSPAZyPic1esPPTzHHxvKkf3M3wibzRYaJ6WcuMgFGBSGg1BNVrBvHqxDQudvF0gPOqV3z4ya2cpO6FHnoJk/s400/server-spaces.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Server spaces</td></tr>
</tbody></table>
<br />
Spaces allow limiting access to tasks using one of the 4 modes:<br />
<ul>
<li>Disabled – tasks are entirely disabled and not available in this particular space</li>
<li>Lock task – a user can only trigger tasks, but not create or edit them</li>
<li>Lock project – a user can edit task schedule, task parameters, but not create tasks, or change the project in a task.</li>
<li>Full access – no restrictions, the only mode where new tasks can be created.</li>
</ul>
<br />
A combination of task access mode with one of the file access modes introduced earlier allows flexibly configuring spaces for different types of users and use cases. See the table below for a few examples:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQvuU_fKjZAtxSZG0MX4lRqKZnQrCFaMBoebCgs_NN0s0RJspuZAikKroljTu810i-a8mtIVTiyZqNUNq0xemg4IKLM9EJwEQ6JYu0HFRyZO3cwuywvvGrnSOF6J-z2vOLsoqMNuC6Vg8/s1600/space-scenarios.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="224" data-original-width="766" height="184" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQvuU_fKjZAtxSZG0MX4lRqKZnQrCFaMBoebCgs_NN0s0RJspuZAikKroljTu810i-a8mtIVTiyZqNUNq0xemg4IKLM9EJwEQ6JYu0HFRyZO3cwuywvvGrnSOF6J-z2vOLsoqMNuC6Vg8/s640/space-scenarios.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Examples of spaces with different roles</td></tr>
</tbody></table>
<br />
<h4>
Password-protected spaces</h4>
A space can be password-protected. In this case, a user should provide correct password in order to access tasks and/or files.<br />
<br />
Performing operations through the API with password-protected spaces also requires providing a password. The <a href="https://github.com/easymorph/server-cmd">command-line client</a> and <a href="https://github.com/easymorph/server-sdk">EasyMorph Server .NET SDK</a> have been updated accordingly.<br />
<br />
<h4>
Remote administration</h4>
By default, the Server settings and space configuration is only allowed from localhost. However, it is now possible to enable remote administration. In this case, it is possible to configure the Server from a remote computer, after providing a valid administrator password.<br />
<br />
<h4>
Start/stop batch scripts</h4>
It is now possible to execute a batch script when the Server service is starting or stopping. Modify <b>onstart.bat</b> or <b>onstop.bat</b> located in <i>C:\Program Files\EasyMorph Server\systemscripts</i> accordingly.<br />
<br />
This capability can be used, for instance, to map additional network drives on server start, or to send an email notification when the Server service is shutting down or rebooting.<br />
<br />Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-70268251914178362872017-12-09T00:19:00.000-05:002018-01-16T20:53:41.521-05:00Conditional workflows in EasyMorphA conditional workflow is a sequence of actions that changes depending on a condition. For instance: IF <i>loaded data has no missing dates</i> THEN <i>export it into database</i> ELSE <i>send an email to Pete</i>. If we depict this workflow graphically, it would look as below:<br />
<br />
When no dates missing, sending an email doesn't happen.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeH7j5-Oz21JZ6MitHfje9ZoHP02kZS3GS0xmxWGdFOA1eoLIo8FJqeHTbGYPCgH__zneFLsWuVw5XFl5ag-eREl6sYUmoAH0tHm7eiC4Sx7Lj83yG41ZpW3VtlZohnRxaj9O4kmDXPVQ/s1600/conditional-workflow2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="320" data-original-width="532" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeH7j5-Oz21JZ6MitHfje9ZoHP02kZS3GS0xmxWGdFOA1eoLIo8FJqeHTbGYPCgH__zneFLsWuVw5XFl5ag-eREl6sYUmoAH0tHm7eiC4Sx7Lj83yG41ZpW3VtlZohnRxaj9O4kmDXPVQ/s1600/conditional-workflow2.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Workflow when no dates missed</td></tr>
</tbody></table>
<br />
When one or more dates are missing, exporting to database doesn't happen.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKj5DWXyb2idPISLebQzkMbd8mCpKfNka1bZc45F5Y_77Ic6_rJG_HLuQ0Lumr8Z5kCKl_T7uKb-ypGDZu8fi30KTnRbKom3GNQcgCpBL7MX5T76a_YoklbfPtjyUx41yFEUejSahqNFY/s1600/conditional-workflow1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="341" data-original-width="545" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKj5DWXyb2idPISLebQzkMbd8mCpKfNka1bZc45F5Y_77Ic6_rJG_HLuQ0Lumr8Z5kCKl_T7uKb-ypGDZu8fi30KTnRbKom3GNQcgCpBL7MX5T76a_YoklbfPtjyUx41yFEUejSahqNFY/s1600/conditional-workflow1.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Workflow when at least one date is missing.</td></tr>
</tbody></table>
<br />
<h3>
Branching </h3>
Such conditional workflow (branching) is arranged in <a href="http://easymorph.com/">EasyMorph</a> using <i>conditionally derived tables</i>. A regular derived table is a table which replicates the resulting state (data) of its source table, and performs certain transformations with it as well as other actions. If you're not familiar with derived tables check out this <a href="http://easymorph.com/learn/derived-tables.html">tutorial article</a>.<br />
<br />
A <i>conditionally</i> derived table does exactly the same, except it only does it when a certain condition is satisfied. If not, then no data is replicated from its source table, and no actions/transformations performed -- all actions and transformations in such table are skipped.<br />
<br />
The example mentioned above would look slightly differently, if we use branching similar to conditionally derived tables in EasyMorph:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG2NYAupyLjca-iQYhSXe-sauaBZNlul_5pToQ87VimfhIloYSAZYKyQAWZhm3RIjOcdHQ4yDVSofoaXykf_hmlt1yIIThfyhiU2XeUp7R9UmT8cKSSWHA622SR9y13nLy73uGjCQ0ad0/s1600/conditional-workflow3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="388" data-original-width="604" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG2NYAupyLjca-iQYhSXe-sauaBZNlul_5pToQ87VimfhIloYSAZYKyQAWZhm3RIjOcdHQ4yDVSofoaXykf_hmlt1yIIThfyhiU2XeUp7R9UmT8cKSSWHA622SR9y13nLy73uGjCQ0ad0/s1600/conditional-workflow3.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Branching using conditionally derivation in EasyMorph.</td></tr>
</tbody></table>
<br />
<br />
Here in this example we have 2 conditionally derived tables. The conditions in the tables are <i>inverse</i> to each other: in one table the condition is "missing a date?" while in the other it's "NOT missing a date?". Therefore, depending on whether the loaded data is incomplete or not, either one table is entirely skipped or the other.<br />
<br />
In an EasyMorph project the example described above would look as follows:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj80u07muuSUSnImUAe2HVvx58IewIhjdfi1z-dX3AmTVRWMh1fuO1WXcfV_up4ntoY3Fah0TFDnQHoyFh0jnEoUeuNQzcA3l_koysmk4maFtt2eHO2DAHnswqVCnHlqXnJCb9i7hyeddM/s1600/conditional-workflow4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="355" data-original-width="863" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj80u07muuSUSnImUAe2HVvx58IewIhjdfi1z-dX3AmTVRWMh1fuO1WXcfV_up4ntoY3Fah0TFDnQHoyFh0jnEoUeuNQzcA3l_koysmk4maFtt2eHO2DAHnswqVCnHlqXnJCb9i7hyeddM/s640/conditional-workflow4.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">EasyMorph project with a conditional workflow.</td></tr>
</tbody></table>
<br />
<br />
The two rightmost derived table are derived conditionally. Notice that one of the tables is skipped -- you can see it has different title bar color, and a derivation icon with red "X". In this table transformations were not executed, and it has no data.<br />
<br />
This project detects if some dates are missing or not by calculating the difference between adjacent dates. If no dates are skipped then the difference with previous date will always be equal to 1 for all dates. If one or more dates are skipped somewhere, then for the following date the distance will be 2 or more. We calculate the max distance, and merge it into the main table as a new column using the "Peek" transformation. Later, prior to exporting, the column is removed.<br />
<br />
In the conditionally derived tables the conditions are as follows:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[Distance, days] = 1</span>, for the upper derived table "Export to DB".<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[Distance, days] <> 1</span>, for the lower derived table "Send an email".<br />
<br />
You can see that conditions are inverse to each other.Therefore, either one or the other derived table is skipped, but never the both are skipped or executed simultaneously.<br />
<br />
<h3>
Conditions</h3>
The "Derive table" transformation has a switch that tells it whether it should be unconditional or conditional. In the latter case, there are three possible conditions for derivation:<br />
<ul>
<li>If expression evaluates to TRUE for <b>each</b> row in table</li>
<li>If the source table is empty</li>
<li>If the source table is NOT empty </li>
</ul>
In the screenshot below you can see properties of the "Derive table" transformation in the conditional mode:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyw5Ih8HHQHd31RzUW4frPhMAeq4R__bfee5qyV1YSSJYJhmbE6qHkKVCK4WwKP8Ptgm_Ke3U58n1K4dsCoFHZwesWW4na0rVI9B3g-r2ubsVwgpnjgfQu6__EPTquGsB8roZ6jK4vtuk/s1600/conditionally+derived+table.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="448" data-original-width="340" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyw5Ih8HHQHd31RzUW4frPhMAeq4R__bfee5qyV1YSSJYJhmbE6qHkKVCK4WwKP8Ptgm_Ke3U58n1K4dsCoFHZwesWW4na0rVI9B3g-r2ubsVwgpnjgfQu6__EPTquGsB8roZ6jK4vtuk/s400/conditionally+derived+table.png" width="302" /></a></div>
<br />
<h3>
Continuation</h3>
If after the branching execution should continue regardless of which of the derived tables was executed, then we need to join the two branched flows back into one flow. This can be done using the "Either table" transformation. The transformation compares two tables and picks one that is not empty. Since a skipped conditionally derived table is always empty (i.e. has no data), the "Either table" transformation will always replicate the resulting dataset of the table where the condition was satisfied and transformation were executed.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc33xxXbHF28Sa_z2_Gasz-PSTx_WYTLqTRwyicM2kr2ZnPzZXE54phGiuD_Zyhm51z2Ut6-ii00zTu7-Aui83r_xXaESZuxeMNTxaSoeZCRdwnvV0l97LhbJv35vcFwe9qD06DCul9gY/s1600/conditional-workflow5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="377" data-original-width="1120" height="214" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc33xxXbHF28Sa_z2_Gasz-PSTx_WYTLqTRwyicM2kr2ZnPzZXE54phGiuD_Zyhm51z2Ut6-ii00zTu7-Aui83r_xXaESZuxeMNTxaSoeZCRdwnvV0l97LhbJv35vcFwe9qD06DCul9gY/s640/conditional-workflow5.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Conditional workflow with continuation.</td></tr>
</tbody></table>
<br />
<br />
In some cases, there is only one branch before continuation. In this case the continuation is derived right from the source table (see below). Make sure that "Either table" is set up to give preference to the conditionally derived table if the source table is never empty.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Kk3c0QqdIwTqZmiIkDl2mUSsowuxChNVWZDybd0Hkh9B7ydktop8voRiqVbFZnesvI9tlF1NAtDdg7juFJo8C5NsGiOV8OcTjTEyhFPDj7v3MXo2eSdYCjaEo2kA7bBLD73ncTk3tUM/s1600/conditional-workflow6.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="593" data-original-width="1127" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Kk3c0QqdIwTqZmiIkDl2mUSsowuxChNVWZDybd0Hkh9B7ydktop8voRiqVbFZnesvI9tlF1NAtDdg7juFJo8C5NsGiOV8OcTjTEyhFPDj7v3MXo2eSdYCjaEo2kA7bBLD73ncTk3tUM/s640/conditional-workflow6.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Workflow with one conditionally derived table and continuation.</td></tr>
</tbody></table>
<br />
The examples about show using conditionally derived tables for IF...THEN...ELSE type of workflows. However, in the same fashion it is possible to arrange SWITCH...CASE type of branching that produces 3 or more branches.<br />
<br />
<br />Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-60841797117664296702017-12-04T22:25:00.000-05:002017-12-05T10:09:47.328-05:00Running PowerShell commands in EasyMorphStarting from version 3.7 EasyMorph includes the "PowerShell" transformation. The transformation allows executing arbitrary PowerShell commands and scripts with a degree of integration with EasyMorph. The integration includes:<br />
<ul>
<li>Using EasyMorph parameters in PowerShell commands</li>
<li>Capturing output and error sequences of a PowerShell command back into EasyMorph </li>
<li>Sending columns values as an input sequence for the PowerShell command pipeline</li>
</ul>
For the transformation and integration to work the host computer must have PowerShell v.3 (or above) installed.<br />
<br />
<h3>
Inserting parameters</h3>
Just like in the "Run Program" transformation, in "Powershell" it's possible to insert project parameters in curly braces right into the command text. For instance in the example below the command copies a file, which is specified by project parameter {Source file}, into folder <span style="font-family: "courier new" , "courier" , monospace;">C:\test</span>. Double quotes added just in case the file path contains spaces.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Copy-Item "{Source file}" c:\test</span><br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVh-fSQulBiQ0BiZefCq84arrY0uj8NGgcR1ja50DBqegc8X3uz4FgX_w0PvVaAtKl2ozyyt_HhCNZ0j4yr1kmgN9XG-0cuOn8lW7UV7uaWT31IwnGl4Y9W6aj82Id44_5xW5mgl_w3B4/s1600/powershell-insert-parameters.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="455" data-original-width="902" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVh-fSQulBiQ0BiZefCq84arrY0uj8NGgcR1ja50DBqegc8X3uz4FgX_w0PvVaAtKl2ozyyt_HhCNZ0j4yr1kmgN9XG-0cuOn8lW7UV7uaWT31IwnGl4Y9W6aj82Id44_5xW5mgl_w3B4/s640/powershell-insert-parameters.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Inserting project parameters into PowerShell command (click to zoom)</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Note that parameter values are inserted only when the text between curly braces is an existing parameter name. If no parameter with such name then the curly braces and text remain as they are and no error generated. This is convenient because PowerShell commands frequently use curly braces for various expressions, therefore you can mix EasyMorph project parameters with PowerShell expressions.<br />
<br />
<span style="font-size: large;"></span><br />
<h3>
Capturing output and errors</h3>
Output sequence of a PowerShell command can be captured back into EasyMorph. However, capturing in "PowerShell" works differently than in "Run Program" where the console output is captured. In the "PowerShell" transformation, capturing means importing the output sequence of the PowerShell command pipeline, not the host console output.<br />
<br />
In the example below a PowerShell command is used to obtain a list of running Windows services, and import it into EasyMorph.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Get-Service |<br />Where-Object {$_.Status -eq "Running"} |</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ForEach-Object -Member DisplayName</span><br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitPIW7cZuVUdVhh1uiDunBHFM0k_LkM79jBU4tscljbXIvIyY0omdHrpJR69a5yv8m-bBYgIHEsy-AUdyCRu6aRHVZAu0BoHT-Mr667yEqk971uSYISQ2QBKfIc6LyR8H9wyS74ynvt70/s1600/powershell-input-sequence.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="457" data-original-width="760" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitPIW7cZuVUdVhh1uiDunBHFM0k_LkM79jBU4tscljbXIvIyY0omdHrpJR69a5yv8m-bBYgIHEsy-AUdyCRu6aRHVZAu0BoHT-Mr667yEqk971uSYISQ2QBKfIc6LyR8H9wyS74ynvt70/s640/powershell-input-sequence.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Capturing the output collection into EasyMorph (click to zoom)</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Notice that because the output sequence is not a collection of basic values (e.g. number or text) but a collection of .NET objects. Therefore <span style="font-family: "courier new" , "courier" , monospace;">ForEach-Object</span> is used to extract a member value of each output object.<br />
<br />
<div class="" style="clear: both; text-align: center;">
</div>
The error collection is captured as separate column, and can be used for diagnostics or arranging a failover logic.<br />
<br />
<h3>
Sending column values as input sequence for PowerShell</h3>
It is possible to send column values as an input sequence for a PowerShell command pipeline. This input sequence is available through special PowerShell variable <b>$input</b>. In the example below a sequence from 1 to 10 is generated in EasyMorph, then sorted using a PowerShell command, and finally captured back into EasyMorph.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$input | Sort-Object -descending</span><br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFmQGEOasUoL2tnPE9xmsr7uL3F0H9A8fwU7TeOugnC_GLV6pk8TzrnePv_NcMIT_by6hfJUQRYT7WRlZgR8APcK9LuCZl3aLnto5WmbegkawTDGru82uUwqdca4fXu2E43I22nhHgklk/s1600/powershell-capture-output.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="460" data-original-width="761" height="386" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFmQGEOasUoL2tnPE9xmsr7uL3F0H9A8fwU7TeOugnC_GLV6pk8TzrnePv_NcMIT_by6hfJUQRYT7WRlZgR8APcK9LuCZl3aLnto5WmbegkawTDGru82uUwqdca4fXu2E43I22nhHgklk/s640/powershell-capture-output.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Input sequence as EasyMorph column (click to zoom).</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Notice that numbers in EasyMorph become integer or float numbers in PowerShell, text values in EasyMorph become text strings in PowerShell. In the screenshot above PowerShell sorted the input sequence as numbers, not as text (otherwise 1 and 10 would've been neighbors). Similarly, boolean values in EasyMorph are converted into booleans in PowerShell, and empty values converted into nulls.<br />
<br />
<h3>
Use cases</h3>
PowerShell is a very versatile scripting language supported by the vast ecosystem of .NET framework. Having EasyMorph integrated with Powershell opens access to large number of <a href="https://technet.microsoft.com/en-us/library/dd772285.aspx">PowerShell cmdlets</a> which help automate various tasks, e.g. advanced file operations, sending emails, computer management and administration.<br />
<br />
EasyMorph transformations can effectively be inserted into PowerShell command pipelines, e.g. generate a list of files in EasyMorph, filter it using EasyMorph expressions, and then send the resulting list into PowerShell.<br />
<br />
Finally, it becomes possible to write custom cmdlets (e.g. in C#) and run them from EasyMorph with EasyMorph parameters or column data.Dmitry Gudkovhttp://www.blogger.com/profile/02480749980533867349noreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-35640192159963583632017-10-13T12:25:00.000-04:002017-10-13T12:27:18.889-04:00Version 3.6.1 is outVersion 3.6.1 has just been released. While it's designated as a minor release, it introduces so many changes that it could've been a major version. The release is synchronized with the release of EasyMorph Server v1.2 as they both share the same <a href="http://easymorph.com/in-memory-engine.html">in-memory engine</a>.<br />
<br />
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd6YB7xQcT5mlGna2cG7FQMUTbNAyQ2I1Vujaszujox5BUGLXo9vH9jLmYSaoe1419_igQbAkOod3NQW7IiMbNXZJTxLCNmW_adlkJCNy94Z4dFNd5V2zrRGHXCTFM4y4N8g99qwojJsw/s1600/released.png" imageanchor="1"><img border="0" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd6YB7xQcT5mlGna2cG7FQMUTbNAyQ2I1Vujaszujox5BUGLXo9vH9jLmYSaoe1419_igQbAkOod3NQW7IiMbNXZJTxLCNmW_adlkJCNy94Z4dFNd5V2zrRGHXCTFM4y4N8g99qwojJsw/s320/released.png" width="320" /></a></div>
<div style="text-align: center;">
<br /></div>
<h4>
.NET requirement upgraded to .NET 4.5.1</h4>
Previously the minimum version of .NET required for EasyMorph was 4.5. Starting from this release it's 4.5.1. The change won't affect the vast majority of EasyMorph users because the 4.5.1 was installed on their computers years ago with one of Windows system updates. For those who for some mysterious reasons missed it here is the link to <a href="https://www.microsoft.com/en-us/download/details.aspx?id=40773">.NET 4.5.1 installer</a>.<br />
<br />
<h4>
Support for quoted line breaks in delimited text files</h4>
We've rewritten the parser for loading text files and now it supports quoted line breaks -- a long awaited feature that allows EasyMorph read correctly some files that it couldn't read previously, and comply with the RFC 4180 standard.<br />
<br />
However, it also means that EasyMorph may now import some text files differently, because now it imports them <i>correctly</i>. To help detect files that can now be imported differently, we included with the installer a command line utility that parses given text file(s) using the old parser and the new parser. If the parsers produce different results it prints a warning. The utility supports wildcards, so you can test with it multiple files at once.<br />
<br />
<h4>
Improved Export to Tableau</h4>
Previously timestamps (i.e. dates with time part) were exported into Tableau as only dates, without the time part. This required implementing a workaround such as exporting the time part as text in a separate column, and then appending it back to dates in Tableau using expressions.<br />
<br />
Starting from this release, if a column contains at least 1 date with time and has type “Date” in the “Export to Tableau” transformation then its Tableau data type set to DATETIME and its values are exported as timestamps.<br />
<br />
Similarly, numbers previously were always exported as floats. Starting from this release, if a column contains only integer numbers its Tableau data type is set to INTEGER and all its values are exported as integers. If the column has at least 1 non-integer value, or value that is outside of the integer range, then entire column is exported as floats.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5dVzvkIUtfylNLJ4RKN1Q5dLD0z-n0TEgFngPgJzz_jYnncox496fUZfnHs37NOBX9R0UpAPW23eGtMRFGe00xb9sPqC4iFUNpGRVapShoFtae_88efBw1tCKbCx1JCmwPNvFeHbcWQk/s1600/tableau-export.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5dVzvkIUtfylNLJ4RKN1Q5dLD0z-n0TEgFngPgJzz_jYnncox496fUZfnHs37NOBX9R0UpAPW23eGtMRFGe00xb9sPqC4iFUNpGRVapShoFtae_88efBw1tCKbCx1JCmwPNvFeHbcWQk/s400/tableau-export.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Integers and timestamps exported to Tableau (click to zoom).</td></tr>
</tbody></table>
<div>
<br /></div>
<h4>
Append mode for Export to Delimited Text File</h4>
<div>
Another long-needed feature is the ability to append data to existing text files in the "Export to delimited text file" transformation. Now it can be used for various kinds of logging. For instance, for collecting records that didn't pass data quality checks.</div>
<div>
<br /></div>
<div>
In the Append mode a text file will be created automatically if it didn't exist.</div>
<div>
<br /></div>
<h4>
Publishing projects to EasyMorph Server</h4>
<div>
All desktop versions of EasyMorph now can publish projects directly to EasyMorph Server v1.2 and up. New button “Publish to server” has been added to the Project menu.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxEmczLHMdKLrfnMpayZGCoZw-lyrTp0xyf60A9Aqtz9I9Y7JjC3v6XeydI4I9X2j-mJh4AB2O2ltRM9wy3w7mFr-me0TRwpNvPq03bwHiOhXy3j715BixRU3lFvra7tIhNyLFjzk6CnU/s1600/ems-publising.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxEmczLHMdKLrfnMpayZGCoZw-lyrTp0xyf60A9Aqtz9I9Y7JjC3v6XeydI4I9X2j-mJh4AB2O2ltRM9wy3w7mFr-me0TRwpNvPq03bwHiOhXy3j715BixRU3lFvra7tIhNyLFjzk6CnU/s400/ems-publising.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Publishing to EasyMorph Server (click to zoom).</td></tr>
</tbody></table>
<br />
<h4>
Comments and date literals in expressions</h4>
<div>
Now it is possible to insert comments in expressions. Start a comment with a double-slash (//). Everything between the double-slash and the end of line will be considered a comment and ignored during expression evaluation. A comment can’t span across multiple rows.</div>
<div>
<br /></div>
<div>
Date literals start with the pound sign (#) and must have the ISO format only, i.e. #YYYY-MM-DD. Time is not supported in date literals. Note that there must be no space between # and year. See the screenshot below:</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFHhl_JzLTuVu4qk1hJNKQeNirBNUe7v9Y0OH0UNtTjeyWGwUjme3-j6Yd7-21RPiAmpMYxS-duHv3ohJQF22rEWK-_J-FYeXZXzwuDppNOAZHaaesnXQaypNC_TzkpOEzuEWvtTjnScw/s1600/date-literals-comments.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFHhl_JzLTuVu4qk1hJNKQeNirBNUe7v9Y0OH0UNtTjeyWGwUjme3-j6Yd7-21RPiAmpMYxS-duHv3ohJQF22rEWK-_J-FYeXZXzwuDppNOAZHaaesnXQaypNC_TzkpOEzuEWvtTjnScw/s1600/date-literals-comments.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Date literals and comments in expressions.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
Links:</div>
<div>
<a href="http://easymorph.com/download.html">Download the latest release of EasyMorph</a></div>
<div>
<a href="http://easymorph.com/download/release-notes/Release_notes_v3.6.1.pdf">Release Notes v3.6.1 (pdf)</a></div>
<div>
<a href="https://www.youtube.com/watch?v=_eD7Ua8-IIc">Web File Manager in EasyMorph Server 1.2 (video)</a></div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-79492682680520154412017-10-12T22:20:00.001-04:002017-10-14T16:01:49.591-04:00Web File Manager in EasyMorph Server v1.2The Web File Manager is the headline feature in <a href="http://easymorph.com/server.html">EasyMorph Server</a> v1.2 released a few days ago. Previously, publishing files to the Server had to be done solely by copying projects and data files to a shared network folder. However, in some cases it can be inconvenient -- the Server can reside on a different network segment or in a cloud where shared folders are inaccessible. Also copying files to a network folder requires setting correct permissions for the folder in order to allow the Server service to access the files.<br />
<br />
Starting from version 1.2 EasyMorph Server has browser-based Web File Manager which allows performing the following file operations:<br />
<ul>
<li>Browse folders</li>
<li>Upload one or multiple files by dragging them into browser</li>
<li>Upload one or multiple files by pressing the “Upload file” button</li>
<li>Download a file by clicking its name</li>
<li>Delete selected file(s)</li>
</ul>
<div>
Watch the video below to see the Web File Manager in action:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/_eD7Ua8-IIc/0.jpg" frameborder="0" height="240" src="https://www.youtube.com/embed/_eD7Ua8-IIc?feature=player_embedded" width="480"></iframe></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Multiple files can be uploaded at once. There is no limit on file size.<br />
<br />
Note that in version 1.2 creating/renaming/deleting folders is not supported yet. This will be available in future releases.<br />
<br />
The Web File Manager has three access modes:<br />
<br />
<ul>
<li>Full access -- all file operations are enabled.</li>
<li>Only upload -- uploading/deleting is enabled. Downloading files not possible.</li>
<li>Only download -- only downloading is possible.</li>
<li>Disabled -- the Files tab is hidden, no file operations possible.</li>
</ul>
<div>
All the file operations are also available through the <a href="https://github.com/easymorph/server-sdk">EasyMorph Server SDK</a> and the <a href="https://github.com/easymorph/server-cmd">command-line API client</a>.</div>
<br />
From a practical perspective, the Web File Manager can complement 3rd party Business Intelligence applications (e.g. Tableau Server or Qlik Server) that are browser-based but don't have built-in capabilities for uploading/downloading data files.</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-67462840415915096062017-08-09T12:44:00.000-04:002017-08-09T12:56:22.524-04:00Version 3.6 is outHere is what's new and exciting in version 3.6:<br />
<br />
<h4>
Multiple file loading</h4>
Previously, loading multiple uniform files (i.e. files with the same type and similar set of columns) required using iterations. This could be challenging for new users unfamiliar with the concept of iterations. Starting from this release, all file import transformations allow loading multiple uniform files at once without using iterations. Loaded files are automatically concatenated into one table. it makes is convenient to load several daily or monthly reports at once.<br />
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1JgT1MNitmO4qFviIodADJsppwIv6_0yNZENwBclSMYu-mbyZLYoDasYvHrJdwv4JxdrKDgnsIXRlXesFuDv4Gc3NQrj8SN20FNOzoOvODQatXPaZSsMnuw5ieGsps4tRFv3KhHVpTEs/s1600/multiloader-selector.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="253" data-original-width="350" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1JgT1MNitmO4qFviIodADJsppwIv6_0yNZENwBclSMYu-mbyZLYoDasYvHrJdwv4JxdrKDgnsIXRlXesFuDv4Gc3NQrj8SN20FNOzoOvODQatXPaZSsMnuw5ieGsps4tRFv3KhHVpTEs/s320/multiloader-selector.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">New mode for loading multiple uniform files at once.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
The files to load can be specified either explicitly by ticking necessary files in list, or by providing a search criterion (substring, wildcard, or regular expression). The source folder path can be specified using a parameter.</div>
<div>
<br /></div>
<div>
<b>Publishing to Tableau Server</b></div>
<div>
Starting from this release the "Export to Tableau" transformation can publish Tableau data extracts right to Tableau Server as data sources.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZMybUm-JYJl-1FC17AvdVfD6HCYXW4Qx65_7H9wMEihCqo67EDEYfa-vVO4ZJ9uUFqRc22o04dCdAYN1D3B1CGl2R3lxrYQHQezZNH040hqKjlBjA09ae6vxN5sAH-786PSNtfCq3nQg/s1600/publish-tableau-server.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="403" data-original-width="344" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZMybUm-JYJl-1FC17AvdVfD6HCYXW4Qx65_7H9wMEihCqo67EDEYfa-vVO4ZJ9uUFqRc22o04dCdAYN1D3B1CGl2R3lxrYQHQezZNH040hqKjlBjA09ae6vxN5sAH-786PSNtfCq3nQg/s320/publish-tableau-server.png" width="273" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Publishing to Tableau Server</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<h4>
Support for fixed width text data</h4>
<div>
We've added two transformations to deal with fixed width text: "Import fixed width text" and "Split fixed width text". The first transformation loads a text file that contains columns of fixed width. It has a preview mode for convenient "slicing".</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkQ9T_i26bYEZYjGb2FRbRM7IJ0habnu66I3vji4IFHgm9OoqyRvue9WN-zsuFv0pV7xrur-dDt1bwMYPljHWhm_D9Kunw6z5f87g95L0hl8CtLbrI-vQG8HG3jw-J12gI24kLQztVm7w/s1600/fixed-width-preview.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="544" data-original-width="1081" height="201" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkQ9T_i26bYEZYjGb2FRbRM7IJ0habnu66I3vji4IFHgm9OoqyRvue9WN-zsuFv0pV7xrur-dDt1bwMYPljHWhm_D9Kunw6z5f87g95L0hl8CtLbrI-vQG8HG3jw-J12gI24kLQztVm7w/s400/fixed-width-preview.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Import fixed width text (click to zoom).</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
The other transformation, "Split fixed width text" allows splitting columns with fixed width text that were already loaded from other data sources (e.g. a database). It has the same preview mode for easy, visual "slicing".</div>
<div>
<br /></div>
<h4>
7 other new transformations</h4>
In total, this release adds 9 new transformations -- more than any previous release ever.<br />
<ul>
<li><b>Fill Right</b> transformation is similar to the “Fill down” transformation. It fills empty cells to the right, in specified row. This transformation helps dealing with spreadsheets where table headers contain merged cells.</li>
<li><b>Download File</b> transformation fetches a file over the internet using the HTTP(S) or the FTP network protocol. It supports basic login/password authorization.</li>
<li><b>Split Delimited Text</b> transformation is similar to the “Import delimited text file” transformation (introduced long time ago), but splits an already loaded column into new columns. It can be used for instance for dealing with CSV data imported from a spreadsheet, or a database.</li>
<li><b>Status transformation</b> is a workflow transformation. When reached it displays a custom message. Typically, it’s used for signaling on project execution progress. For desktop versions of EasyMorph the message text appears in the “Run progress” dialog window. For EasyMorph Server the message appears as task status in the web-console, and also is logged into the task log. Custom statuses of called/iterated projects pop up into their calling projects, which allows displaying iteration progress (e.g. “Processed 15 files out of 132”).</li>
<li><b>Regular Expression</b> transformation find matches for specified <a href="https://en.wikipedia.org/wiki/Regular_expression">regular expression</a>. The transformation works in two modes: find 1st match, and find all matches. In the latter case, new rows can be inserted into the table to store matches beyond 1st. Most common use cases for the transformation include data quality checks and text parsing.</li>
<li><b>Select By Lookup</b> transformation select columns which names exist in another (lookup) table. It is intended for advances scenarios in which a resulting set of columns depends on user input or calculation logic.</li>
<li><b>Convert Data Type</b> transformation converts text dates in one or more columns into number dates. The date format in selected columns can be auto-detected by pressing “Detect” button, if it matches one of 1,000+ formats known to EasyMorph. If no format detected then it should be entered manually in transformation properties. The format specification is the same as for the <a href="http://help.easymorph.com/doku.php?id=syntax:functions:format">format()</a> function.</li>
</ul>
<br />
<h4>
Changes to existing transformations</h4>
<ul>
<li>The Concatenate function of <b>Aggregate</b> now allows specifying a separator and concatenating only distinct values. </li>
<li><b>Running Total</b> transformation has got a <a href="https://en.wikipedia.org/wiki/Ranking#Standard_competition_ranking_.28.221224.22_ranking.29">competition ranking</a> mode in addition to the default dense ranking. </li>
<li><b>Select Matching Database Rows</b> and <b>Delete Matching Database Rows</b> transformations now allow matching on dates (without times). </li>
<li><b>Iterate </b>transformation now runs faster and is more memory-efficient.</li>
</ul>
<div>
<br /></div>
<h4>
The system function</h4>
<div>
A new function has been added: <b>system(keyword)</b>. This function allows obtaining various system values. The keyword must be a text constant. Currently supported keywords:<br />
<br />
<ul>
<li><b>projectpath</b> — The full path to the current project including file name.</li>
<li><b>projectfilename</b> — Current project’s file name only.</li>
<li><b>programpath</b> — The full path to morph.exe.</li>
<li><b>currentpath</b> — The working directory.</li>
<li><b>user</b> — The current user. Replaces function user().</li>
<li><b>callerpath</b> — The full path of the calling project including file name.</li>
<li><b>utcoffset</b> — The UTC offset to the system time zone ( e.g. "-04:00:00").</li>
</ul>
<div>
Example:<br />
<div style="text-align: center;">
<span style="font-family: "courier new" , "courier" , monospace;">system('projectfilepath')</span> returns ‘C:\Documents\myproject.morph’.</div>
</div>
<br />
<h4>
Custom HTML in generated documentation</h4>
Auto-documentation now allows including custom HTML into the head and body of generated HTML page. This can be used for inserting corporate logos, links to other HTML pages, change logs, custom project metadata.</div>
<div>
<br /></div>
<h4>
Miscellaneous </h4>
<ul>
<li>The visual condition builder in Query Editor now has undo/redo buttons. </li>
<li>Added support for SAP HANA SQL dialect (for ODBC connections). </li>
<li>Query preview now have a context menu invoked by right-clicking on datagrid cells. </li>
<li>Expression editor has got a button for inserting date constants. </li>
<li>Project tabs can be switched back and forth by pressing <i>Ctrl+Tab</i> and <i>Ctrl+Shift+Tab</i>. </li>
<li>The “Input” transformation can now be enabled in Plus license as an add-on feature.</li>
</ul>
<div>
<a href="http://easymorph.com/download.html">Download the latest release</a></div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-84796964502505334482017-08-02T12:26:00.002-04:002017-08-02T12:50:33.541-04:00Fuzzy matching and mergingSometimes there are cases when it's hard to match (or merge) two tables because the matched values (e.g. names) are not precisely identical -- for instance when a name has a typo, or extra space between first and last name. Most frequently it happens when one of the tables is a result of a process that involved manual data entry at some point. It could be web-site forms, CRM systems, or spreadsheets. In such cases fuzzy matching comes to rescue. This article explains what this is and how to do it in <a href="http://easymorph.com/">EasyMorph</a>.<br />
<br />
Fuzzy matching can reliably match two words (or in a broader sense, text strings) that are similar but not exactly the same. For instance, "EasyMorph Inc" and "EasyMoprh Inc" clearly mean the same company, but the two names are not identical because the latter has a typo and therefore they won't be matched when doing a VLOOKUP in Excel, or data blending in Tableau.<br />
<br />
In order to deal with such discrepancies the <i>edit distance</i> between two text strings can be calculated. This distance shows the number of edits required in one text in order to obtain the other.<br />
<br />
The <i>Levenshtein distance</i> is the number of inserts, deletions and character replacements required to produce the other word. In our case, the Levenshtein distance between "EasyMoprh Inc" and "EasyMorph Inc" equals 2, because it requires removing "p" before "r" and inserting it after "r" -- so 2 edits in total.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ex9VyxXP3Nyh0n37Qmtb8ewiwhFxeGxefSqx9TozAmKAhzBz8aQ6PiK7-RQbIqnOEFBiFfr4F5XxNdiM8nwKDNHDAK-UO38HKAXJEA1GvKE-Tj3MIO3xVL9wkTqOXamGGr6bSxh_kGg/s1600/levenshtein.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="256" data-original-width="276" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ex9VyxXP3Nyh0n37Qmtb8ewiwhFxeGxefSqx9TozAmKAhzBz8aQ6PiK7-RQbIqnOEFBiFfr4F5XxNdiM8nwKDNHDAK-UO38HKAXJEA1GvKE-Tj3MIO3xVL9wkTqOXamGGr6bSxh_kGg/s1600/levenshtein.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Levenshtein</td></tr>
</tbody></table>
<br />
There is a modified variant of the Levenshtein distance which is called the <i>Damerau-Levenshtein distance</i>. This variant adds swapping two adjacent characters as one more possible edit operation. The Damerau-Levenshtein distance between "EasyMoprh Inc" and "EasyMorph Inc" equals 1 because it only requires swapping "p" and "r" which is 1 edit operation.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3IlvJfCohZJXeeoXuqRs_oBxfmeoXHwZPVaYC9Q9dGGRcMDUasOV3MSLdR9vhCjIGj3yY65OB_5seuJkFVugr13vCeJj259Z-nVcgjHSQNVYluMh7HQ2sSjViFvmv6L5yvCYZkQdPZkE/s1600/damerau-levenshtein.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="198" data-original-width="284" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3IlvJfCohZJXeeoXuqRs_oBxfmeoXHwZPVaYC9Q9dGGRcMDUasOV3MSLdR9vhCjIGj3yY65OB_5seuJkFVugr13vCeJj259Z-nVcgjHSQNVYluMh7HQ2sSjViFvmv6L5yvCYZkQdPZkE/s1600/damerau-levenshtein.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Damerau-Levenshtein</td></tr>
</tbody></table>
<br />
<br />
When fuzzy matching is done using the edit distance two text strings are considered matched when the distance is less than certain threshold (typically 2 or 3). The higher the threshold, the "fuzzier" the matching becomes. Therefore high thresholds are usually not practical as they produce too many false matches. Also, the shorter are matched values, the lower the threshold should be. For instance, the edit distance between "cat" and "dog" is just 3, but these are completely different words.<br />
<br />
In EasyMorph, fuzzy matching is arranged using the "Match" transformation in the "Fuzzy" mode. It calculates the Damerau-Levenshtein distance for text values in two tables.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihnHfmR6DUW9syTIi4HCsnBJz9qH3dkn_hZRZQVc7rSX_p6JBeJ7DvYiUgRhSLJswHh8onDo2vftGdYSG3vzx1gPJxJ0jc3YFCQRSStz5asVRSNz26-T1w0o8ecLL1A4o9-vctSj9hTPw/s1600/match-transformation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="422" data-original-width="339" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihnHfmR6DUW9syTIi4HCsnBJz9qH3dkn_hZRZQVc7rSX_p6JBeJ7DvYiUgRhSLJswHh8onDo2vftGdYSG3vzx1gPJxJ0jc3YFCQRSStz5asVRSNz26-T1w0o8ecLL1A4o9-vctSj9hTPw/s320/match-transformation.png" width="257" /></a></div>
<br />
<br />
The transformation matches text values in one column (with mistyped values) in current table with a column in another table (with correct values). When the edit distance is less than or equal to the specified threshold values are considered matched. If merging other columns is required it can be done using additionally the "Merge" transformation by linking successfully matched values.<br />
<br />
It is also possible to calculate the edit distance in expressions using <span style="font-family: "courier new" , "courier" , monospace;">distance()</span>. The function can be used in simple cases cases when there is only one or two correct values to match with, thus avoiding creating a lookup table.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-85887112892658852572017-07-14T00:08:00.003-04:002017-07-14T00:14:39.463-04:00How to trigger data transformation tasks from TableauSince <a href="http://easymorph.com/server.html">EasyMorph Server</a> is built using web technologies, it makes triggering data transformation (ETL) tasks from analytical systems like Tableau especially simple. Triggering tasks right from dashboards is convenient, because you can refresh underlying data sources from a familiar UI in just one click, plus you get additional bonus feature (explained later in this article).<br />
<br />
Here is how you can do it:<br />
<br />
<h4>
Step 1: Enable HTTP GET requests for the data transformation task</h4>
While it might sound cryptic it's actually a simple thing -- we just tell EasyMorph Server that particular task may be triggered by opening specific URL (hyperlink). This option is disabled by default in order to avoid accidental task triggering, because a task can potentially perform an irreversible action, e.g. overwrite data in a database table or file.<br />
<br />
To enable triggering by opening a hyperlink, go to the task properties in EasyMorph Server web-console and tick "Enable HTTP GET".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxav49_AU6E5gc0T7AwNAIWDdPgija4Eqv05qbAdW1PDZgse1Vx3xQc0PFSFRCLAiMspgu5nrv6Qp9ywo30M7tPGdL7WejmJPZjqjCnSKTGRO76zWEhkGGyNotaaeAPqcGfbNPSxHd6GQ/s1600/enable-reloading.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="409" data-original-width="969" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxav49_AU6E5gc0T7AwNAIWDdPgija4Eqv05qbAdW1PDZgse1Vx3xQc0PFSFRCLAiMspgu5nrv6Qp9ywo30M7tPGdL7WejmJPZjqjCnSKTGRO76zWEhkGGyNotaaeAPqcGfbNPSxHd6GQ/s640/enable-reloading.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Enable triggering EasyMorph Server task by URL (click to zoom).</td></tr>
</tbody></table>
<span id="goog_1141560922"></span><span id="goog_1141560923"></span><br />
<br />
Then copy the generated URL. In the example above the server address is <i>localhost</i>, in your case it would be your EasyMorph Server address. Don't forget to save the change in task properties -- it won't work until you save it.<br />
<br />
<h4>
Step 2: Create URL action in Tableau</h4>
<div>
Now go to Tableau and create a new URL action (menu Dashboard -> Actions -> Add Action -> URL). Give the action a name and paste the URL copied in Step 1.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzILNY4kNegPTRTUSwAXcWMcLcrsDeHY7icWCR6D7VyOqgB4bSgpxadDWaAwjIOz0JGBfJwMh4rijnevuADZLyahndAlkk6pv_-ElIGls1LnWlT7faBEI8nWN4a-IHOUxw-Elxu053xWU/s1600/etl-action-tableau.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="536" data-original-width="502" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzILNY4kNegPTRTUSwAXcWMcLcrsDeHY7icWCR6D7VyOqgB4bSgpxadDWaAwjIOz0JGBfJwMh4rijnevuADZLyahndAlkk6pv_-ElIGls1LnWlT7faBEI8nWN4a-IHOUxw-Elxu053xWU/s400/etl-action-tableau.png" width="374" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Create URL Action in Tableau (click to zoom).</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<h4>
Step 3: Trigger the task from Tableau</h4>
<div>
Now right-click anywhere in your dashboard and trigger the EasyMorph Server task from the right-click menu (actions appear at the bottom). This will open default web-browser showing task execution log updated in real time. Once the task is finished you can go back and refresh the dashboard (in some scenarios it updates automatically). Mission accomplished.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4msbvL_W9_-xmX8V57byx13grmhQY47QKUyOtqu-eUu9CnEnoX8tvhm-mBzwvqqqCwq_qnIcY9m7g2alT7dm2X9H9YWescQ_b1cVqnMHiWWeeEuoKzRcsqrI4EOLmYAic1R-S07SbjSA/s1600/run-task.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="368" data-original-width="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4msbvL_W9_-xmX8V57byx13grmhQY47QKUyOtqu-eUu9CnEnoX8tvhm-mBzwvqqqCwq_qnIcY9m7g2alT7dm2X9H9YWescQ_b1cVqnMHiWWeeEuoKzRcsqrI4EOLmYAic1R-S07SbjSA/s1600/run-task.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Trigger EasyMorph task from Tableau.</td></tr>
</tbody></table>
<div>
<br />
<br /></div>
<h4>
Bonus feature: auto-documentation</h4>
<div>
EasyMorph Server can generate a human-readable description of a task's data transformation logic. This is convenient because it explains in plain English how the source data was processed into the dataset displayed in the dashboard. You can see all the transformations, expressions and filtering conditions, as well as helpful annotations.</div>
<div>
<br /></div>
<div>
Since generated task documentation has a permanent URL it also can be added as an URL action in Tableau.</div>
<div>
<br /></div>
<div>
To obtain the auto-documentation link go to the task properties on EasyMorph Server web-console, and switch to Documentation. Then copy the page URL from the browser address bar.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVTFYyAtgaobcL41whXR1m3kbulc3Z_WfmJXrysw2Rr97N2mw8EgcvBLiJ-fWf1KNVevOW8baP0doLTGMAyREU_UdlGUhxtXBAQGHp0Nmb4QDf9iguE92zwPeY5ESk6cfc5IJ-8GxzmoQ/s1600/task-documentation.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="714" data-original-width="835" height="544" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVTFYyAtgaobcL41whXR1m3kbulc3Z_WfmJXrysw2Rr97N2mw8EgcvBLiJ-fWf1KNVevOW8baP0doLTGMAyREU_UdlGUhxtXBAQGHp0Nmb4QDf9iguE92zwPeY5ESk6cfc5IJ-8GxzmoQ/s640/task-documentation.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Auto-generated ETL documentation (click to zoom).</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
A URL action for documentation is created in the same way as described in the Step 2 above.</div>
<div>
<br /></div>
<div>
Once the URL action is created, you can open the documentation right from the Tableau dashboard.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXGyTiEyJAc7RD0ANbK4OpW1nmO1JNq6W4sUWI7r4zvfLiaIGt_mNafhOo4w0pJCH07wzXFe1JzRRUwChmnPvcwtWHJDEbeyFLlnd71b-e58JJW-bsvcrCXiTN8TZ5NvhHq31WG_-2jXs/s1600/tableau-task-documentation.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="356" data-original-width="317" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXGyTiEyJAc7RD0ANbK4OpW1nmO1JNq6W4sUWI7r4zvfLiaIGt_mNafhOo4w0pJCH07wzXFe1JzRRUwChmnPvcwtWHJDEbeyFLlnd71b-e58JJW-bsvcrCXiTN8TZ5NvhHq31WG_-2jXs/s1600/tableau-task-documentation.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Open task documentation from Tableau.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Go to <a href="http://easymorph.com/server.html">EasyMorph Server web-site</a>.</div>
<div>
<br /></div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-11541632133663255832017-06-29T01:51:00.002-04:002017-06-29T10:59:29.437-04:00Our mid-term roadmap (2017H2-2018H1)We're getting quite a few feature requests recently (which is a great thing!) so it looks like it would be useful to shed some light on our future development plans. We do publish our short term goals on the <a href="http://easymorph.com/download.html">download page</a>. However, our mid-term roadmap has never been disclosed. So here is what we would like to do in the next 12 months:<br />
<br />
<h4>
Persistent editable datasets in projects</h4>
This is supposed to be a very significant change to what EasyMorph is. At this point no data is stored in EasyMorph projects -- they load it from files and databases and output processed data into other files and databases. Once processing is done no data is saved in project. We're going to change that. Similarly to spreadsheets, EasyMorph will be able to have both data AND calculation logic in one file. From a technical standpoint it will be implemented as follows:<br />
<br />
<ul>
<li>Sandboxes will be able to keep datasets <i>permanently</i> -- i.e. data in sandboxes will be stored in projects and survive re-opening projects.</li>
<li>Sandboxes will be <i>editable</i> -- it will be possible to edit data in sandboxes like in spreadsheets (but without formulas).</li>
<li>All import transformations will get the ability to remember (cache) the last loaded dataset, which means that a project can be sent to someone without access to the project's data sources.</li>
</ul>
<div>
While it might be not immediately obvious what the benefits are, they are actually quite significant:</div>
<div>
<ul>
<li>Portability -- no need to keep mapping tables and reference tables in external files. They can be stored and edited right in project.</li>
<li>Shareability -- EM projects become a means to <i>share</i> data. Like with spreadsheets, it allows sending not just some calculation results to someone, but also the source data and the logic that produced the results.</li>
<li>Versatility -- the ability to edit datasets right in EasyMorph means that you can mix manually entered data with calculated data. For instance, edit planned numbers and compare them with actuals in one project. Or update planned numbers based on some actual data. Or perform what-if analysis for different scenarios.</li>
</ul>
<div>
The change would effectively make EasyMorph a hybrid between a spreadsheet and an ETL, or a <i>relational spreadsheet</i>, if you will.<br />
<br /></div>
<h4>
Querybooks</h4>
</div>
<div>
We're going to keep improving queries which will transform into <i>querybooks</i>. A querybook is a set of queries that answer particular questions (one query per tab). Querybooks will be portable, meaning that a querybook can be sent to another user with all necessary data connectors embedded for the queries to work.</div>
<div>
<br /></div>
<div>
Queries themselves will become multi-table, supporting joins and unions (won't be available in the free edition).</div>
<div>
<br /></div>
<h4>
Application data sources</h4>
<div>
So far EasyMorph has been able to import data from databases, files and spreadsheets. However, a new category of data sources will be added -- applications, including cloud applications like Google Analytics, Twitter, Wikipedia, etc.</div>
<div>
<br /></div>
<div>
Application data source won't be available in the free edition.</div>
<div>
<br /></div>
<h4>
Financial and statistical transformations</h4>
<div>
With the release of version 3.6 EasyMorph has 80+ transformations that cover almost all possible basic calculations. But we're not done yet! We will be adding special transformations for financial calculations (payment schedules, compound interest rates, bond yields, etc.) and statistical calculations (e.g. variances, deviations, clustering) with elements of predictive analytics (e.g. linear regressions, extrapolations).</div>
<div>
<br /></div>
<h4>
Visual conditions and business rules</h4>
<div>
Visual conditions that were first introduced for queries in v3.5 will make their way into transformations with conditions that currently use expressions -- Rule, Filter by condition, Trim by condition, Halt and a few more transformations.</div>
<div>
<br />
PS. Lots of things are planned for <a href="http://easymorph.com/server.html">EasyMorph Server</a>, but that's a subject for another post.<br />
PPS. All written above is subject to change without notice.</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-27904784540954072812017-05-21T15:22:00.000-04:002017-05-21T21:55:09.641-04:00Meet EasyMorph ServerToday we're releasing <a href="http://easymorph.com/server.html">EasyMorph Server</a> -- a new product in EasyMorph family. Here I'd like to talk more about why we've made it and what to expect in the future.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<h4>
Why</h4>
The need for a server edition appeared pretty quickly once our customers started performing more transformations on larger datasets. Basically, it was demanded by the switch of usage pattern from personal to team use:<br />
<br />
<ul>
<li>Multiple projects run by multiple users</li>
<li>Data volumes required more RAM and CPU power than a typical personal computer has</li>
<li>Scheduled long-running transformations</li>
<li>Shared source files and data sources</li>
</ul>
<div>
<br /></div>
<h4>
What's this</h4>
EasyMorph Server is a Windows service with a web console and the same columnar in-memory data transformation engine which is employed by desktop editions of EasyMorph. It runs on schedule projects created in any desktop edition (including the free edition, yay!). Project parameters can be assigned (overridden) in Server tasks. Project logs can be viewed right in the web console and are updated in real-time.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigSU71NO7nmUzBg2hOC9ZJoAt4w7d1ZuJT0qrFhKCCFxtMXS5u89zsrx7HeZRRbnp6lpDLypaNzGGTlDe8YyZTQFDxHjSd2kd_DIxN97ACApnFXJddNHA_WPTlZ4t1tIOynuBR9OkR2z4/s1600/server-tasks.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigSU71NO7nmUzBg2hOC9ZJoAt4w7d1ZuJT0qrFhKCCFxtMXS5u89zsrx7HeZRRbnp6lpDLypaNzGGTlDe8YyZTQFDxHjSd2kd_DIxN97ACApnFXJddNHA_WPTlZ4t1tIOynuBR9OkR2z4/s400/server-tasks.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Task list (click to zoom).</td></tr>
</tbody></table>
<br />
<br />
The Server engine performs slightly better than the engine of desktop editions. It runs 10-20% faster and utilizes 10-30% less memory due to server-specific optimizations. For instance, unlike the desktop editions, Server doesn't store some intermediate results since there is no need to display them in UI.<br />
<br />
EasyMorph Server is designed for analytical teams with limited or no IT support. Therefore, it's very easy to install and maintain. The installer package also includes EasyMorph Server Monitor -- a simple utility to manage the service: start it, stop it, view the server log, and change HTTP port for that rare case when the default one is not available.<br />
<br />
<h4>
Documentation</h4>
While the Server doesn't allow creating or editing projects (projects are designed in desktop editions) it still can be helpful for business users that only run projects (created by someone else) because they can view auto-generated documentation on data transformation logic in projects, and understand what happens under the hood when they trigger a data transformation task. For instance, they can see expressions used to calculate columns, filtering conditions, data sources, parameters, etc.<br />
<br />
Since the documentation has a permanent URL it can be accessed (opened) from external applications such as Tableau dashboards or Qlik applications.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbzDnXwfAum5xd-OjK8sO3h8lY-P8wCg5tp_tTpQgleNB42b-cYrZlLDDSp-6s_df6N-nSnfFSs0VxTVJ6Wk56Bl5I3FSAIjQBR1de9qmygjO6nqF3yiF1EN_jBv-8GUyEj-WPKr4ea-U/s1600/tableau-actions.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbzDnXwfAum5xd-OjK8sO3h8lY-P8wCg5tp_tTpQgleNB42b-cYrZlLDDSp-6s_df6N-nSnfFSs0VxTVJ6Wk56Bl5I3FSAIjQBR1de9qmygjO6nqF3yiF1EN_jBv-8GUyEj-WPKr4ea-U/s1600/tableau-actions.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Integration with Tableau dashboard through Actions.</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<h4>
API</h4>
Right from the beginning EasyMorph Server is built around a public RESTful API that can be used by external applications in order to create, modify and trigger Server tasks programmatically. The API enables tight integration with 3rd party tools.<br />
<br />
<h4>
Customization / White-labeling</h4>
The Server's UI is customizable. You can change colors and styles, use your own logo and modify links in the web console's header to match your corporate style. Even more than that -- if you're developing and selling an analytical application you can use EasyMorph Server as native component of your application under an OEM license. Note that desktop EasyMorph (required for creating/editing projects) is not customizable.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNm_UcYtj8gZmde_if5LOXHu6PFCS1S1opssrKPH6fvo2o6AXWFPwN2ePz4Y5syKRdNiJ8wmIRP9qln4gDMBGqIgdcHQQSs2Hji37P70zIv4b-62UozlMJAM6f-qwWJNpg5QAKIYdD3hE/s1600/oem.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNm_UcYtj8gZmde_if5LOXHu6PFCS1S1opssrKPH6fvo2o6AXWFPwN2ePz4Y5syKRdNiJ8wmIRP9qln4gDMBGqIgdcHQQSs2Hji37P70zIv4b-62UozlMJAM6f-qwWJNpg5QAKIYdD3hE/s1600/oem.png" /></a></div>
<br />
<h4>
What's next</h4>
The version 1.0 is, as it's supposed to be for a very first release, somewhat basic :) But it doesn't mean it's going to stay like that -- we have a lot of things on the roadmap:<br />
<br />
Soon, we will be introducing <i>events</i> -- a means to arrange task cascading and non-schedule triggers (e.g. running a task when a file appears/changes). <i>Private groups</i> will allow restricting access to particular tasks. <i>Email integration</i> will appear in one of the future releases and will make automated receiving and processing email attachments possible. <i>Impact tracking</i> will allow logging and analyzing what project created, consumed or modified what file or database table.<br />
<br />
<b>Licensing</b><br />
EasyMorph Server is licensed per server with no limit on users which means that any number of users can access the Server, create and run tasks, and view documentation on them. Together with the ability to run projects created in the free desktop edition this makes EasyMorph Server one of the most affordable department-level ETL systems on the market. More on the licensing on our web-site: <a href="http://easymorph.com/server.html">http://easymorph.com/server.html</a>.<br />
<br />
<i>Dmitry Gudkov</i>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-26362110367432144172017-04-25T23:33:00.001-04:002017-04-30T12:44:58.419-04:00Version 3.5 is outIt's happened, finally we're releasing <a href="http://easymorph.com/download.html">version 3.5</a>! It took us 2 months longer to release it than it was originally scheduled. The visual condition builder turned out to be a more complex task than I initially assumed. Databases deal with dates, non-integer numbers and Boolean values sometimes in very different ways with lots of nuances. The fact that EasyMorph operates with decimals instead of floats, and uses so called OLE Automation dates (same as in Excel) made things even more complicated. But now all the issues have been ironed out (at least those that we're aware of currently). Here is what's new and exciting in version 3.5:<br />
<div>
<br /></div>
<h4>
Visual Condition Builder</h4>
<div>
This is the headliner feature of this version. Now it is possible to design filtering conditions in queries visually. Just drag a field from the list on the left or the preview datagrid, onto the filtering area.</div>
<div>
<br /></div>
<div>
Using a custom WHERE clauses is still possible.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJjHRRjsHUtg18xb26P7zfIUK-QbZNvHelLDKNU9sTZXerqAOx_cV-MQX9P7BwXgq3FQ5uH7i5Ipu_2kkM3k7WrCf6TvURczJJFAnj-ZUxTP1F4fS5e57OqGVkVT4yJyqLOa9C7bztZjo/s1600/2017-04-25+22_34_09-Untitled+-+EasyMorph+Professional.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="449" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJjHRRjsHUtg18xb26P7zfIUK-QbZNvHelLDKNU9sTZXerqAOx_cV-MQX9P7BwXgq3FQ5uH7i5Ipu_2kkM3k7WrCf6TvURczJJFAnj-ZUxTP1F4fS5e57OqGVkVT4yJyqLOa9C7bztZjo/s640/2017-04-25+22_34_09-Untitled+-+EasyMorph+Professional.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
For those who prefer writing SQL queries manually the Query Editor helps saving time -- design a draft query visually, then switch to Custom SQL and the visual query will be automatically converted to a SQL query which you can further modify and run right in EasyMorph.</div>
<div>
<br />
In future releases, visual conditions won't be limited to queries. We are planning to introduce them in EasyMorph transformations as well in addition to formula-based conditions, where applicable.<br />
<br /></div>
<h4>
Delete / update database rows</h4>
<div>
In the previous release (v3.3) we introduced the possibility to export (insert) rows into database tables. With this release we're adding two more transformations that allow deleting particular database rows, thus enabling delete/update workflows:</div>
<div>
<ul>
<li>"Delete database rows" transformation deletes rows specified by a filtering condition of query (column selections are ignored). Using a query in this transformation provides extra convenience as it allows previewing rows which will be actually deleted, before deleting them.</li>
<li>"Delete matching database rows" transformation deletes database rows where key fields match key fields in specified EasyMorph table. Using this transformation you can calculate a list of particular IDs and then delete only database rows with these IDs.</li>
</ul>
<h4>
<br />Transformations for large data volumes</h4>
</div>
<div>
EasyMorph keeps all datasets in memory and this provides a multitude of benefits such as the ability to instantly see results of any transformation, reactive calculation, and re-calculating only transformations affected by a change. At the same time, this approach has a downside – if a dataset doesn't fit RAM entirely it would require using one of two special techniques to process it: <i>partitioning</i> and <i>pre-filtering</i>. Partitioning means that we split the source dataset into smaller parts of the same structure, which are processed one by one (typically using iterations). Pre-filtering means that instead of loading entire dataset and then filtering it, we only load a few fields that are necessary for filtering as well as IDs (primary of foreign keys). Filtering produces a subset of IDs, and for those IDs full records are fetched from the source dataset.</div>
<div>
<ul>
<li>"File splitter" transformation allows splitting (partitioning) large text files into smaller ones without loading them into EasyMorph. The smaller files can later be processed in EasyMorph one by one a usual way, typically using iterations.</li>
<li>"Select matching database rows" transformation loads only database records which ID exists in an EasyMorph table. This transformation can be used for arranging scenarios with pre-filtering described above.</li>
</ul>
<div>
These two transformations allow EasyMorph users to work with tables larger than RAM -- tens and even hundreds of gigabytes.</div>
</div>
<div>
<br /></div>
<h4>
Other new transformations and changes to existing ones</h4>
<div>
<ul>
<li>New transformation: "Filter by search". Allows filtering text values by a substring, or using a regular expression.</li>
<li>"Match" transformation now has Fuzzy Matching mode. In this mode text values are matched if their edit distance (as per <a href="https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance">Damerau-Levenshtein)</a> is within a specified limit. Fuzzy Matching helps match words with typos. For instance it would match "New York" with "New Yrok" or "New Yokr".</li>
<li>"Merge" transformation now has Full Join mode which is similar to SQL full join.</li>
<li>"Export to Excel" transformation now allows creating new sheets in existing spreadsheets.</li>
</ul>
</div>
<div>
<br /></div>
<h4>
New Filter pop-up</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjkPTyDgHSMhbCfn8wlTOxmcQQd5hf6q-V4fW_vaKCAwwbsewHp31tJkuya3faKtH_yxGhIJq9li9_-LUjocRNlt0XQDu5VJS_R7AB3lxfzMQer2XxllfPA0UFODObh6dG9NbuZB5onU/s1600/filter-popup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="363" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjkPTyDgHSMhbCfn8wlTOxmcQQd5hf6q-V4fW_vaKCAwwbsewHp31tJkuya3faKtH_yxGhIJq9li9_-LUjocRNlt0XQDu5VJS_R7AB3lxfzMQer2XxllfPA0UFODObh6dG9NbuZB5onU/s640/filter-popup.png" width="640" /></a></div>
<div>
<br /></div>
Starting from this release double-clicking a column header brings up the new Filter pop-up which allows viewing/searching unique values in the column as well as filtering (according transformation is created automatically). The pop-up has three modes: <br />
<ul>
<li>Selection – explicitly select items that should be kept (removed). </li>
<li>Search – only items that contain particular substring are kept (removed). </li>
<li>Condition – use expression to filter items. Only items where the expression returns TRUE are kept. </li>
</ul>
The Unique Values pop-up used in previous versions is discontinued.<br />
<div>
<br /></div>
<h4>
New data sources</h4>
<ul>
<li>Added support for DB2 (LUW) SQL dialect in ODBC connections.</li>
<li>Added support for Vertica SQL dialect in ODBC connections.</li>
</ul>
<div>
<br /></div>
<h4>
What's next</h4>
<div>
In the future release (3.6) we will continue improving usability and closing remaining gaps in functionality. It will include importing fixed width text, loading multiple files without iterations, and regular expression matching. Version 3.6 is planned for release in June this. However, before that there will be another, no less significant event in May – we're releasing the first ever version of <a href="http://easymorph.com/server.html">EasyMorph Server</a>!</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-66852243640195812622017-02-11T23:31:00.002-05:002017-02-11T23:31:21.124-05:00Visual Condition BuilderHere is a brief overview of the visual condition builder available starting from version 3.5. Besides making importing data to EasyMorph simpler it can be used for generating SQL for use in another applications.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/cimhlIdSVMU/0.jpg" frameborder="0" height="270" src="https://www.youtube.com/embed/cimhlIdSVMU?feature=player_embedded" width="480"></iframe></div>
<br />Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-38761631262736662472017-01-03T23:15:00.000-05:002017-01-03T23:18:07.647-05:00How to apply several filters to two tables simultaneouslySometimes, when performing reconciliation of two tables, it is necessary to apply filters to columns with the same names in both tables simultaneously in order to select identical subset of data. For instance filter exactly the same Year, Region and Product in both tables. Having two identical sets of filters is not so convenient, especially if you need to filter by several columns and select various subsets of data in order to compare them.<br />
<br />
Here is how you can filter data in two tables using only one set of filters:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxRv4Lvx90kqE2oy00Bz63vAjCUdFuk0fbLwVEYDVEQG_1RbAO4-1HjiRpN24FIrsbL28qcAgURz9kCpgOecE52rVtw5b81wRn2hgKNP6D44DyAJ5oMgPnVVSEzY0WfkJQRd4jHR2ZUKs/s1600/2017-01-03+23_11_54-EasyMorph+blog+illustrations.pptx+-+Microsoft+PowerPoint.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxRv4Lvx90kqE2oy00Bz63vAjCUdFuk0fbLwVEYDVEQG_1RbAO4-1HjiRpN24FIrsbL28qcAgURz9kCpgOecE52rVtw5b81wRn2hgKNP6D44DyAJ5oMgPnVVSEzY0WfkJQRd4jHR2ZUKs/s640/2017-01-03+23_11_54-EasyMorph+blog+illustrations.pptx+-+Microsoft+PowerPoint.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table>
<br />
The idea is simple:<br />
<ol>
<li>Append both tables keeping track of data origin.</li>
<li>Apply filters to the united dataset (filtered columns should have the same names in both datasets)</li>
<li>Split the united dataset back into two tables. When filters for the united dataset change, the derived tables will update automatically.</li>
</ol>
This trick can work with more than two datasets, of course.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-20049899790397972252016-12-06T21:19:00.002-05:002016-12-06T21:22:46.628-05:00How to transpose a matrix tableTransposing a matrix table requires two EasyMorph transformations applied one after another -- Unpivot and Pivot:<br />
<br />
<h4>Step 1: Load a sample table.</h4><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRiFm9yc1ST1lQwOAcCgmrAIx676RfBXm5FC03Qc3aPYtgssbiG7Q7ehVKSApN-rJp-3q_Q3aTSOjGa1PD0RiBbAzlh3nq6a7lZfeWy_q-ZrSJkwvf_MNVE-PG-v6zLckWOL1DS1yEJI0/s1600/1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="344" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRiFm9yc1ST1lQwOAcCgmrAIx676RfBXm5FC03Qc3aPYtgssbiG7Q7ehVKSApN-rJp-3q_Q3aTSOjGa1PD0RiBbAzlh3nq6a7lZfeWy_q-ZrSJkwvf_MNVE-PG-v6zLckWOL1DS1yEJI0/s640/1.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table><br />
<h4>Step 2: Unpivot</h4><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR_0w4PaHUGPvQmH3YKiDmYOlNkGqztG9Ns_-Mue-BFY007crlJexm1KDV59by2GaM5LHDxrVGnBinSwCctTqTjouPsJsAt_rjDZNOPpTrJrQbFW32rlMA0ciHILhXJbDY4r4N8IfYNaM/s1600/2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="354" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR_0w4PaHUGPvQmH3YKiDmYOlNkGqztG9Ns_-Mue-BFY007crlJexm1KDV59by2GaM5LHDxrVGnBinSwCctTqTjouPsJsAt_rjDZNOPpTrJrQbFW32rlMA0ciHILhXJbDY4r4N8IfYNaM/s640/2.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table><br />
<h4>Step 3: Pivot using another column for column labels.</h4><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAfmLR1HvFrLfFQCJQ5Z7UBhIT4NohYIk09w-rq1zVyJ6IlYHt37HxsVotxe883r2xr9bbnkyCPE15syfUkbJeOq0KgLIc3GgvE5Td4dDz0OdDwj_OU-GqKsWkdRxWD1xKXJySTO-aK9c/s1600/3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAfmLR1HvFrLfFQCJQ5Z7UBhIT4NohYIk09w-rq1zVyJ6IlYHt37HxsVotxe883r2xr9bbnkyCPE15syfUkbJeOq0KgLIc3GgvE5Td4dDz0OdDwj_OU-GqKsWkdRxWD1xKXJySTO-aK9c/s640/3.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table><br />
Almost there.<br />
<br />
<h4>Step 4: Sort to tidy up the table.</h4><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJUylnqPioqE8JiGKVolxQihFc1AFEbj9r83rwPvH91g4K4fB1FTVS5qgkRCVEt2IqrR_UEjBXgUjFA4Uvu2qeEXtoexFhEtkLjmPScRFejLYfRhY1HkOSFQOivK_TlhFxOP-wBHE99_0/s1600/4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJUylnqPioqE8JiGKVolxQihFc1AFEbj9r83rwPvH91g4K4fB1FTVS5qgkRCVEt2IqrR_UEjBXgUjFA4Uvu2qeEXtoexFhEtkLjmPScRFejLYfRhY1HkOSFQOivK_TlhFxOP-wBHE99_0/s640/4.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(click to zoom)</td></tr>
</tbody></table><br />
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-65698922149079481012016-12-06T08:56:00.002-05:002016-12-06T08:56:54.416-05:00Checking multiple columns for errorsThere is a rather interesting <a href="https://community.easymorph.com/t/highlighting-errors/31">suggestion</a> on EasyMorph Community regarding notifications of error values in datasets. While this is still not available in EasyMorph here is how you can check multiple columns at once if they contain error values:<br />
<br />
Select multiple columns (Ctrl + click) and choose <b>Keep rows -> Errors</b> from the column menu. EasyMorph will automatically create a Filter by Condition transformation with expression like<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">isempty([Column 1]) or isempty([Column b]) or ... or isempty([Column N])</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy5diVFJCUysnr-FvWqzRmOwWEFBuFqx9JU57JzMr_d2-4mZA-0GSg0WW88D6PKCFA7gS5V4yCeTAaqOZ61wWImAqgYC0jCAzumTOiYhJq9kdOI9Nynpps7zMop-Q4noCNFga3ydyrzoQ/s1600/find_errors.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="401" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy5diVFJCUysnr-FvWqzRmOwWEFBuFqx9JU57JzMr_d2-4mZA-0GSg0WW88D6PKCFA7gS5V4yCeTAaqOZ61wWImAqgYC0jCAzumTOiYhJq9kdOI9Nynpps7zMop-Q4noCNFga3ydyrzoQ/s640/find_errors.gif" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">click to zoom</td></tr>
</tbody></table>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-90491179817548819552016-11-30T10:15:00.000-05:002016-11-30T10:15:25.199-05:00Using Synchronize transformationIn order to speed up calculations EasyMorph automatically attempts to execute transformations in independent tables in parallel. Here independent means that transformations in one table don't depend on transformations in another tables. In the example below Table A is completely independent from Table B. Although, in Table B only the first three transformations are independent from Table A, because the 4th transformation (Peek) has dependency of Table A which makes all consequent transformations also dependent.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe-pNDIydXLbD-OV1WMKWdn2flM17Wzz0xbD-0ir2HEtWJYOS4hIl2csA9Upcmbf3t-EGxV0BUdxvRTcajfFXdiDPNfb6-C-rm9_hfeu5TNzdelmSOkblgUVB_wX1xZZapE98EeNoqCuE/s1600/dependent-transformations.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="187" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe-pNDIydXLbD-OV1WMKWdn2flM17Wzz0xbD-0ir2HEtWJYOS4hIl2csA9Upcmbf3t-EGxV0BUdxvRTcajfFXdiDPNfb6-C-rm9_hfeu5TNzdelmSOkblgUVB_wX1xZZapE98EeNoqCuE/s400/dependent-transformations.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">click to zoom</td></tr>
</tbody></table>
<br />
After table 'Census data' is calculated, EasyMorph will start calculating transformations in Table A and Table B in parallel. However, since the Peek transformation in Table B depends on Table A (notice the dotted line from Table A to Peek), it will calculate the first 3 transformations in Table B simultaneously with Table A, then reach Peek and wait until calculation of Table A is finished. After that it will calculate Peek and all the remaining transformations in Table B.<br />
<br />
While automatic parallelization is good for performance it may lead to conflicts in cases when transformations in independent tables attempt to modify the same external file at the same time. Most typically this happens when a project exports data into different sheets of the same spreadsheet. In order to synchronize export transformations (or other side-effect operations) in such cases the <a href="http://help.easymorph.com/doku.php?id=transformations:synchronize">Synchronize</a> transformation should be used. This transformation instructs EasyMorph to wait until another table is calculated. In another words, it inserts a dependency on another table.<br />
<br />
Below is an example where three 1-column tables are exported into three sheets of the same spreadsheet. Notice Synchronize transformations inserted before export transformations in tables "Export to Sheet2" and "Export to Sheet3". They ensure that all export transformations are executed sequentially, even if they are in different tables.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO-mCwmrOr8FUn9UPpS1MAKjQmA3at-dfHB5XBvVUSxjlyhmIPPy9dR_XX8SJ339l34TgGFUmncYg10kISmpbv1Z3jhpHWSZdLcp8pQkL2Ug77dMnpzZUtyjWPbMiAuWXLwOc5e5aWf7c/s1600/synchronize.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="286" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO-mCwmrOr8FUn9UPpS1MAKjQmA3at-dfHB5XBvVUSxjlyhmIPPy9dR_XX8SJ339l34TgGFUmncYg10kISmpbv1Z3jhpHWSZdLcp8pQkL2Ug77dMnpzZUtyjWPbMiAuWXLwOc5e5aWf7c/s400/synchronize.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">click to zoom</td></tr>
</tbody></table>
<br />Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-19834184846532621722016-11-18T10:44:00.000-05:002017-07-19T19:59:07.375-04:00Inner / full joins and union / intersectionUPDATE: The Merge transformation has got the "Full Join" mode starting from version 3.5.<br />
<br />
As you may have already noticed, there is no "Join" transformation in EasyMorph, but instead there is the "Merge" transformation which can work as left join in one of its modes. We intentionally walked away from the traditional representation of multi-modal joins (inner/left/right/full/cross) because relational joins are counter-intuitive and hard to understand not just for business users, but also many developers struggle to join data correctly. You won't believe how many SQL developers fail simple questions about joins during job interviews.<br />
<br />
As of version 3.3 the left join mode is the only join mode in the "Merge" transformation (the other mode is lookup) simply because in vast majority of real-life cases tables have to be joined using a left join. Also having fewer modes to choose from minimizes possible confusion for new users and keeps things simple. However, it doesn't mean that other types of joins can't be arranged in EasyMorph. Here is how to do it:<br />
<br />
<h4>
Inner join</h4>
Inner joins are arranged by simply inserting a "Keep matching" transformation before "Merge". The "Keep matching" transformation should remove rows where key field values don't exist in the other table thus ensuring that merging is made only for key values that exist in both tables.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjauzj0uisYOVJC7D8P9I4V1nzrW_UwG9oBbLnfN7DhvjMhDzi-_IMfNqkhrEHUwcXD8XyrP_sSOARAMfe8n_JfDMaNvSyuny8IBbgjyUS80507kiwTNAWXhyphenhyphen9GX3ogauoOpRFeWG2UWcc/s1600/EasyMorph_Inner_Join.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjauzj0uisYOVJC7D8P9I4V1nzrW_UwG9oBbLnfN7DhvjMhDzi-_IMfNqkhrEHUwcXD8XyrP_sSOARAMfe8n_JfDMaNvSyuny8IBbgjyUS80507kiwTNAWXhyphenhyphen9GX3ogauoOpRFeWG2UWcc/s640/EasyMorph_Inner_Join.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Inner join (click to zoom)</td></tr>
</tbody></table>
<br />
<h4>
Full outer join</h4>
A full join is arranged as a left join with concatenation of rows from the "right" table with key values that are missing in the "left" table and therefore were not not included in the left join.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWCwdAnaaGoQtrPbqaRCml8aoJVQn_gi8I1FrHEQGUnUQCw5EO-TVOF4E2SdAnapOq8KJmYPeeJZscigkud19vzTGHKxDf2VZVXALb9IeQQF5n__8yEFPmWmT5irtpsxFlTRrSfv9L_gA/s1600/EasyMorph_Full_Outer_Join.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="334" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWCwdAnaaGoQtrPbqaRCml8aoJVQn_gi8I1FrHEQGUnUQCw5EO-TVOF4E2SdAnapOq8KJmYPeeJZscigkud19vzTGHKxDf2VZVXALb9IeQQF5n__8yEFPmWmT5irtpsxFlTRrSfv9L_gA/s640/EasyMorph_Full_Outer_Join.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Full join (click to zoom)</td></tr>
</tbody></table>
<br />
<br />
<h4>
Union</h4>
A union of two tables is straight forward in EasyMorph: remove rows that exist in the other table using "Keep mismatching" transformation, and then append both tables using "Append" transformation.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh06jLfBRoL2r-QdpNXGDsnlYnQtGy2UTjknKf1BPgo4vOj2HYwRCW5FhSqfBUtc-nwO9F8bI-Wg4LxNxxrSBZRWkZ0k7aZZAYnuuUGk2tB7XOBOofYSiXuPnAhxeUgVa3e5OLS_VDZgL4/s1600/EasyMorph_Union.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh06jLfBRoL2r-QdpNXGDsnlYnQtGy2UTjknKf1BPgo4vOj2HYwRCW5FhSqfBUtc-nwO9F8bI-Wg4LxNxxrSBZRWkZ0k7aZZAYnuuUGk2tB7XOBOofYSiXuPnAhxeUgVa3e5OLS_VDZgL4/s640/EasyMorph_Union.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Union (click to zoom)</td></tr>
</tbody></table>
<br />
<br />
<h4>
Intersection</h4>
Finally, intersection is achieved using only the "Keep matching" transformation.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5quh4BuRo8QGWd8oJvDw1yBK9Zb4Zra2oZDgUzKlkxz3WmaRP9e2c2SGJz-zBWa6v115S1orTD1KJOD2O1R61fym9nLaEM4hMqiXmGWKnyIlmjWCT6aCPqxaxZY6TCEnl3hxm51-7uOQ/s1600/EasyMorph_Intersection.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5quh4BuRo8QGWd8oJvDw1yBK9Zb4Zra2oZDgUzKlkxz3WmaRP9e2c2SGJz-zBWa6v115S1orTD1KJOD2O1R61fym9nLaEM4hMqiXmGWKnyIlmjWCT6aCPqxaxZY6TCEnl3hxm51-7uOQ/s640/EasyMorph_Intersection.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Intersection (click to zoom)</td></tr>
</tbody></table>
<br />Unknownnoreply@blogger.comtag:blogger.com,1999:blog-5427467232743643410.post-69828905776370285972016-11-02T12:54:00.001-04:002016-11-02T13:05:51.361-04:00Preview: Query Editor and Export to DBVersion 3.3 will have two new major features: Query Editor and Export to Database transformation.<br />
<br />
Here is a brief overview of both features.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/9QQQ2Ow7m_k/0.jpg" frameborder="0" height="532" src="https://www.youtube.com/embed/9QQQ2Ow7m_k?feature=player_embedded" width="640"></iframe></div>
Unknownnoreply@blogger.com