Objectif Lune joins Upland Software.Learn more >

Back to home

DataMapper Improvements

Original Author: Philippe Fontan

As Connect’s popularity grows, so does the variety of data models it must be able to create efficiently. Over
the last year, we’ve seen several implementations that require hundreds and even thousands of fields to be
extracted for each record. Designing the data mapping process for such implementations can become
cumbersome because the DataMapper’s user interface (UI) would become sluggish and would not allow the
user to personalise their view of the data model. The latest version of the DataMapper aims to correct those

The Data Model pane

The Data Model pane is where a user can view all the fields and tables that are being extracted. In order to make it easier to work with a large number of fields, the Data Model pane now includes several improvements that allow each user to personalise their view of the fields and tables in the list.

Grouping/Ungrouping items

Fields can now be grouped into multiple levels. Simply select one or many fields using the usual Windows methods (SHIFT-Click for consecutive fields, CTRL-Click for non-consecutive fields) and then select Add Group from the context menu (or simply press CTRL-SHIFT-Insert). This creates a new group whose name can be set to anything the user wants. Within that group, it is possible to select another series of fields and add a new group, which will then be nested under the first one, and so on.

Once a group has been created, the user can drag and drop any other field in or out of the group. The group can
be collapsed or expanded to hide/show its contents. The user may also ungroup items, and once a group is empty, may delete it altogether.

Expanded groups

Collapsed groups

This makes it much easier to work with related items, as illustrated above. The individual fields that make up a shipping address can be gathered into a Shipping Address group. Then, a Shipping Address group and a Billing Address group can themselves be grouped into a Customer Addresses group. These kinds of fields are usually found in static positions in the data source and once they’ve been extracted, there is little interest in always having them displayed in the Data Model pane. So by collapsing the Customer Addresses group, the user is able to “hide” all the fields that make up each individual address, thereby leaving more space for other fields.

Note! Grouping and ungrouping also works with tables and nested tables. These are considered “group items” just like fields are. The only difference is that any field that belongs to a table may only be added to a group that’s stored inside that same table.

Ordering items

Fields, tables and groups can now be re-ordered to create a fully personalised view into the data model. Multiple fields can be selected at once and dragged & dropped higher or lower in the Data Model pane, regardless of the order in which the data is actually being extracted. Hotkeys and context menu options also allow the user to quickly move fields to the very top or the very bottom of their parent group, table or record.

This allows the user, for instance, to drag to the top of the Data Model some fields that need to be validated visually across several records, thereby avoiding having to scroll through the list of fields when moving from one record to the next. By the same token, items that don’t need to be visible for each record (for instance, the Customer Addresses group mentioned above) can be sent to the bottom of the Data Model, leaving more space available for important fields and tables.

Note! Ordering also applies to tables and groups. However, items inside groups and tables may only be
re-ordered within the same parent item (i.e. a detail field may only be moved up or down inside the
detail table to which it belongs).

Permanent personalisation

The Grouping and Ordering information is now saved inside the data mapping configuration file. So when a user opens a configuration file, it contains the same Grouping and Ordering information that it did when it was last saved.

IMPORTANT NOTE! The Groups and the Order have absolutely no impact on the content and hierarchy of the data that’s being extracted. So even if a detail table belongs to a group named Line Items, which itself belongs to a group named Invoices, the detail table is accessed with the standard record.detail syntax. In other words, the Grouping and Ordering information is purely a user-experience feature that has no impact on the database itself.

Filtering the view

Let’s face it: even with nested groups and a custom field order, it’s sometimes hard to find a specific field or table inside a Data Model that contains hundreds of them. Accordingly, the Data Model pane now allows the user to filter in the items displayed in the list by simply typing a few characters in the Filter box. This immediately narrows down the list of displayed items to all tables/groups/fields whose name contains the characters that were typed.

Filtering items allows the user, with just a few keystrokes, to quickly access any item in the Data Model. Note that the filtering is case-insensitive. For instance, typing “street” in the filter box might reduce the Data Model pane to a list of two fields: ship Street address and bill Street address.
dataMapper 1

The Designer

The Data Model pane is shared between the Designer and the DataMapper. Therefore, all the GUI improvements to the Data Model pane (grouping, ordering, filtering) are also available in the Designer.

One interesting feature is that the user may elect to display the Data Model differently in the Designer and in the DataMapper. This allows each user to personalise the view according to their own preferences or to the task at hand. The Designer’s view is saved inside the Template, just like the DataMapper’s view is saved inside the DM configuration file.

The user may synchronize both views by clicking the Synchronize fields button in the Data Model pane
(accordingly, the button’s description has been changed to Synchronize fields and structure).

Note! When only the structure (i.e. grouping and ordering information) of the data model is different between the Designer and the DataMapper, no warning is issued (i.e. the Synchronize button isn’t displayed in red), as it would be when the list of ields and tables is different. That’s because there is no incompatibility between the data models since they both contain the ame fields and tables. But the user may still click the button to synchronize the structures.

The Steps pane

UI improvements
The Data Model pane is not the only area in the DataMapper that may suffer from too much information. For complex data mapping configurations, the Steps pane may also contain tens, even hundreds of steps that may make the UI sluggish when the user navigates from step to step. Accordingly, the UI has been modified to redraw much more quickly than it used to.

Visually speaking, the dotted lines that used to connect each Step have been reworked to now use full lines that are being drawn onto the display, instead of being blitted as bitmaps. To match the new look, all Step icons have been redesigned accordingly. In addition to those visual elements, the background processes have also been overhauled to reduce the number of refresh operations occurring while working in the DataMapper.

DataMapper 2

While these changes may hardly appear noteworthy, they result in a significantly improved user experience, especially with DM configs that contain a large number of steps, both vertically and horizontally. Scrolling through the Steps in such large configurations is no longer painful.

Finding a Step
Being able to scroll quickly through the list of Steps is one thing, but sometimes a user still has to sort through hundreds of Steps in order to find that specific one that needs to be modified or inspected. This could take a while, even when each Step’s description has been carefully designed to make it uniquely identifiable.

To make it easier, the Steps pane now includes a Find option that allows the user to type a few characters in order to fetch a list of Steps whose description contains those characters (case insensitive). The find operation occurs instantly as each character is being typed, with the results displayed next to the Find box. The user can click on the Next match / Previous match buttons to immediately jump to the next/previous Step in the list of matches.datamapper 3

This improvement also makes the proper naming of each Step all the more relevant since doing so now allows user to quickly access Steps through the Find option.

XML Performance

One recurring issue that has been reported with the DataMapper is the fact that it can be challenging to use large individual XML records when designing a data mapping configuration. Often times, users would have to use stripped down sample data files to design the DM config, simply because the UI became too sluggish if the document contained too many XML elements.

To alleviate those issues, the XML Viewer has been completely overhauled in order to use the same type of display that is available in the TEXT viewer. This translates into a hugely more responsive UI, while maintaining the typical tree display of XML elements and attributes. It is now possible to scroll through extremely large records (i.e. thousands of elements) as easily as with a text file.

But the changes are not limited to the UI; under the hood, the XML data mapping operation has also been greatly improved, with the result that XML data mapping operations should run significantly faster, especially for larger records.

Note! These improvements pertain to the speed and responsiveness of the XML data mapping. They do not mean that the DataMapper can handle individual records larger than what it could process in previous versions. Rather, it means that XML records it could already handle can now be processed much faster. As far as larger individual XML documents are concerned, we are still working on different approaches that will eventually make their way into the application.

XML Boundaries

For XML data streams, it is now possible to set Document boundaries with an XPATH statement, which allows the DataMapper to natively process files that would previously have required an XSLT transform. So for instance, with the following XML structure, it is now possible to specify that each item on the second level is a document boundary:


To do so, the following XPATH statement statement could be used in the Document boundaries:

/root/*[starts-with(name(), 'element')]

Leave a Reply

Your email address will not be published. Required fields are marked *