NI DIAdem Navigator & Data Portal Tutorial

NAVIGATOR Introduction

The DIAdem NAVIGATOR is used to import raw data, connect to an indexing service such as DataFinder so the metadata and content imported can be searched, and it facilitates loading data into memory (Data Portal).   DataPlugins are available to support the importing of hundreds of different raw data file formats.   Imported data is saved to the TDM or TDMS file format and then indexed by a DataFinder.   The NAVIGATOR user interface provides the means to search the DataFinder index to identify data of interest by metadata at the file/root, channel group, or channel level.   Search results may then be loaded into memory and viewed with the Data Portal.  

Clicking on the NAVIGATOR panel, configures the DIAdem user interface area for NAVIGATOR tasks.   DIAdem needs to be connected to a file indexing service.   By default, the 'My DataFinder' DataFinder is installed with DIAdem and is automatically configured as the connected DataFinder.   In multi-user environments, you may have options to also connect to a shared DataFinder, or a DataStore.   You can connect to a different DataFinder or DataStore using the 'File' menu, or the Group Bars.   At the top of the application window, the title will indicate what indexing service DIAdem is currently connected to.  

Configuring Search Areas

Search Areas are the indexed folders that the DataFinder searches for data when a NAVIGATOR search is performed.   When DIAdem is installed, it may configure several folders for search areas.   It is the author's opinion that it is better configure the NAVIGATOR search areas to only index imported data saved to TDMS files, and remove any search areas that index raw data files.   You will achieve the best read/write and analysis performance in DIAdem by only working with indexed TDM/TDMS files.  

To see the current DataFinder search areas, click on the NAVIGATOR panel, and at the bottom of the screen, choose the tab labeled 'File Browser'.   By right clicking on 'Search Areas', you may add a new search area.   IMPORTANT: All imported data saved to TDMS files (.tdms) should be stored on a fast drive that is locally connected to the PC running DIAdem.   This will provide you with the best performance.   To remove a search area, right click on a search area listed below 'Search Areas' and choose the 'Remove Search Area...' option.   After adding and/or removing search areas, you should re-index the DataFinder by selecting the NAVIGATOR menu options 'Settings', 'My DataFinder', 'Configure...'.   In the 'Configure My DataFinder' dialog, click on the 'Indexer' tab and then click the 'Start Now' button.  

Importing Raw Data

DIAdem supports reading measurement data from many different file formats, including binary, text/CSV, and bus log data.   DataPlugins are used to read binary and text/CSV files.   The Bus Log Converter is used to read CAN, LIN, and FlexRay bus log data.   You can easily use the TXT Wizard to create a custom DIAdem DataPlugin for reading text or CSV data files.   In addition to the many binary file types included with a DIAdem installation, you can also visit DIAdem's website online to download a DataPlugin for hundreds of different data acquisition file types.  

NOTE: PC Administrative Rights are typically required for Data PlugIn installations.

Importing Raw Binary Files

When importing a file, the first thing you should do is try to import a raw binary file using the NAVIGATOR panel 'File', 'Open' dialog.   At the bottom right of that dialog, change the file extension filter drop down box to match the file extension of the file you wish to import (if available).   Then at the top of the dialog, browse to the folder where your file resides.   If your file extension is supported, click the 'Load' button at the bottom of the dialog and it will be loaded into memory and visible via the Data Portal.  

If your raw binary file type is not listed in the currently installed DataPlugins, then you need to visit NI's DataPlugin download website (click here) and download the appropriate DataPlugin.   If the downloaded DataPlugin file has a .uri extension, then use Windows file explorer to right-click on the file and choose 'Import' or 'Open'.   If the file extension is .exe, then simply double click on the EXE file and NI's Package Manager will guide you through the installation.  

Importing Raw Text/CSV or Excel Files

The DIAdem Text and Excel DataPlugin Wizards make it easy to import data from text, CSV, and Excel files.   From the Navigator panel, click 'File' and then choose either 'Text DataPlugin Wizard..." or 'Exel DataPlugin Wizard...".   At the end of the process, you can save the import configuration to a new DataPlugin so that it may be used again in the future.  

Import a Excel File

Before starting this tutorial, use Windows file explorer to locate the file 'C:\Users\Public\Documents\National Instruments\DIAdem ####\Data\Example.xls' DO NOT LOAD THIS FILE WITH EXCEL OR ANY OTHER APPLICATION (that will prevent NI DIAdem from loading it).  

This tutorial will demonstrate how to import the contents of an Excel file (.xls) using the 'Excel DataPlugin Wizard'.   The process for the 'Text DataPlugin Wizard' and the 'Excel DataPlugin Wizard' are very similar.  

From the NAVIGATOR panel, click on the 'File' menu and choose the option 'Excel DataPlugin Wizard...'.   A file selection dialog will appear.   Find the file 'Example.xls' and click on the 'Open' button.   The DataPlugin Wizard dialog will appear, and Step 1 'File Structure' will appear as shown below.   Notice that in this Excel sheet, the first two rows consist of heading names for the column, the third row is the units, and the data begins in the fourth row.   The first column is labeled 'Type' and consists of a drop down combo box where you can set the type of row as 'Group Properties', 'Channel Properties', 'Channel Data' or 'Ignore'.  

Set the first two rows as 'Channel Properties', because these values consist of the channel names.   Set the third row as 'Channel Properties' because this is the channel unit (channel units are stored as properties in DIAdem).   The remaining rows from four on should be set to the default of 'Channel Data'.   When configured properly, it will appear as shown in the image below.   Click the 'Next >' button at the bottom of the dialog to continue.  

If the Excel file contained multiple sheets, you could assign the sheet name as a group property, but this is not the case, so click the 'Next >' button at the bottom of the dialog to continue.  

In Step 3 'Channel Properties', DIAdem has correctly assumed that the first row is the channel name, the second row should be the channel property named 'Description', and the third row the channel unit.   Click the 'Next >' button at the bottom of the dialog to continue.  

In Step 4 'Channel Values', you assign the data type for the channel data by column, by changing the value of the drop down combo box in the second unlabeled row from the top.   It is very important to review all columns and insure the correct data type (numeric, text, time or date/time) is selected.   DIAdem will make an assumption on the data type, and the combo box choice will be 'Automatic' followed by the data type in parenthesis (data type).   Note that in the 5th column labeled 'Gear_Idx', row #1 has a value of 1, and the remaining values are NOVALUE.   This is because a blank value is in that cell from row 2 onward.   This is the correct setting.   Similarly, in the 6th column labeled 'Gear', the default data type will be text or 'Automatic (Text)', and the first five values will be the text value 'Gear', followed by NOVALUE for the remaining rows.   Click the 'Next >' button at the bottom of the dialog to continue.  

In the final Step 5 'DataPlugin', you may create a unique name for the DataPlugin you are creating.   It is strongly suggested you include the text 'Excel' in the name, and anything else that will help you know when this DataPlugin should be applied to any future files you wish to import.  

In the section labeled 'X/Y Channel Relations', the checkbox labeled 'First channel of group is x-channel' is checked.   All other columns in the spreadsheet will become Y Channels.   The first column is the X Channel with the label 'Time' and the units of 's' for seconds.   This is the correct setting.   Click the 'Finish' button at the bottom of the dialog to continue.  

The data will be imported with the DataPlugin you created, and it will be visible in the Data Portal.   Use the keys 'Alt-F6' to show the Data Portal if it is not visible.   Note that at the top of the Data Portal, the name of the file or root property 'name' is the Excel file name.   The channel group name of 'DIAdem' corresponds to the Excel sheet name.   The channels within the channel group named 'DIAdem' match the first row from each column in the Excel file.   The file is in DIAdem's memory only at this point.   To save the file, click on the NAVIGATOR panel 'File' menu and choose 'Save As'.   Change the file type to 'National Instruments TDM Streaming (*.tdms') and then click the 'Save' button.  

Importing Raw Bus Log Data

When importing Bus Log Data, it is critical to have the CAN/LIN/FlexRay bus log database files associated with the bus log data, as well as the associated bus numbers that were employed during the recording.   The physical bus lines are assigned a logical bus number between 0 and 255.   A unit under test may have several buses per type (CAN/LIN/FlexRay), all recorded simultaneously by a datalogger.   The most efficient way to perform bus log conversion is to collect the raw bus log files to be converted into a single folder, along with the related bus log databases files.   Include a Rich Text File (.RTF) in the folder with information about the relationship between the bus log database files, and the bus numbers associated with them.  

From the NAVIGATOR panel, click on the 'File' menu and select 'Open Bus Log Converter...".   Choose the file type (1) from the drop down box.   Identify the bus log file (2) to be converted.   Specify the output TDM filename and location at (3).  

Next click on the 'Edit...' button (1) in the 'Bus database configuration' area.   Click on the '+' (2) button in the Edit Bus Database Configuration dialog.   Click on the '+' (3) button in the 'Add Bus to Configuration' dialog and select a bus log database file.   Then configure the 'Bus type' from the drop down box, and then the 'Bus number'.  

If more than one bus log database file is associated with the same 'Bus type' and 'Bus number', then click the '+' (3) button again and select any additional bus log database files.   When finished, click the 'Add Bus to Configuration' 'OK' button.  

If another bus number exists, click on the '+' button in the 'Edit Bus Database Configuration' dialog and repeat the prior process for the additional bus number.   In the image below, you can see one additional CAN bus log file was configured for bus #2 (Bus Name = CAN2) as the second item in the 'Edit Bus Database Configuration' dialog list.   When done, click on the 'Edit Bus Database Configuration' 'OK' button in the dialog.  

You can perform many advanced options with the Bus Log Converter.   What is presented here is what most users will need.   At the bottom of the 'Bus Log Converter' dialog, click the 'Convert' button to begin bus log conversion.   A new file will be created, and the data will be loaded into the Data Portal.   Inspect the conversion results, and if good, then consider saving the conversion settings by clicking the button 'Save As...' in the 'Settings' section.  

If the Bus Log Conversion result is good, then it would be beneficial to add metadata to the file loaded in the Data Portal (see the next section), and then save the file as file extension .TDMS to a NAVIGATOR search folder that is indexed by the DataFinder.  

Browsing Imported Data

DIAdem works most efficiently when raw data is imported, converted to the TDM Model, and then saved to a TDMS (.tdms) binary file in a DataFinder indexed search area.   You can browse files in a search area by clicking on the NAVIGATOR panel, and then at the bottom of the custom NAVIGATOR user interface, clicking on the 'File Browser' tab.  

Expanding the 'Search Areas' icon, you will see a list of the indexed Search Areas (the exact number and names of Search Areas will vary by each installation).   Expanding a Search Area will list any sub folders and imported TDMS files.   Selecting a file and then right clicking on it will cause a pop menu to appear, with a choice 'Load Data' at the top that will read the file into DIAdem's memory and then appear in the Data Portal (Alt-F6 to show the Data Portal).  

Loading Data Into Memory

DIAdem has a graphical user interface (GUI) called the 'Data Portal' designed to make it easy to view and interact with data loaded into memory.   The Data Portal is independent of the other panels, and may be hidden, or made visible by choosing the menu options 'Window', 'Data Portal', or by the key combination 'Alt-F6'.   From the prior tutorial 'Browsing Imported Data', you loaded a previously imported TDMS file.   However, for this tutorial, and nearly every other in the series, the example files that come with the DIAdem installation will be used and referenced.   The instructions below should be used to load the primary tutorial example file 'Example_data.tdm'.  

This tutorial is based on using the DIAdem example file 'Example_data.tdm' located in the folder: 'C:\Program Files\National Instruments\DIAdem ####\Examples\Data\' (replace #### with your DIAdem version).   From the NAVIGATOR panel, choose 'File', 'New' to clear the Data Portal.   Then choose 'File', 'Open...' and the 'Open' dialog will appear.   Navigate to the folder 'C:\Program Files\National Instruments\DIAdem ####\Examples\Data\'.   Change the file filter at the bottom to 'National Instruments TDM Files'.   In the list of files, select the file 'Example_data.tdm'. At the bottom right of the dialog, click on the 'Load' button.   If the Data Portal is not visible at the right of the screen, make it visible by choosing the menu options 'Window', 'Data Portal', or by using the keys 'Alt-F6'.  

The Data Portal Basics

The Data Portal enables you to visualize and interact with the content of a TDMS/TDM data file.   If the Data Portal is not shown, the keys 'Alt-Ft' or choosing the menu 'Window' 'Data Portal' from any panel will show the Data Portal.  

The Data Portal has two tabs named 'Structure' and 'List'.   The 'Structure' tab should be selected, as it is the more intuitive way of viewing the contents of Data Portal.   Below the 'Structure' and 'List' tabs, is a section that shows the file/root, channel group, or channel properties.   Clicking on the file at the top will show the file/root properties.   Clicking on a channel group will show the channel group properties.   Clicking on a channel will show the channel properties.  

Adding Metadata To Imported Data

Adding metadata to imported data saved to TDMS files allows you to categorize the files by key/value pairs of your choice, and then later identify sets of data files based on these categories and values.   Metadata must be assigned to files, one property at a time, and then the file must be saved.   DIAdem calls metadata 'properties'.   You can add properties to the root of a file, the channel group, or a channel.  

To add a property to the file/root, select the filename shown at the top of the Data Portal to display the file/root properties in lower part of the Data Portal.   Then in the open area of the lower property section, right click and from the pop up menu, select 'New'.  

The 'Create Root Properties' dialog will appear.   Assign "Latitude" to the 'Name:', select 'Numeric' for the 'Data type', and assign the value of "30.4065" to the 'Value:' field.   Click the 'OK' button when finished.  

Repeat the process, assigning a "Longitude" property with the numeric value of "-97.7273".   When finished the Data Portal should appear as shown below.  

In order for the two properties added to persist and become included in the index, it would be necessary to save the file just edited, and re-index the search area.   Since this is an example file, don't save the file, but rather clear the Data Portal by accessing the NAVIGATOR panel 'File' menu and selecting 'New'.  

The Time Series Data Mangement System is an application built on NI DIAdem that makes it easy to add metadata to thousands of files in one session.   An always free 'Base Edition' is available for download.  

 

Searching For Data

Searching for data involves using the NAVIGATOR, connected to a DataFinder or DataStore indexing service, with Search Areas configured and indexed. The DataFinder feeds back index information to the NAVIGATOR, allowing you to interactively configure and perform searches.  

This tutorial assumes you have Search Areas configured and TDM/TDMS files in those Search Areas.   If not, then import a few data files, or configure a new Search Area to point to the example TDM/TDMS files that come with the DIAdem installation (folder located at 'C:\PROGRAM FILES\NATIONAL INSTRUMENTS\DIADEM ####\Libr\Data\').   If the Data Portal is visible, hide it by toggling it off with the 'Alt-F6' key combination.  

From the NAVIGATOR panel, click on the 'File Browser' tab at the bottom of the screen.   Configure DIAdem to perform an advanced search by choosing the menu option 'Find', 'Advanced Search'.   Clear out any existing search by choosing the menu option 'Find', 'Delete Query'.   Your DIAdem application should appear approximately like what is shown below.  

The Advanced Search Window is where you configure the search conditions to find metadata consisting of file/root, channel group, or channel properties.   Additionally, you can search by file, channel group, and channel name.  

At the top of the Advanced Search Window are three tabular columns labeled 'Level', 'Property', and 'Value'.   The 'Level' column is where you specify the source of the property to search for, such as the 'File', 'Channel Group' or 'Channel'.   In the first row (not the header), select 'File' under the 'Level' column for a file/root property.   Under the 'Property' column, in the first row click in the '<enter a property>' combo box, and available file/root properties by name will appear in the list.   Choose the 'Fielname' property.   NOTE: Filename, channel group name, channel name and other DIAdem standard properties such as 'folder' and channel 'units' are presented and treated just like custom properties in the user interface.  

To the right of the 'Property' column, is an unlabeled column where you specify the conditions such as <, >, =, etc.   The available options vary depending on the data type of the value.   Choose the equal sign (=).  

In the 'Value' column, you may enter a value directly, or depending on the data type of the value, you can click on the button to the right labeled '...' and a list or range of values will appear as shown below.   Enter into the 'Value' field the asterisk * symbol to search for all files.  

In the upper right of the Advanced Search Window, click the 'Search' button.   This will cause the 'Search Results' tab at the bottom to be activated, and then any search results will be displayed.   When searching by files (DIAdem calls this 'Results as Elements'), the 'Filename' will always appear, and then other properties from the files may appear in additional columns.   You can configure what additional properties from the files are shown by right clicking on the bar above the column titles and choosing 'Configure Results List...' from the pop up menu.   This will cause the 'Configure Search Results List' dialog to appear.   The use of this dialog is self explanatory.  

What You Learned

• The DIAdem NAVIGATOR panel allows you to import raw data, connect to an indexing service such as DataFinder, configure Search Areas (to be indexed), load data into the Data Portal, and perform a search to find file by metadata.
• Hundreds of raw binary data file types can be imported by downloading a DataPlugins from NI's website and installing the DataPlugin into DIAdem.   DIAdem has a Wizard for easily importing text/CSV and Excel data, and saving that import configuration as a custom DataPlugin.   CAN/LIN/FlexRay bus data is imported using the Bus Log Converter.  
• Saving imported raw data to a TDMS file structures the data according to the NI TDM Data Model.   The binary file is extremely efficient for subsequent read/write operations, making file loading/saving, and analysis activities execute quickly.  
• The Data Portal provides an interface where you can visually interact with the contents of a data file loaded into memory.   Properties (metadata) can be viewed, and custom properties may be added at the file/root, channel group, and channel level.