Bulk Import
Bulk Import allows you to import multiple data files and add metadata to them as a group (batch). You may import data files that require a DataPlugin and/or DIAdem's Bus Log Converter. Metadata is added to the data files from the filename(s), and/or the path, and/or Custom key/value pairs. Adding metadata to your files is important because it allows you to categorize and identify your data files later.
If you attempt to import the same file twice, the duplicate will be recognized (by file CRC hash) and you are given the opportunity to overwrite the old file, or abort the import. Another unique hash named 'ImportBatchHash' provides a reference to identify the complete set of files imported. By performing a Navigator search on the 'ImportBatchHash', you can quickly identify a set of imported files.
Example of Analog Data Import
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
If Bus Log Conversion is required, then see the section that follows for more information.
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
All metadata to be added has been specified. Click the 'OK' button to begin import of the files.
.png "click to see a larger image")
If you import files with the same filename as those that have been imported previously, this situation will be automatically detected, and the user will be notified and provided with an opportunity to overwrite those files, or abort.
.png "click to see a larger image")
.png "click to see a larger image")
Click on the screen image above to see some of the metadata that exists for the two files imported in this example.
Bus Log Conversion
Bus Log Conversion is dramatically simplified by using Bulk Import. Bulk Import will scan the CAN/LIN/FlexRay bus numbers and identify those that yield Bus Log data. It will also determine the ID Mode (J1939 or Default) that produces the best conversion results. See Bulk Import vs Bus Log Converter for a detail comparison by example. The screen images below demonstrate how easy Bulk Import is to use to accomplish Bus Log Conversion.
.png "click to see a larger image")
.png "click to see a larger image")
.png "click to see a larger image")
Limitations:
- Requires DIAdem version 2020 or later,
- Currently works with "My DataFinder" only (could be expanded to support any DataFinder)
- Bus log conversion of Field Bus Exchange Format (FIBEX) tested only for FLEXRAY bus log database files (not CAN or LIN), but conversion should work.
- Bus log conversion of Field Bus Exchange Format (FIBEX) with 'MOST' bus log database files is not supported.
- Bus log conversion with AUTOSAR bus log database files (*.arxml) untested. DIAdem supports CAN and CAN-FD formats but no LIN and FLEXRAY formats.
- May need minor customization to optimize import of data types not previously encountered.
New Features by Released Version
Version 20210723
- Significantly simplified the entire Bulk Import process / user interface.
- Source files to be imported must be organized into folders by data type (filename extension).
- Parsing of date/time from filenames is now significantly faster because the date/time masks are read from a file rather than being dynamically generated.
- Source folder parsing interface enhance and made more simple.
- JSON import replaced with editor that builds custom metadata key/value pairs. The custom metadata is automatically saved to a JSON file and may be used with data sets imported later.
- Bus log conversion now supports bus log file 'NI XNET (TDMS)' with bus log database of 'FlexRay', 'CAN', and/or 'LIN'.
Version 20201125
- Bus Log Conversion
Version 20200214
- Added option to import metadata via an external JSON file. This feature allows you to add root/channel group/channel metadata to all files imported, and / or you can add specific metadata to be added to the root, channel group, or channel of each data file (by filename).
Version 20200202
- Remembers the last source and destination folders (even from a prior DIAdem session)
More Bulk Import Examples
NI XNET (TDMS) + FlexRay
NI XNET (TDMS) bus log file and FlexRay bus log database file
.png "click to see larger image")
.png "click to see larger image")
.png "click to see larger image")
Bulk Import Information for Advanced Users
The optional Custom metadata list you build is saved to a JSON file in the same folder as the data and can be used for subsequent data sets you wish to Bulk Import. Each file is imported using a independent instance of DIAdem (worker), so that any issues during the import process will only impact that single file in the batch. A unique hash named 'SourceFileHash' is calculated on the raw data files and automatically added as a file property to each imported data file.
Importing Metadata by JSON File
Study the JSON sample file provided below carefully. It provides insight on how to add root, channel group, and channel properties to all data files, and specific data files. Use a JSON online validator to check the validity of your file before using it. The BulkImport.ZIP file includes sample data files and another JSON files.
{
"AllFiles": {
"root": {
"StringPropKey1": "String_abcABC1230987_01",
"NumericPropKey1": 123.88016012714181,
"IntPropKey1": 32761,
"DateTimePropKey1": "2019-07-28T20:54:01.783Z",
"boolPropKey1": true
},
"groups": {
"Example": {
"StringPropKey2": "String_abcABC1230987_02",
"NumericPropKey2": 123.88016012714182,
"IntPropKey2": 32762,
"DateTimePropKey2": "2019-07-28T20:54:02.783Z",
"boolPropKey2": true,
"channels": {
"RPM": {
"StringPropKey3": "String_abcABC1230987_03",
"NumericPropKey3": 123.88016012714183,
"IntPropKey3": 32763,
"DateTimePropKey3": "2019-07-28T20:54:03.783Z",
"boolPropKey3": true
},
"Torque": {
"StringPropKey4": "String_abcABC1230987_04",
"NumericPropKey4": 123.88016012714184,
"IntPropKey4": 32764,
"DateTimePropKey4": "2019-07-28T20:54:04.783Z",
"boolPropKey4": false
}
}
},
"Noise data": {
"channels": {
"Noise_5": {
"StringPropKey5": "String_abcABC1230987_05",
"NumericPropKey5": 123.88016012714185,
"IntPropKey5": 32765,
"DateTimePropKey5": "2019-07-28T20:54:05.783Z",
"boolPropKey5": true
}
}
}
}
},
"Example": {
"root": {
"StringPropKey6": "String_abcABC1230987_06",
"NumericPropKey6": 123.88016012714186,
"IntPropKey6": 32766,
"DateTimePropKey6": "2019-07-28T20:54:06.783Z",
"boolPropKey6": true
},
"groups": {
"Example": {
"StringPropKey7": "String_abcABC1230987_07",
"NumericPropKey7": 123.88016012714187,
"IntPropKey7": 31767,
"DateTimePropKey7": "2019-07-28T20:54:07.783Z",
"boolPropKey7": true,
"channels": {
"RPM": {
"StringPropKey8": "String_abcABC1230987_08",
"NumericPropKey8": 123.88016012714188,
"IntPropKey8": 31768,
"DateTimePropKey8": "2019-07-28T20:54:08.783Z",
"boolPropKey8": true
},
"Torque": {
"StringPropKey9": "String_abcABC1230987_09",
"NumericPropKey9": 123.88016012714189,
"IntPropKey9": 31769,
"DateTimePropKey9": "2019-07-28T20:54:09.783Z",
"boolPropKey9": true
}
}
},
"Noise data": {
"StringPropKey0": "String_abcABC1230987_00",
"NumericPropKey0": 123.88016012714180,
"IntPropKey0": 32760,
"DateTimePropKey0": "2019-07-28T20:54:00.783Z",
"boolPropKey0": true
}
}
}
}
Use the link below to contact me about acquiring this application.
Bulk Import vs Bus Log Converter
Below is a comparison of bus log conversion using Bulk Import versus DIAdem's Bus Log Converter. DIAdem's Bus Log Converter is a necessary and full featured tool for bus log conversion. However, many bus log conversion tasks may be expedited using Bulk Import.
| Bulk Import | Bus Log Converter |
|---|---|
Bulk Import accepts a folder selection so that multiple bus log files may be converted. .png "click to see a larger image")
|
DIAdem's Bus Log Converter accepts only a single bus log file for conversion. .png "click to see a larger image")
|
All of the bus log database files in the selected folder are automatically identified. .png "click to see a larger image")
|
Begin creating a configuration. .png "click to see a larger image")
|
Select the range of possible CAN/LIN/FlexRay bus numbers that may yield conversion results. The CAN/LIN/FlexRay bus number controls will only be enabled if associated bus log database files are found in the source folder. The user only needs to click the 'Build Bus Log Conversion Configuration' button in order to automatically identify all of the bus numbers and the proper Id Mode (DEFAULT or J1939) for each bus log database file. .png "click to see a larger image")
|
Select a bus log database file. .png "click to see a larger image")
|
Bulk Import is ready to begin bus log conversion of all bus log files in the selected folder based on the configuration automatically generated. |
Select a CAN bus log database file for Bus Number 1. Choose the ID Mode. If you don't know the ID Mode, then you need to attempt the configuration for both the DEFAULT and J1939 modes and review the results to decide on the appropriate selection. .png "click to see a larger image")
|
Bulk Import is ready to begin bus log conversion of all bus log files in the selected folder based on the configuration automatically generated. |
Select a LIN bus log database file for Bus Number 1.
Choose the ID Mode.
If you don't know the ID Mode, then you need to attempt the configuration for both the DEFAULT and J1939 modes and review the results to decide on the appropriate selection.
.png "click to see a larger image")
|
Bulk Import is ready to begin bus log conversion of all bus log files in the selected folder based on the configuration automatically generated. |
Repeat the prior steps in order to build a CAN and LIN configuration for Bus Line #2. If you don't know the ID Mode, then you need to attempt the configuration for both the DEFAULT and J1939 modes and review the results to decide on the appropriate selection. .png "click to see a larger image")
|
Bulk Import is ready to begin bus log conversion of all bus log files in the selected folder based on the configuration automatically generated. |
CAN and LIN configuration for Bus Lines 1 and 2 is now complete. You are now read to bus log convert a single file. .png "click to see a larger image")
|