Peak BigODS Exporter connects ASAM ODS stores to Big Data Lakes

Worldwide a lot of test data is stored on the basis of the ASAM ODS standard. The need to analyze large amounts of this data is becoming increasingly important. At the same time, more and more powerful analysis tools are becoming available in the big data world. However, the “classical” storage options of the ASAM ODS standard are not well suited for these tools. For this reason, ASAM has responded by specifying an “Associate Standard” for ODS that connects existing ODS stores with big data ecosystems. The standard was published by ASAM in the first quarter of 2020.

Peak Solution started implementing the new standard at an early stage. The first version of the-so-called "Peak BigODS Exporter" was made available to customers as a licensed product from mid-2020.

Peak BigODS Exporter separates ODS information into different parts: “Meta Data” include all information regarding the context of a test or a calculation. That is e.g. the exact description of the test object and the respective setup of the test. “Mass Data” represent all information regarding measurement channels or vehicle bus data. “Further data” refer to sound, picture, video or model information, which is also taken into account when exporting.

“Meta Data” is made available for big data analysis tools by exporting them to JSON and Avro files. Reading these file formats is well supported by a wide range of programming languages and big data analysis tools.

Since in many cases it is not necessary to export all information from a classical ODS store to the big data environment, the relevant subset of information can be specified in an XML based “Export Definition File”. This file also allows to understand the semantics of the exported JSON/Avro data and to interpret their content correctly.

The storage format for “Mass data” is Parquet, optionally in two different schemas. It is a format optimized for column-based access. However, in some cases Avro may be more beneficial. Thus it has been added as an alternative format. It is more feasible for a row-based data access.

The export of “Mass data” into Parquet or Avro takes place on the measurements level. If measurements contain only a small amount of information, several of them may be combined into the same Parquet or Avro file.

Depending on the purpose of exporting to the big data environment and the subsequent analysis processes, more than one “Export Definition File” can be defined within the solution. The one to be used can be selected via the configuration settings of the export process.

All in all, companies which want to connect existing or new ODS databases with a Big Data Lake, can do this easily and in a standardized way by using the Peak BigODS Exporter.