QSAS: A Science Analysis System for Space Plasma Data

Data Handling

 

Contents

 

Overview

QSAS allows the ingestion of data from a variety of sources. In all cases, the result of data ingestion is to place one or more data objects onto the Working List.

The Data Selector (the hierarchical view of databases), Open Data File (Direct Import and browse of data files) and Write Data File (data export with join onto common timeline) menu items are accessed through the File pull-down menu on the QSAS Main Window.

Currently supported file formats include Cluster Exchange Format (CEF) files, Common Data Format (CDF) files (which should conform to at least ISTP standard, though others may also be imported) and QSAS Flat ASCII files as specified in the documentation accompanying QSAS. This file format relies on simple ASCII headers (either attached to the file or co-located) which include the capability to specify quite general time formats. Thus many (most) foreign ASCII files can be read by QSAS after writing an appropriate header. Such direct access to files enables variables to be ingested one at a time. The direct file interface allows the data values and metadata to be browsed before import.

Limited PDS format read capability is also provided (.TAB and .LBL files) although helper files are required for most Rosetta RPC files.

The results of calculations performed within QSAS may also be saved by exporting them from the Working List. QSAS supports the export of Cluster Exchange Format (CEF) files, ISTP/Cluster-compliant CDF files and flat ASCII files (tabular or delimited). If multiple time-series objects are exported, QSAS will join them onto a common time line using algorithms chosen by the user.


Data Import

Data may be ingested into QSAS by opening a file directly or using the Data Selector database view.

The Open Data File option brings up a file selector. When a file is opened a list of the variables in the file will be presented, and double clicking a variable or selecting the Get Data button will ingest that data variable along with all its metadata including timetags.  The Window remains open for further imports from the same data file and must be dismissed manually. Data may be restricted to a range of records (or time interval) using the From and To time sliders.

Selecting a Global Attribute will open an object browser display for that attribute. Pressing the Browse Data button will open an object browser display showing the data and a list of cross references from which individual attributes may be selected and displayed.

Data File Browser

The Data Selector is a more powerful interface for frequently-used databases and gives a hierarchical view of the data, which enables the user to determine the availability of data simultaneously from several sources and retrieve data for selected intervals without the needing to know their locations or file names.

Data files that have insufficient or incorrect metadata for use within QSAS may need Helper Files


Helper Files

When either Direct Import or the Data Selector is used to read data QSAS will also search the directory containing the data file for any files with suffix ".qat". All such helper files found in the directory will be read after the data file and before a variable is placed on the working list. The helper files have the same syntax for attributes contained within variable blocks as cef files. Any attribute information in a variable block will be attached to a variable if it has the same name as the variable block identifier. If the attribute is already present it will be replaced - thus a helper file can be used to correcct metadata as well as add it. No other information about a variable can be read from a helper file, and in this respect it differs from a detached header file. Care must be taken with helper files as the attribute information will be available to any file in the same directory and will silently be applied if the file contains a variable with the same name as one of the variable blocks. Multiple helper files may be placed in a directory and all of them will be read. There is no guarantee of reading order, and hence no control over which helper file takes precedence if a conflict arises.

Helper files should be used sparingly and kept in a dedicated directory containing the data set to which they apply. A sample helper file "ThemisFGM.qat" can be found in the examples directory in the QSAS installation. This shows the syntax for modifying metadata for three of the variables in Themis sample FGM cdf files.


Exporting Data

Data objects on the Working List, regardless of whether they have been imported from other data files or created by analysis within QSAS, can be exported to files by choosing Export from the File menu on the Working List. This brings up the Export User Interface as shown. Data Objects to be exported must be dragged or copied and pasted from the Working List (or any other list view showing an object held on the working list) to the export list. Objects on the export list can be grouped into folders.

export window

Time Series data objects will be exported together by joining them onto a SINGLE set of time tags, so the source of these tags must be specified (by default the first time series data object is used as the timetag source). The behaviour of this window is identical to the join window. The user may also specify a time interval for the export (i.e., to export only a subset of the data) using a time series data object, a time interval object, or the Time Editor. Since more than one Time Series object may be exported, they must be joined onto the common time tags using either linear interpolation or boxcar averaging, with data gaps filled or removed (see the Analysis section for more details).

Join options will be applied to any selected items or folders, and different choices may be made for different selections. The time tag and time interval for the exported data apply to all items being exported as QSAS unless Do Not Join is selected. In the case that data are not joined and of different lengths the export window will give an option to pad to the same length. Only a CDF file may have variables of different lengths.

On selecting Export a dialogue appears allowing the user to choose the file name and directory (QSAS will make an automatic choice by default) and provides a check box to indicate whether an identically named file should be overwritten or the export aborted (to allow the user to specify a new, unique name). The bottom selection slot allows the user to choose the file type to be written. If one of the ASCII types is chosen the panel on the right will affect the choices for these  files. Note: at present these options always appear active even if they have no effect on the file type selected.

Export file selector

Tips/FAQ

In the Direct Import window, all quantities identified as variables in the data file are shown. In CDF files, and their QSAS ASCII equivalents, many of these do not vary with records, and include labelling and other information. These do not need to be separately ingested unless the user needs them for other purposes.

When Time Series data are imported the time tag variable (usually denoted "epoch" in CDF files) is automatically ingested at the same time. The time tags will not appear on the working list, but are kept internally as cross references to the data objects.

Foreign ASCII files can usually be imported by first writing a simple ASCII header to describe the variables, file format, and time specification. These headers are described in other documentation.


Last up-dated: October 2016 Tony Allen