QSAS: A Science Analysis System for Space Plasma Data

Data Selection

Overview

The QSAS Data Selection window allows data to be organised into databases. The window presents a hierarchical view of this data, enabling the user to determine the data availability simultaneously from several sources and perform data retrieval for selected time intervals without any knowledge of file locations or names.

Each database comprises a list of folders (and their subfolders) of data, and the same folders can appear in multiple databases to allow different views of the same data holdings. For each folder in a database a binary map file (.xxx.map) is created holding the names of the variables, their intervals of availability and the corresponding data file names. If the user has write permission into the data folder the map is stored with the data (as .qsas.map) to be available to other database views, otherwise a map file is created alongside the database (yyy.qdb) file.

Choosing Data Selector from the File menu of the QSAS Main Window brings up the Data Selection window.

Selecting Open Database from the Database menu will produce a file selection dialog for opening a database (yyy.qdb) file. This will be an ASCII file which has been previously produced using the Create Database menu item. Multiple databases may be open at once, and the Close Database option in the Database menu will close the database associated with any selected folder in the Database Listview.

Selecting the Create Database menu item will show a file selection dialog to choose the top level data folder of a data hierarchy to place in the database. Note that other folders may be added to this database later (see Database Manager below). All data files that are understood by QSAS in the selected folder and hierarchically in all sub-folders will be opened and mapped. When this is complete another window will request a name for the new database. This will be the name used for the database (.qdb) file and also the root item in the Database Listview. Further management of this database is then through the Database Manager.

All operations that may be time consuming will activate the interrupt button and progress bar. An interrupt will be applied at the end of a file operation, so interrupts are more responsive if the data are in smaller files. As a general rule, reading CEF files is an order of magnitude slower than CDF files, and a more responsive system will result for data in this latter format.

Note that the Data Selector is agnostic in terms of the data file formats, so that a single folder may contain data in any format understood by QSAS for the same data products, and continuous data will be ingested for the selected time interval irrespective of the file formats encountered. However, the disadvantage of this is that if the same data interval is held in two formats the data is duplicated in the final object on the Working List since the merge operation of importing across multiple files interlaces data according to the time tags. This can make ingestion unnecessarily slow and care should be taken in managing data folders to avoid this.

The window comprises four main components:-

Database Listview. This presents a hierarchical view of the data available for the selected database and time interval.

Data Selection Listview. This lists the data sources selected for data retrieval.

Database Manager. This is for maintenance of databases for changing holdings.

Time Interval Editor. This enables the user to manipulate the time interval to be analysed/selected.

Each of these individual components is described in more detail below. As a first experiment it may be helpful to try creating a database of the Sample Cluster Data shipped with QSAS, and such a database will be used in the following examples.

Database Listview

Once the database has been opened, a hierarchical list view of the data folders will be shown in the Database listview as in the above screen shot. Opening a folder will show the data availability over the selected time range for each variable in the folder (see screen shot below).

The data availability for a particular data source, is displayed as a single row of cells. The number of cells is determined by the width of the Database Listview window. Thus, the period of time represented by one cell is dependent on the number of cells shown and the currently selected time interval. One can change the resolution by simply editing the time interval selected in the Time Interval Editor and, to a lesser extent, by resizing the window.

The cells are highlighted green when some data is available in the interval covered by the cell and the grey when no data exists in that interval. Each interval of data in the map is considered continuous if it has no gap greater than 5% of the time interval covered by the file. Thus storing data in small files will result in larger data maps, but a finer view of actual data availability, and faster access to short data intervals.

Double clicking on a variable will copy it to the Data Selection list ready for ingesting. This is equivalent to selecting and item in the Database Listview and pressing “>> Add >>”.

Right-clicking on any data source item in the Database Listview will produce a sub-menu with options:

Sort View Up or Sort View Down – will sort the entire Database Listview by name.

Show Files - will popup a view of the data files available for the selected variable. This option has no effect at the folder level.§

Delete – will delete selected folders and variables from the Database Listview. Deleted folders will only be deleted from the database if the Re-Save Database button in the Database Manager is pressed. Individual variables cannot be removed from the database as they are stored at the map level and will return when the database is next opened.

Data Selection Tab

Selected variables appear in the Data Selection list, displayed on the right-hand side of the screen. There are three pushbuttons to the left of the Data Selection list :-

>> Add >> copies a selected item from the Database Listview to the Data Selection list.

Remove << deletes a selected item from the Data Selection list.

Clear << deletes all items from the Data Selection list.

Right-clicking on any variable in the Data Selection list will produce a sub-menu of options :-

Show Available Data Range - will display in the bottom bar of the window the total time interval where data is available for the selected variable.

Show Data Coverage in Selected Range - will display in the bottom bar of the window the percentage of the selected time interval that contains data for the selected variable.

The Get Data button at the bottom of the window will retrieve the data for all the variables in the Data Selection list (together with the associated timetags for the selected time interval, and all associated metadata) and place each of them as an object on the Working List. The Window remains open for further data selection from the same database and must be dismissed manually. The same selection may then be fetched for a different time interval. Each time Get Data is selected all variables in the Data Selection list will be imported, even if they have been fetched before. While data import is happening the interrupt button is active, and a progress bar is shown. Interrupt is done at file level, so interrupts are more responsive if data is stored in smaller files.

Database Manager Tab

Selecting the Database Manager tab replaces the Selection List with a list of buttons for managing databases and their associated data maps.

The top two buttons affect only the binary data map (.map) files associated with selected folders.

Update Data Maps is used to modify the map files describing the data holdings in a folder when data files have been added, removed, or replaced. The time stamp on a data file is used to identify when a file has changed. This action is applied to any selected folders in the Database Listview window. New sub-folders that have been added to the data area are not automatically added when the maps are updated, and must be added explicitly using Add Folder.

Delete Data Maps is used to remove the map file describing the data holdings in a folder. This action is applied to any selected folders in the Database Listview window. It is provided to make it easier to force building a new map file completely from scratch.

The centre four buttons operate on the database and its representation in the Database Listview. A database is selected if any of its folders is selected.

Create Database... duplicates the menu item in the Database Menu. It opens a file navigation dialogue window and creates a new database of the selected directory and its sub-directories.

Add Folder... opens a file navigation dialogue window and adds the selected directory and its sub-directories to the selected databases. Once a folder has been added to the database the new collection can be saved using Re-save Database View.

Remove Folder deletes the folders selected in the Database Listview from the database. These folders are deleted from both the list view and the database file.

Re-save Database View will save the current database state to the database file. This affects folders that have been added, sorted or deleted from the Database Listview.

The bottom two buttons allow deletion of a selected database. A database is selected if any of its folders is selected.

Delete Database will delete the ascii database file (.qdb) associated with a selected database(s), and close the database in the Database Listview. It does not affect the binary map files (.map) that are associated with the data folders, and these will be available to other database views. This option is preferred unless the user really intends to delete the map files since it is much faster to build new database views of a data hierarchy if the folders have already been mapped.
Delete Database and Maps will delete the ascii database file (.qdb) associated with a selected database(s), and close the database in the Database Listview. It will also delete all the binary map files (.map) that are associated with the data folders. Since creating map files takes more time than creating new database views of the folder hierarchy this option should be avoided unless a clean data hierarchy is required.

Time Interval Editor

The Time Interval Editor shows the time interval that is used for displaying data availability and data retrieval. The same time interval editor is now used across QSAS. Using the collapse button it is possible to show just the minimal interval editor of start and stop date/time or an expanded editor which also shows a centre and duration field or the full editor that also shows an outer viewport of the full interval available (shown above).

The time interval displayed will default to the period covered by the open databases. When first opened the viewport and interval are both set to the full range of the data covered by the open databases. Changing the selected interval will update the data availability shown in the Database Listview as well as showing the selection as a grey interval within the viewport bar. This permits the user to change the selection interval without losing visibility of the time range covered by the database.

The viewport bar is normally titled by the full viewport range, but placing the cursor over the bar will display the current time position.

Time intervals can be edited by modifying any of the time fields, and the last two selected fields will be active. The interval may also be changed by dragging the grey interval within the viewport bar, or by dragging either end of the interval.

The time interval editor accepts event sequences (sequences of time intervals) and these permit faster retrieval of data for pre-selected events. The time interval (end event) editor is explained more fully elsewhere.

Tips/FAQ

Managing data hierarchies is faster if data files are stored in hierarchies of folders. Each folder will have its own binary map file describing the data in that folder, and it is more efficient to keep each folder homogeneous (files contain the same data products). This also allows different views of the same data organised by resolution or spacecraft or instrument.
When a database is created the selected folder and all its subfolders are placed in the database. To limit what shows in a database select the lowest level folder applicable when creating the database and then Add Folder to add other folders in turn.
CDF files will map an order of magnitude faster than CEF files, while Get Data is also faster for CDF files.
This interface now supports Themis, MMS and some Rosetta RPC data and correctly handles time varying metadata.