This page describes how the CDF files are generated by the ROC pipelines, and what are the tools required
Table of content
RPW CDF Data Standards
RPW/SOLAR ORBITER CDF DATA
The SOL-SGS-TN-0009 document from ESA gives the convention to be applied by the in situ instrument teams concerning the CDF data products. This document can be found in https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOC+Documents.
RPW/ROC-SGSE CDF DATA
The conventions concerning the RPW data - format, metadata, file naming convention, versioning, etc. - produced on-ground by the ROC-SGSE can be found in the "Data format and metadata definition for the ROC-SGSE" document.
RPW CDF file creation mechanism
Every RPW CDF data files must be produced using the CDF skeleton mechanism (for more information about CDF and skeletons, visit https://cdf.gsfc.nasa.gov/).
Figure below describes the CDF generation mechanism specific to the RPW CDF data files.
The main steps are:
- Each team writes the skeleton CDF files for their RPW sub-system data sets in a specific ASCII format named "skeleton tables". It must be one skeleton table per data set.
- These skeleton tables are then delivered to the ROC using the mechanism defined in the "ROC Engineering Guidelines for External Users" (REGU) document (see ROC-GEN-SYS-NTT-00019-LES, available in ROC Documents).
- After verifications, the skeleton tables are then exported as "master" CDF files by the ROC team. In the same time, copies in the Excel 2007 format are also created. Both "master" CDFs and Excel copies are made available as soon as possible by the ROC, as described in the REGU.
NOTES:
- The name of the skeleton file for each data set is reported in the "Data Definition Schema File" column in the ROC-SGSE and RPW data products lists.
Building CDF file name for science data
The file naming convention for the RPW CDF data files is defined in [RD1].
During data production, the name of the output CDFs will be provided by the ROC pipelines as input arguments to the RCS software (see RCS ICD, available in ROC Documents for more details).
The following table indicates where CDF file name fields come from in the case of the RODP. This convention must comply the SOL-SGS-TN-0009 doc. specification.
| FIELD | ORIGIN |
|---|
| SourceName | "Source_name" global attribute prefix value in the corresponding Master CDF in lower case. (i.e. "solo") |
| Level | "Level" global attributes prefix value in the corresponding Master CDF (e.g. "L1") |
| Descriptor | "Descriptor" global attribute value in the corresponding Master CDF |
| dateTime | See SOL-SGS-TN-0009 |
| Free_field | Not used for RPW |
| VXX | Version XX of the current CDF data file, transmitted by the Software and/or from information inside the Master CDFs. |
CDF skeleton file storing
In the case of the ROC-SGSE
ROC-SGSE skeleton files archive is available in https://gitlab.obspm.fr/ROC/DataPool/tree/master/GSE/ROC-SGSE/CDF.
In the case of the RODP
RODP skeleton files archive is available in https://gitlab.obspm.fr/ROC/DataPool/tree/master/ROADS/RODP/CDF
ROC CDF Skeleton Conversion Tools
Conversion of a CDF skeleton from the Excel forrmat 2007 to the skeleton table ASCII format
In order to convert a given CDF skeleton file in the Excel 2007 format "skeleton.xlsx" into a ASCII skeleton table "skeleton.skt", you need to use the "maser4py" python package.
The maser4py package can be installed using pip tool for Python 3:
pip install maser4py
See https://pypi.python.org/pypi/maser4py for more details.
- The maser4py package only works with Python 3.
- An instance of the maser4py is installed on the roc-dev.obspm.fr server. To run it, you need to activate the CAWA_TEST virtual environment (the same used to test the RCS interface with the validation tool). To do it, execute the "source /opt/roc/bin/load_cawa" from your account in roc-dev. Then enter "load_cawa". If the environament is activated, you should see a (CAWA_TEST) prefix in the prompt.
|
Conversion of a CDF skeleton table to the master CDF binary file
The main way to convert a CDF skeleton table to a master CDF binary file is to use the "skeletoncdf" tool of the NASA CDF standard library (https://cdf.gsfc.nasa.gov/).
Be sure that the NASA GSFC CDF software distribution V3.6 is installed on your system, and that the "skeletoncdf" program can be called from terminal.
(Visit visit [RD2] to download the GSFC CDF soft.)
Note that if the output CDF master filename is not provided, then skeletoncdf program will named it using the "CDF_NAME" parameter defined in the skeleton header.
A copy of the GSFC CDF Soft. dist. V3.6.1 is ready to be used on the roc-dev.obspm.fr.
To load these libraries at login, just enter:
source /usr/local/roc/setup/setup_roc_env.sh
(Be sure to use the BASH shell.)
Then enter:
skeletoncdf
It should be returned something like
Usage: % skeletoncdf [-cdf <cdf-path>] [-[no]delete] [-[no]neg2posfp0]
[-report "<types>"] [-[no]log] [-[no]statistics]
[-cache "<sizes>"] [-[no]fillval] [-zmode <mode>]
[-backward] [-about] <skeleton-path>
Purpose: SkeletonCDF produces a CDF from a skeleton table.
...
For any issue or feedback, please contact the roc team.
The master CDF binary file can also be generated from the skeleton table using the maser4py package. |
Some rules to keep in mind when editing a CDF skeleton file
About CDF variables
- All of the CDF variables shall be zVariables
- The first variable on the CDF file shall always be "Epoch" (becareful that the variable naming is case sensitive).
- Except "Epoch", all of the variables shall be named using capital letters only, shall start with a letter and shall contain alphanumeric plus underscore and hyphens characters only
- Variables of "Data" type shall be real or integer only: CDF_UINT1, CDF_UINT2, CDF_UINT4, CDF_INT1, CDF_INT2, CDF_INT4, CDF_REAL4, and CDF_REAL8.
- The variables shall always be non-fixed "Record variance" (i.e., "T")
- The "Number Elements" parameter of a given variable is always "1", except for CDF_[U]CHAR data type, where it gives the (maximal) number of characters in the string.
- A scalar variable shall always has a 0 dimension (i.e., "Dims = 0)
About variable attributes
- The "DEPEND_0" variable attribute value shall always be "Epoch".
- Depending of variable data type, the FILLVAL attribute value shall comply the ISTP guidelines (see http://spdf.gsfc.nasa.gov/istp_guide/vattributes.html#FILLVAL)
- The CDF data type of the following attributes shall match the corresponding zVariable data type: "VALIDMIN", "VALIDMAX", "SCALEMIN", "SCALEMAX", "FILLVAL".
- Make sure to always give a default value for the following attributes: "VALIDMIN", "VALIDMAX", "SCALEMIN", "SCALEMAX", "FILLVAL". If you do not know which values to give to ["VALIDMIN", "VALIDMAX"] you should use the maximal data type encoding values, but not the FILLVAL value (e.g. for a "CDF_UINT1" data type, [0, 254]). Same, if you do not know which default values for the ["SCALEMIN", "SCALEMAX"] set, you should use the ["VALIDMIN", "VALIDMAX"] values. For the FILLVAL, please use the ISTP guideline values as explained above.
About CDF header
- The DATA ENCODING shall always be "NETWORK"
- The MAJORITY shall always be "COLUMN"
- The FORMAT shall always be "SINGLE"
References
CDF Tools
Several language libraries are available to handle the CDF format.
Here are a non-exhaustive list:
- SpacePy -- http://spacepy.lanl.gov/ - Python library with a dedicated pycdf module. IMPORTANT – pycdf uses the datetime module to handle the "Epoch" variable. The best time resolution reaches by datetime is microsecond, the CDF_TT2000_TIME resolution (i.e., nanosecond) is thus not covered.
- maser4idl – https://github.com/maserlib/maser4idl – IDL routines are available to read/make cdf). Required CDWAlib and the NASA CDF software to work.
Software are also available from the official NASA CDF web page (http://cdf.gsfc.nasa.gov/).