This page describes how the CDF files are generated by the ROC pipelines, and what are the tools required
ROC CDF file-related convention
FOR RPW/SOLAR ORBITER CDF DATA
The SOL-SGS-TN-0009 document from ESA gives the convention to be applied by the in situ instrument teams concerning the CDF. This document can be found in https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOC+Documents.
FOR ROC-SGSE CDF DATA
The conventions concerning the RPW data - format, metadata, file naming convention, versioning, etc. - produced by the ROC-SGSE during the on-ground calibration tests at system level can be found in the "Data format and metadata definition for the ROC-SGSE" document.
ROC CDF FILE CREATION MECHANISM
Every ROC CDF data files must be produced using the CDF skeleton mechanism (for more information about CDF and skeletons, visit https://cdf.gsfc.nasa.gov/).
Figure below describes the CDF generation mechanism specific to the RPW CDF data files.
The main steps are:
- Each team writes the CDF skeleton files for their RPW sub-system data sets in a specific Excel 2007 format. It must be one Excel file per data set.
- From these Excel skeleton files, the teams can generate the corresponding CDF skeleton tables in ASCII format and the master CDF binary files in binary format, using the MASER4PY module (see section "ROC CDF SKELETON CONVERSION PROCEDURE" below).
- The CDF master files can be then used by the software to produce the CDF data files.
NOTES:
- The ROC team will also automatically generate the skeleton tables and master CDF files from the Excel format files.
- The name of the skeleton file for each data set is reported in the "Data Definition Schema File" column in the ROC-SGSE and RPW data products lists.
BUILDING CDF SCIENCE DATA FILE NAMES
The file naming convention for the RPW CDF data files is defined in [RD1].
The ROC pipelines only supplies the path to the output directory to the software, including the RCS.
In consequence, the name of the output CDF science data files shall be named by the software them-self, using the values of global attributes saved in the corresponding master CDF, the parent CDF files (e.g., L1 CDF file) and information transmitted by the software (e.g. date of creation).
For instance, the following table indicates where CDF file name fields come from in the case of the ROC-SGSE.
| Field | Origin |
|---|---|
| SourceName | Copy of the "Source_name" global attributes value in the corresponding Master CDF |
| Level | Copy of the "Level" global attributes value in the corresponding Master CDF |
| Descriptor | Copy of the "Descriptor" global attributes value in the corresponding Master CDF |
| TestId | Copy of the "Test_id" global attributes value in the parent CDF. If more than one parent CDF files is used, they all shall provide the same "Test_id" value. |
| Provider | Copy of the "Provider" global attributes value in the parent CDF. If more than one parent CDF files is used, they all shall provide the same "Provider" value. |
| VXX | Version XX of the current CDF data file, transmitted by the Software and/or from information inside the Master CDFs. |
In all case, the ROC pipelines will automatically check a posteriori the validity of the output file naming, and the consistency with the global attributes saved inside the file.
ROC CDF SKELETON FILE ARCHIVING
- The Excel skeleton files shall be delivered to the ROC team for archiving.
- The Excel skeleton files related to the ROC-SGSE (i.e., on-ground calibration tests data sets) shall be uploaded in the following ROC SVN directory:
https://version-lesia.obspm.fr/repos/ROC/GroundTests/ROC-SGSE/Data/Schemas/CDF/Excel/
- The Excel skeleton files related to the RPW Data Processing Pipeline (RDPP) (i.e., Solar Orbiter mission data sets) shall be uploaded in the following ROC SVN directory:
https://version-lesia.obspm.fr/repos/ROC/DataProcessing/RPW/Data/Schemas/CDF/Excel/
- The Excel skeleton files related to the RPW Low Latency Pipeline (RLLP) shall be uploaded in the following ROC SVN directory:
https://version-lesia.obspm.fr/repos/ROC/DataProcessing/LowLatency/Data/Schemas/CDF/Excel/
ROC CDF SKELETON FILE EXAMPLES
Here are example CDF files for RPW L1 HFR data set :
- Excel Skeleton file: https://version-lesia.obspm.fr/repos/ROC/GroundTests/ROC-SGSE/Data/Schemas/CDF/Excel/ROC-SGSE_L1_RPW-HFR-SURV_V01.xlsx
- ASCII skeleton table file: https://version-lesia.obspm.fr/repos/ROC/GroundTests/ROC-SGSE/Data/Schemas/CDF/Skeleton/ROC-SGSE_L1_RPW-HFR-SURV_V01.skt
- Binary Master CDF file: https://version-lesia.obspm.fr/repos/ROC/GroundTests/ROC-SGSE/Data/Schemas/CDF/Master/ROC-SGSE_L1_RPW-HFR-SURV_V01.cdf
ROC CDF SKELETON CONVERSION PROCEDURE
In order to convert a given Excel skeleton file "skeleton.xlsx" into a ASCII skeleton table "skeleton.skt", you need to use the MASER-PY python package.
The MASER-PY Python 3 package can be downloaded following the instructions given here:
https://sourcesup.renater.fr/wiki/maser/maser_python_package
Read the user manuel, available from the link above, for more information about the module use.
Note that the MASER-PY package program can be also used to generate both the ASCII skeleton and master CDF binary files.
Be sure that the NASA GSFC CDF software distribution V3.6 is installed on your system, and that the "skeletoncdf" program can be called from terminal.
(Visit visit [RD2] to download the GSFC CDF soft.)
Note that if the output CDF master filename is not provided, then skeletoncdf program will named it using the "CDF_NAME" parameter defined in the skeleton header.
A copy of the GSFC CDF Soft. dist. as well as the xlsx2skt module are ready to be used on the roc-dev.obspm.fr.
To load these libraries at login, just enter:
source /usr/local/roc/setup/setup_roc_env.sh
(Be sure to use the BASH shell.)
For instance to test the GSFC CDF skeletoncdf program, juste enter:
skeletoncdf
It should be returned something like
Usage: % skeletoncdf [-cdf <cdf-path>] [-[no]delete] [-[no]neg2posfp0]
[-report "<types>"] [-[no]log] [-[no]statistics]
[-cache "<sizes>"] [-[no]fillval] [-zmode <mode>]
[-backward] [-about] <skeleton-path>
Purpose: SkeletonCDF produces a CDF from a skeleton table.
...
For any issue or feedback, please contact the roc team.
SOME RULES TO KEEP IN MIND WHEN EDITING A CDF SKELETON FILE FOR THE ROC
ABOUT CDF VARIABLES
- All of the CDF variables shall be zVariables
- The first variable on the CDF file shall always be "Epoch" (becareful that the variable naming is case sensitive).
- Except "Epoch", all of the variables shall be named using capital letters only, shall start with a letter and shall contain alphanumeric plus underscore and hyphens characters only
- Variables of "Data" type shall be real or integer only: CDF_UINT1, CDF_UINT2, CDF_UINT4, CDF_INT1, CDF_INT2, CDF_INT4, CDF_REAL4, and CDF_REAL8.
- The variables shall always be non-fixed "Record variance" (i.e., "T")
- The "Number Elements" parameter of a given variable is always "1", except for CDF_[U]CHAR data type, where it gives the (maximal) number of characters in the string.
- A scalar variable shall always has a 0 dimension (i.e., "Dims = 0)
ABOUT VARIABLE ATTRIBUTES
- The "DEPEND_0" variable attribute value shall always be "Epoch".
- Depending of variable data type, the FILLVAL attribute value shall comply the ISTP guidelines (see http://spdf.gsfc.nasa.gov/istp_guide/vattributes.html#FILLVAL)
- The CDF data type of the following attributes shall match the corresponding zVariable data type: "VALIDMIN", "VALIDMAX", "SCALEMIN", "SCALEMAX", "FILLVAL".
- Make sure to always give a default value for the following attributes: "VALIDMIN", "VALIDMAX", "SCALEMIN", "SCALEMAX", "FILLVAL". If you do not know which values to give to ["VALIDMIN", "VALIDMAX"] you should use the maximal data type encoding values, but not the FILLVAL value (e.g. for a "CDF_UINT1" data type, [0, 254]). Same, if you do not know which default values for the ["SCALEMIN", "SCALEMAX"] set, you should use the ["VALIDMIN", "VALIDMAX"] values. For the FILLVAL, please use the ISTP guideline values as explained above.
ABOUT CDF HEADER
- The DATA ENCODING shall always be "NETWORK"
- The MAJORITY shall always be "COLUMN"
- The FORMAT shall always be "SINGLE"
REFERENCES
- [RD2] - CDF Home Page
- [RD3] - CDF ISTP guidelines
REFERENCES ABOUT CDF
Several language libraries are available to handle the CDF format.
Here are a non-exhaustive list:
- CDWALib - http://spdf.gsfc.nasa.gov/CDAWlib.html - IDL library with some
functions to read (read_MyCDF) or produce (IDLmakecdf) CDF files.
- SpacePy - http://spacepy.lanl.gov/ - Python library with a dedicated pycdf module.
- The official CDF web page at NASA GSFC also provides lists of software for CDF (visit http://cdf.gsfc.nasa.gov/).
Finally, the teams can find additional CDF routines in the ROC SVN repos.:
https://version-lesia.obspm.fr/repos/ROC/Resources/Library/Python/CDF/
https://version-lesia.obspm.fr/repos/ROC/Resources/Library/IDL/CDF/