This page reports the Frequently Asked Questions encountered during the RCS implementation activities.
The standards for Solar Orbiter science data are defined in https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/Metadata+Definition+for+Solar+Orbiter+Science+Data.
Especially, this document provides the conventions in terms of file format, naming and data versioning.
All RPW science data must comply these standards.
N.B. Solar Orbiter science data standards partially rely on the ISTP/IACG CDF guidelines (see https://spdf.gsfc.nasa.gov/sp_use_of_cdf.html).
All SOC-provided Solar Orbiter SPICE kernels retrieved by the ROC are made available in https://rpw.lesia.obspm.fr/roc/data/private/solo/soc/spice/ (restricted access).
For more details about:
A "skeleton table" is an ASCII format file used to generate a "master" binary CDF file.
The "master" binary CDF file can be then used as a template to produce series of CDF files with the same structure.
See also "How to generate a "master" CDF binary file?".
The CDF generation principle for RPW is described here.
For more information about the CDF and skeleton/master mechanism, visit http://cdf.gsfc.nasa.gov/.
The MASER4PY Python package allows to generate skeleton table or Master CDF from an Excel 2007 format file (.xlsx).
More details can be found in https://pypi.org/project/maser4py/.
There is no ESA-provided software to check CDF compliance against ESAC Solar Orbiter data standards (SOL-SGS-TN-0009).
However the ROC has started to develop a light Python program to check RPW/Solar Orbiter data compliance.
A first version of this tool can be found in https://gitlab.obspm.fr/ROC/DataPool/-/blob/master/SOLO/RPW/CDF/Tools/check_rpw_l1l2.py (restricted access).
The program must be run under Python 3.6+ and requires spacepy module. Enter 'python check_rpw_l1l2.py –help' to display help message.
This tool is not fully tested yet. Use at your own risk! |
The NASA-provided software SKTeditor (https://spdf.gsfc.nasa.gov/skteditor/) can be used to check compliance of CDF file against NASA ISTP/SPDF data standards.
An instance of the SKTeditor is installed in the roc2-dev.obspm.fr server to run CDF check using the dedicated command line interface (see "How to run the NASA SKTeditor CLI in the roc2-dev.obspm.fr sever?")
From a terminal logged in roc2-dev.obspm.fr server:
First set up the environment by entering:
source /usr/local/lib/cdf/current/bin/definitions.B export CLASSPATH=.:${CDF_JAVA}/classes/cdfjava.jar:${CDF_JAVA}/cdftools/CDFToolsDriver.jar:${CDF_JAVA}/cdfml/cdfml.jar:"$CLASSPATH" export LD_LIBRARY_PATH=.:${CDF_JAVA}/lib:${CDF_LIB}:"$LD_LIBRARY_PATH" export CLASSPATH=/usr/local/cdf/skteditor:"$CLASSPATH" |
N.B.
Then, to check the compliance of a given <cdf_filename>, enter:
java -cp /usr/local/lib/cdf/skteditor/spdfjavaClasses.jar gsfc.spdf.istp.tools.CDFCheck <cdf_filename> |
The mechanism to deliver RCS software to the ROC is presented in the "ROC Engineering Guidelines for External Users" (REGU) document (see ROC-GEN-SYS-NTT-00019-LES, available in ROC Documents).
In summary:
The RCS test data package is required by the ROC Team to check that a given version of the RCS works as expected. Every version of the RCS software must hence be delivered with its test data package.
The RCS test data package must be delivered using the sftp-lesia.obspm.fr server. Table below gives for each team the user account and target folder to be used to deliver the test data package.
Team in charge | RCS name | SFTP user account | Target folder in the SFTP server |
---|---|---|---|
BIAS | BICAS | solbia | /obs/solbia/testdata |
LFR | LFR_CALBUT | sollfr | /obs/sollfr/testdata |
SCM | SCMCAL | solscm | /obs/solscm/testdata |
TDS | TDS_CALBA | soltds | /obs/soltds/testdata |
THR | THR_CALBAR | solthr | /obs/solthr/testdata |
|
The RPW calibration table (RCT) files must be delivered as CDF files in the dedicated SFTP server at LESIA. Table below gives for each team the user account and target folder to be used to deliver the calibration table files.
Team in charge | RCS name | SFTP user account | Target folder in the SFTP server |
---|---|---|---|
BIAS | BICAS | solbia | /obs/solbia/cal |
LFR | LFR_CALBUT | sollfr | /obs/sollfr/cal |
SCM | SCMCAL | solscm | /obs/solscm/cal |
TDS | TDS_CALBA | soltds | /obs/soltds/cal |
THR | THR_CALBAR | solthr | /obs/solthr/cal |
|
The mechanism to deliver CDF skeletons to the ROC is presented in the "ROC Engineering Guidelines for External Users" (REGU) document (see ROC-GEN-SYS-NTT-00019-LES, available in ROC Documents).
In summary:
Every "commit" should be commented to have a traceability of the changes. |
RPW L3 CDF files are not generated by the ROC at LESIA, but they must be delivered to the ROC which will be in charge of validating the files, sharing them in the RPW private/public data servers and deliverying them to Solar Orbiter Archive (SOAR).
There are few rules to follow when deliverying files:
If one these rules are not fullfiled, then the ROC will reject the delivery of RPW L3 CDF files.
|
Information about the ROC software environment can be found in Software environment
Make sure the RCS delivered to LESIA are consistent with the ROC software environment. |
The descriptor file (or descriptor) is JSON format file used by the ROC pipelines to identify and call the RCS in an autonomous way.
A RCS delivered without or with a badly formatted descriptor cannot be run by the ROC pipelines.
See RCS ICD for more details about the descriptor, available in ROC Documents.
The ROC team will always performed verifications of your descriptor just after each RCS delivery.
A JSON format file, as for XML, can be validated against a schema. A JSON schema is following the specifications from http://json-schema.org/draft-04/schema. Dedicated documentation can be found at https://spacetelescope.github.io/understanding-json-schema.
A copy of the RCS descriptor JSON file “cawa-schema.json” can be found in the RCS folder of the https://gitlab.obspm.fr/ROC/DataPool Git repository. Teams are free to download and use this file to verify that their descriptor is compliant.
This schema is also showed in the appendices of the RCS ICD (available in ROC Documents).
N.B. The ROC team plans to make available an instance of its RCS Interface Validation Pipeline (RIVP) on the roc-dev.obspm.fr server. The RIVP is the light version of the ROC pipeline, which can be used to test the execution of RCS in conditions closed to the production.
The ROC pipeline calling interface is described in the RCS ICD (available in ROC Documents).
The RIVP can also be used to test the compliance (see How can be sure that my descriptor file is well formatted? above).
The RCS skeleton CDF files must deliver to the ROC team using Git.
The "DataPool" Git respository to use is hosted on the ROC Gitlab server. Before working with this repository, be sure that you have the right access permissions (if not, please contact the ROC team using the roc.support mailing list). To know how to access to this repository, see next question.
To get a local copy of the "DataPool" repository (i.e., clone), enter the following command from a terminal:
git clone https://gitlab.obspm.fr/ROC/DataPool (HTTPS)
or
git clone git@gitlab.obspm.fr:ROC/DataPool (SSH)
If everything goes right, you should have a local folder "DataPool" in your system.
By default, Git will download the "master" branch only. This branch is protected and contain the skeleton/master CDF files actually used by the ROC. In order to submit their skeleton CDF files, the RCS teams must use the dedicated "rcs" branch. |
To switch to the "rcs" branch, enter the following commands in the given order :
The first command creates the local branch "rcs" and switches to this new branch. The second one synchronizes the local branch with the remote branch "origin/rcs" on the ROC Gitlab server. The third command downloads (i.e., merges) data from the remote branch "origin/rcs" into the local branch "rcs.
NOTES:
|
Any issue related to RPW data produced by ROC (i.e., L1/HK/L1R/L2/ANC, etc.) must be submitted using the dedicated issue management board in the RCS gitlab projects.
The table below lists the URLs of the issue board for each RCS:
RCS name | Issue tracker |
---|---|
BICAS | https://gitlab.obspm.fr/ROC/RCS/BICAS/issues |
LFR_CALBUT | https://gitlab.obspm.fr/ROC/RCS/LFR_CALBUT/issues |
SCMCAL | https://gitlab.obspm.fr/ROC/RCS/SCMCAL/issues |
TDS_CALBA | https://gitlab.obspm.fr/ROC/RCS/TDS_CALBA/issues |
THR_CALBAR | https://gitlab.obspm.fr/ROC/RCS/THR_CALBAR/issues |
Issues related to L1/HK must be assigned to Xavier Bonnin.
Issues related to L1R/L2 must be assigned to Quynh Nhu Nguyen.
Full list of all issues can be read in https://gitlab.obspm.fr/groups/ROC/RCS/-/issues.
The ROC Team can also use the RCS issue boards to submit to the team in charge any problem encountered with RCS. |
The access is restricted and must be requested to roc.support sympa mailing list (i.e., roc dot support at sympa dot obspm dot fr).
The list of acronyms are available here: Acronyms and Definitions
The maser4py is installed on the roc2-dev.obspm.fr server.
To load it, run the command:
source /opt/roc/virtualenvs/maser4py/bin/activate
To check if the program works you can enter:
maser --version
Which will return the version of the program.
To get help:
maser --help
The maser4py module requires NASA CDF library to be installed to work. You can install and load a local copy on your home from https://cdf.gsfc.nasa.gov/. Or you can load the default version installed on the roc2-dev.obspm.fr server, by running the command: source /opt/roc/lib/cdf/current/bin/definition.B (Assuming BASH shell). |