Tests should be run during the RSS3VC planned December 17 to 21, 2018.
After verification, resulting L1/HK CDF will be available from the ROC Web site (private area). Can be used as samples to provide test data (TBC).
Integration tests of RCS software into the ROC pipelines will start on spring 2019. Each team will be contacted individually. The integration should be done in the following order (TBC) :
THR_CALBAR
TDS_CALBA
LFR_CALBUT
SCMCAL
BICAS
A specific pipeline instance will be deployed on the roc-dev server by ROC to test the RCS interface compliance. This instance will be available for teams. The ROC will provide an up-to-date issue of the associated use manual on January 2019.
Reminder:
RCS "ready-for-flight" software versions (full processing+RCS ICD compliance) are supposed to be ready by end of April 2019.
Priority is given to the RODP now. Nevertheless, the ROC also wants to test the RCS integration and run into the pipeline (for ROC-SGSE, RCS execution is the same expect that the descriptor and CDF skeletons are different).
ROC-SGSE will be also used during the SolO mission to perform operations tests and anomaly investigation with LESIA GSE (EM1 or EM2) on-ground.
ROC-RCS16-01: Produce and make available first examples of RODP L1/HK files – by end of 2018 – Action Xavier
ROCDATPRO-92
-
Getting issue details...STATUS
Info and status about data products is now reported in: RPW Data Products
Please take a look and report any error/obsolote info.
ROC-RCS16-02: Check RCS delivery and data products status on Confluence and report any error/inconsistency – before next RCS telecon – Action RCS teams
ROCDATPRO-93
-
Getting issue details...STATUS
2.a
TDS_CALBA data inputs
RCT skeleton and CDF files → Delivered for LFM and SWF. No other RCT file is expected for TDS.
Missing CDF RODP/ROC-SGSE dataset skeletons (see list for TDS_CALBA in RPW Data Products)
Test data to be defined
TDS_CALBA processing status
Processing L1R/L2 OK, expect for E HF
ANT PA HF calibration not implemented (we need ANT PA RCT from THR team).
Related documentation (SRS/SUM)
ROC-RCS16-03: Deliver missing CDF RODP/ROC-SGSE dataset skeletons to ROC – by end of 2018 – Action David
ROCDATPRO-94
-
Getting issue details...STATUS
ROC-RCS16-04: Deliver latest version of TDS_CALBA on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action TDS team
ROCDATPRO-95
-
Getting issue details...STATUS
2.b
BICAS data inputs
RCT - First version delivered for ROC-SGSE - To be done for RODP (same file than ROC-SGSE but different file name and possible metadata)
Missing CDF RODP/ROC-SGSE dataset skeletons (see list for BICAS in RPW Data Products)
Test data to be defined
BICAS processing status
LFR L2-E preliminary - OK
TDS L2-E products missing
Related documentation (SRS/SUM)
ROC-RCS16-05: Deliver missing RODP CDF RCT skeleton to ROC – before next RCS telecon – Action Erik
ROCDATPRO-96
-
Getting issue details...STATUS
ROC-RCS16-06: Deliver missing CDF RODP/ROC-SGSE dataset skeletons to ROC – before next RCS telecon – Action Erik
ROCDATPRO-97
-
Getting issue details...STATUS
ROC-RCS16-07: Re-check BICAS SUM delivered by Erik. Should be issue 1.0 instead of draft – before next RCS telecon – Action Xavier
ROCDATPRO-98
-
Getting issue details...STATUS
ROC-RCS16-08: Deliver latest version of BICAS on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch") – by mid-december 2018 – Action Bias team
ROCDATPRO-99
-
Getting issue details...STATUS
2.c
LFR_CALBUT data inputs
RCT - First version delivered
Missing RODP/ROC-SGSE dataset CDF skeletons (see list for LFR_CALBUT in RPW Data Products)
Test data to be defined
LFR_CALBUT processing status
Processing calibration ASM
Processing BP1/BP2 (L1 files to be processed by ROC - first samples on 01/2019 → can use Python code inside LFR_CALBUT)
Related documentation (SRS/SUM)
Deliver on mid-December 2018 to CNES for PTF
ROC-RCS16-09: Rename SOLO RPW LFR RCT CDF for RODP (see name convention in §4.3.3 of [ROC-PRO-DAT-NTT-00006]) – By end of 2018 – Action Rodrigue
ROCDATPRO-100
-
Getting issue details...STATUS
ROC-RCS16-10: Deliver latest version of LFR_CALBUT on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch") – by mid-december 2018 – Action LFR team
ROCDATPRO-101
-
Getting issue details...STATUS
2.d
SCMCAL data inputs
RCT - First version delivered
RODP/ROC-SGSE dataset CDF skeletons are delivered (see list for SCMCAL in RPW Data Products)
Test data to be defined
SCMCAL processing status
Deliver new version 0.7 - compute all ROC-SGSE datasets and one dataset RODP (with SCM status, LFM status)
Need to implemented missing RODP datasets
Only able to process from L1R. Required L1R CDF generated by other teams.
Related documentation (SRS/SUM)
ROC-RCS16-11: Deliver latest version of SCMCAL on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action SCM team (already done on 19/11/2018)
ROCDATPRO-102
-
Getting issue details...STATUS
2.e
THR_CALBAR data inputs
RCT - Skeleton and CDF not delivered
Missing RODP/ROC-SGSE dataset CDF skeletons (see list for THR_CLABAR in RPW Data Products)
Test data to be defined
THR_CALBAR processing status
B-component calibration to be implemented
Related documentation (SRS/SUM)
ROC-RCS16-12: Deliver latest version of THR_CALBAR on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action THR team
ROCDATPRO-103
-
Getting issue details...STATUS
ROC-RCS16-13: Clarify the situation concerning the expected inputs (RCT and dataset) and deliver to ROC what has been already generated – by end of 2018 – Action THR team
ROCDATPRO-104
-
Getting issue details...STATUS
quality flag excellent level (higher level) never used on Cluster
By default always best level, then reduced depending on the bitmask values
Propagate the quality flag from level to another and adjust if needed
First action is to list the factors that can impact the data quality (see Excel spreadsheets sent by Xavier at each team)
ROC-RCS16-14: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for BIAS – before end of 2018 – Action Bias team
ROCDATPRO-105
-
Getting issue details...STATUS
ROC-RCS16-15: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for TDS – before end of 2018 – Action TDS team
ROCDATPRO-106
-
Getting issue details...STATUS
ROC-RCS16-16: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for LFR – before end of 2018 – Action LFR team
ROCDATPRO-107
-
Getting issue details...STATUS
ROC-RCS16-17: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for SCM – before end of 2018 – Action SCM team
ROCDATPRO-108
-
Getting issue details...STATUS
ROC-RCS16-18: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for THR– before end of 2018 – Action THR team
ROCDATPRO-109
-
Getting issue details...STATUS
Xavier proposes to not include any ancillary data related to the Solar Orbiter spacecraft position, at least into L1 and L2 RPW data CDF files. Ancillary data will be provided by the SOC as both SPICE kernel and CDF format files. (CDF will contain "digest" orbit data.)
Nevertheless, the question concerning the reference frame for instrument coordinates zVariables is still open. At minimum the L2 should include spacecraft - centric RTN coordinates in a Cartesian representation as recommanded in the Solar Orbiter metadata definition document (SOL-SGS-TN-0009). This has to be concluded and expected zVariables added into the SOLO RPW L2 CDF.
5.
Xavier is finishing a first draft of the DPDD for RPW.
This first draft is expected to be provided to the SOC team, prior to the next SOWG on January 2019.
The draft will be sent to the teams for verifications before the end of the year 2018.
ROC-RCS16-19: Send a first draft of the DPDD to the teams for verification – by end of 2018 – Action Xavier
ROCDATPRO-110
-
Getting issue details...STATUS
6.
Next main milestone for RCS to keep in mind is the start of the integration test into the ROC pipelines. It is expected to start around Feb/March 2019 with the receiver then sensor teams.
The next main delivery will be the RCS "ready-for-flight" version still planned by end of April 2019.
Just a rectification concerning you question about the metadata for RPW Calibration Table files.
For now there is no requirement/recommandation from the ROC about the metadata (attributes) to be found inside the RCT CDF. The only requirement is the file format (CDF) and naming that must as defined in the "RPW Data Products" document [ROC-PRO-DAT-NTT-00006] (see latest version in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents).
As said, from the ROC point of view there is no strong argument about forcing specific metadata for RCT CDF, since these files will be directly read from your software using the dedicated ROC_RCS_CAL_PATH env. variable (see "RCS ICD" [ROC-PRO-PIP-ICD-00037-LES] in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents for more details).
The main reason for defining conventions for RCT was having a common format (CDF) and standard file versioning/naming, in order to easily share RCT data between teams but also to deliver "user-friendly" RCT data to the ESAC Solar Orbiter archive. Especially, the ROC will not check and validate the RCT file consistency/compliance against specification. If there is a good reason to add common metadata (e.g., common global attributes related to project, pipeline, author, etc.) or define additional convention (e.g., standard units), we can still discuss. But at this stage of the project, I suggest to first focus on the RPW L2 calibrated data file production.
At the end what is expected by the ROC is two set of RCT CDF files: one to be used for the RODP and one for the ROC-SGSE. The content can be the same, but the name of the RODP RCT CDF files shall comply with the [ROC-PRO-DAT-NTT-00006] convention. For ROC-SGSE RCT CDF file name it is actually free, maybe in the future we will use the same RCT CDF files for both ROC-SGSE and RODP, but for now I recommend to name your file differently, e.g., "ROC-SGSE_CAL_RPW-*.cdf". And, of course, to have as much as possible metadata (attributes) consistent with the content (e.g., not having "ROC-SGSE"-related metadata inside RODP RCT files, and vice-versa.)
When I look at the global attributes inside RCT skeletons (in DataPool git repo) I see global attributes which I assume are inherited from some other datasets, including mentionings of SGSE. There wasn't supposed to be a separate RCT skeleton for RODP too, incidentally?
Indeed. As I said, for now the ROC does not check at all the content of the RCT file, just the file name and delivery location on the ROC server (as explained in the REGU, see ROC-GEN-SYS-NTT-00019-LES in ROC Documents). But we should be careful of defining properly the metadata inside these files, especially ensuring that there is no "ROC-SGSE" metadata into "RODP" RCT files. If there is a really need of checking these files too, then we can discuss of implementing simple automated verification at the ROC level.
thank you for the meeting notes. I am sorry I could not attend. I have a question with the decision/suggestion of not including the Solar Orbiter position in the file (item 4), as I would say that this information is necessary for the self consistency of the data and will help external users to use RPW data in general. I also think that we will need the S/C position & attitude anyway to produce our data in RTN, and that every data product should do the frame transformation in the same way. One solution is that ROC should provide the rotation matrix from S/C axis to RTN, in order that each team can convert from the instrument reference frame to RTN with the same numbers. We should also agree on interpolation method if necessary. Since we need the position information, I don’t see big problem to include them it in the files, though we can discuss if at each record or not.
ESA and MADAWG do not recommend anymore to add Solar Orbiter position/attitude info into the CDF files for in-situ instruments.
My feeling is that:
- For L1 CDF, we should not increase the complexity of the data production since the L1 is our main engineering data level, especially also used by the ROC team to perform operational/monitoring activities. Adding non-instrument data such as solar orbiter position/attitude at this level can: (i) reduce significantly the processing time, (ii) increase the size of the CDF files, (iii) increase the L1 data production dependency to "external" data and, thus, the risk of failures/errors. In another word, I would like to be able of generating and visualizing fluently L1 data, without be possibly "disturbed" with "external" data processing. This will be particularly true during the commissioning phase when the ROC pipeline will have to produce and view as fast as possible L1/HK data.
- For L2 CDF, it is an open question. I also recommend to reduce as much as possible the L2 data processing and CDF content complexity, in order to focus on delivering good quality calibrated L2 data (once again the SOC will provide both SPICE kernels and CDF "digest" ancillary data files to users). The best solution for the ROC would be, thus, to not include the S/C orbit/attitude data in the L2 CDF too.
To compute the S/C to RTN frame transformation at L2 level, there could be two options: (A) the RCS has a direct access to the SPICE library and SolO kernels available at the ROC. (B) The ROC pipeline generates and makes available to the RCS, a specific CDF "digest" dataset containing the orbite/attitude information + the rotation matrix from S/C axis to RTN.
Having separated files (CDF "digest" or SPICE kernels) allows the ROC to maintain the L1 production efficiency, and to shift the computation complexity/latency and possible risk of failure/error at L2 level, which is less critical from an operational point of view.
5 Comments
Xavier Bonnin
Hi Erik,
Just a rectification concerning you question about the metadata for RPW Calibration Table files.
For now there is no requirement/recommandation from the ROC about the metadata (attributes) to be found inside the RCT CDF.
The only requirement is the file format (CDF) and naming that must as defined in the "RPW Data Products" document [ROC-PRO-DAT-NTT-00006] (see latest version in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents).
As said, from the ROC point of view there is no strong argument about forcing specific metadata for RCT CDF, since these files will be directly read from your software using the dedicated ROC_RCS_CAL_PATH env. variable (see "RCS ICD" [ROC-PRO-PIP-ICD-00037-LES] in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents for more details).
The main reason for defining conventions for RCT was having a common format (CDF) and standard file versioning/naming, in order to easily share RCT data between teams but also to deliver "user-friendly" RCT data to the ESAC Solar Orbiter archive. Especially, the ROC will not check and validate the RCT file consistency/compliance against specification.
If there is a good reason to add common metadata (e.g., common global attributes related to project, pipeline, author, etc.) or define additional convention (e.g., standard units), we can still discuss. But at this stage of the project, I suggest to first focus on the RPW L2 calibrated data file production.
At the end what is expected by the ROC is two set of RCT CDF files: one to be used for the RODP and one for the ROC-SGSE. The content can be the same, but the name of the RODP RCT CDF files shall comply with the [ROC-PRO-DAT-NTT-00006] convention. For ROC-SGSE RCT CDF file name it is actually free, maybe in the future we will use the same RCT CDF files for both ROC-SGSE and RODP, but for now I recommend to name your file differently, e.g., "ROC-SGSE_CAL_RPW-*.cdf". And, of course, to have as much as possible metadata (attributes) consistent with the content (e.g., not having "ROC-SGSE"-related metadata inside RODP RCT files, and vice-versa.)
Cheers,
Xavier
Xavier Bonnin
Hi Xavier,
When I look at the global attributes inside RCT skeletons (in DataPool git repo) I see global attributes which I assume are inherited from some other datasets, including mentionings of SGSE. There wasn't supposed to be a separate RCT skeleton for RODP too, incidentally?
Erik P G Johansson
Xavier Bonnin
Indeed.
As I said, for now the ROC does not check at all the content of the RCT file, just the file name and delivery location on the ROC server (as explained in the REGU, see ROC-GEN-SYS-NTT-00019-LES in ROC Documents).
But we should be careful of defining properly the metadata inside these files, especially ensuring that there is no "ROC-SGSE" metadata into "RODP" RCT files.
If there is a really need of checking these files too, then we can discuss of implementing simple automated verification at the ROC level.
X.
Xavier Bonnin
Hi Xavier,
thank you for the meeting notes. I am sorry I could not attend.
I have a question with the decision/suggestion of not including the Solar Orbiter position in the file (item 4), as I would say that this information is necessary for the self consistency of the data and will help external users to use RPW data in general.
I also think that we will need the S/C position & attitude anyway to produce our data in RTN, and that every data product should do the frame transformation in the same way. One solution is that ROC should provide the rotation matrix from S/C axis to RTN, in order that each team can convert from the instrument reference frame to RTN with the same numbers. We should also agree on interpolation method if necessary.
Since we need the position information, I don’t see big problem to include them it in the files, though we can discuss if at each record or not.
Sorry if you discussed this already.
Matthieu
Xavier Bonnin
Hi Matthieu,
ESA and MADAWG do not recommend anymore to add Solar Orbiter position/attitude info into the CDF files for in-situ instruments.
My feeling is that:
- For L1 CDF, we should not increase the complexity of the data production since the L1 is our main engineering data level, especially also used by the ROC team to perform operational/monitoring activities. Adding non-instrument data such as solar orbiter position/attitude at this level can: (i) reduce significantly the processing time, (ii) increase the size of the CDF files, (iii) increase the L1 data production dependency to "external" data and, thus, the risk of failures/errors. In another word, I would like to be able of generating and visualizing fluently L1 data, without be possibly "disturbed" with "external" data processing. This will be particularly true during the commissioning phase when the ROC pipeline will have to produce and view as fast as possible L1/HK data.
- For L2 CDF, it is an open question. I also recommend to reduce as much as possible the L2 data processing and CDF content complexity, in order to focus on delivering good quality calibrated L2 data (once again the SOC will provide both SPICE kernels and CDF "digest" ancillary data files to users). The best solution for the ROC would be, thus, to not include the S/C orbit/attitude data in the L2 CDF too.
To compute the S/C to RTN frame transformation at L2 level, there could be two options:
(A) the RCS has a direct access to the SPICE library and SolO kernels available at the ROC.
(B) The ROC pipeline generates and makes available to the RCS, a specific CDF "digest" dataset containing the orbite/attitude information + the rotation matrix from S/C axis to RTN.
Having separated files (CDF "digest" or SPICE kernels) allows the ROC to maintain the L1 production efficiency, and to shift the computation complexity/latency and possible risk of failure/error at L2 level, which is less critical from an operational point of view.
Xavier