Skip to end of metadata
Go to start of metadata

Goals

  • Status on RCS works

Date

Attendees


  • Xavier Bonnin

  • Jean-Yves Brochot
  • Thomas Chust
  • Erik Johansson
  • Bruno Katra
  • Yuri Khotyaintsev
  • Matthieu Kretzchmar (apologies)
  • Milan Maksimovic (apologies)
  • Lorenzo Matteini
  • Rodrigue Piberne
  • Quynh Nhu Nguyen
  • David Pisa

  • Jan Soucek
  • Andris Vaivads (apologies)

Agenda

  1. ROC pipelines software, data products and documentation status 
  2. RCS software, data products and documentation status
    1. TDS
    2. BIAS
    3. LFR
    4. SCM
    5. THR
  3. Discussion on science verification/validation 
    1. QUALITY_BITMASK/FLAG : Lessons from MMS by Yuri
    2. Proposed implementation for RPW (TBC)
  4. Ancillary parameters in RPW CDF files
  5. RPW Data Product Description Document (DPDD)
  6. Planning
  7. Next RCS telecon
  8. AOB



Discussion items

ItemNotesAction-items
1.
  • ROC-RCS16-01: Produce and make available first examples of RODP L1/HK files – by end of 2018 – Action Xavier ROCDATPRO-92 - Getting issue details... STATUS
2.
  • ROC-RCS16-02: Check RCS delivery and data products status on Confluence and report any error/inconsistency – before next RCS telecon – Action RCS teams ROCDATPRO-93 - Getting issue details... STATUS
2.a
  • TDS_CALBA data inputs
    • RCT skeleton and CDF files → Delivered for LFM and SWF. No other RCT file is expected for TDS.
    • Missing CDF RODP/ROC-SGSE dataset  skeletons (see list for TDS_CALBA in RPW Data Products)
    • Test data to be defined
  • TDS_CALBA processing status
    • Processing L1R/L2 OK, expect for E HF
    • ANT PA HF calibration not implemented (we need ANT PA RCT from THR team).
  • Related documentation (SRS/SUM)
  • ROC-RCS16-03: Deliver missing CDF RODP/ROC-SGSE dataset skeletons to ROC – by end of 2018 – Action David ROCDATPRO-94 - Getting issue details... STATUS
  • ROC-RCS16-04: Deliver latest version of TDS_CALBA on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action TDS team ROCDATPRO-95 - Getting issue details... STATUS
2.b
  • BICAS data inputs
    • RCT - First version delivered for ROC-SGSE  - To be done for RODP (same file than ROC-SGSE but different file name and possible metadata)
    • Missing CDF RODP/ROC-SGSE dataset skeletons (see list for BICAS in RPW Data Products)
    • Test data to be defined
  • BICAS processing status
    • LFR L2-E preliminary - OK
    • TDS L2-E products missing
  • Related documentation (SRS/SUM)
  • ROC-RCS16-05: Deliver missing RODP CDF RCT skeleton to ROC – before next RCS telecon – Action Erik ROCDATPRO-96 - Getting issue details... STATUS
  • ROC-RCS16-06: Deliver missing CDF RODP/ROC-SGSE dataset skeletons to ROC – before next RCS telecon – Action Erik ROCDATPRO-97 - Getting issue details... STATUS
  • ROC-RCS16-07: Re-check BICAS SUM delivered by Erik. Should be issue 1.0 instead of draft – before next RCS telecon – Action Xavier ROCDATPRO-98 - Getting issue details... STATUS
  • ROC-RCS16-08: Deliver latest version of BICAS on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch") – by mid-december 2018 – Action Bias team ROCDATPRO-99 - Getting issue details... STATUS
2.c
  • LFR_CALBUT data inputs
    • RCT - First version delivered
    • Missing RODP/ROC-SGSE dataset CDF skeletons (see list for LFR_CALBUT in RPW Data Products)
    • Test data to be defined
  • LFR_CALBUT processing status
    • Processing calibration ASM
    • Processing BP1/BP2 (L1 files to be processed by ROC - first samples on 01/2019 → can use Python code inside LFR_CALBUT)
  • Related documentation (SRS/SUM)
  • Deliver on mid-December 2018 to CNES for PTF
  • ROC-RCS16-09: Rename SOLO RPW LFR RCT CDF for RODP (see name convention in §4.3.3 of [ROC-PRO-DAT-NTT-00006]) – By end of 2018 – Action Rodrigue ROCDATPRO-100 - Getting issue details... STATUS
  • ROC-RCS16-10: Deliver latest version of LFR_CALBUT on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch") – by mid-december 2018 – Action LFR team ROCDATPRO-101 - Getting issue details... STATUS
2.d
  • SCMCAL data inputs
    • RCT - First version delivered
    • RODP/ROC-SGSE dataset CDF skeletons are delivered (see list for SCMCAL in RPW Data Products)
    • Test data to be defined
  • SCMCAL processing status
    • Deliver new version 0.7 - compute all ROC-SGSE datasets and one dataset RODP (with SCM status, LFM status)
    • Need to implemented missing RODP datasets
    • Only able to process from L1R. Required L1R CDF generated by other teams.
  • Related documentation (SRS/SUM)
  • ROC-RCS16-11: Deliver latest version of SCMCAL on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action SCM team (already done on 19/11/2018) ROCDATPRO-102 - Getting issue details... STATUS
2.e
  • THR_CALBAR data inputs
    • RCT - Skeleton and CDF not delivered
    • Missing RODP/ROC-SGSE dataset CDF skeletons (see list for THR_CLABAR in RPW Data Products)
    • Test data to be defined
  • THR_CALBAR processing status
    • B-component calibration to be implemented
  • Related documentation (SRS/SUM)
  • ROC-RCS16-12: Deliver latest version of THR_CALBAR on the dedicated ROC Git repos. (if possible tagged version on master branch, otherwise at least dev. version on a "dev branch"). – by mid-december 2018 – Action THR team ROCDATPRO-103 - Getting issue details... STATUS
  • ROC-RCS16-13: Clarify the situation concerning the expected inputs (RCT and dataset) and deliver to ROC what has been already generated – by end of 2018 – Action THR team ROCDATPRO-104 - Getting issue details... STATUS
3.
  • See yuri slide (bitmask_quality.pdf)
  • quality flag excellent level (higher level) never used on Cluster
  • By default always best level, then reduced depending on the bitmask values
  • Propagate the quality flag from level to another and adjust if needed
  • First action is to list the factors that can impact the data quality (see Excel spreadsheets sent by Xavier at each team)
  • ROC-RCS16-14: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for BIAS – before end of 2018 – Action Bias team ROCDATPRO-105 - Getting issue details... STATUS
  • ROC-RCS16-15: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for TDS – before end of 2018 – Action TDS team ROCDATPRO-106 - Getting issue details... STATUS
  • ROC-RCS16-16: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for LFR – before end of 2018 – Action LFR team ROCDATPRO-107 - Getting issue details... STATUS
  • ROC-RCS16-17: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for SCM – before end of 2018 –  Action SCM team ROCDATPRO-108 - Getting issue details... STATUS
  • ROC-RCS16-18: Complete if required "rpw_dataset_bitmask_definition_v01.xlsx" Excel spreadsheet with expected content for THR– before end of 2018 – Action THR team ROCDATPRO-109 - Getting issue details... STATUS
4.
  • See Solar Orbiter data convention has changed : see SOL-SGS-TN-0009 issue 2.3 in https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/SOC+Documents. Especially, no recommandation is planned concerning the Solar Orbiter spacecraft position inside CDF.
  • Xavier proposes to not include any ancillary data related to the Solar Orbiter spacecraft position, at least into L1 and L2 RPW data CDF files. Ancillary data will be provided by the SOC as both SPICE kernel and CDF format files. (CDF will contain "digest" orbit data.)
  • Nevertheless, the question concerning the reference frame for instrument coordinates zVariables is still open. At minimum the L2 should include spacecraft - centric RTN coordinates in a Cartesian representation as recommanded in the Solar Orbiter metadata definition document (SOL-SGS-TN-0009). This has to be concluded and expected zVariables added into the SOLO RPW L2 CDF.



5.
  • Xavier is finishing a first draft of the DPDD for RPW.
  • This first draft is expected to be provided to the SOC team, prior to the next SOWG on January 2019.
  • The draft will be sent to the teams for verifications before the end of the year 2018.
  • ROC-RCS16-19: Send a first draft of the DPDD to the teams for verification – by end of 2018 – Action Xavier ROCDATPRO-110 - Getting issue details... STATUS
6.
  • Next main milestone for RCS to keep in mind is the start of the integration test into the ROC pipelines. It is expected to start around Feb/March 2019 with the receiver then sensor teams.
  • The next main delivery will be the RCS "ready-for-flight" version still planned by end of April 2019.

7.
  • Next RCS telecon is planned on at 2pm.

8.

Action items

Open issues

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

Closed issues

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

Attached items

5 Comments

  1. Hi Erik,

    Just a rectification concerning you question about the metadata for RPW Calibration Table files.

    For now there is no requirement/recommandation from the ROC about the metadata (attributes) to be found inside the RCT CDF.
    The only requirement is the file format (CDF) and naming that must as defined in the "RPW Data Products" document [ROC-PRO-DAT-NTT-00006] (see latest version in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents).

    As said, from the ROC point of view there is no strong argument about forcing specific metadata for RCT CDF, since these files will be directly read from your software using the dedicated ROC_RCS_CAL_PATH env. variable (see "RCS ICD" [ROC-PRO-PIP-ICD-00037-LES] in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents for more details).

    The main reason for defining conventions for RCT was having a common format (CDF) and standard file versioning/naming, in order to easily share RCT data between teams but also to deliver "user-friendly" RCT data to the ESAC Solar Orbiter archive. Especially, the ROC will not check and validate the RCT file consistency/compliance against specification.
    If there is a good reason to add common metadata (e.g., common global attributes related to project, pipeline, author, etc.) or define additional convention (e.g., standard units), we can still discuss. But at this stage of the project, I suggest to first focus on the RPW L2 calibrated data file production.

    At the end what is expected by the ROC is two set of RCT CDF files: one to be used for the RODP and one for the ROC-SGSE. The content can be the same, but the name of the RODP RCT CDF files shall comply with the [ROC-PRO-DAT-NTT-00006] convention. For ROC-SGSE RCT CDF file name it is actually free, maybe in the future we will use the same RCT CDF files for both ROC-SGSE and RODP, but for now I recommend to name your file differently, e.g., "ROC-SGSE_CAL_RPW-*.cdf". And, of course, to have as much as possible metadata (attributes) consistent with the content (e.g., not having "ROC-SGSE"-related metadata inside RODP RCT files, and vice-versa.)

    Cheers,
    Xavier

  2. Hi Xavier,

    When I look at the global attributes inside RCT skeletons (in DataPool git repo) I see global attributes which I assume are inherited from some other datasets, including mentionings of SGSE. There wasn't supposed to be a separate RCT skeleton for RODP too, incidentally?

      Erik P G Johansson

  3. Indeed. 
    As I said, for now the ROC does not check at all the content of the RCT file, just the file name and delivery location on the ROC server (as explained in the REGU, see ROC-GEN-SYS-NTT-00019-LES in ROC Documents). 
    But we should be careful of defining properly the metadata inside these files, especially ensuring that there is no "ROC-SGSE" metadata into "RODP" RCT files.
    If there is a really need of checking these files too, then we can discuss of implementing simple automated verification at the ROC level.

    X.

  4. Hi Xavier,

    thank you for the meeting notes. I am sorry I could not attend. 
    I have a question with the decision/suggestion of not including the Solar Orbiter position in the file (item 4), as I would say that this information is necessary for the self consistency of the data and will help external users to use RPW data in general. 
    I also think that we will need the S/C position & attitude anyway to produce our data in RTN, and that every data product should do the frame transformation in the same way. One solution is that ROC should provide the rotation matrix from S/C axis to RTN, in order that each team can convert from the instrument reference frame to RTN with the same numbers. We should also agree on interpolation method if necessary.
    Since we need the position information, I don’t see big problem to include them it in the files, though we can discuss if at each record or not.

    Sorry if you discussed this already.

    Matthieu

  5. Hi Matthieu,

    ESA and MADAWG do not recommend anymore to add Solar Orbiter position/attitude info into the CDF files for in-situ instruments. 

    My feeling is that:

    - For L1 CDF, we should not increase the complexity of the data production since the L1 is our main engineering data level, especially also used by the ROC team to perform operational/monitoring activities. Adding non-instrument data such as solar orbiter position/attitude at this level can: (i) reduce significantly the processing time, (ii) increase the size of the CDF files, (iii) increase the L1 data production dependency to "external" data and, thus, the risk of failures/errors. In another word, I would like to be able of generating and visualizing fluently L1 data, without be possibly "disturbed" with "external" data processing. This will be particularly true during the commissioning phase when the ROC pipeline will have to produce and view as fast as possible L1/HK data.

    - For L2 CDF, it is an open question. I also recommend to reduce as much as possible the L2 data processing and CDF content complexity, in order to focus on delivering good quality calibrated L2 data (once again the SOC will provide both SPICE kernels and CDF "digest" ancillary data files to users). The best solution for the ROC would be, thus, to not include the S/C orbit/attitude data in the L2 CDF too. 

    To compute the S/C to RTN frame transformation at L2 level, there could be two options: 
    (A) the RCS has a direct access to the SPICE library and SolO kernels available at the ROC. 
    (B) The ROC pipeline generates and makes available to the RCS, a specific CDF "digest" dataset containing the orbite/attitude information + the rotation matrix from S/C axis to RTN. 

    Having separated files (CDF "digest" or SPICE kernels) allows the ROC to maintain the L1 production efficiency, and to shift the computation complexity/latency and possible risk of failure/error at L2 level, which is less critical from an operational point of view.

    Xavier