(2018-12-04) RCS telecon #16

Hi Erik,

Just a rectification concerning you question about the metadata for RPW Calibration Table files.

For now there is no requirement/recommandation from the ROC about the metadata (attributes) to be found inside the RCT CDF.
The only requirement is the file format (CDF) and naming that must as defined in the "RPW Data Products" document [ROC-PRO-DAT-NTT-00006] (see latest version in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents).

As said, from the ROC point of view there is no strong argument about forcing specific metadata for RCT CDF, since these files will be directly read from your software using the dedicated ROC_RCS_CAL_PATH env. variable (see "RCS ICD" [ROC-PRO-PIP-ICD-00037-LES] in https://confluence-lesia.obspm.fr/display/ROC/ROC+Documents for more details).

The main reason for defining conventions for RCT was having a common format (CDF) and standard file versioning/naming, in order to easily share RCT data between teams but also to deliver "user-friendly" RCT data to the ESAC Solar Orbiter archive. Especially, the ROC will not check and validate the RCT file consistency/compliance against specification.
If there is a good reason to add common metadata (e.g., common global attributes related to project, pipeline, author, etc.) or define additional convention (e.g., standard units), we can still discuss. But at this stage of the project, I suggest to first focus on the RPW L2 calibrated data file production.

At the end what is expected by the ROC is two set of RCT CDF files: one to be used for the RODP and one for the ROC-SGSE. The content can be the same, but the name of the RODP RCT CDF files shall comply with the [ROC-PRO-DAT-NTT-00006] convention. For ROC-SGSE RCT CDF file name it is actually free, maybe in the future we will use the same RCT CDF files for both ROC-SGSE and RODP, but for now I recommend to name your file differently, e.g., "ROC-SGSE_CAL_RPW-*.cdf". And, of course, to have as much as possible metadata (attributes) consistent with the content (e.g., not having "ROC-SGSE"-related metadata inside RODP RCT files, and vice-versa.)

Cheers,
Xavier

Hi Xavier,

When I look at the global attributes inside RCT skeletons (in DataPool git repo) I see global attributes which I assume are inherited from some other datasets, including mentionings of SGSE. There wasn't supposed to be a separate RCT skeleton for RODP too, incidentally?

  Erik P G Johansson

Indeed. 
As I said, for now the ROC does not check at all the content of the RCT file, just the file name and delivery location on the ROC server (as explained in the REGU, see ROC-GEN-SYS-NTT-00019-LES in ROC Documents). 
But we should be careful of defining properly the metadata inside these files, especially ensuring that there is no "ROC-SGSE" metadata into "RODP" RCT files.
If there is a really need of checking these files too, then we can discuss of implementing simple automated verification at the ROC level.

X.

Hi Xavier,

thank you for the meeting notes. I am sorry I could not attend. 
I have a question with the decision/suggestion of not including the Solar Orbiter position in the file (item 4), as I would say that this information is necessary for the self consistency of the data and will help external users to use RPW data in general. 
I also think that we will need the S/C position & attitude anyway to produce our data in RTN, and that every data product should do the frame transformation in the same way. One solution is that ROC should provide the rotation matrix from S/C axis to RTN, in order that each team can convert from the instrument reference frame to RTN with the same numbers. We should also agree on interpolation method if necessary.
Since we need the position information, I don’t see big problem to include them it in the files, though we can discuss if at each record or not.

Sorry if you discussed this already.

Matthieu

Hi Matthieu,

ESA and MADAWG do not recommend anymore to add Solar Orbiter position/attitude info into the CDF files for in-situ instruments. 

My feeling is that:

- For L1 CDF, we should not increase the complexity of the data production since the L1 is our main engineering data level, especially also used by the ROC team to perform operational/monitoring activities. Adding non-instrument data such as solar orbiter position/attitude at this level can: (i) reduce significantly the processing time, (ii) increase the size of the CDF files, (iii) increase the L1 data production dependency to "external" data and, thus, the risk of failures/errors. In another word, I would like to be able of generating and visualizing fluently L1 data, without be possibly "disturbed" with "external" data processing. This will be particularly true during the commissioning phase when the ROC pipeline will have to produce and view as fast as possible L1/HK data.

- For L2 CDF, it is an open question. I also recommend to reduce as much as possible the L2 data processing and CDF content complexity, in order to focus on delivering good quality calibrated L2 data (once again the SOC will provide both SPICE kernels and CDF "digest" ancillary data files to users). The best solution for the ROC would be, thus, to not include the S/C orbit/attitude data in the L2 CDF too. 

To compute the S/C to RTN frame transformation at L2 level, there could be two options: 
(A) the RCS has a direct access to the SPICE library and SolO kernels available at the ROC. 
(B) The ROC pipeline generates and makes available to the RCS, a specific CDF "digest" dataset containing the orbite/attitude information + the rotation matrix from S/C axis to RTN. 

Having separated files (CDF "digest" or SPICE kernels) allows the ROC to maintain the L1 production efficiency, and to shift the computation complexity/latency and possible risk of failure/error at L2 level, which is less critical from an operational point of view.

Xavier