Page Comparison

Motivation

...

SELECT 
    a.site_id, 
    a.filename,
    --
    "Position",
    "V1" 
FROM metrics a ,
json_table(DATA, '$.records[*]'
COLUMNS(
    "Position"   PATH '$.Position',
    "V1"        NUMBER PATH '$.V1'
)) AS c
WHERE a.site_id =1

...

“FilePusher” can be customized using these parameters:

• url=https://analytics.broadinstitute.org/Metrics?type=vvp
  where the file should be sent to

• filepathRegex=(.*)/(.*AspAllOutputQC.csv)
  which file-names should be picked

• delta=<delta specification>
  specifies ”how” files to be picked

  • pick 1 (or multiple) specific files
    delta=FILES_CSV /seq/tableau_files/VVPVolumeQC/20221205_RACK_QC_083303_AspAllOutputQC.csv

  • pick files timestamped between this and that timestamp
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN 2019-10-19 13:11:46 AND 2019-10-19 13:11:46

  • pick files timestamped in last 10 hours
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN -10h AND NOW

  • pick files timestamped between persisted-in-file-timestamp and now (production setup)
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN /seq/tableau_files/VVPVolumeQC/mytimestamp.txt AND NOW
    The file merely contains a timestamp(2019-10-19 13:11:46) and should be created manually

...