Page Comparison

Motivation

...

• url=https://analytics.broadinstitute.org/Metrics?type=vvp
  where the file should be sent to

• filepathRegex=(.*)/(.*AspAllOutputQC.csv)
  which file-names should be picked

• outer_columns=field1=$1,field2=$2,field3=XYZ
  parse fields out of the filepathRegex and injects them in the JSON (useful when bits of data is encoded in the filename)

• dryrun=true
  run all the delta capturing and regex-parsing and show the data without actually pushing the file

• delta=<delta specification>
  specifies ”how” files to be picked

  • pick 1 (or multiple) specific files
    delta=FILES_CSV /seq/tableau_files/VVPVolumeQC/20221205_RACK_QC_083303_AspAllOutputQC.csv

  • pick files timestamped between this and that timestamp
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN 2019-10-19 13:11:46 AND 2019-10-19 13:11:46

  • pick files timestamped in last 10 hours
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN -10h AND NOW

  • pick files timestamped between persisted-in-file-timestamp and now (production setup)
    delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN /seq/tableau_files/VVPVolumeQC/VVP_etl_timestamp.txt AND NOW
    The file merely contains a timestamp(2019-10-19 13:11:46) and should be created manually

For testing purposes, an individual file can also be manually pushed by using curl:
on Linux

curl -F file="@/seq/tableau_files/VVPVolumeQC/20221205_RACK_QC_083303_AspAllOutputQC.csv" "https://analytics.broadinstitute.org/Metrics?type=vvp"

on Windows

curl -F file="@c:\myfolder\20221205_RACK_QC_083303_AspAllOutputQC.csv" "https://analytics.broadinstitute.org/Metrics?type=vvp"

Final production-grade command would look like this

/broad/software/free/Linux/redhat_6_x86_64/pkgs/jdk1.8.0_121/bin/java -cp /home/unix/analytics/TigerETL3/remote-assembly-1.0.jar analytics.tiger.remote.FilePusher 'dryrun=false' 'url=https://analytics.broadinstitute.org/Metrics?type=vvp' 'filepathRegex=(.*)/(.*AspAllOutputQC.csv)' 'delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN /seq/tableau_files/VVPVolumeQC/VVP_etl_timestamp.txt AND NOW'

...

For testing purposes, an individual file can also be manually pushed by using curl:
on Linux

curl -F file="@/seq/tableau_files/VVPVolumeQC/20221205_RACK_QC_083303_AspAllOutputQC.csv" "https://analytics.broadinstitute.org/Metrics?type=vvp"

on Windows

curl -F file="@c:\myfolder\20221205_RACK_QC_083303_AspAllOutputQC.csv" "https://analytics.broadinstitute.org/Metrics?type=vvp"

Alternative ways of automated “push” are also possible - for example push directly from within Perl script using the "Requests" library - all necessary parameters can be inferred from the curl commands above.