Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Main development work in https://gpinfojira.broadinstitute.org/jira/browse/RPT-5663

This is a tool to help push files to the analytics Metrics web-service.

1. Start with:

/broad/software/free/Linux/redhat_6_x86_64/pkgs/jdk1.8.0_121/bin/java -cp /home/unix/analytics/TigerETL3/remote-assembly-1.0.jar analytics.tiger.remote.FilePusher


2. Add parameters:

dryrun

TRUE if you are just testing (will still show output)
FALSE if you want to push files to the metrics service.

url

This is the actual call to the metrics webservice that should be configured with your destination table and any “outer columns” (columns not being parsed out of your files, such as filename)

(More documentation TBD)

filepathRegex

Regex to pick which files to push

e.g. (.*)/(.*AspAllOutputQC.csv) or

(.*)/([^/]*)_tableau_metrics.txt

where the starting folder is specified in the delta parameter (below).
??? Is a regex needed for files that are listed individually (like if using FILES_CSV)?

delta

tells the service where to look for files

Use FILES_CSV to manually list files

FILES_CSV /home/unix/analytics/SK-3PJO_tableau_metrics.txt,XYZ-File.txt

Use FILES IN FOLDER to specify a folder location (your regular expression above will target files more specifically)

FILES IN FOLDER /seq/tableau_files/VVPVolumeQC

??? are there other existing options here besides FILES_CSV and FILES IN FOLDER?

TIMESTAMPED defines a relative or absolute time range (optional)

push files modified in the last 10 hours:

FILES IN FOLDER /home/unix/analytics TIMESTAMPED BETWEEN -10h AND NOW


push files modified after 2019-10-19:

FILES IN FOLDER /home/unix/analytics TIMESTAMPED BETWEEN 2019-10-19 13:11:46 AND NOW'

push files modified between the previous filepusher run and now. This requires a helper txt file to be created (with an initial timestamp), and file pusher will update it with the current time when run. It then serves as the start delta for the next time the file pusher is run.

FILES IN FOLDER /home/unix/analytics TIMESTAMPED BETWEEN /home/unix/analytics/mytimestamp.txt AND NOW

??? does the timestamp helper need to be in the same location as the pushed files?

3. Put it in a Unix Script

4. Optional - Get Nasko to encode it into a URL

Full examples:

Unix script to look at production VVP files in the directory /seq/tableau_files/VVPVolumeQC using a timestamp helper 

#!/bin/bash
echo "Running production file pusher..."

/broad/software/free/Linux/redhat_6_x86_64/pkgs/jdk1.8.0_121/bin/java -cp /home/unix/analytics/TigerETL3/remote-assembly-1.0.jar analytics.tiger.remote.FilePusher 'dryrun=false'  'url=http://analytics:8090/Metrics?type=FlatFileMetrics&destination_table=ANALYTICS.VVP_QC&outer_columns=FILENAME=$1,RUNMODE=Production' 'filepathRegex=(.*)/(.*AspAllOutputQC.csv)' 'delta=FILES IN FOLDER /seq/tableau_files/VVPVolumeQC TIMESTAMPED BETWEEN /seq/tableau_files/VVPVolumeQC/VVPtimestamp.txt AND NOW'

  • No labels