Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

This web-server supports following services:

  • JqlTableauExtractService - extracts data from JIRA into Tableau as native TDE file.
  • sampleWorkflow - records sampleWorkflow-transitions in DB (used by RQC reports, etc)
  • sampleAnalyticsWorkflow - new version sampleWorkflow
  • labopsjira / BspFreezerLocations - support SRS tools
  • CromwellViewer - monitor pipeline workflows (see also Cromwell API)
  • txtExtract - extract data from datamarts as csv/tsv
  • IlluminaSummary - used by John Walsh to upload summary-file to DWH
  • IlluminaMetrics - used by Walsh to upload "barcodeCounts" files to DWH
  • DependencyVisualizer
  • ETLDashboard - ATTN: Actual ETLs (running on UNIX) are not affected even if Dashboard is down.
  • "generate_sql" service generates automatically a JIRADWH query for extracting data from JIRA
  • JiraWebConnector - alternative interface to get data into Tableau (failed to gain popularity)
  • codeAssistant

  • troubleshooting tools: sqlMonitor, peekattable, LockedObjects, 

Services in red above are being used in production tools, others are for development and maintenance purposes.




CromwellViewer

txtExtract service

created in December 2018 (see RPT-5119, RPT-5179)

Obtain data as a direct extract from Analytics REST API by running this curl command from any UNIX server, Windows command prompt, or Mac terminal.

command: curl -XGET -F file="@<YOUR_SOURCE_FILE>" "http://analytics:8090/api/v2/csvExtract?extractName=<YOUR_EXTRACT_NAME>" -o <RESULTS_FILE>
example: curl -XGET -F file="@/home/unix/connolly/exampleinput_exomecallset2018.csv" "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations" -o exampleoutput_exomecallset2018.csv
  1. In the above example, callSetAggregations is a parameter on the csvExtract service:  "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations"
  2. User prepares a list of records in a CSV file with no headers or extra columns or rows (above example requires project,sample pairs)
  3. The first step of the curl command uploads the input file to the service, which records the data into dataset_items table with a new dataset_id. It then injects that dataset_id into the where clause in the transformation SQL.
  4. If the -o <RESULTS_FILE> is omitted, then the results will be printed out in the standard output (the console).

Extending the service:

A number of configuration files live here: \\neon-cifs\home_unix\analytics\TigerETL3\resources

extractSQL.conf is one of the files, called by the csvExtract service. Within this file is the ETL transformation it uses. Adding new SQL parameters within it allows us to extend to new curl commands.

To extend the service, another section (parameter) must be added to the extractSQL.conf file, specifying the input file parsing and the SQL to produce desired output.

After the conf file is edited, the analytics webserver must be forced to refresh it (otherwise, the webserver will use the version of the file as it was upon webserver restart). You can force an "extractSQL.conf" reload with this http://analytics:8090/api/v2/csvExtract

Troubleshooting

If any of these services gets in trouble, the easiest way to bring it back might be to restart the Web Server (analytics server - useful tools and commands)

  • No labels