/
ANALYTICS Web Server

ANALYTICS Web Server

This web-server supports following services:

Production tools

JqlTableauExtractService

Extracts data from JIRA into Tableau as native HYPER file. https://broadinstitute.atlassian.net/wiki/spaces/AN/pages/613974091/Jira+Extraction+API+v2+Tableau+Edition

Often developed using CodeAssistant.

sampleAnalyticsWorkflow

https://broadinstitute.atlassian.net/wiki/spaces/AN/pages/802029673/SampleAnalyticsWorkflow+SAW+web-service

labopsjira / BspFreezerLocations

Supports SRS tools

Metrics

Service used by John Walsh to upload metrics files (Summary, basecalling, BroadWest, etc) to DWH

CromwellViewer

Monitors pipeline workflows.

Development and Maintenance tools

DependencyVisualizer

Service that visualizes relationships between different things (reports, datamarts, webservices and more).

ETLDashboard

ATTN: Actual ETLs (running on UNIX) are not affected even if Dashboard is down.

http://analytics:8090/etl_runs?latest_run_only=true shows only latest run for each agent.

"generate_sql" service generates automatically a JIRADWH query for extracting data from Jira - http://analytics:8090/api/v2/generate_sql?domain=labopsjira&project_name=SRS

JiraWebConnector

Alternative interface to get data into Tableau (failed to gain popularity)

codeAssistant

Service that helps users create SQL queries from JQL (Jira) queries. 

txtExtract service

extract data from datamarts as csv/tsv. created in December 2018 (see RPT-5119, RPT-5179)

Obtain data as a direct extract from Analytics REST API by running this curl command from any UNIX server, Windows command prompt, or Mac terminal.

command: curl -XGET -F file="@<YOUR_SOURCE_FILE>" "http://analytics:8090/api/v2/csvExtract?extractName=<YOUR_EXTRACT_NAME>" -o <RESULTS_FILE>
example: curl -XGET -F file="@/home/unix/connolly/exampleinput_exomecallset2018.csv" "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations" -o exampleoutput_exomecallset2018.csv
  1. In the above example, callSetAggregations is a parameter on the csvExtract service:  "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations"
  2. User prepares a list of records in a CSV file with no headers or extra columns or rows (above example requires project,sample pairs)
  3. The first step of the curl command uploads the input file to the service, which records the data into dataset_items table with a new dataset_id. It then injects that dataset_id into the where clause in the transformation SQL.
  4. If the -o <RESULTS_FILE> is omitted, then the results will be printed out in the standard output (the console).

Extending the service:

A number of configuration files live here: \\neon-cifs\home_unix\analytics\TigerETL3\resources

extractSQL.conf is one of the files, called by the csvExtract service. Within this file is the ETL transformation it uses. Adding new SQL parameters within it allows us to extend to new curl commands.

To extend the service, another section (parameter) must be added to the extractSQL.conf file, specifying the input file parsing and the SQL to produce desired output.

After the conf file is edited, the analytics webserver must be forced to refresh it (otherwise, the webserver will use the version of the file as it was upon webserver restart). You can force an "extractSQL.conf" reload with this http://analytics:8090/api/v2/csvExtract

Troubleshooting Tools

sqlMonitor, peekattable, LockedObjects, 


Troubleshooting

If any of these services gets in trouble, the easiest way to bring it back might be to restart the Web Server (analytics server - useful tools and commands)