Versions Compared

Version Old Version 5 New Version 6
Changes made by

Atanas Mihalev

Rafael Sanchez (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Services in red above are being used in production tools, others are for development and maintenance purposes.


txtExtract service

created in December 2018 (see RPT-5119, RPT-5179)

Obtain data as a direct extract from Analytics REST API by running this curl command from any UNIX server, Windows command prompt, or Mac terminal.

command: curl -XGET -F file="@<YOUR_SOURCE_FILE>" "http://analytics:8090/api/v2/csvExtract?extractName=<YOUR_EXTRACT_NAME>" -o <RESULTS_FILE>
example: curl -XGET -F file="@/home/unix/connolly/exampleinput_exomecallset2018.csv" "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations" -o exampleoutput_exomecallset2018.csv
  1. In the above example, callSetAggregations is a parameter on the csvExtract service:  "http://analytics:8090/api/v2/csvExtract?extractName=callSetAggregations"
  2. User prepares a list of records in a CSV file with no headers or extra columns or rows (above example requires project,sample pairs)
  3. The first step of the curl command uploads the input file to the service, which records the data into dataset_items table with a new dataset_id. It then injects that dataset_id into the where clause in the transformation SQL.
  4. If the -o <RESULTS_FILE> is omitted, then the results will be printed out in the standard output (the console).

Extending the service:

A number of configuration files live here: \\neon-cifs\home_unix\analytics\TigerETL3\resources

extractSQL.conf is one of the files, called by the csvExtract service. Within this file is the ETL transformation it uses. Adding new SQL parameters within it allows us to extend to new curl commands.

To extend the service, another section (parameter) must be added to the extractSQL.conf file, specifying the input file parsing and the SQL to produce desired output.

After the conf file is edited, the analytics webserver must be forced to refresh it (otherwise, the webserver will use the version of the file as it was upon webserver restart). You can force an "extractSQL.conf" reload with this http://analytics:8090/api/v2/csvExtract

Troubleshooting

If any of these services gets in trouble, the easiest way to bring it back might be to restart the Web Server (analytics server - useful tools and commands)