| Table of Contents | ||||
|---|---|---|---|---|
|
...
“per-barcode” metrics are supported by converting multiple “consensusreadset.xml“ files into JSONs and then merging these into a single “synthetic JSON-array“. These can be recognized by checking for trailing “*” at the end of “domain” field.
...
Turns out some information is not available in the JSON/XML files but can be extracted through the SMRTLink endpoints. Few new domains have been added: “API/runs” and “API/collections”
...
“API/runDataModel“ domain
This is special domain derived from the “apiRoot:/runs/UUID” API, where the “dataModel” field is extracted (turns out it’s an XML), converted into JSON and recorded in PACBIO datamart as “API/runDataModel“ domain. This data is also available in the “DATAROOT/*/*/*.run.metadata.xml“ domain however it would show up there later when cell “movies” start, etc.
...
How files are scraped from the file system - the linux voodoo magic
...
| Code Block | ||
|---|---|---|
| ||
scala> analytics.tiger.utils.AnalyticsDB("analytics.tiger.agents.PacBio.Sodium", analytics.tiger.agents.PacBio.Sodium.perRunETL("r64386e_20220523_180557",Map("override"->"true","verbose"->"true")), toCommit=true)
TIGERETL_RUNID: 4534558
find /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557 -regex ".*\.\(json\|xml\)" => 48 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/bc2012--bc2012/m64386e_220526_091216.bc2012--bc2012.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.ccs_reports.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.lima_guess.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.sts.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/.m64386e_220526_091216.run.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.5mc_report.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/bc2095--bc2095/m64386e_220526_091216.bc2095--bc2095.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/bc2090--bc2090/m64386e_220526_091216.bc2090--bc2090.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/.m64386e_220526_091216.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.unbarcoded.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/bc2011--bc2011/m64386e_220526_091216.bc2011--bc2011.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2012--bc2012/m64386e_220527_172851.bc2012--bc2012.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.ccs_reports.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/.m64386e_220527_172851.run.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.unbarcoded.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/.m64386e_220527_172851.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2095--bc2095/m64386e_220527_172851.bc2095--bc2095.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2090--bc2090/m64386e_220527_172851.bc2090--bc2090.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.lima_guess.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.sts.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2011--bc2011/m64386e_220527_172851.bc2011--bc2011.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.5mc_report.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.unbarcoded.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/bc2012--bc2012/m64386e_220525_014545.bc2012--bc2012.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.lima_guess.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/bc2095--bc2095/m64386e_220525_014545.bc2095--bc2095.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/bc2090--bc2090/m64386e_220525_014545.bc2090--bc2090.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/.m64386e_220525_014545.run.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.5mc_report.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/bc2011--bc2011/m64386e_220525_014545.bc2011--bc2011.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/.m64386e_220525_014545.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.ccs_reports.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.sts.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/bc2012--bc2012/m64386e_220523_181627.bc2012--bc2012.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/.m64386e_220523_181627.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/bc2095--bc2095/m64386e_220523_181627.bc2095--bc2095.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/bc2090--bc2090/m64386e_220523_181627.bc2090--bc2090.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/.m64386e_220523_181627.run.metadata.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.unbarcoded.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.lima_guess.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.5mc_report.json
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/bc2011--bc2011/m64386e_220523_181627.bc2011--bc2011.consensusreadset.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.sts.xml
/seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.ccs_reports.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/*/inputs/*.consensusreadset.xml" -type l -ls | grep /r64386e_20220523_180557/ | cat => 15 files returned
9256292134 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 09:52 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/inputs/480159576/m64386e_220526_091216.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.consensusreadset.xml
9387650676 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 10:02 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/inputs/478310612/m64386e_220523_181627.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.consensusreadset.xml
9294923607 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 10:07 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/inputs/479235094/m64386e_220525_014545.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.consensusreadset.xml
9167679957 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 09:59 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/inputs/481084058/m64386e_220527_172851.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.consensusreadset.xml
9088193591 24 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 10:07 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/inputs/479235094/m64386e_220525_014545.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.consensusreadset.xml
9163607270 32 lrwxrwxrwx 1 pbprod gppacbio 131 Jun 9 09:58 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/cfda101e-bd68-4ae3-a0d0-9e9491e60dd2/call-import_dataset_reports/inputs/481084058/m64386e_220527_172851.unbarcoded.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.unbarcoded.consensusreadset.xml
9088194092 32 lrwxrwxrwx 1 pbprod gppacbio 131 Jun 9 10:56 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/dde0dfad-e14c-47b9-b2da-b37c02c3ab1b/call-import_dataset_reports/inputs/480159576/m64386e_220526_091216.unbarcoded.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.unbarcoded.consensusreadset.xml
9387713582 24 lrwxrwxrwx 1 pbprod gppacbio 150 Jun 9 09:58 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/55615992-df0e-40e8-b131-5b45f7981a3a/call-import_dataset_reports/inputs/1130665717/m64386e_220527_172851.bc2012--bc2012.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2012--bc2012/m64386e_220527_172851.bc2012--bc2012.consensusreadset.xml
9294923510 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 09:59 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/inputs/481084058/m64386e_220527_172851.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/m64386e_220527_172851.consensusreadset.xml
9390327733 32 lrwxrwxrwx 1 pbprod gppacbio 131 Jun 9 10:57 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/76c55248-f8a2-4f04-9868-fa3fbe09fe65/call-import_dataset_reports/inputs/479235094/m64386e_220525_014545.unbarcoded.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/m64386e_220525_014545.unbarcoded.consensusreadset.xml
9390326404 32 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 09:52 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/inputs/480159576/m64386e_220526_091216.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/3_C01/m64386e_220526_091216.consensusreadset.xml
9390326842 32 lrwxrwxrwx 1 pbprod gppacbio 131 Jun 9 10:14 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/05d863a1-8a18-4693-8faa-0153892341b7/call-import_dataset_reports/inputs/478310612/m64386e_220523_181627.unbarcoded.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.unbarcoded.consensusreadset.xml
9167679974 32 lrwxrwxrwx 1 pbprod gppacbio 150 Jun 9 10:01 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/13f78080-40dd-4bb3-bf7a-8a3bb21ff542/call-import_dataset_reports/inputs/1242954991/m64386e_220525_014545.bc2095--bc2095.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/2_B01/bc2095--bc2095/m64386e_220525_014545.bc2095--bc2095.consensusreadset.xml
9387713609 32 lrwxrwxrwx 1 pbprod gppacbio 150 Jun 9 09:59 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/b5e71499-f314-408e-974e-0231e36b7098/call-import_dataset_reports/inputs/-1356847117/m64386e_220527_172851.bc2011--bc2011.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/4_D01/bc2011--bc2011/m64386e_220527_172851.bc2011--bc2011.consensusreadset.xml
9294923539 24 lrwxrwxrwx 1 pbprod gppacbio 120 Jun 9 10:02 /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/inputs/478310612/m64386e_220523_181627.consensusreadset.xml -> /seq/gp_pacbio_prod/smrtlink/userdata/data_root/r64386e_20220523_180557/1_A01/m64386e_220523_181627.consensusreadset.xml
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/barcode.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/per_barcode_reports.datastore.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/per_barcode_reports/a4377b6f-5ed5-45b9-8c6e-da74f67b4719/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/per_barcode_reports/df9c94a6-92d6-4c89-950c-5b34958b6bc0/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/per_barcode_reports/42abca05-d793-43ce-b552-c18fe68ad0ef/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/per_barcode_reports/055d5d05-40d4-4441-ac45-5e05dee0a85d/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/a0823154-4bbd-4b0a-9817-f78742054619/call-pbreports_barcode/execution/task-report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/barcode.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/per_barcode_reports.datastore.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/per_barcode_reports/b7214b4a-f2c7-4a2b-883c-08a98585d239/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/per_barcode_reports/86ff6655-a08b-4222-aa4c-7fd132a2d2ec/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/per_barcode_reports/3739008b-2f93-4855-9ba1-f367c886034a/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/per_barcode_reports/613096b2-0df0-4c7b-8021-aedfdadcfcef/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/c58dc438-f021-426c-89ce-e82ee4728d62/call-pbreports_barcode/execution/task-report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/barcode.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/per_barcode_reports.datastore.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/per_barcode_reports/760e2d75-2397-4316-8a75-facd648bd127/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/per_barcode_reports/198bbcd8-56aa-4628-8241-fcccbcf7e8b7/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/per_barcode_reports/cb3377b8-7db7-430b-aa7e-27a8e7eae0dc/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/per_barcode_reports/eb7a0619-a212-4a0b-9019-be2995cfa6b0/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/b70fbab3-8f00-44b4-a0df-f8e0e607389e/call-pbreports_barcode/execution/task-report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/barcode.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/per_barcode_reports.datastore.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/per_barcode_reports/3fdb0ce5-4004-432c-9903-8ab90e067e35/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/per_barcode_reports/2dc32207-607f-4150-8443-8a3434f6b283/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/per_barcode_reports/f86226fa-e794-40d7-b8c1-476079643dfa/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/per_barcode_reports/22bb414b-6ea7-4060-90bf-a3fc06d395e9/dataset_stats.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_collection_reports/5675af6e-2370-41f2-b4bd-8b41454ed14e/call-pbreports_barcode/execution/task-report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/adapter.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/raw_data.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/control.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/loading.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/detect_cpg_methyl.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/8578251e-cf2b-4a64-bf60-934ea70bdf8c/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/cfda101e-bd68-4ae3-a0d0-9e9491e60dd2 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/cfda101e-bd68-4ae3-a0d0-9e9491e60dd2/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/cfda101e-bd68-4ae3-a0d0-9e9491e60dd2/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/cfda101e-bd68-4ae3-a0d0-9e9491e60dd2/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/dde0dfad-e14c-47b9-b2da-b37c02c3ab1b -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/dde0dfad-e14c-47b9-b2da-b37c02c3ab1b/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/dde0dfad-e14c-47b9-b2da-b37c02c3ab1b/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/dde0dfad-e14c-47b9-b2da-b37c02c3ab1b/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/55615992-df0e-40e8-b131-5b45f7981a3a -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/55615992-df0e-40e8-b131-5b45f7981a3a/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/55615992-df0e-40e8-b131-5b45f7981a3a/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/55615992-df0e-40e8-b131-5b45f7981a3a/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/adapter.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/raw_data.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/control.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/loading.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/detect_cpg_methyl.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/458fbc8c-f5d1-488c-982e-62dc87cfe4f2/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/76c55248-f8a2-4f04-9868-fa3fbe09fe65 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/76c55248-f8a2-4f04-9868-fa3fbe09fe65/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/76c55248-f8a2-4f04-9868-fa3fbe09fe65/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/76c55248-f8a2-4f04-9868-fa3fbe09fe65/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/adapter.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/raw_data.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/control.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/loading.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/detect_cpg_methyl.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/d59cd1d9-d7f0-4283-bcc2-f1f4ef02669c/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/05d863a1-8a18-4693-8faa-0153892341b7 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/05d863a1-8a18-4693-8faa-0153892341b7/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/05d863a1-8a18-4693-8faa-0153892341b7/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/05d863a1-8a18-4693-8faa-0153892341b7/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/13f78080-40dd-4bb3-bf7a-8a3bb21ff542 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/13f78080-40dd-4bb3-bf7a-8a3bb21ff542/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/13f78080-40dd-4bb3-bf7a-8a3bb21ff542/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/13f78080-40dd-4bb3-bf7a-8a3bb21ff542/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/b5e71499-f314-408e-974e-0231e36b7098 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/b5e71499-f314-408e-974e-0231e36b7098/*/execution/*.json" => 2 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/b5e71499-f314-408e-974e-0231e36b7098/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/b5e71499-f314-408e-974e-0231e36b7098/call-import_dataset_reports/execution/ccs.report.json
find /seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34 -path "/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/*/execution/*.json" => 7 files returned
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/adapter.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/raw_data.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/control.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/loading.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/task-report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/detect_cpg_methyl.report.json
/seq/gp_pacbio_prod/smrtlink/userdata/jobs_root/cromwell-executions/sl_dataset_reports/16cab23f-16d9-4784-b544-4d4b1ea41b34/call-import_dataset_reports/execution/ccs.report.json
|
Technical caveats
API-domains are derived via API-calls which appear sensitive to reinstalls. So, API-domains in SODIUM from before Jun-2022 are not available due to SMRTLink reinstall.
This framework is tightly coupled to PacBio’s internal file-structure (unfortunately and inevitably). So, next time PacBio change their SMRTLink version, this solution may have to be fixed accordingly.
All metrics stored in PACBIO datamart are in JSON format. Metrics in XML files are converted into JSON
for each digested metrics file, a special “domain” field is generated - it allows for similar metrics to be grouped and queried via SQL later on
examples shown are for v11 installation on “sodium”. Once “skywalker” is operational switch over should be relatively easy.
ANALYTICS.PACBIO datamart (along with relevant views) is located in this Oracle instance
Code Block db.analytics.url="jdbc:oracle:thin:@//seqprod.broadinstitute.org:1521/seqprod.broadinstitute.org"
username: REPORTING
"ANALYTICS.PACBIO_STAR" view demonstrates how to merge together multiple files (ccs_report, loading, etc) in a flat per (run,cell_well) datasource. It is based on SmrtLink v10, hydrogen data (site_id=1) but techniques used are 100% legit.
Surgically extract fields from metrics-JSON via Oracle JSON
progress of Sodium PacBio flattened metrics ETL can be checked here ETL dashboard
rollback-protection is implemented so that ETL-run is cancelled if seen-before files are removed