knight/datahub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
datahub/site/content/docs/dms/dashboards/hdx-dashboards-notes.md

8.2 KiB
Raw Blame History

HDX Technical Architecture for Quick Dashboards

Notes from analysis and discussion in 2018.

Concepts

  • Bite (View): a description of an individual chart / map / fact and its data (source)
    • bite (for Simon): title, desc, data (compiled), uniqueid, map join info
    • view (Data Package views): title, desc, data sources (on parent data package), transforms, ...
    • compiled view: title, desc, data (compiled)
  • Data source:
    • Single HXL file (Currently, Simon's approach requires that all the data is in a single table so there is always a single data source.)
    • Data Package(s)
  • Creator / Editor: creating and editing the dashboard (given the source datasets)
  • Renderer: given dashboard config render the dashboard

Dashboard Creator

graph LR

datahxl[data+hxl]
layouts[Layout options]
dashboard["Dashboard (config)<br/><br/>(Layout, Data Sources, Selected Bites)"]
editor[Editor]
bites[Bites<br /><em>potential charts, maps etc</em>]

datahxl --suggester--> bites
bites --> editor
layouts --> editor
editor --save--> dashboard

Bite generation

graph LR

data[data with hxl] --> inferbites(("Iterate Recipes<br/>and see what<br/>matches"))
inferbites --> possmatches[List of potential bites]
possmatches --no map info--> done[Bite finished] 
possmatches --lat+lon--> done
possmatches --geo info--> maplink(("Check pcodes<br/> and link<br/>map server url"))
maplink -.-> fuzzy((Fuzzy Matcher))
fuzzy --> done
maplink --> done
maplink --error--> nobite[No Bite]

Extending to non-HXL data

It is easy to extend this to non-HXL data by using base HXL types and inference e.g.

date            => #date
geo             => #geo+lon
geo             => #geo+lat
string/category => #indicator

graph LR

data[data + syntax]
datahxl[data+hxl]
layouts[layout options]
dashboard["Dashboard (config)"]
editor[Editor]
bites[Bites<br /><em>potential charts, maps etc</em>]

data --infer--> datahxl
datahxl --suggester--> bites
bites --> editor
layouts --> editor
editor --save--> dashboard

Dashboard Renderer

Rendering the dashboard involves:

graph LR

bites[Compiled Bites/Views]
renderer["Renderer<br/>(Layout + charting / mapping libs)"]
data[Data]

subgraph Dashboard Config
  bitesconf[Bites/Views Config]
  layoutconf[Layout Config]
end

bitecompiler[Bite/View Compiler]
bitecompiler --> bites

bitesconf --> bitecompiler
data --> bitecompiler

layoutconf --> renderer
bites --> renderer

renderer --> dashboard[HTML Dashboard]

Compiled View generation

See https://docs.datahub.io/developers/views/

Architecture Proposal

  • data loader library
    • File: rows, fields (rows, columns)
  • type inference (?)
    • syntax: table schema infer
    • semantics (not now)
  • data transform library (include hxl support)
  • suggester library
  • renderer library

Interfaces / Objects

  • File
  • (Dataset)
  • Transform
  • Algorithm / Recipe
  • Bite / View
  • Ordered Set of Bites
  • Dashboard

File (and Dataset)

http://okfnlabs.org/blog/2018/02/15/design-pattern-for-a-core-data-library.html

https://github.com/datahq/data.js

File rows descriptor schema schema

Recipe

{
  'id':'chart0001',
  'type':'chart',
  'subType':'row',
  'ingredients':[{'name':'what','tags':['#activity-code-id','#sector']}],
  'criteria':['what > 4', 'what < 11'],
  'variables': ['what', 'count()'],
  'chart':'',
  'title':'Count of {1}',
'priority': 8,
}

Bite / Compiled View

{  
  bite: array [...data for chart...],  
  id: string "...chart bite ID...",  
  priority: number,  
  subtype: string "...bite subtype - row, pie...",  
  title: string "...title of bite...",  
  type: string "...bite type...",  
  uniqueID: string "...unique ID combining bite and data structure",  
}

=>

Dashboard

{
  "title":"",
  "subtext":"",
  "filtersOn":true,
  "filters":[],
  "headlinefigures":0,
  "headlinefigurecharts":[
  ],
  "grid":"grid5",
  "charts":[				
    {
      "data":"https://proxy.hxlstandard.org/data.json?filter01=append&append-dataset01-01=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1FLLwP6nxERjo1xLygV7dn7DVQwQf0_5tIdzrX31HjBA%2Fedit%23gid%3D0&filter02=select&select-query02-01=%23status%3DFunctional&url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1R9zfMTk7SQB8VoEp4XK0xAWtlsQcHgEvYiswZsj9YA4%2Fedit%23gid%3D0",
      "chartID":""
    },      
    {
      "data":"https://proxy.hxlstandard.org/data.json?filter01=append&append-dataset01-01=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1FLLwP6nxERjo1xLygV7dn7DVQwQf0_5tIdzrX31HjBA%2Fedit%23gid%3D0&filter02=select&select-query02-01=%23status%3DFunctional&url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1R9zfMTk7SQB8VoEp4XK0xAWtlsQcHgEvYiswZsj9YA4%2Fedit%23gid%3D0",
      "chartID":""
    }
  ]
}

var config = {
  layout: 2x2 // in city-indicators dashboard is handcrafted in layout
  widgets: [
    {
      elementId / data-id: ...
      view: {
        metadata: { title, sources: "World Bank"}
        resources: rule for creating compiled list of resources. [ { datasetId: ..., resourceId: ..., transform: ...} ]
        specType: 
        viewspec: 
      }
    },
    {
    
    },
  ]
  datasets: [
    list of data package urls ...
  ]
}

Simon's example

https://simonbjohnson.github.io/hdx-iom-dtm/

{
   // metadata for dashboard
  "title":"IOM DTM Example",
  "subtext":" ....",
  "headlinefigures": 3,
  "grid": "grid5", // user chosen layout for dashboard. Choice of 10 grids
  "headlinefigurecharts": [ //widgets - headline widget
    {
      "data": "https://beta.proxy.hxlstandard.org/data/1d0a79/download/africa-dtm-baseline-assessments-topline.csv",
      "chartID": "text0013/#country+name/1" // bite Id
      // elementId: ... // implicit from order in grid ...
    },
    {
      "data": "https://beta.proxy.hxlstandard.org/data/1d0a79/download/africa-dtm-baseline-assessments-topline.csv",
      "chartID": "text0012/#affected+hh+idps/5"
    },
    {
      "data": "https://beta.proxy.hxlstandard.org/data/1d0a79/download/africa-dtm-baseline-assessments-topline.csv",
      "chartID":"text0012/#affected+idps+ind/6"
    }
  ],
  "charts": [	// chart widgets			
    {
      "data": "https://beta.proxy.hxlstandard.org/data/1d0a79/download/africa-dtm-baseline-assessments-topline.csv",
      "chartID": "map0002/#adm1+code/4/#affected+idps+ind/6",
      "scale":"log"  // chart config ... 
    },
    {
      "data": "https://beta.proxy.hxlstandard.org/data/1d0a79/download/africa-dtm-baseline-assessments-topline.csv",
      "chartID": "chart0009/#country+name/1/#affected+idps+ind/6",
      "sort":"descending"
    }
  ]
}

Algorithm

  1. Extract the data references to a common list of datasets and fetch them

  2. You generate compiled data via hxl.js plus own code transforming to final data for charting etc

    function transformChart(rawSourceData (csv parsed), bite) => [ [ ...], [...]] - data for chart

  hxl.js
  custom code

function transformMap

function transformText ...

    https://github.com/SimonbJohnson/hxlbites.js

    https://github.com/SimonbJohnson/hxlbites.js/blob/master/hxlBites.js#L957

    hb.reverse(bite) => compiled bite (see above) (data, chartConfig)

  3. generate dashboard html and compute element ids in actual page element ids computed from grid setup

  4. Now have a final dashboard config

    widgets: [
  {
    data: [ [...], [...]]
    widgetType: text, chart, map ...
    elementId: // element to bind to ... 
  }
]

  5. Now use specific renderer libraries e.g. leaflet, plotly/chartist etc to render out into page

    https://github.com/SimonbJohnson/hxldash/blob/master/js/site.js#L294

Notes

"Source" version of dashboard with data uncompiled.

Compiled version of dashboard with final data inline ...

hxl.js takes an array of arrays ... and outputs array of arrays ...

{
  schema: [...]
  data: [...]
}

Renderer

  • Renderer for the dashboard
  • Renderer for each widget
function createChart(bite, elementId) => svg in bite

Charts

Map

  • Leaflet
  • react-leaflet

Tables

...