knight/datahub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[#796,site,docs][xl]: copy /docs/dms to portaljs

Browse Source
This commit is contained in:
deme
2023-04-24 15:24:31 -03:00
parent 6e90f1897b
commit bfc124473d
59 changed files with 12100 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

109
site/content/docs/dms/data-explorer/datastore-query-builder/index.md Normal file
View File

@@ -0,0 +1,109 @@
---
sidebar: auto
---
# Datastore Query Builder
This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
The code repository is located at github - https://github.com/datopian/datastore-query-builder.
## Usage
Install it:
```
yarn add @datopian/datastore-query-builder
```
Basic usage in a React app:
```JavaScript
import React from 'react'
import { QueryBuilder } from 'datastore-query-builder'
export const MyComponent = props => {
// `resource` is a resource descriptor that must have 'name', 'id' and
// 'schema' properties.
// `action` - this should be a Redux action that expects back the resource
// descriptor with updated 'api' property. It is up to your app to fetch data.
return (
<QueryBuilder resource={resource} filterBuilderAction={action} />
)
}
```
Note that this app doesn't fetch any data - it only builds API URI based on user
selection.
It's easier to learn by examples provided in the `/__fixtures__/` directory.
## Features
* Date Picker - if the resource descriptor has a field with `date` type it will be displayed as a date picker element:
![Date Picker](/static/img/docs/dms/data-explorer/date-picker.png)
## Available Scripts
In the project directory, you can run:
### `yarn cosmos` or `npm run cosmos`
Runs dev server with the fixtures from `__fixtures__` directory. Learn more about `cosmos` - https://github.com/react-cosmos/react-cosmos
### `yarn start` or `npm start`
Runs the app in the development mode.<br/>
Open [http://localhost:3000](http://localhost:3000) to view it in the browser.
The page will reload if you make edits.<br/>
You will also see any lint errors in the console.
### `yarn test` or `npm test`
Launches the test runner in the interactive watch mode.<br/>
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.
### `yarn build:package` or `npm run build:package`
Run this to compile your code so it is installable via yarn/npm.
### `yarn build` or `npm run build`
Builds the app for production to the `build` folder.<br/>
It correctly bundles React in production mode and optimizes the build for the best performance.
The build is minified and the filenames include the hashes.<br/>
Your app is ready to be deployed!
See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.
## Release
When releasing a new version of this module, please, make sure you've built compiled version of the files:
```bash
yarn build:package
# Since this a release, you need to change version number in package.json file.
# E.g., this is a patch release so my `0.3.6` will become `0.3.7`.
# Now commit the changes
git add dist/ package.json
git commit -m "[v0.3.7]: your commit message."
```
Next, you need to tag your commit and add some descriptive message about the release:
```bash
git tag -a v0.3.7 -m "Your release message."
```
Now you can push your commits and tags:
```bash
git push origin branch && git push origin branch --tags
```
The tag will initiate a Github action that will publish the release to NPM.

145
site/content/docs/dms/data-explorer/design.md Normal file
View File

@@ -0,0 +1,145 @@
# Data Explorer Design
>[!note]
Design sketches from Aug 2019. This remains a work in progress though a good part was implemented in the new [Data Explorer](/docs/dms/data-explorer).
## Job Stories
[Preview] As a Data Consumer I want to have a sense of what data there is in a dataset's resources before I download it (or download an individual resource) so that I don't waste my time and get interested
[Preview] As a Data Consumer I want to view (the most important contents) of a resource without downloading it and opening it so i save time (and don't have to get specialist tools)
[Preview - with tweaks] As a Data Consumer I want to be able to display tabular data with geo info on a map so that I can see it in an easily comprehensible way
[Explorer] As a Viewer I want to explore (filter, facet?) a dataset so I can find the data i'm looking for ...
[Explorer - map] As a viewer i want to filt4er down the data i dispaly on the map so that I can see the data i want
[Map / Dash Creator] As a Publisher i want to create a custom map or dashboard so that I can display my data to viewers powerfully
[View the data] As a User, I want to see my city related data (eg, crime, road accidents) on the map so that:
* I can easily understand which area is safe for me.
* I can evaluate different neighbourhoods when planning a move.
As a User from city council, I want to see my city related data (eg, traffic) on the map so that I can take better actions to improve the city (make it safe for citizens).
> is this self-service created, a custom map made by publisher, an auto-generated map (e.g. preview)
[Data Explorer] As a Power User I want to do SQL queries on the datastore so that I can dsiplay / download the results and get insight without having to download into my own tool and do that wrangling
## Architecture
```mermaid
graph LR
subgraph "Filter UI"
simpleselectui[Filter by columns explicitly]
sqlfilterui[SQL UI]
richselectui[Filter and Group by etc in a UI]
end
subgraph Renderers
tableview[Table Renderer]
chartview[Chart Renderer]
mapview[Map Renderer]
end
subgraph Builders
datasetselector[Select datasets to use, potentially with combination]
chartbuilder[Chart Builder - UI to create a chart]
mapbuilder[Map Builder]
end
subgraph APIs
queryapi[Abstract Query API implemented by others]
datastoreapijs[DataStore API wrapper - returns a Data Package with cached data and query as url?]
datajs[Data Package - Data in Memory: Dataset and Table objects]
datajsquery[Query Wrapper Around Dataset with cached data in memory]
end
classDef todo fill:#f9f,stroke:#333,stroke-width:1px
classDef working fill:#00ff00,stroke:#333,stroke-width:1px
class chartbuilder todo;
class chartview,tableview,mapview,simpleselectui working;
```
Filter UI updates Redux Store using a one-way data binding as the ONLY way to modify application state or component state (except internal state of components as needed):
```mermaid
graph TD
FilterUI_Update --> ReduxACTION:UpdateFilters
ReduxACTION:UpdateFilters --> RefetchData
ReduxACTION:UpdateFilters --> updateUIState
RefetchData --store.workingData--> UpdateStore
updateUIState --store.uiState--> UpdateStore
UpdateStore --> RerenderApp
```
## Interfaces to define
```
dataset => data package
query[Query - source data package + cached data + filter state]
workingdataset[Working Dataset in Memory]
chartconfig[]
mapconfig[]
```
### Redux store / top level state
```javascript=
queryuistate: {
// url / data package rarely changes during lifetime of explorer usually
url: datastore url / or an original data package
filters: ...
sqlstatement:
}
// list of datasets / resources we are working with ...
datasets/resources: [
]
layout: [
// this is the switcher layout where you only see one widget at a time
layouttype: chooser; // chooserr aka singleton, stacked, custom ...
views: [list of views in their order]
]
views: [
{
type:
resource:
char
}
]
```
## Research
### Our One
![](https://i.imgur.com/XAdHq26.jpg)
### Redash
![](https://i.imgur.com/6JssnLA.png)
### Metabase
https://github.com/metabase/metabase
![](https://i.imgur.com/bOjIKdE.png)
### CKAN Classic
![](https://i.imgur.com/tGdupkz.png)
![](https://i.imgur.com/fDtjGSk.png)
### Rufus' Data Explorer (2014)
![](https://i.imgur.com/XJMHRes.png)