Merge branch 'main' into basic-example-part-2

## Changes:

- /docs is now a Getting Started page with the first tutorial
- basic-example added

.gitignore

@@ -39,4 +39,8 @@ testem.log
 Thumbs.db
 
 # Next.js
-.next
+.next
+
+# Env
+.env
+**/.env

README.md

@@ -1,5 +1,3 @@
-> :warning: **This documentation is outdated**: In the coming months this repo has been and will continue to experience a major revamping, this is all in the effort of modernizing and expanding the framework, with that said, not everything shown in the documentation below is going to still be aplicable so thread carefully
-
 <h1 align="center">
 🌀 Portal.JS
 <br />
@@ -9,821 +7,48 @@ Rapidly build rich data portals using a modern frontend framework
 * [What is Portal.JS ?](#What-is-Portal.JS)
     * [Features](#Features)
       * [For developers](#For-developers)
-* [Installation and setup](#Installation-and-setup)
+* [Docs](#Docs)
-* [Getting Started](#Getting-Started)
+* [Community](#Community)
-* [Tutorial](#Tutorial)
-    * [Build a single Frictionless dataset portal](#Build-a-single-Frictionless-dataset-portal)
-    * [Build a CKAN powered dataset portal](#Build-a-CKAN-powered-dataset-portal)
-* [Architecture / Reference](#Architecture--Reference)
-    * [Component List](#Component-List)
-        * [UI Components](#UI-Components)
-        * [Dataset Components](#Dataset-Components)
-        * [View Components](#View-Components)
-        * [Search Components](#Search-Components)
-        * [Blog Components](#Blog-Components)
-        * [Misc Components](#Misc-Components)
-    * [Concepts and Terms](#Concepts-and-Terms)
-        * [Dataset](#Dataset)
-        * [Resource](#Resource)
-        * [View Spec](#view-spec)
 * [Appendix](#Appendix)
     * [What happened to Recline?](#What-happened-to-Recline?)
 
 # What is Portal.JS
-🌀 `portal.js` is a framework for rapidly building rich data portal frontends using a modern frontend approach. `portal.js` can be used to present a single dataset or build a full-scale data catalog/portal.
 
-`portal.js` is built in Javascript and React on top of the popular [Next.js](https://nextjs.com/) framework. `portal` assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
+🌀 Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. Portal.JS can be used to present a single dataset or build a full-scale data catalog/portal.
+
+Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com/) framework. Portal.JS assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
 
 ## Features
 
-- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. wordpress) with a common internal API.
+- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. Wordpress) with a common internal API.
-- 👩‍💻 Developer friendly: built with familiar frontend tech Javascript, React etc
+- 👩‍💻 Developer friendly: built with familiar frontend tech (JavaScript, React, Next.js).
-- 🔋 Batteries included: Full set of portal components out of the box e.g. catalog search, dataset showcase, blog etc.
+- 🔋 Batteries included: full set of portal components out of the box e.g. catalog search, dataset showcase, blog, etc.
 - 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
 - 🧱 Extensible: quickly extend and develop/import your own React components
-- 📝 Well documented: full set of documentation plus the documentation of NextJS and Apollo.
+- 📝 Well documented: full set of documentation plus the documentation of Next.js and Apollo.
 
 ### For developers
 
-- 🏗 Build with modern, familiar frontend tech such as Javascript and React.
+- 🏗 Build with modern, familiar frontend tech such as JavaScript and React.
-- 🚀 NextJS framework: so everything in NextJS for free React, SSR, static site generation, huge number of examples and integrations etc.
+- 🚀 Next.js framework: so everything in Next.js for free: Server Side Rendering, Static Site Generation, huge number of examples and integrations, etc.
-  - SSR => unlimited number of pages, SEO etc whilst still using React.
+  - Server Side Rendering (SSR) => Unlimited number of pages, SEO and more whilst still using React.
-  - Static Site Generation (SSG) (good for small sites) => ultra-simple deployment, great performance and lighthouse scores etc
+  - Static Site Generation (SSG) => Ultra-simple deployment, great performance, great lighthouse scores and more (good for small sites)
 
-# Installation and setup
+#### **Check out the [Portal.JS website](https://portaljs.org/) for a gallery of live portals**
-Before installation, ensure your system satisfies the following requirements:
-
-- Node.js 10.13 or later
-- Nextjs 10.0.3
-- MacOS, Windows (including WSL), and Linux are supported
-
-> Note: We also recommend instead of npm using `yarn` instead of `npm`.
-> 
-Portal.js is built with React on top of Nextjs framework, so for a quick setup, you can bootstrap a  Nextjs app and install portal.js as demonstrated in the code below:
-
-```bash=
-## Create a react app
-npx create-next-app
-# or
-yarn create next-app
-```
-After the installation is complete, follow the instructions to start the development server. Try editing pages/index.js and see the result on your browser.
-
-> For more information on how to use create-next-app, you can review the [create-next-app](https://nextjs.org/docs/api-reference/create-next-app) documentation.
-
-Once you have Nextjs created, you can install portal.js:
-
-```bash=
-yarn add https://github.com/datopian/portal.js.git
-```
-
-You're now ready to use portal.js in your next app. To test portal.js, open your `index.js` file in the pages folder. By default you should have some autogenerated code in the `index.js` file:
-
-
-Which outputs a page with the following content:
-
-![](https://i.imgur.com/GVh0P6p.png)
-
-Now, we are going to do some clean up and add a table component. In the `index.js` file, import a [Table]() component from portal as shown below:
-
-```javascript
-import Head from 'next/head'
-import { Table } from 'portal' //import Table component
-import styles from '../styles/Home.module.css'
-
-export default function Home() {
-
-  const columns = [
-    { field: 'id', headerName: 'ID' },
-    { field: 'firstName', headerName: 'First name' },
-    { field: 'lastName', headerName: 'Last name' },
-    { field: 'age', headerName: 'Age' }
-  ];
-
-  const rows = [
-    { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
-    { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
-    { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
-    { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
-    { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
-    { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
-    { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
-  ];
-
-  return (
-    <div className={styles.container}>
-      <Head>
-        <title>Create Portal App</title>
-        <link rel="icon" href="/favicon.ico" />
-      </Head>
-
-      <h1 className={styles.title}>
-        Welcome to <a href="https://nextjs.org">Portal.JS</a>
-      </h1>
-
-      {/* Use table component */}
-      <Table data={rows} columns={columns} />
-
-    </div>
-  )
-}
-```
-
-Now, your page should look like the following:
-
-![](https://i.imgur.com/n0vSjY4.png)
-
-> **Note**: You can learn more about individual portal components, as well as their prop types in the [components reference](#Component-List).
 
 ___
 
-# Getting Started
+# Docs
 
-If you're new to Portal.js we recommend that you start with the step-by-step guide below. You can also check out the following examples of projects built with portal.js.
+Access the Portal.JS documentation at:
 
-* [A portal for a single Frictionless dataset](#Build-a-single-Frictionless-dataset-portal)
+https://portaljs.org/docs
-* [A portal with a CKAN backend](#Build-a-CKAN-powered-dataset-portal)
 
-> The [`examples` directory](https://github.com/datopian/portal.js/tree/main/examples) is regularly updated with different portal examples. 
+- [Examples](https://portaljs.org/docs#examples)
 
-If you have questions about anything related to Portal.js, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions).
+# Community
-___
 
-# Tutorial
+If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
-
-## Build a single Frictionless dataset portal
-This tutorial will guide you through building a portal for a single Frictionless dataset.
-
-[Here’s](https://portal-js.vercel.app/) an example of the final result.
-
-### Setup
-The dataset should be a Frictionless Dataset i.e. it should have a [datapackage.json](https://specs.frictionlessdata.io/data-package/).
-
-Create a frictionless dataset portal app from the template:
-```
-npx create-next-app -e https://github.com/datopian/portal.js/tree/main/examples/dataset-frictionless
-#choose a name for your portal when prompted e.g. your-portal 
-```
-Go into your portal's directory and set the path to your dataset directory that contains the `datapackage.json`:
-```
-cd <your-portal>
-export PORTAL_DATASET_PATH=<path/to/your/dataset>
-```
-Start the server:
-```
-yarn dev
-```
-Visit the Page to view your dataset portal.
-
-## Build a CKAN powered dataset portal
-
-See [the CKAN Portal.JS example](./examples/ckan).
-
-___
-
-
-# Architecture / Reference
-
-## Component List
-
-Portal.js supports many components that can help you build amazing data portals similar to [this](https://catalog-portal-js.vercel.app/) and [this](https://portal-js.vercel.app/). 
-
-In this section, we'll cover all supported components in depth, and help you understand their use as well as the expected properties. 
-
-Components are grouped under the following sections:
-* [UI](https://github.com/datopian/portal.js/tree/main/src/components/ui): Components like Nav bar, Footer, e.t.c 
-* [Dataset](https://github.com/datopian/portal.js/tree/main/src/components/dataset): Components used for displaying a Frictionless dataset and resources
-* [Search](https://github.com/datopian/portal.js/tree/main/src/components/search): Components used for building a search interface for datasets
-* [Blog](https://github.com/datopian/portal.js/tree/main/src/components/blog): Components for building a simple blog for datasets
-* [Views](https://github.com/datopian/portal.js/tree/main/src/components/views): Components like charts, tables, maps for generating data views
-* [Misc](https://github.com/datopian/portal.js/tree/main/src/components/misc): Miscellaneous components like errors, custom links, etc used for extra design. 
-
-### UI Components
-
-In the UI we group all components that can be used for building generic page sections. These are components for building sections like the Navigation bar, Footer, Side pane, Recent datasets, e.t.c.  
-
-#### [Nav Component](https://github.com/datopian/portal.js/blob/main/src/components/ui/Nav.js)
-
-To build a navigation bar, you can use the `Nav` component as demonstrated below:
-
-```javascript
-import { Nav } from 'portal'
-
-export default function Home(){
-
- const navMenu = [{ title: 'Blog', path: '/blog' },
-                 { title: 'Search', path: '/search' }]
- 
- return (
-   <>
-    <Nav logo="/images/logo.png" navMenu={navMenu}/>
-    ...
-   </>
- 
- )
-}
-```
-
-#### Nav Component Prop Types
-
-Nav component accepts two properties:
-* **logo**: A string to an image path. Can be relative or absolute.
-* **navMenu**: An array of objects with title and path. E.g : [{ title: 'Blog', path: '/blog' },{ title: 'Search', path: '/search' }]
-
-
-#### [Recent Component](https://github.com/datopian/portal.js/blob/main/src/components/ui/Recent.js)
-
-The `Recent` component is used to display a list of recent [datasets](#Dataset) in the home page. This useful if you want to display the most recent dataset users have interacted with in your home page. 
-To build a recent dataset section, you can use the `Recent` component as demonstrated below:
-
-```javascript
-import { Recent } from 'portal'
-
-export default function Home() {
-
-  const datasets = [
-    {
-      organization: {
-        name: "Org1",
-        title: "This is the first org",
-        description: "A description of the organization 1"
-      },
-      title: "Data package title",
-      name: "dataset1",
-      description: "description of data package",
-      resources: [],
-    },
-    {
-      organization: {
-        name: "Org2",
-        title: "This is the second org",
-        description: "A description of the organization 2"
-      },
-      title: "Data package title",
-      name: "dataset2",
-      description: "description of data package",
-      resources: [],
-    },
-  ]
-
-  return (
-    <div>
-      {/* Use Recent component */}
-      <Recent datasets={datasets} />
-    </div>
-  )
-}
-
-```
-
-Note: The `Recent` component is hyperlinked with the dataset name of the organization and the dataset name in the following format: 
-
-> `/@<org name>/<dataset name>`
-
-For instance, using the example dataset above, the first component will be link to page:
-
-> `/@org1/dataset1`
-
-and the second will be linked to:
-
-> `/@org2/dataset2` 
-
-This is useful to know when generating dynamic pages for each dataset. 
-
-#### Recent Component Prop Types
-
-The `Recent` component accepts the following properties:
-* **datasets**: An array of [datasets](#Dataset)
-
-### Dataset Components
-
-The dataset component groups together components that can be used for building a dataset UI. These includes components for displaying info about a dataset, resources in a dataset as well as dataset ReadMe.
-
-#### [KeyInfo Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/KeyInfo.js)
-
-The `KeyInfo` components displays key properties like the number of resources, size, format, licences of in a dataset in tabular form. See example in the `Key Info` section [here](https://portal-js.vercel.app/). To use it, you can import the `KeyInfo` component as demonstrated below:
-```javascript
-import { KeyInfo } from 'portal'
-
-export default function Home() {
-
-  const datapackage = {
-    "name": "finance-vix",
-    "title": "VIX - CBOE Volatility Index",
-    "homepage": "http://www.cboe.com/micro/VIX/",
-    "version": "0.1.0",
-    "license": "PDDL-1.0",
-    "sources": [
-      {
-        "title": "CBOE VIX Page",
-        "name": "CBOE VIX Page",
-        "web": "http://www.cboe.com/micro/vix/historical.aspx"
-      }
-    ],
-    "resources": [
-      {
-        "name": "vix-daily",
-        "path": "vix-daily.csv",
-        "format": "csv",
-        "size": 20982,
-        "mediatype": "text/csv",
-      }
-    ]
-  }
-
-  return (
-    <div>
-      {/* Use KeyInfo component */}
-      <KeyInfo descriptor={datapackage} resources={datapackage.resources} />
-    </div>
-  )
-}
-
-```
-
-#### KeyInfo Component Prop Types
-
-KeyInfo component accepts two properties:
-* **descriptor**: A [Frictionless data package descriptor](https://specs.frictionlessdata.io/data-package/#descriptor)
-* **resources**: An [Frictionless data package resource](https://specs.frictionlessdata.io/data-resource/#introduction)
-
-
-#### [ResourceInfo Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/ResourceInfo.js)
-
-The `ResourceInfo` components displays key properties like the name, size, format, modification dates, as well as a download link in a resource object. See an example of a `ResourceInfo` component in the `Data Files` section [here](https://portal-js.vercel.app/).
-
-You can import and use the `ResourceInfo` component as demonstrated below:
-```javascript
-import { ResourceInfo } from 'portal'
-
-export default function Home() {
-
-  const resources = [
-    {
-      "name": "vix-daily",
-      "path": "vix-daily.csv",
-      "format": "csv",
-      "size": 20982,
-      "mediatype": "text/csv",
-    },
-    {
-      "name": "vix-daily 2",
-      "path": "vix-daily2.csv",
-      "format": "csv",
-      "size": 2082,
-      "mediatype": "text/csv",
-    }
-  ]
-
-  return (
-    <div>
-      {/* Use Recent component */}
-      <ResourceInfo resources={resources} />
-    </div>
-  )
-}
-```
-
-#### ResourceInfo Component Prop Types
-
-ResourceInfo component accepts a single property:
-* **resources**: An [Frictionless data package resource](https://specs.frictionlessdata.io/data-resource/#introduction)
-
-
-#### [ReadMe Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/Readme.js)
-
-The `ReadMe` component is used for displaying a compiled dataset Readme in a readable format. See example in the `README` section [here](https://portal-js.vercel.app/).
-
-> Note: By compiled ReadMe, we mean ReadMe that has been converted to plain string using a package like [remark](https://www.npmjs.com/package/remark).
-
-You can import and use the `ReadMe` component as demonstrated below:
-```javascript
-import { ReadMe } from 'portal'
-import remark from 'remark'
-import html from 'remark-html'
-import { useEffect, useState } from 'react'
-
-
-const readMeMarkdown = `
-    CBOE Volatility Index (VIX) time-series dataset including daily open, close,
-    high and low. The CBOE Volatility Index (VIX) is a key measure of market
-    expectations of near-term volatility conveyed by S&P 500 stock index option
-    prices introduced in 1993.
-
-    ## Data
-
-    From the [VIX FAQ][faq]:
-
-    > In 1993, the Chicago Board Options Exchange® (CBOE®) introduced the CBOE
-    > Volatility Index®, VIX®, and it quickly became the benchmark for stock market
-    > volatility. It is widely followed and has been cited in hundreds of news
-    > articles in the Wall Street Journal, Barron's and other leading financial
-    > publications. Since volatility often signifies financial turmoil, VIX is
-    > often referred to as the "investor fear gauge".
-
-    [faq]: http://www.cboe.com/micro/vix/faq.aspx
-
-    ## License
-
-    No obvious statement on [historical data page][historical]. Given size and
-    factual nature of the data and its source from a US company would imagine this
-    was public domain and as such have licensed the Data Package under the Public
-    Domain Dedication and License (PDDL).
-
-    [historical]: http://www.cboe.com/micro/vix/historical.aspx
-`
-
-export default function Home() {
-  const [readMe, setreadMe] = useState("")
-
-  useEffect(() => {
-    async function processReadMe() {
-      const processed = await remark()
-        .use(html)
-        .process(readMeMarkdown)
-      setreadMe(processed.toString())
-    }
-    processReadMe()
-  }, [])
-
-  return (
-    <div>
-      <ReadMe readme={readMe} />
-    </div>
-  )
-}
-
-```
-
-#### ReadMe Component Prop Types
-
-The `ReadMe` component accepts a single property:
-* **readme**: A string of a compiled ReadMe in html format. 
-
-### [View Components](https://github.com/datopian/portal.js/tree/main/src/components/views)
-
-View components is a set of components that can be used for displaying dataset views like charts, tables, maps, e.t.c. 
-
-#### [Chart Component](https://github.com/datopian/portal.js/blob/main/src/components/views/Chart.js)
-
-The `Chart` components exposes different chart components like Plotly Chart, Vega charts, which can be used for showing graphs. See example in the `Graph` section [here](https://portal-js.vercel.app/). 
-To use a chart component, you need to compile and pass a view spec as props to the chart component. 
-Each Chart type have their specific spec, as explained in this [doc](https://specs.frictionlessdata.io/views/#graph-spec).
-
-In the example below, we assume there's a compiled Plotly spec:
-
-```javascript
-import { PlotlyChart } from 'portal'
-
-export default function Home({plotlySpec}) {
-
-  return (
-    < div >
-      <PlotlyChart spec={plotlySpec} />
-    </div>
-  )
-}
-
-```
-> Note: You can compile views using the [datapackage-render](https://github.com/datopian/datapackage-views-js) library, as demonstrated in [this example](https://github.com/datopian/portal.js/blob/main/examples/dataset-frictionless/lib/utils.js).
-
-
-#### Chart Component Prop Types
-
-KeyInfo component accepts two properties:
-* **spec**: A compiled view spec depending on the chart type. 
-
-#### [Table Component](https://github.com/datopian/portal.js/blob/main/examples/dataset-frictionless/components/Table.js)
-
-The `Table` component is used for displaying dataset resources as a tabular grid. See example in the `Data Preview` section [here](https://portal-js.vercel.app/). 
-To use a Table component, you have to pass an array of data and columns as demonstrated below:
-
-```javascript
-import { Table } from 'portal' //import Table component
-
-export default function Home() {
-
-  const columns = [
-    { field: 'id', headerName: 'ID' },
-    { field: 'firstName', headerName: 'First name' },
-    { field: 'lastName', headerName: 'Last name' },
-    { field: 'age', headerName: 'Age' }
-  ];
-
-  const data = [
-    { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
-    { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
-    { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
-    { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
-    { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
-    { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
-    { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
-  ];
-
-  return (
-    <Table data={data} columns={columns} />
-  )
-}
-
-```
-> Note: Under the hood, Table component uses the [DataGrid Material UI table](https://material-ui.com/components/data-grid/), and as such all supported params in data and columns are supported. 
-
-
-#### Table Component Prop Types
-
-Table component accepts two properties:
-* **data**: An array of column names with properties: e.g [{field: "col1", headerName: "col1"}, {field: "col2", headerName: "col2"}]
-* **columns**: An array of data objects e.g. [ {col1: 1, col2: 2}, {col1: 5, col2: 7} ]
-
-
-### [Search Components](https://github.com/datopian/portal.js/tree/main/src/components/search)
-
-Search components groups together components that can be used for creating a search interface. This includes search forms, search item as well as search result list. 
-
-#### [Form Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Form.js)
-
-The search`Form` component is a simple search input and submit button. See example of a search form [here](https://catalog-portal-js.vercel.app/search). 
-
-The search `form` requires a submit handler (`handleSubmit`). This handler function receives the search term, and handles actual search. 
-
-In the example below, we demonstrate how to use the `Form` component.
-
-```javascript
-import { Form } from 'portal'
-
-export default function Home() {
-
-  const handleSearchSubmit = (searchQuery) => {
-    // Write your custom code to perform search in db
-    console.log(searchQuery);
-  }
-
-  return (
-    <Form
-      handleSubmit={handleSearchSubmit} />
-  )
-}
-```
-
-#### Form Component Prop Types
-
-The `Form` component accepts a single property:
-* **handleSubmit**: A function that receives the search text, and can be customize to perform the actual search. 
-
-#### [Item Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Item.js)
-
-The search`Item` component can be used to display a single search result. 
-
-In the example below, we demonstrate how to use the `Item` component.
-
-```javascript
-import { Item } from 'portal'
-
-export default function Home() {
-  const datapackage = {
-    "name": "finance-vix",
-    "title": "VIX - CBOE Volatility Index",
-    "homepage": "http://www.cboe.com/micro/VIX/",
-    "version": "0.1.0",
-    "description": "This is a test organization description",
-    "resources": [
-      {
-        "name": "vix-daily",
-        "path": "vix-daily.csv",
-        "format": "csv",
-        "size": 20982,
-        "mediatype": "text/csv",
-      }
-    ]
-  }
-
-  return (
-    <Item dataset={datapackage} />
-  )
-}
-```
-
-#### Item Component Prop Types
-
-The `Item` component accepts a single property:
-* **dataset**: A [Frictionless data package descriptor](https://specs.frictionlessdata.io/data-package/#descriptor)
-
-
-#### [ItemTotal Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Item.js)
-
-The search`ItemTotal` is a simple component for displaying the total search result 
-
-In the example below, we demonstrate how to use the `ItemTotal` component.
-
-```javascript
-import { ItemTotal } from 'portal'
-
-export default function Home() {
-  //do some custom search to get results
-  const search = (text) => {
-    return [{ name: "data1" }, { name: "data2" }]
-  }
-  //get the total result count
-  const searchTotal = search("some text").length
-
-  return (
-    <ItemTotal count={searchTotal} />
-  )
-}
-```
-
-#### ItemTotal Component Prop Types
-
-The `ItemTotal` component accepts a single property:
-* **count**: An integer of the total number of results. 
-
-
-### [Blog Components](https://github.com/datopian/portal.js/tree/main/src/components/blog)
-
-These are group of components for building a portal blog. See example of portal blog [here](https://catalog-portal-js.vercel.app/blog)
-
-#### [PostList Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-The `PostList` component is used to display a list of blog posts with the title and a short excerpts from the content.
-
-In the example below, we demonstrate how to use the `PostList` component.
-
-```javascript
-import { PostList } from 'portal'
-
-export default function Home() {
-
-  const posts = [
-    { title: "Blog post 1", excerpt: "This is the first blog excerpts in this list." },
-    { title: "Blog post 2", excerpt: "This is the second blog excerpts in this list." },
-    { title: "Blog post 3", excerpt: "This is the third blog excerpts in this list." },
-  ]
-  return (
-    <PostList posts={posts} />
-  )
-}
-```
-
-#### PostList Component Prop Types
-
-The `PostList` component accepts a single property:
-* **posts**: An array of post list objects with the following properties:
-    ```javascript
-    [
-        { 
-          title: "The title of the blog post",
-          excerpt: "A short excerpt from the post content",
-        },
-    ]
-    ```
-
-#### [Post Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-The `Post` component is used to display a blog post. See an example of a blog post [here](https://catalog-portal-js.vercel.app/blog/nyt-pa-platformen-opdateringsfrekvens-og-andres-data)
-
-In the example below, we demonstrate how to use the `Post` component.
-
-```javascript
-import { Post } from 'portal'
-import * as dayjs from 'dayjs' //For converting UTC time to relative format
-import relativeTime from 'dayjs/plugin/relativeTime'
-
-dayjs.extend(relativeTime)
-
-export default function Home() {
-
-  const post = {
-    title: "This is a sample blog post",
-    content: `<h1>A simple header</h1>
-    The PostList component is used to display a list of blog posts
-              with the title and a short excerpts from the content.
-              In the example below, we demonstrate how to use the PostList component.`,
-    createdAt: dayjs().to(dayjs(1620649596902)),
-    featuredImage: "https://pixabay.com/get/ge9a766d1f7b5fe0eccbf0f439501a2cf2b191997290e7ab15e6a402574acc2fdba48a82d278dca3547030e0202b7906d_640.jpg"
-  }
-
-  return (
-    <Post post={post} />
-  )
-}
-```
-
-#### Post Component Prop Types
-
-The `Post` component accepts a single property:
-* **post**: An object with the following properties:
-```javascript
-  {
-    title: <The title of the blog post>
-    content: <The body of the blog post. Can be plain text or html>
-    createdAt: <The utc date when the post was last modified>
-    featuredImage: < Url/relative url to post cover image>
-  }
-```
-
-
-### [Misc Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-These are group of miscellaneous/extra components for extending your portal. They include components like Errors, custom links, etc.
-
-#### [Error Component](https://github.com/datopian/portal.js/blob/main/src/components/misc/Error.js)
-
-The `Error` component is used to display a custom error message.
-
-In the example below, we demonstrate how to use the `Error` component.
-
-```javascript
-import { Error } from 'portal'
-
-export default function Home() {
-
-  return (
-    <Error message="An error occured when loading the file!" />
-  )
-}
-```
-
-#### Error Component Prop Types
-
-The `Error` component accepts a single property:
-* **message**: A string with the error message to display. 
-    
-    
-#### [Custom Component](https://github.com/datopian/portal.js/blob/main/src/components/misc/Error.js)
-
-The `CustomLink` component is used to create a link with a consistent style to other portal components. 
-
-In the example below, we demonstrate how to use the `CustomLink` component.
-
-```javascript
-import { CustomLink } from 'portal'
-
-export default function Home() {
-
-  return (
-    <CustomLink url="/blog" title="Goto Blog" />
-  )
-}
-```
-
-#### CustomLink Component Prop Types
-
-The `CustomLink` component accepts the following properties:
-
-* **url**: A string. The relative or absolute url of the link. 
-* **title**: A string. The title of the link
-
-
-___
-
-## Concepts and Terms
-In this section, we explain some of the terms and concepts used throughtout the portal.js documentation. 
-> Some of these concepts are part of official specs, and when appropriate, we'll link to the sources where you can get more details. 
-### Dataset
-A dataset extends the [Frictionless data package](https://specs.frictionlessdata.io/data-package/#metadata) to add an extra organization property. The organization property describes the organization the dataset belongs to, and it should have the following properties:
-```javascript
-organization = {
- name: "some org name",
- title: "Some optional org title",
- description: "A description of the organization"
-}
-```
-An example of dataset with organization properties is given below:
-```javascript
-datasets = [{
-    organization: {
-     name: "some org name",
-     title: "Some optional org title",
-     description: "A description of the organization"
-    },
-    title: "Data package title",
-    name:  "Data package name",
-    description: "description of data package",
-    resources: [...],
-    licences: [...],
-    sources: [...]
-    }
-]
-```
-
-### Resource
-TODO
-
-### view spec
-
----
-
-## Deploying portal build to github pages
-
-[Deploying single frictionless dataset to Github](https://portaljs.org/publish)
-
-## Showcases
-
-### Single Dataset with Default Theme
-
-![Single Dataset Example](./examples/dataset-frictionless/assets/demo.gif)
-
----
 
 # Appendix
 

examples/basic-example/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+  "extends": "next/core-web-vitals"
+}

examples/basic-example/.gitignore

@@ -0,0 +1,36 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

examples/basic-example/README.md

@@ -0,0 +1,34 @@
+This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
+
+[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
+
+The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

examples/basic-example/components/DRD.tsx

@@ -0,0 +1,21 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@flowershow/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('./Table')),
+  mermaid: Mermaid,
+  // Excel: dynamic(() => import('../components/Excel')),
+  // TODO: try and make these dynamic ...
+  Vega: dynamic(() => import('./Vega')),
+  VegaLite: dynamic(() => import('./VegaLite')),
+  LineChart: dynamic(() => import('./LineChart')),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/basic-example/components/DebouncedInput.tsx

@@ -30,4 +30,3 @@ const DebouncedInput = ({
 };
 
 export default DebouncedInput;
-

examples/basic-example/components/LineChart.tsx

@@ -1,4 +1,4 @@
-import { VegaLite } from "react-vega";
+import VegaLite from "./VegaLite";
 
 export default function LineChart({
   data = [],
@@ -15,10 +15,10 @@ export default function LineChart({
   const spec = {
     $schema: "https://vega.github.io/schema/vega-lite/v5.json",
     title,
-    width: "container" as "container",
+    width: 500,
     height: 300,
     mark: {
-      type: "line" as "line",
+      type: "line",
       color: "black",
       strokeWidth: 1,
       tooltip: true,
@@ -28,7 +28,7 @@ export default function LineChart({
     },
     selection: {
       grid: {
-        type: "interval" as "interval",
+        type: "interval",
         bind: "scales",
       },
     },
@@ -36,14 +36,14 @@ export default function LineChart({
       x: {
         field: "x",
         timeUnit: "year",
-        type: "temporal" as "temporal",
+        type: "temporal",
       },
       y: {
         field: "y",
-        type: "quantitative" as "temporal",
+        type: "quantitative",
       },
     },
   };
 
-  return <VegaLite data={vegaData} spec={spec} />;
+  return <VegaLite fullWidth={fullWidth} data={vegaData} spec={spec} />;
 }

examples/basic-example/components/Table.tsx

@@ -20,15 +20,16 @@ import {
 
 import React, { useEffect, useMemo, useState } from "react";
 
-import loadUrlProxied from "../../lib/loadUrlProxied";
+import parseCsv from "../lib/parseCsv";
-import parseCsv from "../../lib/parseCsv";
 import DebouncedInput from "./DebouncedInput";
+import loadData from "../lib/loadData";
 
 const Table = ({
   data: ogData = [],
   cols: ogCols = [],
   csv = "",
   url = "",
+  fullWidth = false,
 }) => {
   if (csv) {
     const out = parseCsv(csv);
@@ -68,7 +69,7 @@ const Table = ({
 
   useEffect(() => {
     if (url) {
-      loadUrlProxied(url).then((data) => {
+      loadData(url).then((data) => {
         const { rows, fields } = parseCsv(data);
         setData(rows);
         setCols(fields);
@@ -77,7 +78,7 @@ const Table = ({
   }, [url]);
 
   return (
-    <div>
+    <div className={`${fullWidth ? "w-[90vw] ml-[calc(50%-45vw)]" : "w-full"}`}>
       <DebouncedInput
         value={globalFilter ?? ""}
         onChange={(value) => setGlobalFilter(String(value))}

examples/basic-example/components/Vega.tsx

@@ -1,4 +1,6 @@
+//  Wrapper for the Vega component
 import { Vega as VegaOg } from "react-vega";
+
 export default function Vega(props) {
-  return <VegaOg className="w-full" {...props} />;
+  return <VegaOg {...props} />;
 }

examples/basic-example/components/VegaLite.tsx

@@ -0,0 +1,6 @@
+//  Wrapper for the Vega Lite component
+import { VegaLite as VegaLiteOg } from "react-vega";
+
+export default function VegaLite(props) {
+  return <VegaLiteOg {...props} />;
+}

examples/basic-example/content/index.md

@@ -0,0 +1,8 @@
+# My Dataset
+
+Built with PortalJS 
+
+## Table 
+
+<Table url="data_1.csv" />
+

examples/basic-example/content/my-dataset/README.md

@@ -0,0 +1,11 @@
+# Data
+
+This is the README.md this project.
+
+## Table 
+
+<Table url="data_1.csv" />
+ 
+## Vega Lite Line Chart from URL 
+
+<VegaLite spec={ { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "data": {"url": "data_2.csv"}, "width": 550, "height": 250, "mark": "line", "encoding": { "x": {"field": "Time", "type": "temporal"}, "y": {"field": "Anomaly (deg C)", "type": "quantitative"}, "tooltip": {"field": "Anomaly (deg C)", "type": "quantitative"} } } } />

examples/basic-example/lib/loadData.tsx

@@ -0,0 +1,5 @@
+export default async function loadData(url: string) {
+  const response = await fetch(url)
+  const data = await response.text()
+  return data
+}

examples/basic-example/next.config.js

@@ -0,0 +1,7 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  reactStrictMode: true,
+  swcMinify: true,
+}
+
+module.exports = nextConfig

examples/basic-example/package.json

@@ -0,0 +1,49 @@
+{
+  "name": "basic-example",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@flowershow/core": "^0.4.10",
+    "@flowershow/remark-callouts": "^1.0.0",
+    "@flowershow/remark-embed": "^1.0.0",
+    "@flowershow/remark-wiki-link": "^1.1.2",
+    "@heroicons/react": "^2.0.17",
+    "@opentelemetry/api": "^1.4.0",
+    "@tanstack/react-table": "^8.8.5",
+    "@types/node": "18.16.0",
+    "@types/react": "18.2.0",
+    "@types/react-dom": "18.2.0",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "gray-matter": "^4.0.3",
+    "hastscript": "^7.2.0",
+    "mdx-mermaid": "2.0.0-rc7",
+    "next": "13.2.1",
+    "next-mdx-remote": "^4.4.1",
+    "papaparse": "^5.4.1",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-vega": "^7.6.0",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
+    "remark-gfm": "^3.0.1",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
+}

examples/basic-example/pages/[...path].tsx

@@ -0,0 +1,42 @@
+import { GetStaticProps } from 'next';
+import { promises as fs } from 'fs';
+import path from 'path';
+import parse from '../lib/markdown';
+import DRD from '../components/DRD';
+
+export const getServerSideProps = async (context) => {
+  const indexFile = path.join(process.cwd(), '/content/' + context.params.path.join('/') + '/index.md');
+  const readme = await fs.readFile(indexFile, 'utf8');
+  let { mdxSource, frontMatter } = await parse(readme, '.mdx');
+  return {
+    props: {
+      mdxSource,
+      frontMatter,
+    },
+  };
+};
+
+export default function DatasetPage({ mdxSource, frontMatter }) {
+  return (
+    <div className="prose mx-auto">
+      <header>
+        <div className="mb-6">
+          <>
+            <h1>{frontMatter.title}</h1>
+            {frontMatter.author && (
+              <div className="-mt-6">
+                <p className="opacity-60 pl-1">{frontMatter.author}</p>
+              </div>
+            )}
+            {frontMatter.description && (
+              <p className="description">{frontMatter.description}</p>
+            )}
+          </>
+        </div>
+      </header>
+      <main>
+        <DRD source={mdxSource} />
+      </main>
+    </div>
+  );
+}

examples/basic-example/pages/_app.tsx

@@ -0,0 +1,6 @@
+import '../styles/globals.css'
+import type { AppProps } from 'next/app'
+
+export default function App({ Component, pageProps }: AppProps) {
+  return <Component {...pageProps} />
+}

examples/basic-example/pages/api/get-data-file.ts

@@ -0,0 +1,20 @@
+// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
+import type { NextApiRequest, NextApiResponse } from 'next'
+import { promises as fs } from 'fs';
+import path from 'path';
+
+export default async function handler(
+  req: NextApiRequest,
+  res: NextApiResponse<string>
+) {
+  const contentDir = path.join(process.cwd(), '/content');
+  const datasets = await fs.readdir(contentDir);
+  const query = req.query;
+  const { fileName } = query;
+  const dataFile = path.join(
+    process.cwd(),
+    '/content/' + datasets[0] + '/' + fileName
+  );
+  const data = await fs.readFile(dataFile, 'utf8');
+  res.status(200).send(data)
+}

examples/basic-example/pages/index.tsx

@@ -0,0 +1,42 @@
+import { GetStaticProps } from 'next';
+import { promises as fs } from 'fs';
+import path from 'path';
+import parse from '../lib/markdown';
+import DRD from '../components/DRD';
+
+export const getStaticProps = async (context) => {
+  const indexFile = path.join(process.cwd(), '/content/index.md');
+  const readme = await fs.readFile(indexFile, 'utf8');
+  let { mdxSource, frontMatter } = await parse(readme, '.mdx');
+  return {
+    props: {
+      mdxSource,
+      frontMatter,
+    },
+  };
+};
+
+export default function DatasetPage({ mdxSource, frontMatter }) {
+  return (
+    <div className="prose mx-auto">
+      <header>
+        <div className="mb-6">
+          <>
+            <h1>{frontMatter.title}</h1>
+            {frontMatter.author && (
+              <div className="-mt-6">
+                <p className="opacity-60 pl-1">{frontMatter.author}</p>
+              </div>
+            )}
+            {frontMatter.description && (
+              <p className="description">{frontMatter.description}</p>
+            )}
+          </>
+        </div>
+      </header>
+      <main>
+        <DRD source={mdxSource} />
+      </main>
+    </div>
+  );
+}

examples/basic-example/postcss.config.js

@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}

examples/basic-example/public/data_1.csv

@@ -0,0 +1,3 @@
+Year,Temp Anomaly
+1850,-0.418
+2020,0.923

examples/basic-example/public/data_2.csv

@@ -0,0 +1,173 @@
+Time,Anomaly (deg C),Lower confidence limit (2.5%),Upper confidence limit (97.5%)
+1850,-0.41765878,-0.589203,-0.24611452
+1851,-0.2333498,-0.41186792,-0.054831687
+1852,-0.22939907,-0.40938243,-0.04941572
+1853,-0.27035445,-0.43000934,-0.110699534
+1854,-0.29163003,-0.43282393,-0.15043613
+1855,-0.2969512,-0.43935776,-0.15454465
+1856,-0.32035372,-0.46809322,-0.1726142
+1857,-0.46723005,-0.61632216,-0.31813794
+1858,-0.3887657,-0.53688604,-0.24064532
+1859,-0.28119546,-0.42384982,-0.13854107
+1860,-0.39016518,-0.5389766,-0.24135375
+1861,-0.42927712,-0.5972301,-0.26132414
+1862,-0.53639776,-0.7037096,-0.36908585
+1863,-0.3443432,-0.5341645,-0.1545219
+1864,-0.4654367,-0.6480974,-0.282776
+1865,-0.33258784,-0.5246526,-0.14052312
+1866,-0.34126064,-0.52183825,-0.16068307
+1867,-0.35696334,-0.55306214,-0.16086453
+1868,-0.35196072,-0.52965826,-0.17426313
+1869,-0.31657043,-0.47642276,-0.15671812
+1870,-0.32789087,-0.46867347,-0.18710826
+1871,-0.3685807,-0.5141493,-0.22301209
+1872,-0.32804197,-0.4630833,-0.19300064
+1873,-0.34133235,-0.4725396,-0.21012507
+1874,-0.3732512,-0.5071426,-0.2393598
+1875,-0.37562594,-0.514041,-0.23721085
+1876,-0.42410994,-0.56287116,-0.28534868
+1877,-0.101108834,-0.22982001,0.027602348
+1878,-0.011315193,-0.13121258,0.10858219
+1879,-0.30363432,-0.43406433,-0.1732043
+1880,-0.31583208,-0.44015095,-0.19151321
+1881,-0.23224552,-0.35793498,-0.10655605
+1882,-0.29553008,-0.4201501,-0.17091006
+1883,-0.3464744,-0.4608177,-0.23213111
+1884,-0.49232006,-0.6026686,-0.38197154
+1885,-0.47112358,-0.5830682,-0.35917896
+1886,-0.42090362,-0.5225382,-0.31926903
+1887,-0.49878576,-0.61655986,-0.3810117
+1888,-0.37937889,-0.49332377,-0.265434
+1889,-0.24989556,-0.37222093,-0.12757017
+1890,-0.50685817,-0.6324095,-0.3813068
+1891,-0.40131494,-0.5373699,-0.26525995
+1892,-0.5075585,-0.64432853,-0.3707885
+1893,-0.49461925,-0.6315314,-0.35770702
+1894,-0.48376393,-0.6255681,-0.34195974
+1895,-0.4487516,-0.58202064,-0.3154826
+1896,-0.28400728,-0.4174015,-0.15061308
+1897,-0.25980017,-0.39852425,-0.12107607
+1898,-0.48579213,-0.6176492,-0.35393503
+1899,-0.35543364,-0.48639694,-0.22447036
+1900,-0.23447904,-0.3669676,-0.10199049
+1901,-0.29342857,-0.42967388,-0.15718324
+1902,-0.43898427,-0.5754281,-0.30254042
+1903,-0.5333264,-0.66081935,-0.40583345
+1904,-0.5975614,-0.7288325,-0.46629035
+1905,-0.40775132,-0.5350291,-0.28047356
+1906,-0.3191393,-0.45052385,-0.18775477
+1907,-0.5041577,-0.6262818,-0.38203365
+1908,-0.5138707,-0.63748026,-0.3902612
+1909,-0.5357649,-0.6526296,-0.41890016
+1910,-0.5310242,-0.6556868,-0.40636164
+1911,-0.5392051,-0.66223973,-0.4161705
+1912,-0.47567302,-0.5893311,-0.36201498
+1913,-0.46715254,-0.5893755,-0.34492958
+1914,-0.2625924,-0.38276345,-0.1424214
+1915,-0.19184391,-0.32196194,-0.06172589
+1916,-0.42020997,-0.5588941,-0.28152588
+1917,-0.54301953,-0.6921192,-0.3939199
+1918,-0.42458433,-0.58198184,-0.26718682
+1919,-0.32551822,-0.48145813,-0.1695783
+1920,-0.2985808,-0.44860035,-0.14856121
+1921,-0.24067703,-0.38175339,-0.09960067
+1922,-0.33922812,-0.46610323,-0.21235302
+1923,-0.31793055,-0.444173,-0.1916881
+1924,-0.3120622,-0.4388317,-0.18529275
+1925,-0.28242525,-0.4147755,-0.15007503
+1926,-0.12283547,-0.25264767,0.006976739
+1927,-0.22940508,-0.35135695,-0.10745319
+1928,-0.20676155,-0.33881804,-0.074705064
+1929,-0.39275664,-0.52656746,-0.25894582
+1930,-0.1768054,-0.29041144,-0.06319936
+1931,-0.10339768,-0.2126916,0.0058962475
+1932,-0.14546166,-0.25195515,-0.0389682
+1933,-0.32234442,-0.4271004,-0.21758842
+1934,-0.17433685,-0.27400395,-0.07466974
+1935,-0.20605922,-0.30349734,-0.10862111
+1936,-0.16952093,-0.26351926,-0.07552261
+1937,-0.01919893,-0.11975875,0.08136089
+1938,-0.012200732,-0.11030374,0.08590227
+1939,-0.040797167,-0.14670466,0.065110326
+1940,0.07593584,-0.04194966,0.19382134
+1941,0.038129337,-0.16225387,0.23851255
+1942,0.0014060909,-0.1952124,0.19802457
+1943,0.0064140745,-0.19959097,0.21241911
+1944,0.14410514,-0.054494828,0.3427051
+1945,0.043088365,-0.15728289,0.24345961
+1946,-0.1188128,-0.2659574,0.028331792
+1947,-0.091205545,-0.23179041,0.04937931
+1948,-0.12466127,-0.25913337,0.009810844
+1949,-0.14380224,-0.2540775,-0.033526987
+1950,-0.22662179,-0.33265698,-0.12058662
+1951,-0.06115397,-0.15035024,0.028042298
+1952,0.015354565,-0.08293597,0.11364509
+1953,0.07763074,-0.020529618,0.1757911
+1954,-0.11675021,-0.20850271,-0.024997713
+1955,-0.19730993,-0.28442997,-0.1101899
+1956,-0.2631656,-0.33912563,-0.18720557
+1957,-0.035334926,-0.10056862,0.029898768
+1958,-0.017632553,-0.083074555,0.04780945
+1959,-0.048004825,-0.11036375,0.0143540995
+1960,-0.115487024,-0.17416587,-0.056808177
+1961,-0.019997388,-0.07078052,0.030785747
+1962,-0.06405444,-0.11731443,-0.010794453
+1963,-0.03680589,-0.09057008,0.016958294
+1964,-0.30586675,-0.34949213,-0.26224136
+1965,-0.2043879,-0.25357357,-0.15520222
+1966,-0.14888458,-0.19839221,-0.09937696
+1967,-0.11751631,-0.16062479,-0.07440783
+1968,-0.1686323,-0.21325313,-0.124011464
+1969,-0.031366713,-0.07186544,0.009132013
+1970,-0.08510657,-0.12608096,-0.04413217
+1971,-0.20593274,-0.24450706,-0.16735843
+1972,-0.0938271,-0.13171694,-0.05593726
+1973,0.04993336,0.013468528,0.086398184
+1974,-0.17253734,-0.21022376,-0.1348509
+1975,-0.11075424,-0.15130512,-0.07020335
+1976,-0.21586166,-0.25588378,-0.17583954
+1977,0.10308852,0.060056705,0.14612034
+1978,0.0052557723,-0.034576867,0.04508841
+1979,0.09085813,0.062358618,0.119357646
+1980,0.19607207,0.162804,0.22934014
+1981,0.25001204,0.21939126,0.28063282
+1982,0.034263328,-0.005104665,0.07363132
+1983,0.22383861,0.18807402,0.2596032
+1984,0.04800471,0.011560736,0.08444869
+1985,0.04972978,0.015663471,0.08379609
+1986,0.09568697,0.064408,0.12696595
+1987,0.2430264,0.21218552,0.27386728
+1988,0.28215173,0.2470353,0.31726816
+1989,0.17925027,0.14449838,0.21400215
+1990,0.36056247,0.32455227,0.39657268
+1991,0.33889654,0.30403617,0.3737569
+1992,0.124896795,0.09088206,0.15891153
+1993,0.16565846,0.12817313,0.2031438
+1994,0.23354977,0.19841294,0.2686866
+1995,0.37686616,0.34365577,0.41007656
+1996,0.2766894,0.24318004,0.31019878
+1997,0.4223085,0.39009082,0.4545262
+1998,0.57731646,0.54304415,0.6115888
+1999,0.32448497,0.29283476,0.35613516
+2000,0.3310848,0.29822788,0.36394167
+2001,0.48928034,0.4580683,0.5204924
+2002,0.5434665,0.51278186,0.57415116
+2003,0.5441702,0.5112426,0.5770977
+2004,0.46737072,0.43433833,0.5004031
+2005,0.60686255,0.5757053,0.6380198
+2006,0.5725527,0.541973,0.60313237
+2007,0.5917013,0.56135315,0.6220495
+2008,0.46564984,0.43265733,0.49864236
+2009,0.5967817,0.56525564,0.6283077
+2010,0.68037146,0.649076,0.7116669
+2011,0.53769773,0.5060012,0.5693943
+2012,0.5776071,0.5448553,0.6103589
+2013,0.6235754,0.5884838,0.6586669
+2014,0.67287165,0.63890487,0.7068384
+2015,0.82511437,0.79128706,0.8589417
+2016,0.93292713,0.90176356,0.96409065
+2017,0.84517425,0.81477475,0.87557375
+2018,0.762654,0.731052,0.79425603
+2019,0.8910726,0.85678726,0.92535794
+2020,0.9227938,0.8882121,0.9573755
+2021,0.6640137,0.5372486,0.79077876

examples/basic-example/public/vercel.svg

@@ -0,0 +1,4 @@
+<svg width="283" height="64" viewBox="0 0 283 64" fill="none" 
+    xmlns="http://www.w3.org/2000/svg">
+    <path d="M141.04 16c-11.04 0-19 7.2-19 18s8.96 18 20 18c6.67 0 12.55-2.64 16.19-7.09l-7.65-4.42c-2.02 2.21-5.09 3.5-8.54 3.5-4.79 0-8.86-2.5-10.37-6.5h28.02c.22-1.12.35-2.28.35-3.5 0-10.79-7.96-17.99-19-17.99zm-9.46 14.5c1.25-3.99 4.67-6.5 9.45-6.5 4.79 0 8.21 2.51 9.45 6.5h-18.9zM248.72 16c-11.04 0-19 7.2-19 18s8.96 18 20 18c6.67 0 12.55-2.64 16.19-7.09l-7.65-4.42c-2.02 2.21-5.09 3.5-8.54 3.5-4.79 0-8.86-2.5-10.37-6.5h28.02c.22-1.12.35-2.28.35-3.5 0-10.79-7.96-17.99-19-17.99zm-9.45 14.5c1.25-3.99 4.67-6.5 9.45-6.5 4.79 0 8.21 2.51 9.45 6.5h-18.9zM200.24 34c0 6 3.92 10 10 10 4.12 0 7.21-1.87 8.8-4.92l7.68 4.43c-3.18 5.3-9.14 8.49-16.48 8.49-11.05 0-19-7.2-19-18s7.96-18 19-18c7.34 0 13.29 3.19 16.48 8.49l-7.68 4.43c-1.59-3.05-4.68-4.92-8.8-4.92-6.07 0-10 4-10 10zm82.48-29v46h-9V5h9zM36.95 0L73.9 64H0L36.95 0zm92.38 5l-27.71 48L73.91 5H84.3l17.32 30 17.32-30h10.39zm58.91 12v9.69c-1-.29-2.06-.49-3.2-.49-5.81 0-10 4-10 10V51h-9V17h9v9.2c0-5.08 5.91-9.2 13.2-9.2z" fill="#000"/>
+</svg>

examples/basic-example/styles/Home.module.css

@@ -0,0 +1,129 @@
+.container {
+  padding: 0 2rem;
+}
+
+.main {
+  min-height: 100vh;
+  padding: 4rem 0;
+  flex: 1;
+  display: flex;
+  flex-direction: column;
+  justify-content: center;
+  align-items: center;
+}
+
+.footer {
+  display: flex;
+  flex: 1;
+  padding: 2rem 0;
+  border-top: 1px solid #eaeaea;
+  justify-content: center;
+  align-items: center;
+}
+
+.footer a {
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  flex-grow: 1;
+}
+
+.title a {
+  color: #0070f3;
+  text-decoration: none;
+}
+
+.title a:hover,
+.title a:focus,
+.title a:active {
+  text-decoration: underline;
+}
+
+.title {
+  margin: 0;
+  line-height: 1.15;
+  font-size: 4rem;
+}
+
+.title,
+.description {
+  text-align: center;
+}
+
+.description {
+  margin: 4rem 0;
+  line-height: 1.5;
+  font-size: 1.5rem;
+}
+
+.code {
+  background: #fafafa;
+  border-radius: 5px;
+  padding: 0.75rem;
+  font-size: 1.1rem;
+  font-family: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono,
+    Bitstream Vera Sans Mono, Courier New, monospace;
+}
+
+.grid {
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  flex-wrap: wrap;
+  max-width: 800px;
+}
+
+.card {
+  margin: 1rem;
+  padding: 1.5rem;
+  text-align: left;
+  color: inherit;
+  text-decoration: none;
+  border: 1px solid #eaeaea;
+  border-radius: 10px;
+  transition: color 0.15s ease, border-color 0.15s ease;
+  max-width: 300px;
+}
+
+.card:hover,
+.card:focus,
+.card:active {
+  color: #0070f3;
+  border-color: #0070f3;
+}
+
+.card h2 {
+  margin: 0 0 1rem 0;
+  font-size: 1.5rem;
+}
+
+.card p {
+  margin: 0;
+  font-size: 1.25rem;
+  line-height: 1.5;
+}
+
+.logo {
+  height: 1em;
+  margin-left: 0.5rem;
+}
+
+@media (max-width: 600px) {
+  .grid {
+    width: 100%;
+    flex-direction: column;
+  }
+}
+
+@media (prefers-color-scheme: dark) {
+  .card,
+  .footer {
+    border-color: #222;
+  }
+  .code {
+    background: #111;
+  }
+  .logo img {
+    filter: invert(1);
+  }
+}

examples/basic-example/styles/globals.css

@@ -1,5 +1,16 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
 @import "@flowershow/remark-callouts/styles.css";
 
+.w-5 {
+  width: 1.25rem
+}
+
+.h-5 {
+  height: 1.25rem
+}
+
 /* mathjax */
 .math-inline > mjx-container > svg {
   display: inline;
@@ -65,3 +76,30 @@ html {
     visibility: hidden;
   }
 }
+
+html,
+body {
+  padding: 0;
+  margin: 0;
+  font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen,
+    Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif;
+}
+
+a {
+  color: inherit;
+  text-decoration: none;
+}
+
+* {
+  box-sizing: border-box;
+}
+
+@media (prefers-color-scheme: dark) {
+  html {
+    color-scheme: dark;
+  }
+  body {
+    color: white;
+    background: black;
+  }
+}

examples/basic-example/tailwind.config.js

@@ -0,0 +1,12 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    './app/**/*.{js,ts,jsx,tsx,mdx}',
+    './pages/**/*.{js,ts,jsx,tsx,mdx}',
+    './components/**/*.{js,ts,jsx,tsx,mdx}',
+  ],
+  theme: {
+    extend: {},
+  },
+  plugins: [require('@tailwindcss/typography')],
+};

examples/basic-example/tsconfig.json

@@ -0,0 +1,20 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": false,
+    "forceConsistentCasingInFileNames": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "preserve",
+    "incremental": true
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", "middleware.ts"],
+  "exclude": ["node_modules"]
+}

examples/ckan-2021/README.md

@@ -1,262 +1,8 @@
-<h1 align="center">
+# 🌀 PortalJS example with CKAN and Apollo
 
-🌀 Portal.JS<br/>
+**🚩 UPDATE April 2023: This example is now deprecated - though still works!. Please use the [new CKAN examples](https://github.com/datopian/portaljs/tree/main/examples)**
-The javascript framework for<br/>
-data portals
 
-</h1>
+This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.org/blog/example-ckan-2021)
-
-🌀 `Portal` is a framework for rapidly building rich data portal frontends using a modern frontend approach (javascript, React, SSR).
-
-`Portal` assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN][]. `portal` is built in Javascript and React on top of the popular [Next.js][] framework.
-
-[ckan]: https://ckan.org/
-[next.js]: https://nextjs.com/
-
-Live DEMO: https://catalog-portal-js.vercel.app
-
-## Features
-
-- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. wordpress) with a common internal API.
-- 👩‍💻 Developer friendly: built with familiar frontend tech Javascript, React etc
-- 🔋 Batteries included: Full set of portal components out of the box e.g. catalog search, dataset showcase, blog etc.
-- 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
-- 🧱 Extensible: quickly extend and develop/import your own React components
-- 📝 Well documented: full set of documentation plus the documentation of NextJS and Apollo.
-
-### For developers
-
-- 🏗 Build with modern, familiar frontend tech such as Javascript and React.
-- 🚀 NextJS framework: so everything in NextJS for free React, SSR, static site generation, huge number of examples and integrations etc.
-  - SSR => unlimited number of pages, SEO etc whilst still using React.
-  - Static Site Generation (SSG) (good for small sites) => ultra-simple deployment, great performance and lighthouse scores etc
-- 📋 Typescript support
-
-## Getting Started
-
-### Setup
-
-Install a recent version of Node. You'll need Node 10.13 or later.
-
-### Create a Portal app
-
-To create a Portal app, open your terminal, cd into the directory you'd like to create the app in, and run the following command:
-
-```console
-npm init portal-app my-data-portal
-```
-
-> NB: Under the hood, this uses the tool called create-next-app, which bootstraps a Next.js app for you. It uses this template through the --example flag.
->
-> If it doesn’t work, please open an issue.
-
-## Guide
-
-### Styling 🎨
-
-We use Tailwind as a CSS framework. Take a look at `/styles/index.css` to see what we're importing from Tailwind bundle. You can also configure Tailwind using `tailwind.config.js` file.
-
-Have a look at Next.js support of CSS and ways of writing CSS:
-
-https://nextjs.org/docs/basic-features/built-in-css-support
-
-### Backend
-
-So far the app is running with mocked data behind. You can connect CMS and DMS backends easily via environment variables:
-
-```console
-$ export DMS=http://ckan:5000
-$ export CMS=http://myblog.wordpress.com
-```
-
-> Note that we don't yet have implementations for the following CKAN features:
->
-> - Activities
-> - Auth
-> - Groups
-> - Facets
-
-### Routes
-
-These are the default routes set up in the "starter" app.
-
-- Home `/`
-- Search `/search`
-- Dataset `/@org/dataset`
-- Resource `/@org/dataset/r/resource`
-- Organization `/@org`
-- Collection (aka group in CKAN) (?) - suggest to merge into org
-- Static pages, eg, `/about` etc. from CMS or can do it without external CMS, e.g., in Next.js
-
-### New Routes
-
-TODO
-
-### Data fetching
-
-We use Apollo client which allows us to query data with GraphQL. We have setup CKAN API for the demo (it uses demo.ckan.org as DMS):
-
-http://portal.datopian1.now.sh/
-
-Note that we don't have Apollo Server but we connect CKAN API using [`apollo-link-rest`](https://www.apollographql.com/docs/link/links/rest/) module. You can see how it works in [lib/apolloClient.ts](https://github.com/datopian/portal/blob/master/lib/apolloClient.ts) and then have a look at [pages/\_app.tsx](https://github.com/datopian/portal/blob/master/pages/_app.tsx).
-
-For development/debugging purposes, we suggest installing the Chrome extension - https://chrome.google.com/webstore/detail/apollo-client-developer-t/jdkknkkbebbapilgoeccciglkfbmbnfm.
-
-#### i18n configuration
-
-Portal.js is configured by default to support both `English` and `French` subpath for language translation. But for subsequent users, this following steps can be used to configure i18n for other languages;
-
-1.  Update `next.config.js`, to add more languages to the i18n locales
-
-```js
-i18n: {
-  locales: ['en', 'fr', 'nl-NL'], // add more language to the list
-  defaultLocale: 'en',  // set the default language to use
-},
-```
-
-2. Create a folder for the language in `locales` --> `locales/en-Us`
-
-3. In the language folder, different namespace files (json) can be created for each translation. For the `index.js` use-case, I named it `common.json`
-
-```json
-// locales/en/common.json
-{
-   "title" : "Portal js in English",
-}
-
-// locales/fr/common.json
-{
-   "title" : "Portal js in French",
-}
-```
-
-4. To use on pages using Server-side Props.
-
-```js
-import { loadNamespaces } from './_app';
-import useTranslation from 'next-translate/useTranslation';
-
-const Home: React.FC = ()=> {
-  const { t } = useTranslation();
-  return (
-    <div>{t(`common:title`)}</div> // we use common and title base on the common.json data
-  );
-};
-
-export const getServerSideProps: GetServerSideProps = async ({ locale }) => {
-      ........  ........
-  return {
-    props : {
-      _ns:  await loadNamespaces(['common'], locale),
-    }
-  };
-};
-
-```
-
-5. Go to the browser and view the changes using language subpath like this `http://localhost:3000` and `http://localhost:3000/fr`. **Note** The subpath also activate chrome language Translator
-
-#### Pre-fetch data in the server-side
-
-When visiting a dataset page, you may want to fetch the dataset metadata in the server-side. To do so, you can use `getServerSideProps` function from NextJS:
-
-```javascript
-import { GetServerSideProps } from 'next';
-import { initializeApollo } from '../lib/apolloClient';
-import gql from 'graphql-tag';
-
-const QUERY = gql`
-  query dataset($id: String) {
-    dataset(id: $id) @rest(type: "Response", path: "package_show?{args}") {
-      result
-    }
-  }
-`;
-
-...
-
-export const getServerSideProps: GetServerSideProps = async (context) => {
-  const apolloClient = initializeApollo();
-
-  await apolloClient.query({
-    query: QUERY,
-    variables: {
-      id: 'my-dataset'
-    },
-  });
-
-  return {
-    props: {
-      initialApolloState: apolloClient.cache.extract(),
-    },
-  };
-};
-```
-
-This would fetch the data from DMS and save it in the Apollo cache so that we can query it again from the components.
-
-#### Access data from a component
-
-Consider situation when rendering a component for org info on the dataset page. We already have pre-fetched dataset metadata that includes `organization` property with attributes such as `name`, `title` etc. We can now query only organization part for our `Org` component:
-
-```javascript
-import { useQuery } from '@apollo/react-hooks';
-import gql from 'graphql-tag';
-
-export const GET_ORG_QUERY = gql`
-  query dataset($id: String) {
-    dataset(id: $id) @rest(type: "Response", path: "package_show?{args}") {
-      result {
-        organization {
-          name
-          title
-          image_url
-        }
-      }
-    }
-  }
-`;
-
-export default function Org({ variables }) {
-  const { loading, error, data } = useQuery(
-    GET_ORG_QUERY,
-    {
-      variables: { id: 'my-dataset' }
-    }
-  );
-
-  ...
-
-  const { organization } = data.dataset.result;
-
-  return (
-    <>
-      {organization ? (
-        <>
-          <img
-            src={
-              organization.image_url
-            }
-            className="h-5 w-5 mr-2 inline-block"
-          />
-          <Link href={`/@${organization.name}`}>
-            <a className="font-semibold text-primary underline">
-              {organization.title || organization.name}
-            </a>
-          </Link>
-        </>
-      ) : (
-        ''
-      )}
-    </>
-  );
-}
-```
-
-#### Add a new data source
-
-TODO
 
 ## Developers
 
@@ -303,4 +49,5 @@ yarn run e2e
 
 ### Key Pages
 
-See https://tech.datopian.com/frontend/
+See https://datahub.io/docs/dms/frontend/
+

examples/ckan-example/.eslintrc.json

@@ -1,6 +1,5 @@
 {
   "extends": [
-    "plugin:@nrwl/nx/react-typescript",
     "next",
     "next/core-web-vitals",
     "../../.eslintrc.json"

examples/ckan-example/README.md

@@ -1,17 +1,46 @@
 This is a repo intended to serve as an example of a data catalog that get its data from a CKAN Instance.
 
-- Creating a new file inside o `examples` with `create-next-app` like so:
 ```
-npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/ --example-path examples/ckan-example
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/ckan-example
+cd <app-name>
 ```
-- Inside `<app-name>` go to the `project.json` file and replace all instances of `ckan-example` with `<app-name>`
+
-- Set the `DMS` env variable to the Url of the CKAN Instance Ex: `export DMS=https://demo.dev.datopian.com`
+- This project uses CKAN as a backend, so you need to point the project to the CKAN Url desired, you can do so by setting up the `DMS` env variable in your terminal or adding a `.env` file with the following content:
+
+```
+DMS=<ckan url>
+```
+
 - Run the app using:
+
 ```
-nx serve <app-name>
+npm run dev
 ```
+
 Congratulations, you now have something similar to this running on `http://localhost:4200`
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252297726865408/image.png?width=853&height=461)
 If yo go to any one of those pages by clicking on `More info` you will see something similar to this
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252298074988595/image.png?width=853&height=461)
 
+## Deployment
+
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fckan-example&env=DMS&envDescription=URL%20For%20the%20CKAN%20Backend%20Ex%3A%20https%3A%2F%2Fdemo.dev.datopian.com)
+
+By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own github/gitlab/bitbucket account and automatically deploy everything.
+
+
+
+## Extra commands
+
+You can also build the project for production with
+
+```
+npm run build
+```
+
+And run using the production build like so:
+
+```
+npm run start
+```
+

examples/ckan-example/next.config.js

@@ -1,9 +1,3 @@
-// eslint-disable-next-line @typescript-eslint/no-var-requires
-const { withNx } = require('@nrwl/next/plugins/with-nx');
-
-/**
- * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
- **/
 const nextConfig = {
   publicRuntimeConfig: {
     DMS: process.env.DMS ? process.env.DMS : '',
@@ -18,11 +12,6 @@ const nextConfig = {
       ],
     };
   },
-  nx: {
-    // Set this to true if you would like to use SVGR
-    // See: https://github.com/gregberge/svgr
-    svgr: false,
-  },
 };
 
-module.exports = withNx(nextConfig);
+module.exports = nextConfig;

examples/ckan-example/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "my-app",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@heroicons/react": "^2.0.17",
+    "@types/node": "18.16.0",
+    "@types/react": "18.0.38",
+    "@types/react-dom": "18.0.11",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "next": "13.3.1",
+    "next-seo": "^6.0.0",
+    "octokit": "^2.0.14",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-markdown": "^8.0.7",
+    "remark-gfm": "^3.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
+}

examples/ckan-example/postcss.config.js

@@ -1,15 +1,6 @@
-const { join } = require('path');
-
-// Note: If you use library-specific PostCSS/Tailwind configuration then you should remove the `postcssConfig` build
-// option from your application's configuration (i.e. project.json).
-//
-// See: https://nx.dev/guides/using-tailwind-css-in-react#step-4:-applying-configuration-to-libraries
-
 module.exports = {
   plugins: {
-    tailwindcss: {
+    tailwindcss: {},
-      config: join(__dirname, 'tailwind.config.js'),
-    },
     autoprefixer: {},
   },
-};
+}

examples/ckan-example/tailwind.config.js

@@ -1,17 +1,15 @@
-const { createGlobPatternsForDependencies } = require('@nrwl/react/tailwind');
-const { join } = require('path');
-
 /** @type {import('tailwindcss').Config} */
 module.exports = {
   content: [
-    join(
+    "./app/**/*.{js,ts,jsx,tsx,mdx}",
-      __dirname,
+    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
-      '{src,pages,components}/**/*!(*.stories|*.spec).{ts,tsx,html}'
+    "./components/**/*.{js,ts,jsx,tsx,mdx}",
-    ),
-    ...createGlobPatternsForDependencies(__dirname),
   ],
   theme: {
     extend: {},
   },
-  plugins: [],
+  plugins: [
-};
+    require('@tailwindcss/typography')
+  ],
+}
+

examples/ckan-example/tsconfig.json

@@ -1,23 +1,20 @@
 {
-  "extends": "../../tsconfig.base.json",
   "compilerOptions": {
-    "jsx": "preserve",
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
     "allowJs": true,
-    "esModuleInterop": true,
+    "skipLibCheck": true,
-    "allowSyntheticDefaultImports": true,
     "strict": false,
     "forceConsistentCasingInFileNames": true,
     "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
     "resolveJsonModule": true,
     "isolatedModules": true,
-    "incremental": true,
+    "jsx": "preserve",
-    "types": ["jest", "node"]
+    "incremental": true
   },
-  "include": ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "next-env.d.ts"],
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
-  "exclude": [
+  "exclude": ["node_modules"]
-    "node_modules",
-    "jest.config.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.ts"
-  ]
 }

examples/simple-example/.eslintrc.json

@@ -1,9 +1,7 @@
 {
   "extends": [
-    "plugin:@nrwl/nx/react-typescript",
     "next",
-    "next/core-web-vitals",
+    "next/core-web-vitals"
-    "../../.eslintrc.json"
   ],
   "ignorePatterns": ["!**/*", ".next/**/*"],
   "overrides": [

examples/simple-example/README.md

@@ -1,17 +1,75 @@
 This is a repo intended to serve as a simple example of a data catalog that get its data from a series of github repos, you can init an example just like this one by.
 
-- Creating a new file inside o `examples` with `create-next-app` like so:
+- Creating a new project with `create-next-app` like so:
+
 ```
-npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/ --example-path examples/simple-example
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/simple-example
+cd <app-name>
 ```
-- Inside `<app-name>` go to the `project.json` file and replace all instances of `simple-example` with `<app-name>`
+
+- This project uses the github api, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
+
+```
+GITHUB_PAT=<github token>
+```
+
 - Edit the file `datasets.json` to your liking, some examples can be found inside this [repo](https://github.com/datasets)
 - Run the app using:
-```
+
-nx serve <app-name>
+```
-```
+npm run dev
-Congratulations, you now have something similar to this running on `http://localhost:4200`
+```
-![](https://i.imgur.com/JrDLycF.png)
+
-If yo go to any one of those pages by clicking on `More info` you will see something similar to this
+Congratulations, you now have something similar to this running on `http://localhost:3000`
-![](https://i.imgur.com/cpKMS80.png)
+![](https://i.imgur.com/jAljJ9C.png)
+If yo go to any one of those pages by clicking on `More info` you will see something similar to this
+![](https://i.imgur.com/AoJd4O0.png)
+
+## Deployment
+
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fsimple-example)
+
+By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own github/gitlab/bitbucket account and automatically deploy everything.
+
+
+## Structure of `datasets.json`
+
+The `datasets.json` file is simply a list of datasets, below you can see a minimal example of a dataset
+
+```json
+{
+  "owner": "fivethirtyeight",
+  "repo": "data",
+  "branch": "master",
+  "files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
+  "readme": "nba-raptor/README.md"
+}
+```
+
+It has
+
+- A `owner` which is going to be the github repo owner
+- A `repo` which is going to be the github repo name
+- A `branch` which is going to be the branch to which we need to get the files and the readme
+- A list of `files` which is going to be a list of paths with files that you want to show to the world
+- A `readme` which is going to be the path to your data description, it can also be a subpath eg: `example/README.md`
+
+You can also add
+
+- A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
+- A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
+
+## Extra commands
+
+You can also build the project for production with
+
+```
+npm run build
+```
+
+And run using the production build like so:
+
+```
+npm run start
+```
 

examples/simple-example/components/MDLayout.tsx

@@ -1,85 +0,0 @@
-import FrictionlessViewFactory from "./drd/FrictionlessView";
-import Table from "./drd/Table";
-
-/* eslint import/no-default-export: off */
-function DatapackageLayout({ children, project, excerpt }) {
-  const { metadata } = project;
-
-  const title = metadata.title;
-  const resources = metadata.resources;
-  const views = metadata.views;
-
-  const FrictionlessView = FrictionlessViewFactory({ views, resources });
-
-  return (
-    <article className="docs prose text-primary dark:text-primary-dark dark:prose-invert prose-headings:font-headings prose-a:break-words mx-auto p-6">
-      <header>
-        {title && <h1 className="mb-4">{title}</h1>}
-        <a
-          className="font-semibold mb-4"
-          target="_blank"
-          href={project.github_repo}
-        >
-          @{project.owner} / {project.name}
-        </a>
-        {excerpt && <p className="text-md">{excerpt}</p>}
-      </header>
-      <section className="mt-10">
-        {views.map((view, i) => {
-          return (
-            <div key={`visualization-${i}`}>
-              <FrictionlessView viewId={i} />
-            </div>
-          );
-        })}
-      </section>
-      <section className="mt-10">
-        <h2>Data files</h2>
-        <table className="table-auto">
-          <thead>
-            <tr>
-              <th>File</th>
-              <th>Title</th>
-              <th>Format</th>
-            </tr>
-          </thead>
-          <tbody>
-            {resources.map((r) => {
-              return (
-                <tr key={`resources-list-${r.name}`}>
-                  <td>
-                    <a
-                      target="_blank"
-                      href={`https://github.com/${project.owner}/${project.name}/blob/main/${r.path}`}
-                    >
-                      {r.path}
-                    </a>
-                  </td>
-                  <td>{r.title}</td>
-                  <td>{r.format.toUpperCase()}</td>
-                </tr>
-              );
-            })}
-          </tbody>
-        </table>
-        {resources.slice(0, 5).map((resource) => {
-          return (
-            <div key={`resource-preview-${resource.name}`} className="mt-10">
-              <h3>{resource.title || resource.name || resource.path}</h3>
-              <Table url={resource.path} />
-            </div>
-          );
-        })}
-      </section>
-      <hr />
-      <section>
-        <h2>Read me</h2>
-        {children}
-      </section>
-    </article>
-  );
-}
-
-export default function MDLayout({ children, layout, ...props }) {
-  return <DatapackageLayout project={props.project} excerpt={props.excerpt}>{children}</DatapackageLayout>;
-}

examples/simple-example/components/drd/DRD.tsx

@@ -1,40 +0,0 @@
-import { MDXRemote } from "next-mdx-remote";
-import dynamic from "next/dynamic";
-import { Mermaid } from "@flowershow/core";
-
-import FrictionlessViewFactory from "./FrictionlessView";
-
-// Custom components/renderers to pass to MDX.
-// Since the MDX files aren't loaded by webpack, they have no knowledge of how
-// to handle import statements. Instead, you must include components in scope
-// here.
-const components = {
-  Table: dynamic(() => import("./Table")),
-  mermaid: Mermaid,
-  // Excel: dynamic(() => import('../components/Excel')),
-  // TODO: try and make these dynamic ...
-  Vega: dynamic(() => import("./Vega")),
-  VegaLite: dynamic(() => import("./VegaLite")),
-  LineChart: dynamic(() => import("./LineChart")),
-} as any;
-
-export default function DRD({
-  source,
-  frictionless = {
-    views: [],
-    resources: [],
-  },
-}: {
-  source: any;
-  frictionless?: any;
-}) {
-  //  dynamic() can't be used inside of React rendering
-  //  as it needs to be marked in the top level of the
-  //  module for preloading to work
-  components.FrictionlessView = FrictionlessViewFactory({
-    views: frictionless.views,
-    resources: frictionless.resources,
-  });
-
-  return <MDXRemote {...source} components={components} />;
-}

examples/simple-example/components/drd/FrictionlessView.tsx

@@ -1,55 +0,0 @@
-// FrictionlessView is a factory because we have to
-// set the views and resources lists before using it
-
-import { convertSimpleToVegaLite } from "../../lib/viewSpecConversion";
-import VegaLite from "./VegaLite";
-
-export default function FrictionlessViewFactory({
-  views = [],
-  resources = [],
-}): ({
-  viewId,
-  fullWidth,
-}: {
-  viewId: number;
-  fullWidth?: boolean;
-}) => JSX.Element {
-  return ({ viewId, fullWidth = false }) => {
-    if (!(viewId in views)) {
-      console.error(`View ${viewId} not found`);
-      return <></>;
-    }
-    const view = views[viewId];
-
-    let resource;
-    if (resources.length > 1) {
-      resource = resources.find((r) => r.name === view.resourceName);
-    } else {
-      resource = resources[0];
-    }
-
-    if (!resource) {
-      console.error(`Resource not found for view id ${viewId}`);
-      return <></>;
-    }
-
-    let vegaSpec;
-    switch (view.specType) {
-      case "simple":
-        vegaSpec = convertSimpleToVegaLite(view, resource);
-        break;
-      // ... other conversions
-    }
-
-    vegaSpec.data = { url: resource.path };
-
-    return (
-      <VegaLite
-        fullWidth={fullWidth}
-        spec={vegaSpec}
-        actions={{ editor: false }}
-        downloadFileName={resource.name}
-      />
-    );
-  };
-}

examples/simple-example/components/drd/VegaLite.tsx

@@ -1,4 +0,0 @@
-import { VegaLite as VegaOg } from "react-vega";
-export default function Vega(props) {
-  return <VegaOg  className="w-full" {...props} />;
-}

examples/simple-example/datasets.json

@@ -1,7 +1,44 @@
 [
-  { "owner": "datasets", "repo": "oil-prices"},
+  {
-  { "owner": "datasets", "repo": "investor-flow-of-funds-us"},
+    "owner": "datasets",
-  { "owner": "datasets", "repo": "browser-stats"},
+    "branch": "main",
-  { "owner": "datasets", "repo": "glacier-mass-balance"},
+    "repo": "oil-prices",
-  { "owner": "datasets", "repo": "bond-yields-us-10y"}
+    "files": [
+      "data/brent-daily.csv",
+      "data/brent-monthly.csv",
+      "data/brent-weekly.csv",
+      "data/brent-year.csv",
+      "data/wti-daily.csv",
+      "data/wti-monthly.csv",
+      "data/wti-weekly.csv",
+      "data/wti-year.csv"
+    ],
+    "readme": "README.md"
+  },
+  {
+    "owner": "datasets",
+    "branch": "main",
+    "repo": "investor-flow-of-funds-us",
+    "files": [
+      "data/monthly.csv",
+      "data/weekly.csv"
+    ],
+    "readme": "README.md"
+  },
+  {
+    "owner": "fivethirtyeight",
+    "repo": "data",
+    "branch": "master",
+    "description": "Data about bad drivers",
+    "name": "Bad Drivers",
+    "files": ["bad-drivers/bad-drivers.csv"],
+    "readme": "bad-drivers/README.md"
+  },
+  {
+    "owner": "fivethirtyeight",
+    "repo": "data",
+    "branch": "master",
+    "files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
+    "readme": "nba-raptor/README.md"
+  }
 ]

examples/simple-example/jest.config.ts

@@ -1,11 +0,0 @@
-/* eslint-disable */
-export default {
-  displayName: 'simple-example',
-  preset: '../../jest.preset.js',
-  transform: {
-    '^(?!.*\\.(js|jsx|ts|tsx|css|json)$)': '@nrwl/react/plugins/jest',
-    '^.+\\.[tj]sx?$': ['babel-jest', { presets: ['@nrwl/next/babel'] }],
-  },
-  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx'],
-  coverageDirectory: '../../coverage/examples/simple-example',
-};

examples/simple-example/lib/loadUrlProxied.tsx

@@ -1,11 +0,0 @@
-import axios from "axios";
-
-export default function loadUrlProxied(url: string) {
-  // HACK: duplicate of Excel code - maybe refactor
-  // if url is external may have CORS issue so we proxy it ...
-  if (url.startsWith("http")) {
-    const PROXY_URL = "/api/proxy";
-    url = PROXY_URL + "?url=" + encodeURIComponent(url);
-  }
-  return axios.get(url).then((res) => res.data);
-}

examples/simple-example/lib/octokit.ts

@@ -0,0 +1,147 @@
+import { Octokit } from 'octokit';
+
+export interface GithubProject {
+  owner: string;
+  repo: string;
+  branch: string;
+  files: string[];
+  readme: string;
+  description?: string;
+  name?: string;
+}
+
+export async function getProjectReadme(
+  owner: string,
+  repo: string,
+  branch: string,
+  readme: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.getContent({
+      owner,
+      repo,
+      path: readme,
+      ref: branch,
+    });
+    const data = response.data as { content?: string };
+    const fileContent = data.content ? data.content : '';
+    if (fileContent === '') {
+      return null;
+    }
+    const decodedContent = Buffer.from(fileContent, 'base64').toString();
+    return decodedContent;
+  } catch (error) {
+    console.log(error);
+    return null;
+  }
+}
+
+export async function getLastUpdated(
+  owner: string,
+  repo: string,
+  branch: string,
+  readme: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.listCommits({
+      owner,
+      repo,
+      path: readme,
+      ref: branch,
+    });
+    return response.data[0].commit.committer.date;
+  } catch (error) {
+    console.log(error);
+    return null;
+  }
+}
+export async function getProjectMetadata(
+  owner: string,
+  repo: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.get({
+      owner,
+      repo,
+    });
+    return response.data;
+  } catch (error) {
+    console.log(error);
+    return null;
+  }
+}
+
+export async function getRepoContents(
+  owner: string,
+  repo: string,
+  branch: string,
+  files: string[],
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const contents = [];
+    for (const path of files) {
+      const response = await octokit.rest.repos.getContent({
+        owner,
+        repo,
+        ref: branch,
+        path: path,
+      });
+      const data = response.data as { download_url?: string, name: string, size: number };
+      contents.push({ download_url: data.download_url, name: data.name, size: data.size});
+    }
+    return contents;
+  } catch (error) {
+    console.log(error);
+    return null;
+  }
+}
+
+export async function getProject(project: GithubProject, github_pat?: string) {
+  const projectMetadata = await getProjectMetadata(
+    project.owner,
+    project.repo,
+    github_pat
+  );
+  if (!projectMetadata) {
+    return null;
+  }
+  const projectReadme = await getProjectReadme(
+    project.owner,
+    project.repo,
+    project.branch,
+    project.readme,
+    github_pat
+  );
+  if (!projectReadme) {
+    return null;
+  }
+  const projectData = await getRepoContents(
+    project.owner,
+    project.repo,
+    project.branch,
+    project.files,
+    github_pat
+  );
+  if (!projectData) {
+    return null;
+  }
+  const projectBase = project.readme.split('/').length > 1
+      ? project.readme.split('/').slice(0, -1).join('/')
+      : '/'
+  const last_updated = await getLastUpdated(
+    project.owner,
+    project.repo,
+    project.branch,
+    projectBase,
+    github_pat
+  );
+  return { ...projectMetadata, files: projectData, readmeContent: projectReadme, last_updated, base_path: projectBase };
+}

examples/simple-example/lib/project.ts

@@ -1,60 +0,0 @@
-import * as crypto from "crypto";
-import axios from "axios";
-import { Octokit } from "octokit"
-
-export default class Project {
-  id: string;
-  name: string;
-  owner: string;
-  github_repo: string;
-  readme: string;
-  metadata: any;
-  repo_metadata: any;
-
-  constructor(owner: string, name: string) {
-    this.name = name;
-    this.owner = owner;
-    this.github_repo = `https://github.com/${owner}/${name}`;
-
-    //  TODO: using the GitHub repo to set the id is not a good idea
-    //  since repos can be renamed and then we are going to end up with
-    //  a duplicate
-    const encodedGHRepo = Buffer.from(this.github_repo, "utf-8").toString();
-    this.id = crypto.createHash("sha1").update(encodedGHRepo).digest("hex");
-  }
-
-  initFromGitHub = async () => {
-    const octokit = new Octokit()
-    //  TODO: what if the repo doesn't exist?
-    await this.getFileContent("README.md")
-      .then((content) => (this.readme = content))
-      .catch((e) => (this.readme = null));
-
-    await this.getFileContent("datapackage.json")
-      .then((content) => (this.metadata = content))
-      .catch((e) => (this.metadata = {}));
-
-    const github_metadata = await octokit.rest.repos.get({ owner: this.owner, repo: this.name })
-    this.repo_metadata = github_metadata.data ? github_metadata.data : null
-  };
-
-  getFileContent = (path, branch = "main") => {
-    return axios
-      .get(
-        `https://raw.githubusercontent.com/${this.owner}/${this.name}/${branch}/${path}`
-      )
-      .then((res) => res.data);
-  };
-
-  serialize() {
-    return JSON.parse(JSON.stringify(this));
-  }
-
-  static async getFromGitHub(owner: string, name: string) {
-    const project = new Project(owner, name);
-    await project.initFromGitHub();
-
-    return project;
-  }
-}
-

examples/simple-example/lib/viewSpecConversion.ts

@@ -1,47 +0,0 @@
-export function convertSimpleToVegaLite(view, resource) {
-  const x = resource.schema.fields.find((f) => f.name === view.spec.group);
-  const y = resource.schema.fields.find((f) => f.name === view.spec.series[0]);
-
-  const xType = inferVegaType(x.type);
-  const yType = inferVegaType(y.type);
-
-  let vegaLiteSpec = {
-    $schema: "https://vega.github.io/schema/vega-lite/v5.json",
-    mark: {
-      type: view.spec.type,
-      color: "black",
-      strokeWidth: 1,
-      tooltip: true,
-    },
-    title: view.title,
-    width: 500,
-    height: 300,
-    selection: {
-      grid: {
-        type: "interval",
-        bind: "scales",
-      },
-    },
-    encoding: {
-      x: {
-        field: x.name,
-        type: xType,
-      },
-      y: {
-        field: y.name,
-        type: yType,
-      },
-    },
-  };
-
-  return vegaLiteSpec;
-}
-
-const inferVegaType = (fieldType) => {
-  switch (fieldType) {
-    case "date":
-      return "Temporal";
-    case "number":
-      return "Quantitative";
-  }
-};

examples/simple-example/next.config.js

@@ -1,23 +1,7 @@
-// eslint-disable-next-line @typescript-eslint/no-var-requires
-const { withNx } = require('@nrwl/next/plugins/with-nx');
-
-/**
- * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
- **/
 const nextConfig = {
   async rewrites() {
     return {
       beforeFiles: [
-        {
-          source: "/@org/:org/:project/:file(\.\+\\\.\.\+\$)",
-          destination:
-            '/api/proxy?url=https://raw.githubusercontent.com/:org/:project/main/:file',
-        },
-        {
-          source: "/@:org/:project/:file(\.\+\\\.\.\+\$)",
-          destination:
-            '/api/proxy?url=https://raw.githubusercontent.com/:org/:project/main/:file',
-        },
         {
           source: '/@:org/:project*',
           destination: '/@org/:org/:project*',
@@ -25,11 +9,9 @@ const nextConfig = {
       ],
     };
   },
-  nx: {
+  serverRuntimeConfig: {
-    // Set this to true if you would like to use SVGR
+    github_pat: process.env.GITHUB_PAT ? process.env.GITHUB_PAT : null,
-    // See: https://github.com/gregberge/svgr
-    svgr: true,
   },
 };
 
-module.exports = withNx(nextConfig);
+module.exports = nextConfig;

examples/simple-example/package.json

@@ -1,7 +1,32 @@
 {
-  "name": "simple-example",
+  "name": "my-app",
-  "version": "1.0.0",
+  "version": "0.1.0",
-  "description": "",
+  "private": true,
-  "author": "",
+  "scripts": {
-  "license": "ISC"
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@types/node": "18.16.0",
+    "@types/react": "18.0.38",
+    "@types/react-dom": "18.0.11",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "next": "13.3.1",
+    "next-seo": "^6.0.0",
+    "octokit": "^2.0.14",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-markdown": "^8.0.7",
+    "remark-gfm": "^3.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
 }

examples/simple-example/pages/@org/[org]/[...path].tsx

@@ -0,0 +1,120 @@
+import Head from 'next/head';
+import { useRouter } from 'next/router';
+
+import { NextSeo } from 'next-seo';
+import { promises as fs } from 'fs';
+import path from 'path';
+import getConfig from 'next/config';
+import { getProject, GithubProject } from '../../../lib/octokit';
+import ReactMarkdown from 'react-markdown';
+import remarkGfm from 'remark-gfm';
+import Link from 'next/link';
+
+export default function ProjectPage({ project }) {
+  return (
+    <>
+      <NextSeo title={`PortalJS - @${project.repo_config.owner}/${project.repo_config.repo}${project.base_path !== '/' ? '/' + project.base_path : ''}`} />
+      <main className="prose mx-auto my-8">
+        <Link href='/'>Back to homepage</Link>
+        <h1 className="mb-0">Data</h1>
+        <div className="inline-block min-w-full py-2 align-middle">
+          <table className="min-w-full divide-y divide-gray-300">
+            <thead>
+              <tr>
+                <th
+                  scope="col"
+                  className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                >
+                  Name
+                </th>
+                <th
+                  scope="col"
+                  className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                >
+                  Size
+                </th>
+              </tr>
+            </thead>
+            <tbody className="divide-y divide-gray-200">
+              {project.files.map((file) => (
+                <tr key={file.download_url}>
+                  <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
+                    <a href={file.download_url}>{file.name}</a>
+                  </td>
+                  <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
+                    {file.size} Bytes
+                  </td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        </div>
+
+        <h1>Readme</h1>
+        <ReactMarkdown remarkPlugins={[remarkGfm]}>
+          {project.readmeContent}
+        </ReactMarkdown>
+      </main>
+    </>
+  );
+}
+
+// Generates `/posts/1` and `/posts/2`
+export async function getStaticPaths() {
+  const jsonDirectory = path.join(
+    process.cwd(),
+    'datasets.json'
+  );
+  const repos = await fs.readFile(jsonDirectory, 'utf8');
+
+  return {
+    paths: JSON.parse(repos).map((repo) => {
+      const projectPath =
+        repo.readme.split('/').length > 1
+          ? repo.readme.split('/').slice(0, -1)
+          : null;
+      let path = [repo.repo];
+      if (projectPath) {
+        projectPath.forEach((element) => {
+          path.push(element);
+        });
+      }
+      return {
+        params: { org: repo.owner, path },
+      };
+    }),
+    fallback: false, // can also be true or 'blocking'
+  };
+}
+
+export async function getStaticProps({ params }) {
+  const jsonDirectory = path.join(
+    process.cwd(),
+    'datasets.json'
+  );
+  const reposFile = await fs.readFile(jsonDirectory, 'utf8');
+  const repos: GithubProject[] = JSON.parse(reposFile);
+  const repo = repos.find((_repo) => {
+    const projectPath =
+      _repo.readme.split('/').length > 1
+        ? _repo.readme.split('/').slice(0, -1)
+        : null;
+    let path = [_repo.repo];
+    if (projectPath) {
+      projectPath.forEach((element) => {
+        path.push(element);
+      });
+    }
+    return (
+      _repo.owner == params.org &&
+      JSON.stringify(path) === JSON.stringify(params.path)
+    );
+  });
+  const github_pat = getConfig().serverRuntimeConfig.github_pat;
+  const project = await getProject(repo, github_pat);
+  return {
+    props: {
+      project: { ...project, repo_config: repo },
+    },
+  };
+}

examples/simple-example/pages/@org/[org]/[project]/index.tsx

@@ -1,110 +0,0 @@
-import Head from 'next/head';
-import { useRouter } from 'next/router';
-
-import DRD from '../../../../components/drd/DRD';
-import parse from '../../../../lib/markdown';
-import Project from '../../../../lib/project';
-import { NextSeo } from 'next-seo';
-import MDLayout from 'examples/simple-example/components/MDLayout';
-import { promises as fs } from 'fs';
-import path from 'path';
-
-function CollectionsLayout({ children, ...frontMatter }) {
-  const { title, date, description } = frontMatter;
-
-  return (
-    <article className="docs prose text-primary dark:text-primary-dark dark:prose-invert prose-headings:font-headings prose-a:break-words mx-auto p-6">
-      <header>
-        <div className="mb-6">
-          {date && (
-            <p className="text-sm text-zinc-400 dark:text-zinc-500">
-              <time dateTime={date}>{date}</time>
-            </p>
-          )}
-          {title && <h1 className="mb-2">{title}</h1>}
-          {description && <p className="text-xl mt-0">{description}</p>}
-        </div>
-      </header>
-      <section>{children}</section>
-    </article>
-  );
-}
-
-export default function ProjectPage({
-  mdxSource,
-  frontMatter,
-  excerpt,
-  project,
-}) {
-  const router = useRouter();
-
-  return (
-    <>
-      <NextSeo title={`PortalJS - @${project.owner}/${project.name}`} />
-      <Head>
-        {/* 
-          On index files, add trailling slash to the base path 
-          see notes: https://github.com/datopian/datahub-next/issues/69
-        */}
-        <base href={router.asPath.split('#')[0] + '/'} />
-      </Head>
-      <main>
-        <MDLayout
-          layout={frontMatter.layout}
-          excerpt={excerpt}
-          project={project}
-          {...frontMatter}
-        >
-          <DRD
-            source={mdxSource}
-            frictionless={{
-              views: project.metadata?.views,
-              resources: project.metadata?.resources,
-            }}
-          />
-        </MDLayout>
-      </main>
-    </>
-  );
-}
-
-// Generates `/posts/1` and `/posts/2`
-export async function getStaticPaths() {
-  const jsonDirectory = path.join(process.cwd(), '/examples/simple-example/datasets.json');
-  const repos = await fs.readFile(jsonDirectory, 'utf8');
-
-  return {
-    paths: JSON.parse(repos).map(repo => ({ params: { org: repo.owner, project: repo.repo}})),
-    fallback: false, // can also be true or 'blocking'
-  }
-}
-
-export async function getStaticProps({ params }) {
-  const { org: orgName, project: projectName } = params;
-
-  const project = await Project.getFromGitHub(orgName, projectName);
-
-  //  Defaults to README
-  let content = project.readme;
-
-  if (content === null) {
-    return {
-      notFound: true,
-    };
-  }
-
-  let { mdxSource, frontMatter, excerpt } = await parse(content, '.mdx');
-
-  if (project.metadata?.resources) {
-    frontMatter.layout = 'datapackage';
-  }
-
-  return {
-    props: {
-      mdxSource,
-      frontMatter,
-      excerpt,
-      project: project.serialize(),
-    },
-  };
-}

examples/simple-example/pages/_app.tsx

@@ -1,7 +1,6 @@
 import { AppProps } from 'next/app';
 import Head from 'next/head';
 import './styles.css';
-import "../styles/global.css";
 
 function CustomApp({ Component, pageProps }: AppProps) {
   return (

examples/simple-example/pages/api/proxy.ts

@@ -1,26 +0,0 @@
-import axios from "axios";
-
-export default function handler(req, res) {
-  if (!req.query.url) {
-    res.status(200).send({
-      error: true,
-      info: "No url to proxy in query string i.e. ?url=...",
-    });
-    return;
-  }
-  axios({
-    method: "get",
-    url: req.query.url,
-    responseType: "stream",
-  })
-    .then((resp) => {
-      resp.data.pipe(res);
-    })
-    .catch((err) => {
-      res.status(400).send({
-        error: true,
-        info: err.message,
-        detailed: err,
-      });
-    });
-}

examples/simple-example/pages/index.tsx

@@ -1,35 +1,20 @@
-import parse from '../lib/markdown';
-import Project from '../lib/project';
 import { promises as fs } from 'fs';
 import path from 'path';
-import Link from 'next/link';
+import { getProject } from '../lib/octokit';
+import getConfig from 'next/config';
 
 export async function getStaticProps() {
   const jsonDirectory = path.join(
     process.cwd(),
-    '/examples/simple-example/datasets.json'
+    '/datasets.json'
   );
   const repos = await fs.readFile(jsonDirectory, 'utf8');
+  const github_pat = getConfig().serverRuntimeConfig.github_pat;
 
   const projects = await Promise.all(
-    JSON.parse(repos).map(async (repo) => {
+    (JSON.parse(repos)).map(async (repo) => {
-      const project = await Project.getFromGitHub(repo.owner, repo.repo);
+      const project = await getProject(repo, github_pat);
-
+      return { ...project, repo_config: repo };
-      //  Defaults to README
-      const content = project.readme ? project.readme : '';
-
-      let { mdxSource, frontMatter, excerpt } = await parse(content, '.mdx');
-
-      if (project.metadata?.resources) {
-        frontMatter.layout = 'datapackage';
-      }
-
-      return {
-        mdxSource,
-        frontMatter,
-        excerpt,
-        project: project.serialize(),
-      };
     })
   );
   return {
@@ -53,37 +38,13 @@ export function Datasets({ projects }) {
   return (
     <div className="bg-white">
       <div className="mx-auto max-w-7xl px-6 py-16 sm:py-24 lg:px-8">
-        <h2 className="text-2xl font-bold leading-10 tracking-tight text-indigo-500">
+        <h2 className="text-2xl font-bold leading-10 tracking-tight">
           My Datasets
         </h2>
         <p className="mt-6 max-w-2xl text-base leading-7 text-gray-600">
           Here is a list of all my datasets for easy access and sharing
         </p>
         <div className="mt-20">
-          {/*
-          <dl className="space-y-16 sm:grid sm:grid-cols-2 sm:gap-x-6 sm:gap-y-16 sm:space-y-0 lg:grid-cols-3 lg:gap-x-10">
-            {projects.map((project) => (
-              <div>
-                <dt className="text-base font-semibold leading-7 text-gray-900">
-                  <Link
-                    href={`@${project.project.owner}/${project.project.name}`}
-                  >
-                    {project.project.owner}/{project.project.name}
-                  </Link>
-                </dt>
-                <dt className="text-base font-semibold leading-7 text-indigo-600">
-                  <a
-                    href={`https://github.com/${project.project.owner}/${project.project.name}`}
-                  >
-                    Github repo
-                  </a>
-                </dt>
-                <dd className="mt-2 text-base leading-7 text-gray-600">
-                  {project.excerpt !== '' ? project.excerpt : 'No description'}
-                </dd>
-              </div>
-            ))}
-          </dl> */}
           <div className="-mx-4 -my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
             <div className="inline-block min-w-full py-2 align-middle sm:px-6 lg:px-8">
               <table className="min-w-full divide-y divide-gray-300">
@@ -93,7 +54,13 @@ export function Datasets({ projects }) {
                       scope="col"
                       className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
                     >
-                      Dataset name
+                      Name
+                    </th>
+                    <th
+                      scope="col"
+                      className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                    >
+                      Repo
                     </th>
                     <th
                       scope="col"
@@ -115,24 +82,26 @@ export function Datasets({ projects }) {
                 </thead>
                 <tbody className="divide-y divide-gray-200">
                   {projects.map((project) => (
-                    <tr>
+                    <tr key={project.id}>
                       <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
-                        <a href={project.project.repo_metadata.html_url}>
+                        {project.repo_config.name
-                          {project.project.owner}/{project.project.name}
+                          ? project.repo_config.name
-                        </a>
+                          : project.full_name + (project.base_path === '/' ? '' : '/' + project.base_path)}
+                      </td>
+                      <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
+                        <a href={project.html_url}>{project.full_name}</a>
                       </td>
                       <td className="px-3 py-4 text-sm text-gray-500">
-                        {project.project.repo_metadata.description}
+                        {project.repo_config.description
+                          ? project.repo_config.description
+                          : project.description}
                       </td>
                       <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
-                        {formatter.format(
+                        {formatter.format(new Date(project.last_updated))}
-                          new Date(project.project.repo_metadata.updated_at)
-                        )}
                       </td>
                       <td className="relative whitespace-nowrap py-4 pl-3 pr-4 text-right text-sm font-medium sm:pr-0">
                         <a
-                          href={`/@${project.project.owner}/${project.project.name}`}
+                          href={`/@${project.repo_config.owner}/${project.repo_config.repo}/${project.base_path === '/' ? '' : project.base_path}`}
-                          className="text-indigo-600 hover:text-indigo-900"
                         >
                           More info
                         </a>

examples/simple-example/postcss.config.js

@@ -1,15 +1,6 @@
-const { join } = require('path');
-
-// Note: If you use library-specific PostCSS/Tailwind configuration then you should remove the `postcssConfig` build
-// option from your application's configuration (i.e. project.json).
-//
-// See: https://nx.dev/guides/using-tailwind-css-in-react#step-4:-applying-configuration-to-libraries
-
 module.exports = {
   plugins: {
-    tailwindcss: {
+    tailwindcss: {},
-      config: join(__dirname, 'tailwind.config.js'),
-    },
     autoprefixer: {},
   },
-};
+}

examples/simple-example/project.json

@@ -1,69 +0,0 @@
-{
-  "name": "simple-example",
-  "$schema": "../../node_modules/nx/schemas/project-schema.json",
-  "sourceRoot": "examples/simple-example",
-  "projectType": "application",
-  "targets": {
-    "build": {
-      "executor": "@nrwl/next:build",
-      "outputs": ["{options.outputPath}"],
-      "defaultConfiguration": "production",
-      "options": {
-        "root": "examples/simple-example",
-        "outputPath": "dist/examples/simple-example"
-      },
-      "configurations": {
-        "development": {
-          "outputPath": "examples/simple-example"
-        },
-        "production": {}
-      }
-    },
-    "serve": {
-      "executor": "@nrwl/next:server",
-      "defaultConfiguration": "development",
-      "options": {
-        "buildTarget": "simple-example:build",
-        "dev": true
-      },
-      "configurations": {
-        "development": {
-          "buildTarget": "simple-example:build:development",
-          "dev": true
-        },
-        "production": {
-          "buildTarget": "simple-example:build:production",
-          "dev": false
-        }
-      }
-    },
-    "export": {
-      "executor": "@nrwl/next:export",
-      "options": {
-        "buildTarget": "simple-example:build:production"
-      }
-    },
-    "test": {
-      "executor": "@nrwl/jest:jest",
-      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
-      "options": {
-        "jestConfig": "examples/simple-example/jest.config.ts",
-        "passWithNoTests": true
-      },
-      "configurations": {
-        "ci": {
-          "ci": true,
-          "codeCoverage": true
-        }
-      }
-    },
-    "lint": {
-      "executor": "@nrwl/linter:eslint",
-      "outputs": ["{options.outputFile}"],
-      "options": {
-        "lintFilePatterns": ["examples/simple-example/**/*.{ts,tsx,js,jsx}"]
-      }
-    }
-  },
-  "tags": []
-}

examples/simple-example/specs/index.spec.tsx

@@ -1,11 +0,0 @@
-import React from 'react';
-import { render } from '@testing-library/react';
-
-import Index from '../pages/index';
-
-describe('Index', () => {
-  it('should render successfully', () => {
-    const { baseElement } = render(<Index />);
-    expect(baseElement).toBeTruthy();
-  });
-});

examples/simple-example/tailwind.config.js

@@ -1,21 +1,15 @@
-const { createGlobPatternsForDependencies } = require('@nrwl/react/tailwind');
-const { join } = require('path');
-
 /** @type {import('tailwindcss').Config} */
 module.exports = {
   content: [
-    "node_modules/@flowershow/core/dist/*.js",
+    "./app/**/*.{js,ts,jsx,tsx,mdx}",
-    "node_modules/@flowershow/core/*.js",
+    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
-    join(
+    "./components/**/*.{js,ts,jsx,tsx,mdx}",
-      __dirname,
-      '{src,pages,components}/**/*!(*.stories|*.spec).{ts,tsx,html}'
-    ),
-    ...createGlobPatternsForDependencies(__dirname),
   ],
   theme: {
     extend: {},
   },
   plugins: [
-    require('@tailwindcss/typography'),
+    require('@tailwindcss/typography')
   ],
-};
+}
+

examples/simple-example/tsconfig.json

@@ -1,50 +1,20 @@
 {
-  "extends": "../../tsconfig.base.json",
   "compilerOptions": {
-    "jsx": "preserve",
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
     "allowJs": true,
-    "esModuleInterop": true,
+    "skipLibCheck": true,
-    "allowSyntheticDefaultImports": true,
     "strict": false,
     "forceConsistentCasingInFileNames": true,
     "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
     "resolveJsonModule": true,
     "isolatedModules": true,
-    "incremental": true,
+    "jsx": "preserve",
-    "types": [
+    "incremental": true
-      "jest",
-      "node"
-    ]
   },
-  "target": "es2020",
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
-  "lib": [
+  "exclude": ["node_modules"]
-    "dom",
-    "dom.iterable",
-    "esnext"
-  ],
-  "allowJs": true,
-  "skipLibCheck": true,
-  "strict": false,
-  "forceConsistentCasingInFileNames": true,
-  "noEmit": true,
-  "incremental": true,
-  "esModuleInterop": true,
-  "module": "esnext",
-  "moduleResolution": "node",
-  "resolveJsonModule": true,
-  "isolatedModules": true,
-  "jsx": "preserve",
-  "include": [
-    "**/*.ts",
-    "**/*.tsx",
-    "**/*.js",
-    "**/*.jsx",
-    "next-env.d.ts"
-  ],
-  "exclude": [
-    "node_modules",
-    "jest.config.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.ts"
-  ]
 }

examples/simple-example/tsconfig.spec.json

@@ -1,24 +0,0 @@
-{
-  "extends": "./tsconfig.json",
-  "compilerOptions": {
-    "outDir": "../../dist/out-tsc",
-    "module": "commonjs",
-    "types": ["jest", "node"],
-    "jsx": "react"
-  },
-  "paths": {
-      "@/*": ["./*"]
-  },
-  "include": [
-    "jest.config.ts",
-    "src/**/*.test.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.tsx",
-    "src/**/*.spec.tsx",
-    "src/**/*.test.js",
-    "src/**/*.spec.js",
-    "src/**/*.test.jsx",
-    "src/**/*.spec.jsx",
-    "src/**/*.d.ts"
-  ]
-}

package.json

@@ -4,62 +4,7 @@
   "license": "MIT",
   "scripts": {},
   "private": true,
-  "dependencies": {
+  "dependencies": {},
-    "@apollo/client": "^3.7.11",
-    "@apollo/react-hooks": "^4.0.0",
-    "@emotion/react": "^11.10.6",
-    "@emotion/styled": "^11.10.6",
-    "@flowershow/core": "^0.4.9",
-    "@flowershow/markdowndb": "^0.1.0",
-    "@flowershow/remark-callouts": "^1.0.0",
-    "@flowershow/remark-embed": "^1.0.0",
-    "@flowershow/remark-wiki-link": "^1.0.1",
-    "@headlessui/react": "^1.7.13",
-    "@heroicons/react": "^2.0.17",
-    "@mui/icons-material": "^5.11.16",
-    "@mui/material": "^5.11.16",
-    "@mui/x-data-grid": "^6.1.0",
-    "@opentelemetry/api": "^1.4.0",
-    "@tailwindcss/typography": "^0.5.9",
-    "@tanstack/react-table": "^8.8.5",
-    "apollo-cache-inmemory": "^1.6.6",
-    "apollo-link": "^1.2.14",
-    "apollo-link-rest": "^0.9.0",
-    "filesize": "^10.0.7",
-    "gray-matter": "^4.0.3",
-    "html-react-parser": "^3.0.15",
-    "markdown-it": "^13.0.1",
-    "next": "^13.2.1",
-    "next-mdx-remote": "^4.4.1",
-    "next-seo": "^6.0.0",
-    "next-translate": "^2.0.5",
-    "nock": "^13.3.0",
-    "octokit": "^2.0.14",
-    "papaparse": "^5.4.1",
-    "plotly.js-basic-dist": "^2.20.0",
-    "prop-types": "^15.8.1",
-    "react": "18.2.0",
-    "react-dom": "18.2.0",
-    "react-next-github-btn": "^1.2.1",
-    "react-plotly.js": "^2.6.0",
-    "react-plotlyjs": "^0.4.4",
-    "react-vega": "^7.6.0",
-    "rehype-autolink-headings": "^6.1.1",
-    "rehype-katex": "^6.0.2",
-    "rehype-prism-plus": "^1.5.1",
-    "rehype-slug": "^5.1.0",
-    "remark-footnotes": "^4.0.1",
-    "remark-gfm": "^3.0.1",
-    "remark-math": "^5.1.1",
-    "remark-slug": "^7.0.1",
-    "remark-smartypants": "^2.0.0",
-    "remark-toc": "^8.0.1",
-    "slugify": "^1.6.6",
-    "timeago.js": "^4.0.2",
-    "tslib": "^2.3.0",
-    "vega": "^5.24.0",
-    "xlsx": "^0.18.5"
-  },
   "devDependencies": {
     "@babel/preset-react": "^7.14.5",
     "@nrwl/cypress": "15.9.2",
@@ -83,7 +28,6 @@
     "@types/react-dom": "18.0.11",
     "@typescript-eslint/eslint-plugin": "^5.36.1",
     "@typescript-eslint/parser": "^5.36.1",
-    "autoprefixer": "10.4.13",
     "babel-jest": "^29.4.1",
     "cypress": "^12.2.0",
     "eslint": "~8.15.0",
@@ -97,11 +41,9 @@
     "jest": "^29.4.1",
     "jest-environment-jsdom": "^29.4.1",
     "nx": "15.9.2",
-    "postcss": "8.4.21",
     "prettier": "^2.6.2",
     "react-test-renderer": "18.2.0",
     "swc-loader": "0.1.15",
-    "tailwindcss": "3.2.7",
     "ts-jest": "^29.0.5",
     "ts-node": "10.9.1",
     "typescript": "~4.9.5"

site/README.md

@@ -1,4 +1,4 @@
-This the Portal.JS website.
+This the PortalJS website.
 
 It is built on [Next.js](https://nextjs.org/).
 

site/components/ButtonLink.tsx

@@ -0,0 +1,26 @@
+import Link from 'next/link';
+
+export default function ButtonLink({
+  style = 'primary',
+  className = '',
+  href = '',
+  children,
+}) {
+  let styleClassName = '';
+
+  if (style == 'primary') {
+    styleClassName = 'text-primary bg-blue-400 hover:bg-blue-300';
+  } else if (style == 'secondary') {
+    styleClassName =
+      'text-secondary border !border-secondary hover:text-primary hover:bg-blue-300';
+  }
+
+  return (
+    <Link
+      href={href}
+      className={`inline-block h-12 px-6 py-3 border border-transparent text-base font-medium rounded-md focus:outline-none focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-sky-300/50 active:bg-sky-500 ${styleClassName} ${className}`}
+    >
+      {children}
+    </Link>
+  );
+}

site/components/Community.tsx

@@ -0,0 +1,99 @@
+import Container from './Container';
+import DiscordIcon from './icons/DiscordIcon';
+import EmailIcon from './icons/EmailIcon';
+import GitHubIcon from './icons/GitHubIcon';
+
+import { siteConfig } from '@/config/siteConfig';
+import { getContributorsCount, getRepoInfo } from '@/lib/getGitHubData';
+import { useEffect, useState } from 'react';
+
+const Stat = ({ title, value, ...props }) => {
+  return (
+    <div {...props}>
+      <span className="text-6xl font-bold text-secondary">{value}</span>
+      <p className="text-lg font-medium">{title}</p>
+    </div>
+  );
+};
+
+const IconButton = ({ Icon, text, href, ...props }) => {
+  return (
+    <div {...props}>
+      <a
+        className="rounded border border-secondary px-5 py-3 text-primary dark:text-primary-dark flex items-center hover:bg-secondary hover:text-primary dark:hover:text-primary transition-all duration-200"
+        href={href}
+      >
+        <Icon className="w-6 h-6 mr-2" />
+        {text}
+      </a>
+    </div>
+  );
+};
+
+export default function Community() {
+  const [repoInfo, setRepoInfo] = useState<any>();
+  const [contributorsCount, setContributorsCount] = useState('');
+
+  useEffect(() => {
+    //  This runs on client side and it's unlikely that users
+    //  will exceed the GitHub API  usage limit,  but added a
+    //  handling for that just in case.
+
+    getRepoInfo().then((res) => {
+      if (res.success) {
+        res.info.then((data) => setRepoInfo(data));
+      } else {
+        //  If the request fail e.g API usage limit, use
+        //  a placeholder
+        setRepoInfo({ stargazers_count: '+2k' });
+      }
+    });
+
+    getContributorsCount().then((res) => {
+      if (res.success) {
+        setContributorsCount(res.count);
+      } else {
+        setContributorsCount('+70');
+      }
+    });
+  }, []);
+
+  return (
+    <Container>
+      <h2 className="text-3xl font-bold text-primary dark:text-primary-dark ">
+        Community
+      </h2>
+      <p className="text-lg mt-8 ">
+        We are growing. Get in touch or become a contributor!
+      </p>
+      <div className="flex justify-center mt-12">
+        <Stat
+          title="Stars on GitHub"
+          value={repoInfo?.stargazers_count}
+          className="mr-10"
+        />
+        <Stat title="Contributors" value={contributorsCount} />
+      </div>
+      <div className="flex flex-wrap justify-center mt-12">
+        <IconButton
+          Icon={GitHubIcon}
+          text="Star PortalJS on GitHub"
+          className="sm:mr-4 mb-4 w-full sm:w-auto"
+          href={siteConfig.github}
+        />
+        <IconButton
+          Icon={DiscordIcon}
+          text="Join the Discord server"
+          className="sm:mr-4 mb-4 w-full sm:w-auto"
+          href={siteConfig.discord}
+        />
+        <IconButton
+          Icon={EmailIcon}
+          text="Subscribe to the PortalJS newsletter"
+          className="w-full sm:w-auto"
+          href="#hero"
+        />
+      </div>
+    </Container>
+  );
+}

site/components/Container.tsx

@@ -0,0 +1,7 @@
+export default function Container({ children }) {
+  return (
+    <div className="lg:max-w-8xl mx-auto px-4 lg:px-8 xl:px-12 mb-32">
+      {children}
+    </div>
+  );
+}

site/components/Features.tsx

@@ -1,3 +1,5 @@
+import Container from './Container';
+
 const features: { title: string; description: string; icon: string }[] = [
   {
     title: 'Unified sites',
@@ -37,10 +39,12 @@ const features: { title: string; description: string; icon: string }[] = [
 
 export default function Features() {
   return (
-    <div className="lg:max-w-8xl mx-auto px-4 lg:px-8 xl:px-12">
+    <Container>
-      <h2 className="text-3xl font-bold">How Portal.JS works?</h2>
+      <h2 className="text-3xl font-bold text-primary dark:text-primary-dark">
+        How PortalJS works?
+      </h2>
       <p className="text-lg mt-8">
-        Portal.JS is built in JavaScript and React on top of the popular Next.js
+        PortalJS is built in JavaScript and React on top of the popular Next.js
         framework, assuming a "decoupled" approach where the frontend is a
         separate service from the backend and interacts with backend(s) via an
         API. It can be used with any backend and has out of the box support for
@@ -55,7 +59,7 @@ export default function Features() {
             <div className="absolute -inset-px rounded-xl border-2 border-transparent opacity-0 [background:linear-gradient(var(--quick-links-hover-bg,theme(colors.sky.50)),var(--quick-links-hover-bg,theme(colors.sky.50)))_padding-box,linear-gradient(to_top,theme(colors.blue.300),theme(colors.blue.400),theme(colors.blue.500))_border-box] group-hover:opacity-100 dark:[--quick-links-hover-bg:theme(colors.slate.800)]" />
             <div className="relative overflow-hidden rounded-xl p-6">
               <img src={feature.icon} alt="" className="h-24 w-auto" />
-              <h2 className="mt-4 font-display text-base text-slate-900 dark:text-white">
+              <h2 className="mt-4 font-display text-base text-primary dark:text-primary-dark">
                 <span className="absolute -inset-px rounded-xl" />
                 {feature.title}
               </h2>
@@ -66,6 +70,6 @@ export default function Features() {
           </div>
         ))}
       </div>
-    </div>
+    </Container>
   );
 }

site/components/Gallery.tsx

@@ -0,0 +1,73 @@
+import Container from './Container';
+import GalleryItem from './GalleryItem';
+
+const items = [
+  {
+    title: 'Open Data Northern Ireland',
+    href: 'https://www.opendatani.gov.uk/',
+    image: '/images/showcases/odni.png',
+    description: 'Government Open Data Portal',
+  },
+  {
+    title: 'Birmingham City Observatory',
+    href: 'https://www.cityobservatory.birmingham.gov.uk/',
+    image: '/images/showcases/birmingham.png',
+    description: 'Government Open Data Portal',
+  },
+  {
+    title: 'UAE Open Data',
+    href: 'https://opendata.fcsc.gov.ae/',
+    image: '/images/showcases/uae.png',
+    description: 'Government Open Data Portal',
+    sourceUrl: 'https://github.com/FCSCOpendata/frontend',
+  },
+  {
+    title: 'Brazil Open Data',
+    href: 'https://dados.gov.br/',
+    image: '/images/showcases/brazil.png',
+    description: 'Government Open Data Portal',
+  },
+  {
+    title: 'Datahub Open Data',
+    href: 'https://opendata.datahub.io/',
+    image: '/images/showcases/datahub.png',
+    description: 'Demo Data Portal by DataHub',
+  },
+  {
+    title: 'Example: Simple Data Catalog',
+    href: 'https://example.portaljs.org/',
+    image: '/images/showcases/example-simple-catalog.png',
+    description: 'Simple data catalog',
+    sourceUrl:
+      'https://github.com/datopian/portaljs/tree/main/examples/simple-example',
+    docsUrl: '/docs/example-data-catalog',
+  },
+  {
+    title: 'Example: Portal with CKAN',
+    href: 'https://ckan-example.portaljs.org/',
+    image: '/images/showcases/example-ckan.png',
+    description: 'Simple portal with data coming from CKAN',
+    sourceUrl:
+      'https://github.com/datopian/portaljs/tree/main/examples/ckan-example',
+    docsUrl: '/docs/example-ckan',
+  },
+];
+
+export default function Gallery() {
+  return (
+    <Container>
+      <h2
+        className="text-3xl font-bold text-primary dark:text-primary-dark"
+        id="gallery"
+      >
+        Gallery
+      </h2>
+      <p className="text-lg mt-8">Discover what's being powered by PortalJS</p>
+      <div className="not-prose my-12 grid grid-cols-1 gap-6 md:grid-cols-2 lg:grid-cols-3">
+        {items.map((item) => {
+          return <GalleryItem item={item} />;
+        })}
+      </div>
+    </Container>
+  );
+}

site/components/GalleryItem.tsx

@@ -0,0 +1,106 @@
+const IconBeaker = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    stroke-width="1.5"
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      d="M9.75 3.104v5.714a2.25 2.25 0 01-.659 1.591L5 14.5M9.75 3.104c-.251.023-.501.05-.75.082m.75-.082a24.301 24.301 0 014.5 0m0 0v5.714c0 .597.237 1.17.659 1.591L19.8 15.3M14.25 3.104c.251.023.501.05.75.082M19.8 15.3l-1.57.393A9.065 9.065 0 0112 15a9.065 9.065 0 00-6.23-.693L5 14.5m14.8.8l1.402 1.402c1.232 1.232.65 3.318-1.067 3.611A48.309 48.309 0 0112 21c-2.773 0-5.491-.235-8.135-.687-1.718-.293-2.3-2.379-1.067-3.61L5 14.5"
+    />
+  </svg>
+);
+
+const IconDocs = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    strokeWidth={1.5}
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      strokeLinecap="round"
+      strokeLinejoin="round"
+      d="M12 6.042A8.967 8.967 0 006 3.75c-1.052 0-2.062.18-3 .512v14.25A8.987 8.987 0 016 18c2.305 0 4.408.867 6 2.292m0-14.25a8.966 8.966 0 016-2.292c1.052 0 2.062.18 3 .512v14.25A8.987 8.987 0 0018 18a8.967 8.967 0 00-6 2.292m0-14.25v14.25"
+    />
+  </svg>
+);
+
+const IconCode = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    strokeWidth={1.5}
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      strokeLinecap="round"
+      strokeLinejoin="round"
+      d="M17.25 6.75L22.5 12l-5.25 5.25m-10.5 0L1.5 12l5.25-5.25m7.5-3l-4.5 16.5"
+    />
+  </svg>
+);
+
+const ActionButton = ({ title, Icon, href, className = '' }) => (
+  <a
+    title={title}
+    target="_blank"
+    href={href}
+    className={`rounded-full p-2 hover:bg-secondary transition-all duration-250 ${className}`}
+  >
+    <Icon />
+  </a>
+);
+
+export default function GalleryItem({ item }) {
+  return (
+    <a
+      className="rounded overflow-hidden group relative border-1 shadow-lg"
+      target="_blank"
+      href={item.href}
+    >
+      <div
+        className="bg-cover bg-no-repeat bg-top aspect-video w-full group-hover:blur-sm group-hover:scale-105 transition-all duration-200"
+        style={{ backgroundImage: `url(${item.image})` }}
+      >
+        <div className="w-full h-full bg-black opacity-0 group-hover:opacity-50 transition-all duration-200"></div>
+      </div>
+      <div>
+        <div className="opacity-0 group-hover:opacity-100 absolute top-0 bottom-0 right-0 left-0 transition-all duration-200 px-2 flex items-center justify-center">
+          <div className="text-center text-primary-dark">
+            <span className="text-xl font-semibold">{item.title}</span>
+            <p className="text-base font-medium">{item.description}</p>
+            <div className="flex justify-center mt-2">
+              <ActionButton Icon={IconBeaker} title="Demo" href={item.href} />
+              {item.docsUrl && (
+                <ActionButton
+                  Icon={IconDocs}
+                  title="Documentation"
+                  href={item.docsUrl}
+                  className="mx-5"
+                />
+              )}
+              {item.sourceUrl && (
+                <ActionButton
+                  Icon={IconCode}
+                  title="Source code"
+                  href={item.sourceUrl}
+                />
+              )}
+
+              {/* Maybe: Blog post */}
+            </div>
+          </div>
+        </div>
+      </div>
+    </a>
+  );
+}

site/components/Hero.tsx

@@ -1,6 +1,7 @@
 import clsx from 'clsx';
 import Highlight, { defaultProps } from 'prism-react-renderer';
 import { Fragment, useRef } from 'react';
+import ButtonLink from './ButtonLink';
 import NewsletterForm from './NewsletterForm';
 
 const codeLanguage = 'javascript';
@@ -32,7 +33,10 @@ export function Hero() {
   const el = useRef(null);
 
   return (
-    <div className="overflow-hidden -mb-32 mt-[-4.5rem] pb-32 pt-[4.5rem] lg:mt-[-4.75rem] lg:pt-[4.75rem]">
+    <div
+      className="overflow-hidden -mb-32 mt-[-4.5rem] pb-32 pt-[4.5rem] lg:mt-[-4.75rem] lg:pt-[4.75rem]"
+      id="hero"
+    >
       <div className="py-16 sm:px-2 lg:relative lg:py-20 lg:px-0">
         {/* Commented code on line 37, 39 and 113 will reenable the two columns hero */}
         {/* <div className="mx-auto grid max-w-2xl grid-cols-1 items-center gap-y-16 gap-x-8 px-4 lg:max-w-8xl lg:grid-cols-2 lg:px-8 xl:gap-x-16 xl:px-12"> */}
@@ -45,12 +49,20 @@ export function Hero() {
               </h1>
             </div>
             <p className="mt-4 text-xl tracking-tight text-slate-400">
-              Portal.JS is a framework for rapidly building rich data portal
+              Rapidly build rich data portals using a modern frontend framework.
-              frontends using a modern frontend approach. It can be used to
-              present a single dataset or build a full-scale data
-              catalog/portal.
             </p>
-            <NewsletterForm />
+
+            <ButtonLink className="mt-8" href="/docs">
+              Get started
+            </ButtonLink>
+
+            <ButtonLink className="ml-3" href="#gallery" style="secondary">
+              Gallery
+            </ButtonLink>
+
+            <div className="md:max-w-md mx-auto">
+              <NewsletterForm />
+            </div>
             <p className="my-10 text-l tracking-wide">
               <span>A project of</span>
               <a

site/components/Layout.tsx

@@ -1,5 +1,6 @@
 import { siteConfig } from '@/config/siteConfig';
 import { NextSeo } from 'next-seo';
+import { useTheme } from 'next-themes';
 import Link from 'next/link';
 import { useCallback, useEffect, useState } from 'react';
 
@@ -58,6 +59,7 @@ export default function Layout({
   tableOfContents?;
 }) {
   // const { toc } = children.props;
+  const { theme, setTheme } = useTheme();
 
   const currentSection = useTableOfContents(tableOfContents);
 
@@ -87,17 +89,21 @@ export default function Layout({
         >
           Built by{' '}
           <img
-            src="/datopian-logo.png"
+            src={
+              theme === 'dark'
+                ? '/images/datopian-light-logotype.svg'
+                : '/images/datopian-dark-logotype.svg'
+            }
             alt="Datopian Logo"
             className="h-6 ml-2"
           />
         </a>
       </footer>
       {/** TABLE OF CONTENTS */}
-      {tableOfContents.length > 0 && (siteConfig.tableOfContents) && (
+      {tableOfContents.length > 0 && siteConfig.tableOfContents && (
         <div className="hidden xl:fixed xl:right-0 xl:top-[4.5rem] xl:block xl:w-1/5 xl:h-[calc(100vh-4.5rem)] xl:flex-none xl:overflow-y-auto xl:py-16 xl:pr-6 xl:mb-16">
           <nav aria-labelledby="on-this-page-title" className="w-56">
-            <h2 className="font-display text-md font-medium text-slate-900 dark:text-white">
+            <h2 className="font-display text-md font-medium text-primary dark:text-primary-dark">
               On this page
             </h2>
             <ol className="mt-4 space-y-3 text-sm">
@@ -108,7 +114,7 @@ export default function Layout({
                       href={`#${section.id}`}
                       className={
                         isActive(section)
-                          ? 'text-sky-500'
+                          ? 'text-secondary'
                           : 'font-normal text-slate-500 hover:text-slate-700 dark:text-slate-400 dark:hover:text-slate-300'
                       }
                     >

site/components/MDXPage.tsx

@@ -11,7 +11,7 @@ export default function MDXPage({ source, frontMatter }) {
   };
 
   return (
-    <div className="prose mx-auto prose-a:text-primary dark:prose-a:text-primary-dark prose-strong:text-primary dark:prose-strong:text-primary-dark prose-code:text-primary dark:prose-code:text-primary-dark prose-headings:text-primary dark:prose-headings:text-primary-dark prose text-primary dark:text-primary-dark prose-headings:font-headings dark:prose-invert prose-a:break-words">
+    <div className="prose mx-auto prose-a:text-primary dark:prose-a:text-primary-dark prose-strong:text-primary dark:prose-strong:text-primary-dark prose-headings:text-primary dark:prose-headings:text-primary-dark text-primary dark:text-primary-dark prose-headings:font-headings dark:prose-invert prose-a:break-words">
       <header>
         <div className="mb-6">
           {/* Default layout */}

site/components/NavItem.tsx

@@ -29,12 +29,12 @@ export default function NavItem({ item }) {
         {Object.prototype.hasOwnProperty.call(item, "href") ? (
           <Link
             href={item.href}
-            className="text-slate-500 inline-flex items-center mr-2 px-1 pt-1 text-sm font-medium hover:text-slate-600"
+            className="text-slate-600 dark:text-slate-400 inline-flex items-center mr-2 px-1 pt-1 text-sm font-medium hover:text-slate-500"
           >
             {item.name}
           </Link>
         ) : (
-          <div className="text-slate-500 inline-flex items-center mr-2 px-1 pt-1 text-sm font-medium hover:text-slate-600 fill-slate-500 hover:fill-slate-600">
+          <div className="text-slate-600 dark:text-slate-400 inline-flex items-center mr-2 px-1 pt-1 text-sm font-medium hover:text-slate-500 fill-slate-500 hover:fill-slate-600">
             {item.name}
           </div>
         )}

site/components/NewsletterForm.tsx

@@ -22,24 +22,7 @@ export default function NewsletterForm() {
             data-type="subscription"
             className="mt-3 sm:flex"
           >
-            <div className="sib-input sib-form-block !p-0 block w-full sm:flex-auto sm:w-32">
+            <div className="sib-input sib-form-block !p-0 block w-full sm:flex-auto sm:w-64 mt-3 sm:mt-0">
-              <div className="form__entry entry_block w-full">
-                <label htmlFor="name" className="sr-only entry__label">
-                  Name
-                </label>
-                <input
-                  id="NAME"
-                  name="NAME"
-                  type="text"
-                  required
-                  placeholder="Your name"
-                  className="input entry__field !w-full px-2 py-3 text-base rounded-md bg-slate-200 dark:bg-slate-800 placeholder-gray-500 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-blue-400 focus:ring-offset-gray-900"
-                />
-
-                <label className="entry__error entry__error--primary text-red-400 text-sm"></label>
-              </div>
-            </div>
-            <div className="sib-input sib-form-block !p-0 block w-full sm:flex-auto sm:w-64 mt-3 sm:mt-0 sm:ml-3">
               <div className="form__entry entry_block w-full">
                 <label htmlFor="email" className="sr-only entry__label">
                   Email address
@@ -68,7 +51,7 @@ export default function NewsletterForm() {
               >
                 <path d="M460.116 373.846l-20.823-12.022c-5.541-3.199-7.54-10.159-4.663-15.874 30.137-59.886 28.343-131.652-5.386-189.946-33.641-58.394-94.896-95.833-161.827-99.676C261.028 55.961 256 50.751 256 44.352V20.309c0-6.904 5.808-12.337 12.703-11.982 83.556 4.306 160.163 50.864 202.11 123.677 42.063 72.696 44.079 162.316 6.031 236.832-3.14 6.148-10.75 8.461-16.728 5.01z" />
               </svg>
-              Notify me
+              Notify Me
             </button>
             <input
               type="text"
@@ -79,10 +62,7 @@ export default function NewsletterForm() {
             <input type="hidden" name="locale" value="en" />
           </form>
         </div>
-        <div
+        <div id="error-message" className="sib-form-message-panel !border-none">
-          id="error-message"
-          className="sib-form-message-panel !border-none"
-        >
           <div className="sib-form-message-panel__text sib-form-message-panel__text--center !text-red-400 justify-center">
             <svg
               viewBox="0 0 512 512"

site/components/icons/DiscordIcon.tsx

@@ -0,0 +1,14 @@
+export default function DiscordIcon(props) {
+  return (
+    <svg
+      xmlns="http://www.w3.org/2000/svg"
+      width="16"
+      height="16"
+      fill="currentColor"
+      viewBox="0 0 16 16"
+      {...props}
+    >
+      <path d="M13.545 2.907a13.227 13.227 0 0 0-3.257-1.011.05.05 0 0 0-.052.025c-.141.25-.297.577-.406.833a12.19 12.19 0 0 0-3.658 0 8.258 8.258 0 0 0-.412-.833.051.051 0 0 0-.052-.025c-1.125.194-2.22.534-3.257 1.011a.041.041 0 0 0-.021.018C.356 6.024-.213 9.047.066 12.032c.001.014.01.028.021.037a13.276 13.276 0 0 0 3.995 2.02.05.05 0 0 0 .056-.019c.308-.42.582-.863.818-1.329a.05.05 0 0 0-.01-.059.051.051 0 0 0-.018-.011 8.875 8.875 0 0 1-1.248-.595.05.05 0 0 1-.02-.066.051.051 0 0 1 .015-.019c.084-.063.168-.129.248-.195a.05.05 0 0 1 .051-.007c2.619 1.196 5.454 1.196 8.041 0a.052.052 0 0 1 .053.007c.08.066.164.132.248.195a.051.051 0 0 1-.004.085 8.254 8.254 0 0 1-1.249.594.05.05 0 0 0-.03.03.052.052 0 0 0 .003.041c.24.465.515.909.817 1.329a.05.05 0 0 0 .056.019 13.235 13.235 0 0 0 4.001-2.02.049.049 0 0 0 .021-.037c.334-3.451-.559-6.449-2.366-9.106a.034.034 0 0 0-.02-.019Zm-8.198 7.307c-.789 0-1.438-.724-1.438-1.612 0-.889.637-1.613 1.438-1.613.807 0 1.45.73 1.438 1.613 0 .888-.637 1.612-1.438 1.612Zm5.316 0c-.788 0-1.438-.724-1.438-1.612 0-.889.637-1.613 1.438-1.613.807 0 1.451.73 1.438 1.613 0 .888-.631 1.612-1.438 1.612Z" />
+    </svg>
+  );
+}

site/components/icons/EmailIcon.tsx

@@ -0,0 +1,14 @@
+export default function EmailIcon(props) {
+  return (
+    <svg
+      fill="currentColor"
+      viewBox="0 0 2150 2150"
+      xmlns="http://www.w3.org/2000/svg"
+      {...props}
+    >
+      <path
+        d="M1920 428.266v1189.54l-464.16-580.146-88.203 70.585 468.679 585.904H83.684l468.679-585.904-88.202-70.585L0 1617.805V428.265l959.944 832.441L1920 428.266ZM1919.932 226v52.627l-959.943 832.44L.045 278.628V226h1919.887Z"
+      />
+    </svg>
+  );
+}

site/components/icons/GitHubIcon.tsx

@@ -0,0 +1,7 @@
+export default function GitHubIcon(props) {
+  return (
+    <svg aria-hidden="true" viewBox="0 0 16 16" fill="currentColor" {...props}>
+      <path d="M8 0C3.58 0 0 3.58 0 8C0 11.54 2.29 14.53 5.47 15.59C5.87 15.66 6.02 15.42 6.02 15.21C6.02 15.02 6.01 14.39 6.01 13.72C4 14.09 3.48 13.23 3.32 12.78C3.23 12.55 2.84 11.84 2.5 11.65C2.22 11.5 1.82 11.13 2.49 11.12C3.12 11.11 3.57 11.7 3.72 11.94C4.44 13.15 5.59 12.81 6.05 12.6C6.12 12.08 6.33 11.73 6.56 11.53C4.78 11.33 2.92 10.64 2.92 7.58C2.92 6.71 3.23 5.99 3.74 5.43C3.66 5.23 3.38 4.41 3.82 3.31C3.82 3.31 4.49 3.1 6.02 4.13C6.66 3.95 7.34 3.86 8.02 3.86C8.7 3.86 9.38 3.95 10.02 4.13C11.55 3.09 12.22 3.31 12.22 3.31C12.66 4.41 12.38 5.23 12.3 5.43C12.81 5.99 13.12 6.7 13.12 7.58C13.12 10.65 11.25 11.33 9.47 11.53C9.76 11.78 10.01 12.26 10.01 13.01C10.01 14.08 10 14.94 10 15.21C10 15.42 10.15 15.67 10.55 15.59C13.71 14.53 16 11.53 16 8C16 3.58 12.42 0 8 0Z" />
+    </svg>
+  );
+}

site/content/blog/example-ckan-2021.md

@@ -1,14 +1,20 @@
-Live DEMOs:
+---
+title: "PortalJS example 1: Create a full-featured custom data portal frontend for CKAN with PortalJS"
+authors: ['Luccas Mateus']
+date: 2021-04-20
+---
 
-- https://catalog-portal-js.vercel.app
+We have created a full data portal demo using PortalJS all backed by a CKAN instance storing data and metadata, you can see below a screenshot of the homepage and of an individual dataset page.
-- https://ckan-enterprise-frontend.vercel.app/
+
+![](https://i.imgur.com/ai0VLS4.png)
+![](https://i.imgur.com/3RhXOW4.png)
 
 ## Create a Portal app for CKAN
 
 To create a Portal app, run the following command in your terminal:
 
 ```console
-npx create-next-app -e https://github.com/datopian/portal.js/tree/main/examples/ckan
+npx create-next-app -e https://github.com/datopian/portaljs/tree/main/examples/ckan
 ```
 
 > NB: Under the hood, this uses the tool called create-next-app, which bootstraps an app for you based on our CKAN example.
@@ -69,7 +75,7 @@ For development/debugging purposes, we suggest installing the Chrome extension -
 
 ### I18n configuration
 
-Portal.js is configured by default to support both `English` and `French` subpath for language translation. But for subsequent users, this following steps can be used to configure i18n for other languages;
+PortalJS is configured by default to support both `English` and `French` subpath for language translation. But for subsequent users, this following steps can be used to configure i18n for other languages;
 
 1.  Update `next.config.js`, to add more languages to the i18n locales
 

site/content/config.js

@@ -1,8 +1,8 @@
 const config = {
   title:
-    "Portal.JS",
+    "PortalJS",
   description:
-    "Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. portal.js can be used to present a single dataset or build a full-scale data catalog/portal.",
+    "PortalJS is a framework for rapidly building rich data portal frontends using a modern frontend approach. PortalJS can be used to present a single dataset or build a full-scale data catalog/portal.",
   theme: {
     default: "dark",
     toggleIcon: "/images/theme-button.svg",
@@ -12,34 +12,33 @@ const config = {
   authorUrl: "https://datopian.com/",
   navbarTitle: {
     // logo: "/images/logo.svg",
-    text: "🌀 Portal.JS",
+    text: "🌀 PortalJS",
     // version: "Alpha",
   },
   navLinks: [
     { name: "Docs", href: "/docs" },
-    { name: "Components", href: "/docs/components" },
+    // { name: "Components", href: "/docs/components" },
-    { name: "Learn", href: "/learn" },
+    { name: "Blog", href: "/blog" },
-    { name: "Gallery", href: "/gallery" },
+    // { name: "Gallery", href: "/gallery" },
-    { name: "Data Literate", href: "/data-literate" },
+    // { name: "Data Literate", href: "/data-literate" },
-    { name: "DL Demo", href: "/data-literate/demo" },
+    // { name: "DL Demo", href: "/data-literate/demo" },
-    { name: "Excel Viewer", href: "/excel-viewer" },
+    // { name: "Excel Viewer", href: "/excel-viewer" },
-    { name: "GitHub", href: "https://github.com/datopian/portal.js" },
   ],
   footerLinks: [],
   nextSeo: {
     openGraph: {
       type: "website",
       title:
-        "Portal.JS - Rapidly build rich data portals using a modern frontend framework",
+        "PortalJS - rapidly build rich data portals using a modern frontend framework.",
       description:
-        "Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. portal.js can be used to present a single dataset or build a full-scale data catalog/portal.",
+        "PortalJS is a framework for rapidly building rich data portal frontends using a modern frontend approach. PortalJS can be used to present a single dataset or build a full-scale data catalog and portal.",
       locale: "en_US",
       images: [
         {
-          url: "https://datahub.io/static/img/opendata/product.png",  //  TODO
+          url: "/homepage-screenshot.png",  //  TODO
-          alt: "Portal.JS - Rapidly build rich data portals using a modern frontend framework",
+          alt: "PortalJS - rapidly build rich data portals using a modern frontend framework.",
-          width: 1200,
+          width: 1280,
-          height: 627,
+          height: 720,
           type: "image/jpg",
         },
       ],
@@ -51,7 +50,7 @@ const config = {
     },
   },
   github: "https://github.com/datopian/portaljs",
-  discord: "https://discord.gg/An7Bu5x8",
+  discord: "https://discord.gg/EeyfGrGu4U",
   tableOfContents: true,
   // analytics: "xxxxxx",
   // editLinkShow: true,

site/content/data-literate/demo.mdx

@@ -13,7 +13,7 @@ You can see the raw source of this page here: https://raw.githubusercontent.com/
 We can have github-flavored markdown including markdown tables, auto-linked links and checklists:
 
 ```
-https://github.com/datopian/portal.js
+https://github.com/datopian/portaljs
 
 | a | b |
 |---|---|
@@ -23,7 +23,7 @@ https://github.com/datopian/portal.js
 * [ ] a second thing to do
 ```
 
-https://github.com/datopian/portal.js
+https://github.com/datopian/portaljs
 
 | a | b |
 |---|---|

site/content/docs.md

@@ -1,132 +0,0 @@
-# 🌀 Portal.JS: The JavaScript framework for data portals
-
-🌀 `portal.js` is a framework for rapidly building rich data portal frontends using a modern frontend approach. `portal.js` can be used to present a single dataset or build a full-scale data catalog/portal.
-
-`portal.js` is built in Javascript and React on top of the popular [Next.js](https://nextjs.com/) framework. `portal` assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
-
-## Features
-
-- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. wordpress) with a common internal API.
-- 👩‍💻 Developer friendly: built with familiar frontend tech Javascript, React etc
-- 🔋 Batteries included: Full set of portal components out of the box e.g. catalog search, dataset showcase, blog etc.
-- 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
-- 🧱 Extensible: quickly extend and develop/import your own React components
-- 📝 Well documented: full set of documentation plus the documentation of NextJS and Apollo.
-
-### For developers
-
-- 🏗 Build with modern, familiar frontend tech such as Javascript and React.
-- 🚀 NextJS framework: so everything in NextJS for free React, SSR, static site generation, huge number of examples and integrations etc.
-  - SSR => unlimited number of pages, SEO etc whilst still using React.
-  - Static Site Generation (SSG) (good for small sites) => ultra-simple deployment, great performance and lighthouse scores etc
-
-## Installation and setup
-
-Before installation, ensure your system satisfies the following requirements:
-
-- Node.js 10.13 or later
-- Nextjs 10.0.3
-- MacOS, Windows (including WSL), and Linux are supported
-
-> Note: We also recommend instead of npm using `yarn` instead of `npm`.
-> 
-Portal.js is built with React on top of Nextjs framework, so for a quick setup, you can bootstrap a  Nextjs app and install portal.js as demonstrated in the code below:
-
-```bash=
-## Create a react app
-npx create-next-app
-# or
-yarn create next-app
-```
-After the installation is complete, follow the instructions to start the development server. Try editing pages/index.js and see the result on your browser.
-
-> For more information on how to use create-next-app, you can review the [create-next-app](https://nextjs.org/docs/api-reference/create-next-app) documentation.
-
-Once you have Nextjs created, you can install portal.js:
-
-```bash=
-yarn add https://github.com/datopian/portal.js.git
-```
-
-You're now ready to use portal.js in your next app. To test portal.js, open your `index.js` file in the pages folder. By default you should have some autogenerated code in the `index.js` file:
-
-
-Which outputs a page with the following content:
-
-![](https://i.imgur.com/GVh0P6p.png)
-
-Now, we are going to do some clean up and add a table component. In the `index.js` file, import a [Table]() component from portal as shown below:
-
-```javascript
-import Head from 'next/head'
-import { Table } from 'portal' //import Table component
-import styles from '../styles/Home.module.css'
-
-export default function Home() {
-
-  const columns = [
-    { field: 'id', headerName: 'ID' },
-    { field: 'firstName', headerName: 'First name' },
-    { field: 'lastName', headerName: 'Last name' },
-    { field: 'age', headerName: 'Age' }
-  ];
-
-  const rows = [
-    { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
-    { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
-    { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
-    { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
-    { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
-    { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
-    { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
-  ];
-
-  return (
-    <div className={styles.container}>
-      <Head>
-        <title>Create Portal App</title>
-        <link rel="icon" href="/favicon.ico" />
-      </Head>
-
-      <h1 className={styles.title}>
-        Welcome to <a href="https://nextjs.org">Portal.JS</a>
-      </h1>
-
-      {/* Use table component */}
-      <Table data={rows} columns={columns} />
-
-    </div>
-  )
-}
-```
-
-Now, your page should look like the following:
-
-![](https://i.imgur.com/n0vSjY4.png)
-
-> **Note**: You can learn more about individual portal components, as well as their prop types in the [components reference](/docs/components).
-
-
-## Next Steps
-
-You can check out the following examples built with Portal.js.
-
-* [A portal for a single Frictionless dataset](/learn/ckan)
-* [A portal with a CKAN backend](/learn/single-frictionless-dataset)
-
-> The [`examples` directory](https://github.com/datopian/portal.js/tree/main/examples) is regularly updated with different portal examples. 
-
-You can also look at the full list of the available components that are provided by Portal.JS in [Components](/docs/components).
-
-
-## Reference Information
-
-* [Full list of the available components that are provided by Portal.JS](/docs/components)
-* [Reference](/docs/references)
-
-
-## Getting Help
-
-If you have questions about anything related to Portal.js, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions).
-
-

site/content/docs/components.md

@@ -1,589 +0,0 @@
-# Components Reference
-
-Portal.js supports many components that can help you build amazing data portals similar to [this](https://catalog-portal-js.vercel.app/) and [this](https://portal-js.vercel.app/). 
-
-In this section, we'll cover all supported components in depth, and help you understand their use as well as the expected properties. 
-
-Components are grouped under the following sections:
-* [UI](https://github.com/datopian/portal.js/tree/main/src/components/ui): Components like Nav bar, Footer, e.t.c 
-* [Dataset](https://github.com/datopian/portal.js/tree/main/src/components/dataset): Components used for displaying a Frictionless dataset and resources
-* [Search](https://github.com/datopian/portal.js/tree/main/src/components/search): Components used for building a search interface for datasets
-* [Blog](https://github.com/datopian/portal.js/tree/main/src/components/blog): Components for building a simple blog for datasets
-* [Views](https://github.com/datopian/portal.js/tree/main/src/components/views): Components like charts, tables, maps for generating data views
-* [Misc](https://github.com/datopian/portal.js/tree/main/src/components/misc): Miscellaneos components like errors, custom links, etc used for extra design. 
-
-### UI Components
-
-In the UI we group all components that can be used for building generic page sections. These are components for building sections like the Navigation bar, Footer, Side pane, Recent datasets, e.t.c.  
-
-#### [Nav Component](https://github.com/datopian/portal.js/blob/main/src/components/ui/Nav.js)
-
-To build a navigation bar, you can use the `Nav` component as demonstrated below:
-
-```javascript
-import { Nav } from 'portal'
-
-export default function Home(){
-
- const navMenu = [{ title: 'Blog', path: '/blog' },
-                 { title: 'Search', path: '/search' }]
- 
- return (
-   <>
-    <Nav logo="/images/logo.png" navMenu={navMenu}/>
-    ...
-   </>
- 
- )
-}
-```
-
-#### Nav Component Prop Types
-
-Nav component accepts two properties:
-* **logo**: A string to an image path. Can be relative or absolute.
-* **navMenu**: An array of objects with title and path. E.g : {"[{ title: 'Blog', path: '/blog' },{ title: 'Search', path: '/search' }]"}
-
-
-#### [Recent Component](https://github.com/datopian/portal.js/blob/main/src/components/ui/Recent.js)
-
-The `Recent` component is used to display a list of recent [datasets](#Dataset) in the home page. This useful if you want to display the most recent dataset users have interacted with in your home page. 
-To build a recent dataset section, you can use the `Recent` component as demonstrated below:
-
-```javascript
-import { Recent } from 'portal'
-
-export default function Home() {
-
-  const datasets = [
-    {
-      organization: {
-        name: "Org1",
-        title: "This is the first org",
-        description: "A description of the organization 1"
-      },
-      title: "Data package title",
-      name: "dataset1",
-      description: "description of data package",
-      resources: [],
-    },
-    {
-      organization: {
-        name: "Org2",
-        title: "This is the second org",
-        description: "A description of the organization 2"
-      },
-      title: "Data package title",
-      name: "dataset2",
-      description: "description of data package",
-      resources: [],
-    },
-  ]
-
-  return (
-    <div>
-      {/* Use Recent component */}
-      <Recent datasets={datasets} />
-    </div>
-  )
-}
-
-```
-
-Note: The `Recent` component is hyperlinked with the dataset name of the organization and the dataset name in the following format: 
-
-> `/@<org name>/<dataset name>`
-
-For instance, using the example dataset above, the first component will be link to page:
-
-> `/@org1/dataset1`
-
-and the second will be linked to:
-
-> `/@org2/dataset2` 
-
-This is useful to know when generating dynamic pages for each dataset. 
-
-#### Recent Component Prop Types
-
-The `Recent` component accepts the following properties:
-* **datasets**: An array of [datasets](#Dataset)
-
-### Dataset Components
-
-The dataset component groups together components that can be used for building a dataset UI. These includes components for displaying info about a dataset, resources in a dataset as well as dataset ReadMe.
-
-#### [KeyInfo Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/KeyInfo.js)
-
-The `KeyInfo` components displays key properties like the number of resources, size, format, licences of in a dataset in tabular form. See example in the `Key Info` section [here](https://portal-js.vercel.app/). To use it, you can import the `KeyInfo` component as demonstrated below:
-```javascript
-import { KeyInfo } from 'portal'
-
-export default function Home() {
-
-  const datapackage = {
-    "name": "finance-vix",
-    "title": "VIX - CBOE Volatility Index",
-    "homepage": "http://www.cboe.com/micro/VIX/",
-    "version": "0.1.0",
-    "license": "PDDL-1.0",
-    "sources": [
-      {
-        "title": "CBOE VIX Page",
-        "name": "CBOE VIX Page",
-        "web": "http://www.cboe.com/micro/vix/historical.aspx"
-      }
-    ],
-    "resources": [
-      {
-        "name": "vix-daily",
-        "path": "vix-daily.csv",
-        "format": "csv",
-        "size": 20982,
-        "mediatype": "text/csv",
-      }
-    ]
-  }
-
-  return (
-    <div>
-      {/* Use KeyInfo component */}
-      <KeyInfo descriptor={datapackage} resources={datapackage.resources} />
-    </div>
-  )
-}
-
-```
-
-#### KeyInfo Component Prop Types
-
-KeyInfo component accepts two properties:
-* **descriptor**: A [Frictionless data package descriptor](https://specs.frictionlessdata.io/data-package/#descriptor)
-* **resources**: An [Frictionless data package resource](https://specs.frictionlessdata.io/data-resource/#introduction)
-
-
-#### [ResourceInfo Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/ResourceInfo.js)
-
-The `ResourceInfo` components displays key properties like the name, size, format, modification dates, as well as a download link in a resource object. See an example of a `ResourceInfo` component in the `Data Files` section [here](https://portal-js.vercel.app/).
-
-You can import and use the`ResourceInfo` component as demonstrated below:
-```javascript
-import { ResourceInfo } from 'portal'
-
-export default function Home() {
-
-  const resources = [
-    {
-      "name": "vix-daily",
-      "path": "vix-daily.csv",
-      "format": "csv",
-      "size": 20982,
-      "mediatype": "text/csv",
-    },
-    {
-      "name": "vix-daily 2",
-      "path": "vix-daily2.csv",
-      "format": "csv",
-      "size": 2082,
-      "mediatype": "text/csv",
-    }
-  ]
-
-  return (
-    <div>
-      {/* Use Recent component */}
-      <ResourceInfo resources={resources} />
-    </div>
-  )
-}
-```
-
-#### ResourceInfo Component Prop Types
-
-ResourceInfo component accepts a single property:
-* **resources**: An [Frictionless data package resource](https://specs.frictionlessdata.io/data-resource/#introduction)
-
-
-#### [ReadMe Component](https://github.com/datopian/portal.js/blob/main/src/components/dataset/Readme.js)
-
-The `ReadMe` component is used for displaying a compiled dataset Readme in a readable format. See example in the `README` section [here](https://portal-js.vercel.app/).
-
-> Note: By compiled ReadMe, we mean ReadMe that has been converted to plain string using a package like [remark](https://www.npmjs.com/package/remark).
-
-You can import and use the`ReadMe` component as demonstrated below:
-```javascript
-import { ReadMe } from 'portal'
-import remark from 'remark'
-import html from 'remark-html'
-import { useEffect, useState } from 'react'
-
-
-const readMeMarkdown = `
-    CBOE Volatility Index (VIX) time-series dataset including daily open, close,
-    high and low. The CBOE Volatility Index (VIX) is a key measure of market
-    expectations of near-term volatility conveyed by S&P 500 stock index option
-    prices introduced in 1993.
-
-    ## Data
-
-    From the [VIX FAQ][faq]:
-
-    > In 1993, the Chicago Board Options Exchange® (CBOE®) introduced the CBOE
-    > Volatility Index®, VIX®, and it quickly became the benchmark for stock market
-    > volatility. It is widely followed and has been cited in hundreds of news
-    > articles in the Wall Street Journal, Barron's and other leading financial
-    > publications. Since volatility often signifies financial turmoil, VIX is
-    > often referred to as the "investor fear gauge".
-
-    [faq]: http://www.cboe.com/micro/vix/faq.aspx
-
-    ## License
-
-    No obvious statement on [historical data page][historical]. Given size and
-    factual nature of the data and its source from a US company would imagine this
-    was public domain and as such have licensed the Data Package under the Public
-    Domain Dedication and License (PDDL).
-
-    [historical]: http://www.cboe.com/micro/vix/historical.aspx
-`
-
-export default function Home() {
-  const [readMe, setreadMe] = useState("")
-
-  useEffect(() => {
-    async function processReadMe() {
-      const processed = await remark()
-        .use(html)
-        .process(readMeMarkdown)
-      setreadMe(processed.toString())
-    }
-    processReadMe()
-  }, [])
-
-  return (
-    <div>
-      <ReadMe readme={readMe} />
-    </div>
-  )
-}
-
-```
-
-#### ReadMe Component Prop Types
-
-The `ReadMe` component accepts a single property:
-* **readme**: A string of a compiled ReadMe in html format. 
-
-### [View Components](https://github.com/datopian/portal.js/tree/main/src/components/views)
-
-View components is a set of components that can be used for displaying dataset views like charts, tables, maps, e.t.c. 
-
-#### [Chart Component](https://github.com/datopian/portal.js/blob/main/src/components/views/Chart.js)
-
-The `Chart` components exposes different chart components like Plotly Chart, Vega charts, which can be used for showing graphs. See example in the `Graph` section [here](https://portal-js.vercel.app/). 
-To use a chart component, you need to compile and pass a view spec as props to the chart component. 
-Each Chart type have their specific spec, as explained in this [doc](https://specs.frictionlessdata.io/views/#graph-spec).
-
-In the example below, we assume there's a compiled Plotly spec:
-
-```javascript
-import { PlotlyChart } from 'portal'
-
-export default function Home({plotlySpec}) {
-
-  return (
-    < div >
-      <PlotlyChart spec={plotlySpec} />
-    </div>
-  )
-}
-
-```
-> Note: You can compile views using the [datapackage-render](https://github.com/datopian/datapackage-views-js) library, as demonstrated in [this example](https://github.com/datopian/portal.js/blob/main/examples/dataset-frictionless/lib/utils.js).
-
-
-#### Chart Component Prop Types
-
-KeyInfo component accepts two properties:
-* **spec**: A compiled view spec depending on the chart type. 
-
-#### [Table Component](https://github.com/datopian/portal.js/blob/main/examples/dataset-frictionless/components/Table.js)
-
-The `Table` component is used for displaying dataset resources as a tabular grid. See example in the `Data Preview` section [here](https://portal-js.vercel.app/). 
-To use a Table component, you have to pass an array of data and columns as demonstrated below:
-
-```javascript
-import { Table } from 'portal' //import Table component
-
-export default function Home() {
-
-  const columns = [
-    { field: 'id', headerName: 'ID' },
-    { field: 'firstName', headerName: 'First name' },
-    { field: 'lastName', headerName: 'Last name' },
-    { field: 'age', headerName: 'Age' }
-  ];
-
-  const data = [
-    { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
-    { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
-    { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
-    { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
-    { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
-    { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
-    { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
-  ];
-
-  return (
-    <Table data={data} columns={columns} />
-  )
-}
-
-```
-> Note: Under the hood, Table component uses the [DataGrid Material UI table](https://material-ui.com/components/data-grid/), and as such all supported params in data and columns are supported. 
-
-
-#### Table Component Prop Types
-
-Table component accepts two properties:
-* **data**: An array of column names with properties: e.g {'[{field: "col1", headerName: "col1"}, {field: "col2", headerName: "col2"}]'}
-* **columns**: An array of data objects e.g. {'[ {col1: 1, col2: 2}, {col1: 5, col2: 7} ]'}
-
-
-### [Search Components](https://github.com/datopian/portal.js/tree/main/src/components/search)
-
-Search components groups together components that can be used for creating a search interface. This includes search forms, search item as well as search result list. 
-
-#### [Form Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Form.js)
-
-The search`Form` component is a simple search input and submit button. See example of a search form [here](https://catalog-portal-js.vercel.app/search). 
-
-The search `form` requires a submit handler (`handleSubmit`). This handler function receives the search term, and handles actual search. 
-
-In the example below, we demonstrate how to use the `Form` component.
-
-```javascript
-import { Form } from 'portal'
-
-export default function Home() {
-
-  const handleSearchSubmit = (searchQuery) => {
-    // Write your custom code to perform search in db
-    console.log(searchQuery);
-  }
-
-  return (
-    <Form
-      handleSubmit={handleSearchSubmit} />
-  )
-}
-```
-
-#### Form Component Prop Types
-
-The `Form` component accepts a single property:
-* **handleSubmit**: A function that receives the search text, and can be customize to perform the actual search. 
-
-#### [Item Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Item.js)
-
-The search`Item` component can be used to display a single search result. 
-
-In the example below, we demonstrate how to use the `Item` component.
-
-```javascript
-import { Item } from 'portal'
-
-export default function Home() {
-  const datapackage = {
-    "name": "finance-vix",
-    "title": "VIX - CBOE Volatility Index",
-    "homepage": "http://www.cboe.com/micro/VIX/",
-    "version": "0.1.0",
-    "description": "This is a test organization description",
-    "resources": [
-      {
-        "name": "vix-daily",
-        "path": "vix-daily.csv",
-        "format": "csv",
-        "size": 20982,
-        "mediatype": "text/csv",
-      }
-    ]
-  }
-
-  return (
-    <Item dataset={datapackage} />
-  )
-}
-```
-
-#### Item Component Prop Types
-
-The `Item` component accepts a single property:
-* **dataset**: A [Frictionless data package descriptor](https://specs.frictionlessdata.io/data-package/#descriptor)
-
-
-#### [ItemTotal Component](https://github.com/datopian/portal.js/blob/main/src/components/search/Item.js)
-
-The search`ItemTotal` is a simple component for displaying the total search result 
-
-In the example below, we demonstrate how to use the `ItemTotal` component.
-
-```javascript
-import { ItemTotal } from 'portal'
-
-export default function Home() {
-  //do some custom search to get results
-  const search = (text) => {
-    return [{ name: "data1" }, { name: "data2" }]
-  }
-  //get the total result count
-  const searchTotal = search("some text").length
-
-  return (
-    <ItemTotal count={searchTotal} />
-  )
-}
-```
-
-#### ItemTotal Component Prop Types
-
-The `ItemTotal` component accepts a single property:
-* **count**: An integer of the total number of results. 
-
-
-### [Blog Components](https://github.com/datopian/portal.js/tree/main/src/components/blog)
-
-These are group of components for building a portal blog. See example of portal blog [here](https://catalog-portal-js.vercel.app/blog)
-
-#### [PostList Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-The `PostList` component is used to display a list of blog posts with the title and a short excerpts from the content.
-
-In the example below, we demonstrate how to use the `PostList` component.
-
-```javascript
-import { PostList } from 'portal'
-
-export default function Home() {
-
-  const posts = [
-    { title: "Blog post 1", excerpt: "This is the first blog excerpts in this list." },
-    { title: "Blog post 2", excerpt: "This is the second blog excerpts in this list." },
-    { title: "Blog post 3", excerpt: "This is the third blog excerpts in this list." },
-  ]
-  return (
-    <PostList posts={posts} />
-  )
-}
-```
-
-#### PostList Component Prop Types
-
-The `PostList` component accepts a single property:
-* **posts**: An array of post list objects with the following properties:
-    ```javascript
-    [
-        { 
-          title: "The title of the blog post",
-          excerpt: "A short excerpt from the post content",
-        },
-    ]
-    ```
-
-#### [Post Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-The `Post` component is used to display a blog post. See an example of a blog post [here](https://catalog-portal-js.vercel.app/blog/nyt-pa-platformen-opdateringsfrekvens-og-andres-data)
-
-In the example below, we demonstrate how to use the `Post` component.
-
-```javascript
-import { Post } from 'portal'
-import * as dayjs from 'dayjs' //For converting UTC time to relative format
-import relativeTime from 'dayjs/plugin/relativeTime'
-
-dayjs.extend(relativeTime)
-
-export default function Home() {
-
-  const post = {
-    title: "This is a sample blog post",
-    content: `<h1>A simple header</h1>
-    The PostList component is used to display a list of blog posts
-              with the title and a short excerpts from the content.
-              In the example below, we demonstrate how to use the PostList component.`,
-    createdAt: dayjs().to(dayjs(1620649596902)),
-    featuredImage: "https://pixabay.com/get/ge9a766d1f7b5fe0eccbf0f439501a2cf2b191997290e7ab15e6a402574acc2fdba48a82d278dca3547030e0202b7906d_640.jpg"
-  }
-
-  return (
-    <Post post={post} />
-  )
-}
-```
-
-#### Post Component Prop Types
-
-The `Post` component accepts a single property:
-* **post**: An object with the following properties:
-```javascript
-  {
-    title: <The title of the blog post>
-    content: <The body of the blog post. Can be plain text or html>
-    createdAt: <The utc date when the post was last modified>
-    featuredImage: < Url/relative url to post cover image>
-  }
-```
-
-
-### [Misc Components](https://github.com/datopian/portal.js/tree/main/src/components/misc)
-
-These are group of miscellaneous/extra components for extending your portal. They include components like Errors, custom links, etc.
-
-#### [Error Component](https://github.com/datopian/portal.js/blob/main/src/components/misc/Error.js)
-
-The `Error` component is used to display a custom error message.
-
-In the example below, we demonstrate how to use the `Error` component.
-
-```javascript
-import { Error } from 'portal'
-
-export default function Home() {
-
-  return (
-    <Error message="An error occured when loading the file!" />
-  )
-}
-```
-
-#### Error Component Prop Types
-
-The `Error` component accepts a single property:
-* **message**: A string with the error message to display. 
-    
-    
-#### [Custom Component](https://github.com/datopian/portal.js/blob/main/src/components/misc/Error.js)
-
-The `CustomLink` component is used to create a link with a consistent style to other portal components. 
-
-In the example below, we demonstrate how to use the `CustomLink` component.
-
-```javascript
-import { CustomLink } from 'portal'
-
-export default function Home() {
-
-  return (
-    <CustomLink url="/blog" title="Goto Blog" />
-  )
-}
-```
-
-#### CustomLink Component Prop Types
-
-The `CustomLink` component accepts the following properties:
-
-* **url**: A string. The relative or absolute url of the link. 
-* **title**: A string. The title of the link
-

site/content/docs/dms/authentication.md

@@ -0,0 +1,249 @@
+# Authentication
+
+## Introduction
+
+The core function of authentication is to **Identify** Users of the Portal (in a federated way) so we can base access on their identity.
+
+There are 3 major conceptual components: Identity, Accounts and Sessions which come together in the following stages:
+
+* **Root Identity Determination:** Determine Identity often via Delegation
+* **Sessions:** Persistence of the identity in the web application in a secure way (without new identity determination on each request! I don't want to have to login via third party service every time)
+* **Account (aka profile):** Storing Related Account/Profile Information in our application (not in third party identity) eg. email, name (other preferences)
+  * This will get auto-created usually at first Identification
+  * In limited case this can be seen as a cache of info from Identity system (e.g. your email)
+  * However often richer info that is app specific that is generated (relevant for personalization)
+
+### Root Identity Determination options :key:
+
+The identity determination can be done in multiple ways. In this article we're considering following 3 options that we believe are widely used:
+
+- Password authentication - traditional username and password pair
+- Single Sign-on (SSO) via protocols such as OAuth, SAML, OpenID Connect
+- One-time password (OTP) via email or SMS (aka passwordless connection)
+
+#### Password authentication
+
+Traditional way of authentication of users. When signing up user provides at least username and password pair which is then stored in a database for future authentication processes. Normally, additional information such as email address, full name etc. is also requested when registering.
+
+Examples of password authentication in popular services:
+
+- GitHub - https://github.com/join
+- GitLab - https://gitlab.com/users/sign_up
+- NPM - https://www.npmjs.com/signup
+
+#### Single Sign-on (SSO)
+
+The way of delegating identity determination process to some third-party service. Normally, popular social network services are used, e.g., Google, Facebook, Twitter etc. SSO implementations can be done using OAuth or SAML protocols. In addition, there is OpenID Connect protocol which is an extension of OAuth2.0.
+
+- OAuth
+  - JWT based
+  - JSON based
+  - 'webby'
+- SAML
+  - XML based
+  - SOAP based
+  - 'enterprisey'
+
+List of OAuth providers:
+
+https://en.wikipedia.org/wiki/List_of_OAuth_providers
+
+Examples of SSO in popular projects:
+
+- https://datahub.io/login
+- https://vercel.com/signup
+
+#### One-time password (OTP)
+
+Also known as dynamic password, OTP also solves limitations of traditional password authentication method. Usually, the one time passwords are received via email or SMS.
+
+### Account (aka profile) 
+
+- Storage of user profile information (email, fullname, gravatar etc.)
+- Retrieving user profile information via API
+- Updating profile
+- Deleting profile
+
+### Sessions
+
+- Log out: DePersisting the Session
+- Invalidating all Sessions: e.g. if a security issue
+- Sessions outside of browsers
+
+## Key Job Stories
+
+When a user signs in, I want to know her/his identity so that I can limit access and editing based on who she/he is.
+
+When a user visits the data portal for the first time, I want to provide him/her a way to register easily/quickly so that more people uses the data portal.
+
+When I visit the data portal for the first time, I want to sign up using my existing social network account so that I don't need to remember yet another credentials.
+
+When I'm using the CLI app (or anything else outside browser), I want to be able to login so that I can work from the terminal (e.g., have write access: editing datasets etc.).
+
+[More job stories](#more-job-stories).
+
+## CKAN 2 (CKAN Classic)
+
+### Basic CKAN authentication
+
+In classic system, we have basic CKAN authentication. Below is how registration page looks like:
+
+![CKAN Classic register page](/static/img/docs/dms/ckan-register.png)
+
+Registration flow in CKAN Classic:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: fill in the form and submit
+  ckan->>ckan: check access (if user can create user)
+  ckan->>ckan: parse params
+  ckan->>ckan: check recaptcha
+  ckan->>ckan: call 'user_create' action
+  ckan->>ckan.model: add a new user into db
+  ckan->>ckan: create an activity
+  ckan->>ckan: log the user
+  ckan->>user: redirect to dashboard
+```
+
+We can extend basic CKAN authentication with:
+
+- LDAP
+  - https://extensions.ckan.org/extension/ldap/
+  - https://github.com/NaturalHistoryMuseum/ckanext-ldap
+- OAuth - see below
+- SAML - https://extensions.ckan.org/extension/saml2/
+
+### CKAN Classic as OAuth client
+
+CKAN Classic can also be used as OAuth client:
+
+- https://github.com/conwetlab/ckanext-oauth2 - this is the only one that's maintained.
+- https://github.com/etalab/ckanext-oauth2 - outdated, the one above is based on this.
+- https://github.com/okfn/ckanext-oauth - last commit 9 years ago.
+- https://github.com/ckan/ckanext-oauth2waad - Windows Azure Active Directory specific and outdated.
+
+How it works:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: request for login via OAuth provider
+  ckan->>ckan.oauth: raise 401 and call `challenge` function
+  ckan.oauth->>user: redirect the user to the 3rd party log in page
+  user->>3rdparty: perform login
+  3rdparty->>ckan.oauth: redirect to /oauth2/callback with token
+  ckan.oauth->>3rdparty: call `authenticate` with token
+  3rdparty->>ckan.oauth: return user info
+  ckan.oauth->>ckan: if doesn't exist save that info in db or update it
+  ckan.oauth->>ckan.oauth: add cookies
+  ckan.oauth->>user: redirect to dashboard
+```
+
+## CKAN 3 (Next Gen)
+
+We have considered some of popular and/or modern solutions for identity management that we can implement in CKAN 3:
+
+https://docs.google.com/spreadsheets/d/1qXZyzAbA2NtpnoSZRJ2K_EbaWJnvxkrKVzQ_2rD5eQw/edit#gid=0
+
+Shortlist based on scores from the spreadsheet above:
+
+- Auth0
+- AuthN
+- Ory/Kratos
+
+Recommendation:
+
+All projects from the shortlist can be considered for a project. It worth to give a try for each of them and find out what works best for your project's needs. Testing out Auth0 should be straightforward and take less than an hour. AuthN and Ory/Kratos would require to build docker images and to run it locally but overall it should not be time consuming.
+
+### Existing work
+
+In datahub.io we have implemented SSO via Google/Github. Below is sequence diagram showing the auth flow with datopian/auth + frontend express app (similar to CKAN 3 frontend):
+
+```mermaid
+sequenceDiagram
+
+  frontend.login->>auth.authenticate: authenticate(jwt=None,next=/success/...)
+  auth.authenticate->>frontend.login: failed + here are urls for logging on 3rd party including success
+  frontend.login->>user: login form with login urls to 3rd party including next url in state
+  user->>3rdparty: login
+  3rdparty->>auth.oauth_response: success
+  auth.oauth_response->>frontend.success: redirect to next url
+  frontend.success->>auth.authenticate: with valid jwt
+  auth.authenticate->>frontend.success: valid + here is profile
+  frontend.success->>frontend.success: decode jwt, check it, then see localstorage
+  frontend.success->>frontend.dashboard: redirect to dashboard
+```
+
+## CKAN 2 to CKAN 3 (aka Next Gen)
+
+How does this conceptual framework map to an evolution of CKAN 2 to CKAN 3?
+
+```mermaid
+graph TD
+
+subgraph "CKAN Classic"
+  Signup["Classic signup, e.g., self-service or by sysadmin"]
+  Login["Classic login if you're using the classic UI"]
+  OAuth["OAuth2(ORY/Hydra)"]
+end
+
+subgraph "Authentication service (ORY/Kratos)"
+  SSO["Social Sign-On: Github, Google, Facebook"]
+  CC["CKAN Classic"]
+  Admins["Sysadmin users"]
+  Curators["Data curators"]
+  Users["Regular users"]
+end
+
+subgraph "Frontend v3"
+  SignupFront["Signup via Kratos"]
+  LoginFront["Login via Kratos"]
+end
+
+SignupFront --"Regular user"--> SSO
+LoginFront --"Regular user"--> SSO
+
+LoginFront --"Data curator"--> CC
+
+CC --> Admins
+CC --> Curators
+SSO --> Users
+
+CC --"Redirect"--> OAuth
+OAuth --> Login
+```
+
+Sequence diagram of login process:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)
+
+From ORY/Kratos:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)
+
+
+Kratos to Hydra in CKAN Classic:
+
+WIP
+
+Questions
+
+* Does CKAN Classic allow us to store arbitrary account information (are there "extras")
+* How would we avoid having to support identity persistence, delegation etc in both NG frontend and Classic Admin UI?
+  * Can we share cookies (e.g. via using subdomains)
+* How is login, identity determination etc done at least for frontend in DataHub.io
+* Should account UI really be in NG frontend vs Classic Admin UI?
+* how can we handle "invite a user" to my org set up ... (it's basically post processing after sign up ...)
+
+## Appendix
+
+### More job stories
+
+When a user visits the data portal, I want to provide multiple options for him/her to sign up so that I have more users registered and using the data portal.
+
+When a user needs to change his/her profile info, I want to make sure it is possible, so that I have the up-to-date information about users.
+
+When my personal info (email etc.) is changed, I want to edit it in my profile so that I provide up-to-date information about me and I receive messages (eg, notifications) properly.
+
+When I decide to stop using the data portal, I want to be able to delete my account, so that my personal details aren't stored in the service that I don't need anymore.

site/content/docs/dms/blob-storage.md

@@ -0,0 +1,215 @@
+# Blob Storage
+
+## Introduction
+
+DMS and data portals often need to *store* data as well as metadata. As such, they require a system for doing this. This page focuses on Blob Storage aka Bulk or Raw storage (see [storage](/docs/dms/storage) page for an overview of all types of storage).
+
+Blob storage is for storing "blobs" of data, that is a raw stream of bytes like files on a filesystem. For blob storage think local filesystem or cloud storage like S3, GCS, etc.
+
+Blob Storage in a DMS can be provided via:
+
+* Local file system: storing on disk or storage directly connected to the instance
+* Cloud storage like S3, Google Cloud Storage, Azure storage etc
+
+Today, cloud storage would be the default in most cases.
+
+### Features
+
+* Storage: Persistent, cost-efficient storage
+* Download: Fast, reliable download (possibly even with support for edge distribution)
+* Upload: reliable and rapid upload 
+  * Direct upload to (cloud) storage by clients i.e. without going via the DMS. Why? Because cloud storage has many features that it would be costly replicate (e.g. multipart, resumable etc), excellent performance and reliability for upload. It also cuts out the middleman of the DMS backend thereby saving bandwidth, reducing load on the DMS backend and improving performance
+  * Upload UI: having an excellent UI for doing upload. NB: this UI is considered part of the [publish feature](/docs/dms/publish)
+* Cloud: integrate with cloud storage
+* Permissions: restricting access to data stored in blob storage based on the permissions of the DMS. For example, if Joe does not have access to a dataset on the DMS he should not be able to access associated blob data in the storage system
+
+## Flows
+
+### Direct to Cloud Upload
+
+Want: Direct upload to cloud storage ... But you need to authorize that ... So give them a token from your app
+
+A sequence diagram illustrating the process for a direct to cloud upload:
+
+```mermaid
+sequenceDiagram
+
+participant Browser as Client (Browser / Code)
+participant Authz as Authz Server
+participant BitStore as Storage Access Token Service
+participant Storage as Cloud Storage
+
+  Browser->>Authz: Give me a BitStore access token
+  Authz->>Browser: Token
+  Browser->>BitStore: Get a signed upload URL (access token, file metdata)
+  BitStore->>Browser: Signed URL
+  Browser->>Storage: Upload file (signed URL)
+  Storage->>Browser: OK (storage metadata)
+```
+
+Here's a more elaborate version showing storage of metadata into the MetaStore afterwards (and skipping the Authz service):
+
+```mermaid
+sequenceDiagram
+
+  participant browser as Client (Browser / Code)
+  participant vfts as MetaStore
+  participant bitstore as Storage Access Token Service
+  participant storage as Cloud Storage
+  
+  browser->>browser: Select files to upload
+  browser->>browser: calculate file hashes (if doing content addressable)
+  browser->bitstore: get signed URLs(file1.csv URL, file2.csv URL, auth info)
+  bitstore->>browser: signed URLs
+  browser->>storage: upload file1.csv
+  storage->>browser: OK
+  browser->>storage: upload file2.csv
+  storage->>browser: OK
+  browser->>browser: Compose datapackage.json
+  browser->>vfts: create dataset(datapackage.json, file1.csv pointer, file2.csv pointer, jwt token, ...)
+  vfts->>browser: OK
+```
+
+## CKAN 2 (Classic)
+
+Blob Storage is known as the FileStore in CKAN v2 and below. The default is local disk storage.
+
+There is support for cloud storage via a variety of extensions the most prominent of which is `ckanext-cloudstorage`: https://github.com/TkTech/ckanext-cloudstorage
+
+There are a variety of issues:
+
+* Cloud storage is not a first class citizen in CKAN: CKAN defaults to local file storage but cloud storage is the default in the world and has much better scalability, performance as well as integratability with cloud deployment
+* The FileStore interface definition has a poor separation of concerns (for example, blob storage file paths is set in the FileStore component not in core CKAN) which makes it hard / hacky to extend and use for key use cases e.g. versioning.
+* `ckanext-cloudstorage` (the default cloud storage extension) is ok but has many issues e.g.
+  * No direct to cloud upload: it uses CKAN backend as a middleman so all data must go via ckan backend
+  * Implements its own (sometimes unreliable) version of multipart upload (which means additional code which isn't as reliable as cloud storage providers interface)
+  * No access to advanced features such as resumability etc
+
+Generally, we at Datopian have seen a lot of issues around multipart / large file upload stability with clients and are still seeing issues when a lot of large files are uploaded via scripts. Fixing and refactoring code related to storage is very costly, and tends to result in client specific "hacks".
+
+## CKAN v3
+
+An approach to blob storage that leverages cloud blob storage directly (i.e. without having to upload and serve all files via the CKAN web server), unlocking the performance characteristics of the storage backend directly. It is designed with a microservice approach and supports direct to cloud uploads and downloads. The key components are listed in the next section. You can read more about the overall design approach in the [design section below](#Design).
+
+It is backwards compatible with CKAN v2 and has been successfully deployed with CKAN v2.8 and v2.9.
+
+**Status: Production.**
+
+### Components
+
+* [ckanext-blob-storage](https://github.com/datopian/ckanext-blob-storage) (formerly known as ckanext-external-storage)
+  * Hooking CKAN to Giftless replacing resource storage
+  * Depends on giftless-client and ckanext-authz-service
+  * Doesn't implement IUploader - completely overrides upload / download routes for resources
+* [Giftless](https://github.com/datopian/giftless) - Git LFS compatible implementation for storage with some extras on top. This hands out access tokens to store data in cloud storage.
+  * Docs at https://giftless.datopian.com
+  * Backends for Azure, Google Cloud Storage and local
+  * Multipart support (on top of standard LFS protocol)
+  * Accepts JWT tokens for authentication and authorization
+* [ckanext-authz-service](https://github.com/datopian/ckanext-authz-service/) - This extension uses CKAN’s built-in authentication and authorization capabilities to: a) Generate JWT tokens and provide them via CKAN’s Web API to clients and b) Validate JWT tokens.
+  * Allows hooking CKAN's authentication and authorization capabilities to generate signed JWT tokens, to integrate with external systems
+  * Not specific for Giftless, but this is what it was built for
+* [ckanext-asset-storage](https://github.com/datopian/ckanext-asset-storage) - this takes care of storing non-data assets e.g. organization images etc.
+  * CKAN IUploader for assets (not resources!)
+  * Pluggable backends - currently local and Azure
+  * Much cleaner than older implementations (ckanext-cloudstorage etc.)
+
+Clients:
+
+* [giftless-client-py](https://github.com/datopian/giftless-client) - Python client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools
+* [giftless-client-js](https://github.com/datopian/giftless-client-js) - Javascript client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools for creating uploaders in the UI
+
+## Design
+
+### Purpose
+
+The goal of this project is to create a more **_flexible_** system for storing **_data files_** (AKA “resources”) for **_CKAN_ and _other implementations_** of a data portal so that CKAN can support versioning, large file upload (and great file upload UX), plug easily into cloud and local file storage backends and, in general, is easy to customize both for storage layer and for CKAN client code of that layer
+
+### Features
+
+* Do one thing and do it well: provide an API to store and retrieve files from storage, in a way that is pluggable into a micro-services based application and to existing CKAN (2.8 / 2.9)
+* Does not force, and in fact is not aware of, a specific file naming logic (i.e. resource file names could be based on a user given name, a content hash, a revision ID or any mixture of these - it is up to the using system to decide)
+* Does not force a specific storage backend; Should support Amazon S3, Azure Storage and local file storage in some way initially but in general backend should be pluggable
+* Does not force a specific authentication scheme; Expects a signed JWT token, does not care who signed it and how the user got authenticated
+* Does not force complex authorization scheme; Leave it to external system to do complex authorization if needed;
+  * By default, the system can work in an “admin party” mode where all authenticated users have full access to all files. This will be “good enough” for many DMS implementations including CKAN.
+  * Potentially, allow plugging in a more complex authorization logic that relies on JWT claims to perform granular authorization checks
+
+### For Data Files (i.e. Blobs)
+
+This system is about storing and providing access to blobs, or streams of bytes; It is not about providing access to the data stored within (i.e. it is not meant to replace CKAN’s datastore).
+
+### For CKAN – whilst not necessarily CKAN Specific
+
+While the system’s design should not be CKAN specific in any way, our current client needs require us to provide a CKAN extension that integrates with this system.
+
+CKAN’s current IUploader interface has been identified to be too narrow to provide the functionality required by complex projects (resource versioning, direct cloud uploads and downloads, large file support and multipart support). While some of these needs could be and have been “hacked” through the IUploader interface, the implementations have been over complex and hard to debug.
+
+Our goal should be to provide a CKAN extension that provides the following functionality directly:
+
+* Uploading and downloading resource files directly from the client if supported by the storage backend
+  *   Multipart upload support if supported by storage backend
+  *   Handling of signed URLs for uploads and private downloads
+  *   Client side code for handling multipart uploads
+  *   TBD: If storage backend does not support direct uploads / downloads, fall back to …
+
+In addition, this extension should provide an API for other extensions to do things like:
+
+*   Set the file naming scheme (We need this for ckanext-versions)
+*   Lower level file access, e.g. move and delete files. We may need this in the future to optimize storage and deduplicate files as proposed for ckanext-versions
+
+In addition, this extension must “play nice” with common CKAN features such as the datastore extension and related datapusher / xloader extensions.
+
+### Usable For other DMS implementations
+
+There should be nothing in this system, except for the CKAN extension described above, that is specific to CKAN. That will allow to re-use and re-integrate this system as a micro-service in other DMS implementations such as ckan-ng and others.
+In fact, the core part of this system should be a generic, abstract storage service with a light authorization layer. This could make it useful in a host of situations where storage micro-service is needed.
+
+### High Level Principles
+
+Common Principles
+
+* Uploads and downloads directly from cloud provides to browser
+* Signed uploads / downloads - for private / authorized only data access
+* Support for AWS, Azure and potentially GCP storage
+* Support for local (non cloud) storage, potentially through a system like [https://min.io/](https://min.io/)
+* Multipart / large file upload support (a few GB in size should be supported for Gates)
+* Not opinionated about file naming / paths; Allow users to set file locations under some pre-defined patchs / buckets
+* Client side support - browser widgets / code for uploading and downloading files / multipart uploads directly to different backends
+* Well-documented flow for using from API (not browser)
+* Provided API for deleting and moving files
+* Provided API for accessing storage-level metadata (e.g. file MD5) (do we need this could be useful for processes that do things like deduplicate storage)
+* Provided API for managing storage-level object level settings (e.g. “Content-disposition” / “Content-type” headers, etc.)
+* Authorization based on some kind of portable scheme (JWT)
+
+CKAN integration specific (implemented as a CKAN extension)
+
+* JWT generation based on current CKAN user permissions
+* Client widgets integration (or CKAN specific widgets) in right places in CKAN templates
+* Hook into resource upload / download / deletion controllers in CKAN
+* API to allow other extensions to control storage level object metadata (headers, path)
+* API to allow other extensions to hook into lifecycle events - upload completion, download request, deletion etc.
+
+
+### Components
+
+The Decoupled Storage solution should be split into several parts, with some parts being independent of others:
+
+* [External] Cloud Storage service (or API similar if local file system) e.g. S3, GCS, Azure Storage, Min.io (for local file system)
+* Cloud Storage Access Service
+* [External] Permissions Service for granting general permission tokens that give access to Cloud Storage Access Service
+  * JWT tokens can be generated by any party that has the right signing key. Thus, we can initially do without this if JWT signing is implemented as part of the CKAN extension
+* Browser based Client for Cloud Storage (compatible with #1 and with different cloud vendors)
+* CKAN extension that wraps the two parts above to provide a storage solution for CKAN
+
+### Questions
+
+* What is file structure in cloud ... i.e. What is the file path for uploaded files? Options:
+  * Client chooses a name/path
+  * Content addressable i.e. the name is given by the content? How? Use a hash.]
+    * Beauty of that: standard way to name things. The same thing has the same name (modulo collisions)
+    * Goes with versioning => same file = same name, diff file = diff name
+* And do you enforce that from your app
+  * Request for token needs to include the destination file path

site/content/docs/dms/ckan-client-guide/index.md

@@ -0,0 +1,503 @@
+# CKAN Client Guide
+
+Guide to interacting with [CKAN](/docs/dms/ckan) for power users such as data scientists, data engineers and data wranglers.
+
+This guide is about adding and managing data in CKAN programmatically and it assumes:
+
+* You are familiar with key concepts like metadata, data, etc.
+* You are working programmatically with a programming language such as Python, JavaScript or R (_coming soon_).
+
+## Frictionless Formats
+
+Clients use [Frictionless formats](https://specs.frictionlessdata.io/) by default for describing dataset and resource objects passed to client methods. Internally, we then use the a *CKAN {'<=>'} Frictionless Mapper* (both [in JavaScript]( https://github.com/datopian/frictionless-ckan-mapper-js ) and [in Python](https://github.com/frictionlessdata/frictionless-ckan-mapper)) to convert objects to CKAN formats before calling the API. **Thus, you can use _Frictionless Formats_ by default with the client**.
+
+>[!tip]As CKAN moves to Frictionless to default this will gradually become unnecessary.
+
+## Quick start
+
+Most of this guide has Python programming language in mind, including its [convention regading using _snake case_ for instances and methods names](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles).
+
+If needed, you can adapt the instructions to JavaScript and R (coming soon) by using _camel case_ instead — for example, if in the Python code we have  `client.push_blob(…)`, in JavaScript it would be `client.pushBlob(…)`.
+
+### Prerequisites
+
+Install the client for your language of choice:
+
+* Python: https://github.com/datopian/ckan-client-py#install
+* JavaScript: https://github.com/datopian/ckan-client-js#install
+* R: _coming soon_
+
+### Create a client
+
+#### Python
+
+```python
+from ckanclient import Client
+
+
+api_key = '771a05ad-af90-4a70-beea-cbb050059e14'
+api_url = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+lfs_url = 'http://localhost:9419'
+
+client = Client(api_url, organization, dataset, lfs_url)
+```
+
+#### JavaScript
+
+```javascript
+const { Client } = require('ckanClient')
+
+apiKey = '771a05ad-af90-4a70-beea-cbb050059e14'
+apiUrl = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+
+const client = Client(apiKey, organization, dataset, apiUrl)
+```
+
+### Upload a resource
+
+That is to say, upload a file, implicitly creating a new dataset.
+
+#### Python
+
+```python
+from frictionless import describe
+
+
+resource = describe('my-data.csv')
+client.push_blob(resource)
+```
+
+### Create a new empty Dataset with metadata
+
+#### Python
+
+```python
+client.create('my-data')
+client.push(resource)
+```
+
+### Adding a resource to an existing Dataset
+
+>[!note]Not implemented yet.
+
+
+```python
+client.create('my-data')
+client.push_resource(resource)
+```
+
+### Edit a Dataset's metadata
+
+>[!note]Not implemented yet.
+
+
+```python
+dataset = client.retrieve('sample-dataset')
+client.update_metadata(
+    dataset,
+    metadata: {'maintainer_email': 'sample@datopian.com'}
+)
+```
+
+For details of metadata see the [metadata reference below](#metadata-reference).
+
+## API - Porcelain
+
+### `Client.create`
+
+Expects as a single argument: a _string_, or a _dict_ (in Python), or an _object_ (in JavaScript). This argument is either a valid dataset name or dictionary with metadata for the dataset in Frictionless format.
+
+### `Client.push`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a dataset metadata in Frictionless format.
+
+### `Client.retrieve`
+
+Expects a single argument: a string with a dataset name or uniquer ID. Returns a Frictionless resource as a _dict_ (in Python) or as an _Promisse .&lt;object&gt;_ (in JavaScript).
+
+### `Client.push_blob`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a Frictionless resource.
+
+## API - Plumbing
+
+### `Client.action`
+
+This method bridges access to the CKAN API _action endpoint_.
+
+#### In Python
+
+Arguments:
+
+| Name                 | Type       | Default    | Description                                                  |
+| -------------------- | ---------- | ---------- | ------------------------------------------------------------ |
+| `name`               | `str`      | (required) | The action name, for example, `site_read`, `package_show`…   |
+| `payload`            | `dict`     | (required) | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `http_get`           | `bool`     | `False`    | Optional, if `True` will make `GET` request, otherwise `POST`. |
+| `transform_payload`  | `function` | `None`     | Function to mutate the `payload` before making the request (useful to convert to and from CKAN and Frictionless formats). |
+| `transform_response` | `function` | `None`     | function to mutate the response data before returning it (useful to convert to and from CKAN and Frictionless formats). |
+
+>[!note]The CKAN API uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you can pass `frictionless_ckan_mapper.frictionless_to_ckan` as `transform_payload`, and `frictionless_ckan_mapper.ckan_to_frictionless` as `transform_response`.
+
+
+#### In JavaScript
+
+Arguments:
+
+| Name         | Type                | Default            | Description                                                  |
+| ------------ | ------------------- | ------------------ | ------------------------------------------------------------ |
+| `actionName` | <code>string</code> | (required)         | The action name, for example, `site_read`, `package_show`…   |
+| `payload`    | <code>object</code> | (required)         | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `useHttpGet` | <code>object</code> | <code>false</code> | Optional, if `True` will make `GET` request, otherwise `POST`. |
+
+>[!note]The JavaScript implementation uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you need to convert from Frictionless to CKAN before calling `action` , and from CKAN to Frictionless after calling `action`.
+
+## Metadata reference
+
+>[!info]Your site may have custom metadata that differs from the example set below.
+
+
+### Profile
+
+**(`string`)** Defaults to _data-resource_.
+
+The profile of this descriptor.
+
+Every Package and Resource descriptor has a profile. The default profile, if none is declared, is `data-package` for Package and `data-resource` for Resource.
+
+#### Examples
+
+- `{"profile":"tabular-data-package"}`
+
+- `{"profile":"http://example.com/my-profiles-json-schema.json"}`
+
+### Name
+
+**(`string`)**
+
+An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+
+This is ideally a url-usable and human-readable name. Name `SHOULD` be invariant, meaning it `SHOULD NOT` change when its parent descriptor is updated.
+
+#### Example
+
+- `{"name":"my-nice-name"}`
+
+### Path
+
+A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+
+The dereferenced value of each referenced data source in `path` `MUST` be commensurate with a native, dereferenced representation of the data the resource describes. For example, in a *Tabular* Data Resource, this means that the dereferenced value of `path` `MUST` be an array.
+
+#### Validation
+
+##### It must satisfy one of these conditions
+
+###### Path
+
+**(`string`)**
+
+A fully qualified URL, or a POSIX file path..
+
+Implementations need to negotiate the type of path provided, and dereference the data accordingly.
+
+**Examples**
+
+- `{"path":"file.csv"}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+**(`array`)**
+
+**Examples**
+
+- `["file.csv"]`
+
+- `["http://example.com/file.csv"]`
+
+#### Examples
+
+- `{"path":["file.csv","file2.csv"]}`
+
+- `{"path":["http://example.com/file.csv","http://example.com/file2.csv"]}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+### Data
+
+Inline data for this resource.
+
+### Schema
+
+**(`object`)**
+
+A schema for this resource.
+
+### Title
+
+**(`string`)**
+
+A human-readable title.
+
+#### Example
+
+- `{"title":"My Package Title"}`
+
+### Description
+
+**(`string`)**
+
+A text description. Markdown is encouraged.
+
+#### Example
+
+- `{"description":"# My Package description\nAll about my package."}`
+
+### Home Page
+
+**(`string`)**
+
+The home on the web that is related to this data package.
+
+#### Example
+
+- `{"homepage":"http://example.com/"}`
+
+### Sources
+
+**(`array`)**
+
+The raw sources for this resource.
+
+#### Example
+
+- `{"sources":[{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]}`
+
+### Licenses
+
+**(`array`)**
+
+The license(s) under which the resource is published.
+
+This property is not legally binding and does not guarantee that the package is licensed under the terms defined herein.
+
+#### Example
+
+- `{"licenses":[{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]}`
+
+### Format
+
+**(`string`)**
+
+The file format of this resource.
+
+`csv`, `xls`, `json` are examples of common formats.
+
+#### Example
+
+- `{"format":"xls"}`
+
+### Media Type
+
+**(`string`)**
+
+The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+
+#### Example
+
+- `{"mediatype":"text/csv"}`
+
+### Encoding
+
+**(`string`)** Defaults to _utf-8_.
+
+The file encoding of this resource.
+
+#### Example
+
+- `{"encoding":"utf-8"}`
+
+### Bytes
+
+**(`integer`)**
+
+The size of this resource in bytes.
+
+#### Example
+
+- `{"bytes":2082}`
+
+### Hash
+
+**(`string`)**
+
+The MD5 hash of this resource. Indicate other hashing algorithms with the {'{algorithm}'}:{'{hash}'} format.
+
+#### Examples
+
+- `{"hash":"d25c9c77f588f5dc32059d2da1136c02"}`
+
+- `{"hash":"SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"}`
+
+## Generating templates
+
+You can use [`jsv`](https://github.com/datopian/jsv) to generate a template  script in Python, JavaScript, and R.
+
+To install it:
+
+```
+$ npm install -g git+https://github.com/datopian/jsv.git
+```
+
+### Python
+
+```
+$ jsv data-resource.json --output py
+```
+
+**Output**
+```python
+dataset_metadata = {
+    "profile": "data-resource",  # The profile of this descriptor.
+    # [example] "profile": "tabular-data-package"
+    # [example] "profile": "http://example.com/my-profiles-json-schema.json"
+    "name": "my-nice-name",  # An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+    "path": ["file.csv","file2.csv"],  # A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+    # [example] "path": ["http://example.com/file.csv","http://example.com/file2.csv"]
+    # [example] "path": "http://example.com/file.csv"
+    "data": None,  # Inline data for this resource.
+    "schema": None,  # A schema for this resource.
+    "title": "My Package Title",  # A human-readable title.
+    "description": "# My Package description\nAll about my package.",  # A text description. Markdown is encouraged.
+    "homepage": "http://example.com/",  # The home on the web that is related to this data package.
+    "sources": [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}],  # The raw sources for this resource.
+    "licenses": [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}],  # The license(s) under which the resource is published.
+    "format": "xls",  # The file format of this resource.
+    "mediatype": "text/csv",  # The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+    "encoding": "utf-8",  # The file encoding of this resource.
+    # [example] "encoding": "utf-8"
+    "bytes": 2082,  # The size of this resource in bytes.
+    "hash": "d25c9c77f588f5dc32059d2da1136c02",  # The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+    # [example] "hash": "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+}
+```
+
+
+### JavaScript
+
+```
+$ jsv data-resource.json --output js
+```
+
+**Output**
+```javascript
+const datasetMetadata = {
+  // The profile of this descriptor.
+  profile: "data-resource",
+  // [example] profile: "tabular-data-package"
+  // [example] profile: "http://example.com/my-profiles-json-schema.json"
+  // An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+  name: "my-nice-name",
+  // A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+  path: ["file.csv", "file2.csv"],
+  // [example] path: ["http://example.com/file.csv","http://example.com/file2.csv"]
+  // [example] path: "http://example.com/file.csv"
+  // Inline data for this resource.
+  data: null,
+  // A schema for this resource.
+  schema: null,
+  // A human-readable title.
+  title: "My Package Title",
+  // A text description. Markdown is encouraged.
+  description: "# My Package description\nAll about my package.",
+  // The home on the web that is related to this data package.
+  homepage: "http://example.com/",
+  // The raw sources for this resource.
+  sources: [
+    {
+      title: "World Bank and OECD",
+      path: "http://data.worldbank.org/indicator/NY.GDP.MKTP.CD",
+    },
+  ],
+  // The license(s) under which the resource is published.
+  licenses: [
+    {
+      name: "odc-pddl-1.0",
+      path: "http://opendatacommons.org/licenses/pddl/",
+      title: "Open Data Commons Public Domain Dedication and License v1.0",
+    },
+  ],
+  // The file format of this resource.
+  format: "xls",
+  // The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+  mediatype: "text/csv",
+  // The file encoding of this resource.
+  encoding: "utf-8",
+  // [example] encoding: "utf-8"
+  // The size of this resource in bytes.
+  bytes: 2082,
+  // The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+  hash: "d25c9c77f588f5dc32059d2da1136c02",
+  // [example] hash: "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+};
+```
+
+### R
+
+```
+$ jsv data-resource.json --output r
+```
+
+**Output** 
+```r
+# The profile of this descriptor.
+profile <- "data-resource"
+# [example] profile <- "tabular-data-package"
+# [example] profile <- "http://example.com/my-profiles-json-schema.json"
+# An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+name <- "my-nice-name"
+# A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+path <- ["file.csv","file2.csv"]
+# [example] path <- ["http://example.com/file.csv","http://example.com/file2.csv"]
+# [example] path <- "http://example.com/file.csv"
+# Inline data for this resource.
+data <- NA
+# A schema for this resource.
+schema <- NA
+# A human-readable title.
+title <- "My Package Title"
+# A text description. Markdown is encouraged.
+description <- "# My Package description\nAll about my package."
+# The home on the web that is related to this data package.
+homepage <- "http://example.com/"
+# The raw sources for this resource.
+sources <- [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]
+# The license(s) under which the resource is published.
+licenses <- [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]
+# The file format of this resource.
+format <- "xls"
+# The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+mediatype <- "text/csv"
+# The file encoding of this resource.
+encoding <- "utf-8"
+# [example] encoding <- "utf-8"
+# The size of this resource in bytes.
+bytes <- 2082L
+# The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+hash <- "d25c9c77f588f5dc32059d2da1136c02"
+# [example] hash <- "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+
+```
+
+
+## Design Principles
+
+The client **should** use Frictionless formats by default for describing dataset and resource objects passed to client methods.
+
+In addition, where more than metadata is needed (e.g., we need to access the data stream, or get the schema) we expect the _Dataset_ and _Resource_ objects to follow the [Frictionless Data Lib pattern](https://github.com/frictionlessdata/project/blob/master/rfcs/0004-frictionless-data-lib-pattern.md).

site/content/docs/dms/ckan-enterprise/index.md

@@ -0,0 +1,108 @@
+# CKAN Enterprise
+
+## Introduction
+
+CKAN Enterprise is our name for what we plan would become our standard "base" distribution for CKAN going forward:
+
+* It is a CKAN standard code base with micro-services.
+* Enterprise grade data catalog and portal targeted at Gov (open data portals) and Enterprise (Data Catalogs +).
+* It is also known as [Datopian DMS](https://www.datopian.com/datopian-dms/).
+
+## Roadmap 2021 and beyond
+
+|                   | Current                                                                                    | CKAN Enterprise                                                 |
+|-------------------|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
+| Raw storage       | Filestore                                                                                  | Giftless                                                        |
+| Data Loader (db)  | DataPusher extension                                                                       | Aircan                                                          |
+| Data Storage (db) | Postgres                                                                                   | Any database engine. By default, Postgres                       |
+| Data API (read)   | Built-in DataStore extension's API including SQL endpoint                                  | GraphQL based standalone micro-service                          |
+| Frontend (public) | Build-in frontend into CKAN Classic python app (some projects are using nodejs app)        | PortalJS or nodejs app                                          |
+| Data Explorer     | ReclineJS (some projects that uses nodejs app for frontend have React based Data Explorer) | GraphQL based Data Explorer                                     |
+| Auth              | Traditional login/password + extendable with CKAN Classic extensions                       | SSO with default Google, Github, Facebook and Microsoft options |
+| Permissions       | CKAN Classic based permissions                                                             | Existing permissions exposed via JWT based authz API            |
+
+## Timeline 2021
+
+To develop a base distribution of CKAN Enterprise, we want to build a demo project with the features from the roadmap. This way we can:
+
+* understand its advantages/limitations;
+* compare against other instances of CKAN;
+* demonstrate for the potential clients.
+
+High level overview of the planned features with ETA:
+
+| Name                          | Description                          | Effort | ETA |
+| ----------------------------- | ------------------------------------ | ------ | --- |
+| [Init](#Init)                 | Select CKAN version and deploy to DX | xs     | Q2  |
+| [Blobstore](#Blobstore)       | Integrate Giftless for raw storage   | s      | Q2  |
+| [Versioning](#Versioning)     | Develop/integrate new versioning sys | l      | Q3  |
+| [DataLoader](#DataLoader)     | Develop/integrate Aircan             | xl     | Q3  |
+| [Data API](#Data-API)         | Integrate new Data API (read)        | m      | Q2  |
+| [Frontend](#Frontend)         | Build a theme using PortalJS         | s      | Q2  |
+| [DataExplorer](#DataExplorer) | Integrate into PortalJS              | s      | Q2  |
+| [Permissions](#Permissions)   | Develop permissions in read frontend | m      | Q4  |
+| [Auth](#Auth)                 | Integrate                            | s      | Q4  |
+
+### Init
+
+Initialize a new project for development of CKAN Enterprise.
+
+Tasks:
+
+* Boot project in Datopian-DX cluster
+* Use CKAN v2.8.x (latest patch) or 2.9.x
+* Don't setup DataPusher
+* Namespace: `ckan-enterprise`
+* Domain: `enterprise.ckan.datopian.com`
+
+### Blobstore
+
+See [blob storage](/docs/dms/blob-storage#ckan-v3)
+
+### Versioning
+
+See [versioning](/docs/dms/versioning#ckan-v3)
+
+### DataLoader
+
+See [DataLoader](/docs/dms/load)
+
+### Data API
+
+* Install new [Data API service](https://github.com/datopian/data-api) in the project
+* Install Hasura service in the project
+* Set it up to work with DB of CKAN Enterprise
+* Read more about Data API [here](/docs/dms/data-api#read-api-3)
+
+Notes:
+
+* We could experiment and use various features of Hasura, eg:
+  * Setting up row/column limits per user role (permissions)
+  * Subscriptions to auto load new data rows
+
+### Frontend
+
+PortalJS for the read frontend of CKAN Enterprise. [Read more](/docs/dms/frontend/#frontend).
+
+### DataExplorer
+
+A new Data Explorer based on GraphQL API: https://github.com/datopian/data-explorer-graphql
+
+### Permissions
+
+See [permissions](/docs/dms/permissions#permissions-authorization).
+
+### Auth
+
+Next generation, Kratos based, authentication (mostly SSO with no Traditional login by default) with following options out of the box:
+
+* GitHub
+* Google
+* Facebook
+* Microsoft
+
+Easy to add:
+
+* Discord
+* GitLab
+* Slack