.vscode/extensions.json

@@ -0,0 +1,8 @@
+{
+  "recommendations": [
+    "nrwl.angular-console",
+    "esbenp.prettier-vscode",
+    "firsttris.vscode-jest-runner",
+    "dbaeumer.vscode-eslint"
+  ]
+}

CONTRIBUTING.md

@@ -4,7 +4,7 @@ title: Developer docs for contributors
 
 ## Our repository
 
-https://github.com/datopian/datahub
+https://github.com/datopian/portaljs
 
 Structure:
 
@@ -17,7 +17,7 @@ Structure:
 
 ## How to contribute
 
-You can start by checking our [issues board](https://github.com/datopian/datahub/issues).
+You can start by checking our [issues board](https://github.com/datopian/portaljs/issues).
 
 If you'd like to work on one of the issues you can:
 
@@ -35,7 +35,7 @@ If you'd like to work on one of the issues you can:
 If you have an idea for improvement, and it doesn't have a corresponding issue yet, simply submit a new one.
 
 > [!note]
-> Join our [Discord channel](https://discord.gg/KZSf3FG4EZ) do discuss existing issues and to ask for help.
+> Join our [Discord channel](https://discord.gg/rTxfCutu) do discuss existing issues and to ask for help.
 
 ## Nx
 

README.md

@@ -1,51 +1,31 @@
-<p align="center">
-  Bugs, issues and suggestions re PortalJS framework
-  <br />
-  <br /><a href="https://discord.gg/xfFDMPU9dC"><img src="https://dcbadge.vercel.app/api/server/xfFDMPU9dC" /></a>
-</p>
+<h1 align="center">
+🌀 Portal.JS
+<br />
+Rapidly build rich data portals using a modern frontend framework
+</h1>
 
-## PortalJS framework
+* [What is Portal.JS ?](#What-is-Portal.JS)
+    * [Features](#Features)
+      * [For developers](#For-developers)
+* [Docs](#Docs)
+* [Community](#Community)
+* [Appendix](#Appendix)
+    * [What happened to Recline?](#What-happened-to-Recline?)
 
-This repo and issue tracker are for
+# What is Portal.JS
 
-- PortalJS 🌀 - https://www.portaljs.com/
-- DataHub Cloud ☁️ - https://datahub.io/
+🌀 Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. Portal.JS can be used to present a single dataset or build a full-scale data catalog/portal.
 
-### Issues
+Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com/) framework. Portal.JS assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
 
-Found a bug: 👉 https://github.com/datopian/portaljs/issues/new
-
-### Discussions
-
-Got a suggestion, a question, want some support or just want to shoot the breeze 🙂
-
-Head to the discussion forum: 👉 https://github.com/datopian/portaljs/discussions
-
-### Chat on Discord
-
-If you would prefer to get help via live chat check out our discord 👉
-
-[Discord](https://discord.gg/xfFDMPU9dC)
-
-### Docs
-
-- For PortalJS go to https://www.portaljs.com/opensource
-- For DataHub Cloud – https://datahub.io/docs
-
-## PortalJS Cloud 🌀
-
-PortalJS Cloud 🌀 is a platform for rapidly creating rich data portal and publishing systems using a modern frontend approach. PortalJS Cloud can be used to publish a single dataset or build a full-scale data catalog/portal.
-
-PortalJS Cloud is built in JavaScript and React on top of the popular [Next.js](https://nextjs.org) framework. PortalJS Cloud assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/), GitHub, Frictionless Data Packages and more.
-
-### Features
+## Features
 
 - 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. Wordpress) with a common internal API.
 - 👩‍💻 Developer friendly: built with familiar frontend tech (JavaScript, React, Next.js).
 - 🔋 Batteries included: full set of portal components out of the box e.g. catalog search, dataset showcase, blog, etc.
 - 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
 - 🧱 Extensible: quickly extend and develop/import your own React components
-- 📝 Well documented: full set of documentation plus the documentation of Next.js.
+- 📝 Well documented: full set of documentation plus the documentation of Next.js and Apollo.
 
 ### For developers
 
@@ -53,3 +33,25 @@ PortalJS Cloud is built in JavaScript and React on top of the popular [Next.js](
 - 🚀 Next.js framework: so everything in Next.js for free: Server Side Rendering, Static Site Generation, huge number of examples and integrations, etc.
   - Server Side Rendering (SSR) => Unlimited number of pages, SEO and more whilst still using React.
   - Static Site Generation (SSG) => Ultra-simple deployment, great performance, great lighthouse scores and more (good for small sites)
+
+#### **Check out the [Portal.JS website](https://portaljs.org/) for a gallery of live portals**
+
+___
+
+# Docs
+
+Access the Portal.JS documentation at:
+
+https://portaljs.org/docs
+
+- [Examples](https://portaljs.org/docs#examples)
+
+# Community
+
+If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
+
+# Appendix
+
+## What happened to Recline?
+
+Portal.JS used to be Recline(JS). If you are looking for the old Recline codebase it still exists:  see the [`recline` branch](https://github.com/datopian/portal.js/tree/recline). If you want context for the rename see [this issue](https://github.com/datopian/portal.js/issues/520).

examples/ckan-2021/README.md

@@ -2,7 +2,7 @@
 
 **🚩 UPDATE April 2023: This example is now deprecated - though still works!. Please use the [new CKAN examples](https://github.com/datopian/portaljs/tree/main/examples)**
 
-This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.com/blog/example-ckan-2021)
+This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.org/blog/example-ckan-2021)
 
 ## Developers
 

examples/ckan-ssg/README.md

@@ -1,7 +1,7 @@
 This is a repo intended to serve as an example of a data catalog that get its data from a CKAN Instance.
 
 ```
-npx create-next-app <app-name> --example https://github.com/datopian/datahub/tree/main/examples/ckan-ssg
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/ckan-example
 cd <app-name>
 ```
 
@@ -19,7 +19,7 @@ npm run dev
 
 Congratulations, you now have something similar to this running on `http://localhost:4200`
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252297726865408/image.png?width=853&height=461)
-If you go to any one of those pages by clicking on `More info` you will see something similar to this
+If yo go to any one of those pages by clicking on `More info` you will see something similar to this
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252298074988595/image.png?width=853&height=461)
 
 ## Deployment

examples/dataset-frictionless/README.md

@@ -1,6 +1,6 @@
 This example creates a portal/showcase for a single dataset. The dataset should be a [Frictionless dataset (data package)][fd] i.e. there should be a `datapackage.json`.
 
-[fd]: https://specs.frictionlessdata.io/data-package/
+[fd]: https://frictionlessdata.io/data-packages/
 
 ## How to use
 

examples/fivethirtyeight/README.md

@@ -1,9 +1,3 @@
-# PortalJS Demo replicating the FiveThirtyEight data portal
-
-## 👉 https://fivethirtyeight.portaljs.org 👈
-
-Here's a blog post we wrote about it: https://www.datopian.com/blog/fivethirtyeight-replica
-
 This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
 
 You might be asking why we did that, there are three main reasons:

examples/fivethirtyeight/components/Layout.tsx

@@ -59,7 +59,7 @@ export default function Layout({ children }: { children: React.ReactNode }) {
           <div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
             <a
               className="hover:opacity-75 transition"
-              href="https://portaljs.com"
+              href="https://portaljs.org"
             >
               Built with 🌀PortalJS
             </a>
@@ -77,7 +77,7 @@ export default function Layout({ children }: { children: React.ReactNode }) {
             <li>
               <a
                 className="hover:opacity-75 transition"
-                href="https://portaljs.com"
+                href="https://portaljs.org"
               >
                 PortalJS
               </a>

examples/github-backed-catalog/README.md

@@ -6,7 +6,7 @@ A `datasets.json` file is used to specify which datasets are going to be part of
 
 The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
 
-You can read more about it on the [Data catalog with data on GitHub](https://portaljs.com/docs/examples/github-backed-catalog) blog post.
+You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
 
 ## Demo
 

examples/github-backed-catalog/pages/index.tsx

@@ -40,7 +40,7 @@ export function Datasets({ projects }) {
               <Link
                 target="_blank"
                 className="underline"
-                href="https://portaljs.com/"
+                href="https://portaljs.org/"
               >
                 🌀 PortalJS
               </Link>

examples/learn/README.md

@@ -1 +1 @@
-PortalJS Learn Example - https://portaljs.com/docs
+PortalJS Learn Example - https://portaljs.org/docs

examples/openspending/README.md

@@ -6,7 +6,7 @@ A `datasets.json` file is used to specify which datasets are going to be part of
 
 The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
 
-You can read more about it on the [Data catalog with data on GitHub](https://portaljs.com/docs/examples/github-backed-catalog) blog post.
+You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
 
 ## Demo
 

examples/openspending/components/_shared/Footer.tsx

@@ -17,7 +17,7 @@ export default function Footer() {
           </a>
         </div>
         <div className="flex gap-x-2 items-center mx-auto h-20">
-          <p className="mt-8 text-base text-slate-500 md:mt-0">Built with <a href="https://portaljs.com" target="_blank" className='text-xl font-medium'>🌀 PortalJS</a></p>
+          <p className="mt-8 text-base text-slate-500 md:mt-0">Built with <a href="https://portaljs.org" target="_blank" className='text-xl font-medium'>🌀 PortalJS</a></p>
         </div>
       </div>
     </footer>

examples/openspending/content/stories/where-the-european-structural-and-investment-funds-go.mdx

@@ -127,4 +127,4 @@ Based on the bar chart above we can conclude that the following 3 countries have
 2. Poland - EUR ~68b.
 3. Italy - EUR ~35b.
 
-_This data story was created by using Datopian's PortalJS framework. You can learn more about the framework by visiting https://portaljs.com/_
+_This data story was created by using Datopian's PortalJS framework. You can learn more about the framework by visiting https://portaljs.org/_

examples/turing/README.md

@@ -1,6 +1,6 @@
 This demo data portal is designed for https://hatespeechdata.com. It catalogs datasets annotated for hate speech, online abuse, and offensive language which are useful for training a natural language processing system to detect this online abuse.
 
-The site is built on top of [PortalJS](https://portaljs.com/). It catalogs datasets and lists of offensive keywords. It also includes static pages. All of these are stored as markdown files inside the `content` folder.
+The site is built on top of [PortalJS](https://portaljs.org/). It catalogs datasets and lists of offensive keywords. It also includes static pages. All of these are stored as markdown files inside the `content` folder.
 
 - .md files inside `content/datasets/` will appear on the dataset list section of the homepage and be searchable as well as having a individual page in `datasets/<file name>`
 - .md files inside `content/keywords/` will appear on the list of offensive keywords section of the homepage as well as having a individual page in `keywords/<file name>`

examples/turing/components/Footer.jsx

@@ -21,7 +21,7 @@ export function Footer() {
           <Container.Inner>
             <div className="flex flex-col items-center justify-between gap-6 sm:flex-row">
               <p className="text-sm font-medium text-zinc-800 dark:text-zinc-200">
-                Built with <a href='https://portaljs.com'>PortalJS 🌀</a>
+                Built with <a href='https://portaljs.org'>PortalJS 🌀</a>
               </p>
               <p className="text-sm text-zinc-400 dark:text-zinc-500">
                 &copy; {new Date().getFullYear()} Leon Derczynski. All rights

packages/ckan/package.json

@@ -2,7 +2,7 @@
   "name": "@portaljs/ckan",
   "version": "0.1.0",
   "type": "module",
-  "description": "https://portaljs.com",
+  "description": "https://portaljs.org",
   "keywords": [
     "data portal",
     "data catalog",

packages/components/.storybook/preview.ts

@@ -1,16 +1,9 @@
 import 'tailwindcss/tailwind.css'
 import '../src/index.css'
+
+
 import type { Preview } from '@storybook/react';
 
-window.process = {
- ...window.process,
- env:{
-  ...window.process?.env,
-
-  
- }
-};
-
 const preview: Preview = {
   parameters: {
     actions: { argTypesRegex: '^on[A-Z].*' },

packages/components/CHANGELOG.md

@@ -1,121 +1,5 @@
 # @portaljs/components
 
-## 1.2.3
-
-### Patch Changes
-
-- [`62dbc35d`](https://github.com/datopian/portaljs/commit/62dbc35d3b39ea7409949340214ca83a448ee999) Thanks [@olayway](https://github.com/olayway)! - LineChart: break lines at invalid / missing values (don't connect if there are gaps in values).
-
-## 1.2.2
-
-### Patch Changes
-
-- [`eeb480e8`](https://github.com/datopian/datahub/commit/eeb480e8cff2d11072ace55ad683a65f54f5d07a) Thanks [@olayway](https://github.com/olayway)! - Adjust `xAxisTimeUnit` property in LineChart to allow for passing `yearmonth`.
-
-## 1.2.1
-
-### Patch Changes
-
-- [`836b143a`](https://github.com/datopian/datahub/commit/836b143a3178b893b1aae3fb511d795dd3a63545) Thanks [@olayway](https://github.com/olayway)! - Fix: make tileLayerName in Map optional.
-
-## 1.2.0
-
-### Minor Changes
-
-- [#1338](https://github.com/datopian/datahub/pull/1338) [`63d9e3b7`](https://github.com/datopian/datahub/commit/63d9e3b7543c38154e6989ef1cc1d694ae9fc4f8) Thanks [@olayway](https://github.com/olayway)! - Support for plotting multiple series in LineChart component.
-
-## 1.1.0
-
-### Minor Changes
-
-- [#1122](https://github.com/datopian/datahub/pull/1122) [`8e349678`](https://github.com/datopian/datahub/commit/8e3496782c022b0653e07f217c6b315ba84e0e61) Thanks [@willy1989cv](https://github.com/willy1989cv)! - Map: allow users to choose a base layer setting
-
-## 1.0.1
-
-### Patch Changes
-
-- [#1170](https://github.com/datopian/datahub/pull/1170) [`9ff25ed7`](https://github.com/datopian/datahub/commit/9ff25ed7c47c8c02cc078c64f76ae35d6754c508) Thanks [@lucasmbispo](https://github.com/lucasmbispo)! - iFrame component: change height
-
-## 1.0.0
-
-### Major Changes
-
-- [#1103](https://github.com/datopian/datahub/pull/1103) [`48cd812a`](https://github.com/datopian/datahub/commit/48cd812a488a069a419d8ecc67f24f94d4d1d1d6) Thanks [@demenech](https://github.com/demenech)! - Components API tidying up and storybook docs improvements.
-
-## 0.6.0
-
-### Minor Changes
-
-- [`a044f56e`](https://github.com/datopian/portaljs/commit/a044f56e3cbe0519ddf9d24d78b0bb7eac917e1c) Thanks [@luccasmmg](https://github.com/luccasmmg)! - Added plotly components
-
-## 0.5.10
-
-### Patch Changes
-
-- [#1083](https://github.com/datopian/portaljs/pull/1083) [`86a2945e`](https://github.com/datopian/portaljs/commit/86a2945ee68dfcea0299984ca9cc9070d68fe1c2) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created integration with datastore api for table component
-
-## 0.5.9
-
-### Patch Changes
-
-- [#1081](https://github.com/datopian/portaljs/pull/1081) [`2bbf3134`](https://github.com/datopian/portaljs/commit/2bbf3134896df3ecc66560bdf95bece143614c7b) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed error to remove anchor from document
-
-## 0.5.8
-
-### Patch Changes
-
-- [#1079](https://github.com/datopian/portaljs/pull/1079) [`058d2367`](https://github.com/datopian/portaljs/commit/058d23678a024890f8a6d909ded9fc8fc11cf145) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Changed the download behaviour of the bucket viewer component and removed loading component while downloading
-
-## 0.5.7
-
-### Patch Changes
-
-- [#1077](https://github.com/datopian/portaljs/pull/1077) [`6d7acd27`](https://github.com/datopian/portaljs/commit/6d7acd27ed9299cbcc14eab906f2f0eb414656b8) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created property to present a component while is loading the download of the file and fixed download bug on pagination
-
-## 0.5.6
-
-### Patch Changes
-
-- [#1075](https://github.com/datopian/portaljs/pull/1075) [`26dcffc2`](https://github.com/datopian/portaljs/commit/26dcffc279057f80a579134e862085ba042c06c3) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed problem presenting the download component in the first load of the bucket viewer
-
-## 0.5.5
-
-### Patch Changes
-
-- [#1073](https://github.com/datopian/portaljs/pull/1073) [`cf24042a`](https://github.com/datopian/portaljs/commit/cf24042a910567e98eeb75ade42ce0149bdb62d1) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed filter by startDate error
-
-## 0.5.4
-
-### Patch Changes
-
-- [#1071](https://github.com/datopian/portaljs/pull/1071) [`27c99add`](https://github.com/datopian/portaljs/commit/27c99adde8fa36ad2c2e03f227f93aa62454eefa) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Added pagination and filter properties for the BucketViewer component
-
-## 0.5.3
-
-### Patch Changes
-
-- [#1066](https://github.com/datopian/portaljs/pull/1066) [`dd03a493`](https://github.com/datopian/portaljs/commit/dd03a493beca5459d1ef447b2df505609fc64e95) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created Iframe component
-
-## 0.5.2
-
-### Patch Changes
-
-- [#1063](https://github.com/datopian/portaljs/pull/1063) [`b13e3ade`](https://github.com/datopian/portaljs/commit/b13e3ade3ccefe7dffe84f824bdedd3e512ce499) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created auto zoom configuration for the map component
-
-## 0.5.1
-
-### Patch Changes
-
-- [#1061](https://github.com/datopian/portaljs/pull/1061) [`4ddfc112`](https://github.com/datopian/portaljs/commit/4ddfc1126a3f0b8137ea47a08a36c56b7373b8f6) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created the style property in the Map component
-
-## 0.5.0
-
-### Minor Changes
-
-- [#1055](https://github.com/datopian/portaljs/pull/1055) [`712f4a3b`](https://github.com/datopian/portaljs/commit/712f4a3b0f074e654879bb75059f51e06b422b32) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Creation of BucketViewer component to show the data of public buckets
-
-- [#1057](https://github.com/datopian/portaljs/pull/1057) [`61c750b7`](https://github.com/datopian/portaljs/commit/61c750b7e11fe52bf04d25f192440ee1bb307404) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Exporting BucketViewer to be accessed out of the folder
-
 ## 0.4.0
 
 ### Minor Changes

packages/components/README.md

@@ -1,7 +1,7 @@
 # PortalJS React Components
 
 **Storybook:** https://storybook.portaljs.org  
-**Docs**: https://portaljs.com/opensource  
+**Docs**: https://portaljs.org/docs  
 
 ## Usage
 

packages/components/package.json

@@ -1,8 +1,8 @@
 {
   "name": "@portaljs/components",
-  "version": "1.2.3",
+  "version": "0.4.0",
   "type": "module",
-  "description": "https://portaljs.com",
+  "description": "https://portaljs.org",
   "keywords": [
     "data portal",
     "data catalog",
@@ -29,8 +29,6 @@
     "@githubocto/flat-ui": "^0.14.1",
     "@heroicons/react": "^2.0.17",
     "@planet/maps": "^8.1.0",
-    "@react-pdf-viewer/core": "3.6.0",
-    "@react-pdf-viewer/default-layout": "3.6.0",
     "@tanstack/react-table": "^8.8.5",
     "ag-grid-react": "^30.0.4",
     "chroma-js": "^2.4.2",
@@ -39,19 +37,19 @@
     "next-mdx-remote": "^4.4.1",
     "ol": "^7.4.0",
     "papaparse": "^5.4.1",
-    "pdfjs-dist": "2.15.349",
-    "plotly.js": "^2.30.1",
     "postcss-url": "^10.1.3",
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-hook-form": "^7.43.9",
     "react-leaflet": "^4.2.1",
-    "react-plotly.js": "^2.6.0",
     "react-query": "^3.39.3",
     "react-vega": "^7.6.0",
     "vega": "5.25.0",
     "vega-lite": "5.1.0",
     "vitest": "^0.31.4",
+    "@react-pdf-viewer/core": "3.6.0",
+    "@react-pdf-viewer/default-layout": "3.6.0",
+    "pdfjs-dist": "2.15.349",
     "xlsx": "^0.18.5"
   },
   "devDependencies": {

packages/components/src/components/BucketViewer.tsx

@@ -1,222 +0,0 @@
-import { CSSProperties, ReactNode, useEffect, useState } from 'react';
-import LoadingSpinner from './LoadingSpinner';
-
-export interface BucketViewerFilterSearchedDataEvent {
-  startDate?: Date;
-  endDate?: Date;
-}
-
-export interface BucketViewerProps {
-  onLoadTotalNumberOfItems?: (total: number) => void;
-  domain: string;
-  downloadConfig?: {
-    hoverOfTheFileComponent?: ReactNode;
-  };
-  suffix?: string;
-  className?: string;
-  paginationConfig?: BucketViewerPaginationConfig;
-  filterState?: BucketViewerFilterSearchedDataEvent;
-  dataMapperFn: (rawData: Response) => Promise<BucketViewerData[]>;
-}
-
-export interface BucketViewerPaginationConfig {
-  containerClassName?: string;
-  containerStyles?: CSSProperties;
-  itemsPerPage: number;
-}
-
-export interface BucketViewerData {
-  fileName: string;
-  downloadFileUri: string;
-  dateProps?: {
-    date: Date;
-    dateFormatter?: (date: Date) => string;
-  };
-}
-
-export function BucketViewer({
-  domain,
-  suffix,
-  dataMapperFn,
-  className,
-  filterState,
-  paginationConfig,
-  downloadConfig,
-  onLoadTotalNumberOfItems,
-}: BucketViewerProps) {
-  suffix = suffix ?? '/';
-
-  const { hoverOfTheFileComponent } = downloadConfig ?? {};
-  const [isLoading, setIsLoading] = useState<boolean>(false);
-  const [showDownloadComponentOnLine, setShowDownloadComponentOnLine] =
-    useState(-1);
-  const [currentPage, setCurrentPage] = useState<number>(0);
-  const [lastPage, setLastPage] = useState<number>(0);
-  const [bucketFiles, setBucketFiles] = useState<BucketViewerData[]>([]);
-  const [paginatedData, setPaginatedData] = useState<BucketViewerData[]>([]);
-  const [filteredData, setFilteredData] = useState<BucketViewerData[]>([]);
-
-  useEffect(() => {
-    setIsLoading(true);
-    fetch(`${domain}${suffix}`)
-      .then((res) => dataMapperFn(res))
-      .then((data) => {
-        setBucketFiles(data);
-        setFilteredData(data);
-      })
-      .finally(() => setIsLoading(false));
-  }, [domain, suffix]);
-
-  useEffect(() => {
-    if (paginationConfig) {
-      const startIndex = paginationConfig
-        ? currentPage * paginationConfig.itemsPerPage
-        : 0;
-
-      const endIndex = paginationConfig
-        ? startIndex + paginationConfig.itemsPerPage
-        : 0;
-
-      setLastPage(
-        Math.ceil(filteredData.length / paginationConfig.itemsPerPage) - 1
-      );
-      setPaginatedData(filteredData.slice(startIndex, endIndex));
-    }
-  }, [currentPage, filteredData]);
-
-  useEffect(() => {
-    if (onLoadTotalNumberOfItems) onLoadTotalNumberOfItems(filteredData.length);
-  }, [filteredData]);
-
-  useEffect(() => {
-    if (!filterState) return;
-
-    if (filterState.startDate && filterState.endDate) {
-      setFilteredData(
-        bucketFiles.filter(({ dateProps }) =>
-          dateProps
-            ? dateProps.date.getTime() >= filterState.startDate.getTime() &&
-              dateProps.date.getTime() <= filterState.endDate.getTime()
-            : true
-        )
-      );
-    } else if (filterState.startDate) {
-      setFilteredData(
-        bucketFiles.filter(({ dateProps }) =>
-          dateProps
-            ? dateProps.date.getTime() >= filterState.startDate.getTime()
-            : true
-        )
-      );
-    } else if (filterState.endDate) {
-      setFilteredData(
-        bucketFiles.filter(({ dateProps }) =>
-          dateProps
-            ? dateProps.date.getTime() <= filterState.endDate.getTime()
-            : true
-        )
-      );
-    } else {
-      setFilteredData(bucketFiles);
-    }
-  }, [filterState]);
-
-  return isLoading ? (
-    <div className="w-full flex items-center justify-center h-[300px]">
-      <LoadingSpinner />
-    </div>
-  ) : bucketFiles ? (
-    <>
-      {...(paginationConfig && bucketFiles ? paginatedData : filteredData)?.map(
-        (data, i) => (
-          <ul
-            onClick={() => {
-              const a: HTMLAnchorElement = document.createElement('a');
-              a.href = data.downloadFileUri;
-              a.target = `_blank`;
-              a.download = data.fileName;
-              document.body.appendChild(a);
-              a.click();
-              document.body.removeChild(a);
-            }}
-            key={i}
-            onMouseEnter={() => setShowDownloadComponentOnLine(i)}
-            onMouseLeave={() => setShowDownloadComponentOnLine(undefined)}
-            className={`${
-              className ??
-              'mb-2 border-b-[2px] border-b-[red] hover:cursor-pointer'
-            }`}
-          >
-            {hoverOfTheFileComponent && showDownloadComponentOnLine === i ? (
-              hoverOfTheFileComponent
-            ) : (
-              <></>
-            )}
-            <div className="flex justify-between w-full items-center">
-              <div>
-                <li>{data.fileName}</li>
-                {data.dateProps && data.dateProps.dateFormatter ? (
-                  <li>{data.dateProps.dateFormatter(data.dateProps.date)}</li>
-                ) : (
-                  <></>
-                )}
-              </div>
-            </div>
-          </ul>
-        )
-      )}
-      {paginationConfig ? (
-        <ul
-          className={
-            paginationConfig.containerClassName
-              ? paginationConfig.containerClassName
-              : 'flex justify-end gap-x-[0.5rem] w-full'
-          }
-          style={paginationConfig.containerStyles ?? {}}
-        >
-          <li>
-            <button
-              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
-              disabled={currentPage === 0}
-              onClick={() => setCurrentPage(0)}
-            >
-              First
-            </button>
-          </li>
-          <li>
-            <button
-              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
-              onClick={() => setCurrentPage(currentPage - 1)}
-              disabled={currentPage === 0}
-            >
-              Previous
-            </button>
-          </li>
-          <label>{currentPage + 1}</label>
-
-          <li>
-            <button
-              onClick={() => setCurrentPage(currentPage + 1)}
-              disabled={currentPage >= lastPage}
-              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
-            >
-              Next
-            </button>
-          </li>
-
-          <li>
-            <button
-              onClick={() => setCurrentPage(lastPage)}
-              disabled={currentPage >= lastPage}
-              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
-            >
-              Last
-            </button>
-          </li>
-        </ul>
-      ) : (
-        <></>
-      )}
-    </>
-  ) : null;
-}

packages/components/src/components/Catalog.tsx

@@ -7,12 +7,7 @@ export function Catalog({
   datasets,
   facets,
 }: {
-  datasets: {
-    _id: string | number;
-    metadata: { title: string; [k: string]: string | number };
-    url_path: string;
-    [k: string]: any;
-  }[];
+  datasets: any[];
   facets: string[];
 }) {
   const [indexFilter, setIndexFilter] = useState('');
@@ -61,7 +56,7 @@ export function Catalog({
     //Then check if the selectedValue for the given facet is included in the dataset metadata
     .filter((dataset) => {
       //Avoids a server rendering breakage
-      if (!watch() || Object.keys(watch()).length === 0) return true;
+      if (!watch() || Object.keys(watch()).length === 0) return true
       //This will filter only the key pairs of the metadata values that were selected as facets
       const datasetFacets = Object.entries(dataset.metadata).filter((entry) =>
         facets.includes(entry[0])
@@ -91,7 +86,9 @@ export function Catalog({
           className="p-2 ml-1 text-sm shadow border border-block"
           {...register(elem[0] + '.selectedValue')}
         >
-          <option value="">Filter by {elem[0]}</option>
+          <option value="">
+            Filter by {elem[0]}
+          </option>
           {(elem[1] as { possibleValues: string[] }).possibleValues.map(
             (val) => (
               <option
@@ -105,10 +102,10 @@ export function Catalog({
           )}
         </select>
       ))}
-      <ul className="mb-5 pl-6 mt-5 list-disc">
+      <ul className='mb-5 pl-6 mt-5 list-disc'>
         {filteredDatasets.map((dataset) => (
-          <li className="py-2" key={dataset._id}>
-            <a className="font-medium underline" href={dataset.url_path}>
+          <li className='py-2' key={dataset._id}>
+            <a className='font-medium underline' href={dataset.url_path}>
               {dataset.metadata.title
                 ? dataset.metadata.title
                 : dataset.url_path}
@@ -119,3 +116,4 @@ export function Catalog({
     </>
   );
 }
+

packages/components/src/components/Excel.tsx

@@ -4,14 +4,12 @@ import { read, utils } from 'xlsx';
 import { AgGridReact } from 'ag-grid-react';
 import 'ag-grid-community/styles/ag-grid.css';
 import 'ag-grid-community/styles/ag-theme-alpine.css';
-import { Data } from '../types/properties';
 
 export type ExcelProps = {
-  data: Required<Pick<Data, 'url'>>;
+  url: string;
 };
 
-export function Excel({ data }: ExcelProps) {
-  const url = data.url;
+export function Excel({ url }: ExcelProps) {
   const [isLoading, setIsLoading] = useState<boolean>(false);
   const [activeSheetName, setActiveSheetName] = useState<string>();
   const [workbook, setWorkbook] = useState<any>();

packages/components/src/components/FlatUiTable.tsx

@@ -2,7 +2,6 @@ import { QueryClient, QueryClientProvider, useQuery } from 'react-query';
 import Papa from 'papaparse';
 import { Grid } from '@githubocto/flat-ui';
 import LoadingSpinner from './LoadingSpinner';
-import { Data } from '../types/properties';
 
 const queryClient = new QueryClient();
 
@@ -37,25 +36,30 @@ export async function parseCsv(file: string, parsingConfig): Promise<any> {
 }
 
 export interface FlatUiTableProps {
-  data: Data;
-  uniqueId?: number;
+  url?: string;
+  data?: { [key: string]: number | string }[];
+  rawCsv?: string;
+  randomId?: number;
   bytes: number;
   parsingConfig: any;
 }
 export const FlatUiTable: React.FC<FlatUiTableProps> = ({
+  url,
   data,
-  uniqueId,
+  rawCsv,
   bytes = 5132288,
   parsingConfig = {},
 }) => {
-  uniqueId = uniqueId ?? Math.random();
+  const randomId = Math.random();
   return (
     // Provide the client to your App
     <QueryClientProvider client={queryClient}>
       <TableInner
         bytes={bytes}
+        url={url}
         data={data}
-        uniqueId={uniqueId}
+        rawCsv={rawCsv}
+        randomId={randomId}
         parsingConfig={parsingConfig}
       />
     </QueryClientProvider>
@@ -63,32 +67,33 @@ export const FlatUiTable: React.FC<FlatUiTableProps> = ({
 };
 
 const TableInner: React.FC<FlatUiTableProps> = ({
+  url,
   data,
-  uniqueId,
+  rawCsv,
+  randomId,
   bytes,
   parsingConfig,
 }) => {
-  const url = data.url;
-  const csv = data.csv;
-  const values = data.values;
-
-  if (values) {
+  if (data) {
     return (
       <div className="w-full" style={{ height: '500px' }}>
-        <Grid data={values} />
+        <Grid data={data} />
       </div>
     );
   }
   const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
-    ['dataCsv', url, uniqueId],
+    ['dataCsv', url, randomId],
     () => getCsv(url as string, bytes),
     { enabled: !!url }
   );
   const { data: parsedData, isLoading: isParsing } = useQuery(
-    ['dataPreview', csvString, uniqueId],
+    ['dataPreview', csvString, randomId],
     () =>
-      parseCsv(csv ? (csv as string) : (csvString as string), parsingConfig),
-    { enabled: csv ? true : !!csvString }
+      parseCsv(
+        rawCsv ? (rawCsv as string) : (csvString as string),
+        parsingConfig
+      ),
+    { enabled: rawCsv ? true : !!csvString }
   );
   if (isParsing || isDownloadingCSV)
     <div className="w-full flex justify-center items-center h-[500px]">

packages/components/src/components/Iframe.tsx

@@ -1,17 +0,0 @@
-import { CSSProperties } from 'react';
-import { Data } from '../types/properties';
-
-export interface IframeProps {
-  data: Required<Pick<Data, 'url'>>;
-  style?: CSSProperties;
-}
-
-export function Iframe({ data, style }: IframeProps) {
-  const url = data.url;
-  return (
-    <iframe
-      src={url}
-      style={style ?? { width: `100%`, height: `600px` }}
-    ></iframe>
-  );
-}

packages/components/src/components/LineChart.tsx

@@ -2,40 +2,35 @@ import { useEffect, useState } from 'react';
 import LoadingSpinner from './LoadingSpinner';
 import { VegaLite } from './VegaLite';
 import loadData from '../lib/loadData';
-import { Data } from '../types/properties';
 
 type AxisType = 'quantitative' | 'temporal';
-type TimeUnit = 'year' | 'yearmonth' | undefined; // or ...
+type TimeUnit = 'year' | undefined; // or ...
 
 export type LineChartProps = {
-  data: Omit<Data, 'csv'>;
+  data: Array<Array<string | number>> | string | { x: string; y: number }[];
   title?: string;
-  xAxis: string;
+  xAxis?: string;
   xAxisType?: AxisType;
-  xAxisTimeUnit?: TimeUnit;
-  yAxis: string | string[];
+  xAxisTimeUnit: TimeUnit;
+  yAxis?: string;
   yAxisType?: AxisType;
   fullWidth?: boolean;
-  symbol?: string;
 };
 
 export function LineChart({
-  data,
+  data = [],
+  fullWidth = false,
   title = '',
-  xAxis,
+  xAxis = 'x',
   xAxisType = 'temporal',
   xAxisTimeUnit = 'year', //  TODO: defaults to undefined would probably work better... keeping it as it's for compatibility purposes
-  yAxis,
+  yAxis = 'y',
   yAxisType = 'quantitative',
-  symbol,
 }: LineChartProps) {
-  const url = data.url;
-  const values = data.values;
   const [isLoading, setIsLoading] = useState<boolean>(false);
 
   //  By default, assumes data is an Array...
   const [specData, setSpecData] = useState<any>({ name: 'table' });
-  const isMultiYAxis = Array.isArray(yAxis);
 
   const spec = {
     $schema: 'https://vega.github.io/schema/vega-lite/v5.json',
@@ -47,17 +42,8 @@ export function LineChart({
       color: 'black',
       strokeWidth: 1,
       tooltip: true,
-      invalid: "break-paths"
     },
     data: specData,
-    ...(isMultiYAxis
-      ? {
-          transform: [
-            { fold: yAxis, as: ['key', 'value'] },
-            { filter: 'datum.value != null && datum.value != ""' }
-          ],
-        }
-      : {}),
     selection: {
       grid: {
         type: 'interval',
@@ -71,35 +57,20 @@ export function LineChart({
         type: xAxisType,
       },
       y: {
-        field: isMultiYAxis ? 'value' : yAxis,
+        field: yAxis,
         type: yAxisType,
       },
-      ...(symbol
-        ? {
-            color: {
-              field: symbol,
-              type: 'nominal',
-            },
-          }
-        : {}),
-      ...(isMultiYAxis
-        ? {
-            color: {
-              field: 'key',
-              type: 'nominal',
-            },
-          }
-        : {}),
     },
   } as any;
 
   useEffect(() => {
-    if (url) {
+    //  If data is string, assume it's a URL
+    if (typeof data === 'string') {
       setIsLoading(true);
 
       //  Manualy loading the data allows us to do other kinds
       //  of stuff later e.g. load a file partially
-      loadData(url).then((res: any) => {
+      loadData(data).then((res: any) => {
         setSpecData({ values: res, format: { type: 'csv' } });
         setIsLoading(false);
       });
@@ -107,8 +78,12 @@ export function LineChart({
   }, []);
 
   var vegaData = {};
-  if (values) {
-    vegaData = { table: values };
+  if (Array.isArray(data)) {
+    var dataObj;
+    dataObj = data.map((r) => {
+      return { x: r[0], y: r[1] };
+    });
+    vegaData = { table: dataObj };
   }
 
   return isLoading ? (
@@ -116,6 +91,6 @@ export function LineChart({
       <LoadingSpinner />
     </div>
   ) : (
-    <VegaLite data={vegaData} spec={spec} />
+    <VegaLite fullWidth={fullWidth} data={vegaData} spec={spec} />
   );
 }

packages/components/src/components/Map.tsx

@@ -1,8 +1,7 @@
-import { CSSProperties, useEffect, useState } from 'react';
+import { useEffect, useState } from 'react';
 import LoadingSpinner from './LoadingSpinner';
 import loadData from '../lib/loadData';
 import chroma from 'chroma-js';
-import { GeospatialData } from '../types/properties';
 import {
   MapContainer,
   TileLayer,
@@ -12,34 +11,10 @@ import {
 
 import 'leaflet/dist/leaflet.css';
 import * as L from 'leaflet';
-import providers from '../lib/tileLayerPresets';
-
-type VariantKeys<T> = T extends { variants: infer V }
-  ? {
-      [K in keyof V]: K extends string
-        ? `${K}` | `${K}.${VariantKeys<V[K]>}`
-        : never;
-    }[keyof V]
-  : never;
-
-type ProviderVariantKeys<T> = {
-  [K in keyof T]: K extends string
-    ? `${K}` | `${K}.${VariantKeys<T[K]>}`
-    : never;
-}[keyof T];
-
-type TileLayerPreset = ProviderVariantKeys<typeof providers> | 'custom';
-
-interface TileLayerSettings extends L.TileLayerOptions {
-  url?: string;
-  variant?: string | any;
-}
 
 export type MapProps = {
-  tileLayerName?: TileLayerPreset;
-  tileLayerOptions?: TileLayerSettings | undefined;
   layers: {
-    data: GeospatialData;
+    data: string | GeoJSON.GeoJSON;
     name: string;
     colorScale?: {
       starting: string;
@@ -50,29 +25,14 @@ export type MapProps = {
           propNames: string[];
         }
       | boolean;
+    _id?: number;
   }[];
   title?: string;
   center?: { latitude: number | undefined; longitude: number | undefined };
   zoom?: number;
-  style?: CSSProperties;
-  autoZoomConfiguration?: {
-    layerName: string;
-  };
 };
 
-const tileLayerDefaultName = process?.env
-  .NEXT_PUBLIC_MAP_TILE_LAYER_NAME as TileLayerPreset;
-
-const tileLayerDefaultOptions = Object.keys(process?.env)
-  .filter((key) => key.startsWith('NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_'))
-  .reduce((obj, key) => {
-    obj[key.split('NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_')[1]] = process.env[key];
-    return obj;
-  }, {}) as TileLayerSettings;
-
 export function Map({
-  tileLayerName = tileLayerDefaultName || 'OpenStreetMap',
-  tileLayerOptions,
   layers = [
     {
       data: null,
@@ -84,116 +44,23 @@ export function Map({
   center = { latitude: 45, longitude: 45 },
   zoom = 2,
   title = '',
-  style = {},
-  autoZoomConfiguration = undefined,
 }: MapProps) {
   const [isLoading, setIsLoading] = useState<boolean>(false);
   const [layersData, setLayersData] = useState<any>([]);
 
-  /*
-    tileLayerDefaultOptions
-    extract all environment variables thats starts with NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_.
-      the variables names are the same as the TileLayer object properties:
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_url:
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_attribution
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_accessToken
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_id
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_ext
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_bounds
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_maxZoom
-      - NEXT_PUBLIC_MAP_TILE_LAYER_OPTION_minZoom
-      see TileLayerOptions inteface
-     */
-
-  //tileLayerData prioritizes properties passed through component over those passed through .env variables
-  tileLayerOptions = Object.assign(tileLayerDefaultOptions, tileLayerOptions);
-
-  let provider = {
-    url: tileLayerOptions.url,
-    options: tileLayerOptions,
-  };
-
-  if (tileLayerName != 'custom') {
-    var parts = tileLayerName.split('.');
-    var providerName = parts[0];
-    var variantName: string = parts[1];
-
-    //make sure to declare a variant if url depends on a variant: assume first
-    if (providers[providerName].url?.includes('{variant}') && !variantName)
-      variantName = Object.keys(providers[providerName].variants)[0];
-
-    if (!providers[providerName]) {
-      throw 'No such provider (' + providerName + ')';
-    }
-
-    provider = {
-      url: providers[providerName].url,
-      options: providers[providerName].options,
-    };
-
-    // overwrite values in provider from variant.
-    if (variantName && 'variants' in providers[providerName]) {
-      if (!(variantName in providers[providerName].variants)) {
-        throw 'No such variant of ' + providerName + ' (' + variantName + ')';
-      }
-      var variant = providers[providerName].variants[variantName];
-      var variantOptions;
-      if (typeof variant === 'string') {
-        variantOptions = {
-          variant: variant,
-        };
-      } else {
-        variantOptions = variant.options;
-      }
-      provider = {
-        url: variant.url || provider.url,
-        options: L.Util.extend({}, provider.options, variantOptions),
-      };
-    }
-
-    var attributionReplacer = function (attr) {
-      if (attr.indexOf('{attribution.') === -1) {
-        return attr;
-      }
-      return attr.replace(
-        /\{attribution.(\w*)\}/g,
-        function (match: any, attributionName: string) {
-          match;
-          return attributionReplacer(
-            providers[attributionName].options.attribution
-          );
-        }
-      );
-    };
-
-    provider.options.attribution = attributionReplacer(
-      provider.options.attribution
-    );
-  }
-
-  var tileLayerData = L.Util.extend(
-    {
-      url: provider.url,
-    },
-    provider.options,
-    tileLayerOptions
-  );
-
   useEffect(() => {
     const loadDataPromises = layers.map(async (layer) => {
-      const url = layer.data.url;
-      const geojson = layer.data.geojson;
       let layerData: any;
 
-      if (url) {
+      if (typeof layer.data === 'string') {
         //  If "data" is string, assume it's a URL
         setIsLoading(true);
-        layerData = await loadData(url).then((res: any) => {
+        layerData = await loadData(layer.data).then((res: any) => {
           return JSON.parse(res);
         });
       } else {
         //  Else, expect raw GeoJSON
-        layerData = geojson;
+        layerData = layer.data;
       }
 
       if (layer.colorScale) {
@@ -225,12 +92,10 @@ export function Map({
     </div>
   ) : (
     <MapContainer
-      key={layersData}
       center={[center.latitude, center.longitude]}
       zoom={zoom}
       scrollWheelZoom={false}
       className="h-80 w-full"
-      style={style ?? {}}
       // @ts-ignore
       whenReady={(map: any) => {
         //  Enable zoom using scroll wheel
@@ -250,28 +115,12 @@ export function Map({
         };
 
         if (title) info.addTo(map.target);
-        if (!autoZoomConfiguration) return;
-
-        let layerToZoomBounds = L.latLngBounds(L.latLng(0, 0), L.latLng(0, 0));
-
-        layers.forEach((layer) => {
-          if (layer.name === autoZoomConfiguration.layerName) {
-            const data = layersData.find(
-              (layerData) => layerData.name === layer.name
-            )?.data;
-
-            if (data) {
-              layerToZoomBounds = L.geoJSON(data).getBounds();
-              return;
-            }
-          }
-        });
-
-        map.target.fitBounds(layerToZoomBounds);
       }}
     >
-      {tileLayerData.url && <TileLayer {...tileLayerData} />}
-
+      <TileLayer
+        attribution='&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors'
+        url="https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"
+      />
       <LayersControl position="bottomright">
         {layers.map((layer) => {
           const data = layersData.find(

packages/components/src/components/PdfViewer.tsx

@@ -1,24 +1,22 @@
 // Core viewer
 import { Viewer, Worker, SpecialZoomLevel } from '@react-pdf-viewer/core';
 import { defaultLayoutPlugin } from '@react-pdf-viewer/default-layout';
-import { Data } from '../types/properties';
 
 // Import styles
 import '@react-pdf-viewer/core/lib/styles/index.css';
 import '@react-pdf-viewer/default-layout/lib/styles/index.css';
 
 export interface PdfViewerProps {
-  data: Required<Pick<Data, 'url'>>;
+  url: string;
   layout: boolean;
   parentClassName?: string;
 }
 
 export function PdfViewer({
-  data,
+  url,
   layout = false,
-  parentClassName = 'h-screen',
+  parentClassName,
 }: PdfViewerProps) {
-  const url = data.url;
   const defaultLayoutPluginInstance = defaultLayoutPlugin();
   return (
     <Worker workerUrl="https://unpkg.com/pdfjs-dist@2.15.349/build/pdf.worker.js">

packages/components/src/components/Plotly.tsx

@@ -1,9 +0,0 @@
-import Plot, { PlotParams } from "react-plotly.js";
-
-export const Plotly: React.FC<PlotParams> = (props) => {
-  return (
-    <div>
-      <Plot {...props} />
-    </div>
-  );
-};

packages/components/src/components/PlotlyBarChart.tsx

@@ -1,153 +0,0 @@
-import { QueryClient, QueryClientProvider, useQuery } from 'react-query';
-import { Plotly } from './Plotly';
-import Papa, { ParseConfig } from 'papaparse';
-import LoadingSpinner from './LoadingSpinner';
-import { Data } from '../types/properties';
-
-const queryClient = new QueryClient();
-
-async function getCsv(url: string, bytes: number) {
-  const response = await fetch(url, {
-    headers: {
-      Range: `bytes=0-${bytes}`,
-    },
-  });
-  const data = await response.text();
-  return data;
-}
-
-async function parseCsv(
-  file: string,
-  parsingConfig: ParseConfig
-): Promise<any> {
-  return new Promise((resolve, reject) => {
-    Papa.parse(file, {
-      ...parsingConfig,
-      header: true,
-      dynamicTyping: true,
-      skipEmptyLines: true,
-      transform: (value: string): string => {
-        return value.trim();
-      },
-      complete: (results: any) => {
-        return resolve(results);
-      },
-      error: (error: any) => {
-        return reject(error);
-      },
-    });
-  });
-}
-
-export interface PlotlyBarChartProps {
-  data: Data;
-  uniqueId?: number;
-  bytes?: number;
-  parsingConfig?: ParseConfig;
-  xAxis: string;
-  yAxis: string;
-  // TODO: commented out because this doesn't work. I believe
-  // this would only make any difference on charts with multiple
-  // traces.
-  // lineLabel?: string;
-  title?: string;
-}
-
-export const PlotlyBarChart: React.FC<PlotlyBarChartProps> = ({
-  data,
-  bytes = 5132288,
-  parsingConfig = {},
-  xAxis,
-  yAxis,
-  // lineLabel,
-  title = '',
-}) => {
-  const uniqueId = Math.random();
-  return (
-    // Provide the client to your App
-    <QueryClientProvider client={queryClient}>
-      <PlotlyBarChartInner
-        data={data}
-        uniqueId={uniqueId}
-        bytes={bytes}
-        parsingConfig={parsingConfig}
-        xAxis={xAxis}
-        yAxis={yAxis}
-        // lineLabel={lineLabel ?? yAxis}
-        title={title}
-      />
-    </QueryClientProvider>
-  );
-};
-
-const PlotlyBarChartInner: React.FC<PlotlyBarChartProps> = ({
-  data,
-  uniqueId,
-  bytes,
-  parsingConfig,
-  xAxis,
-  yAxis,
-  // lineLabel,
-  title,
-}) => {
-  if (data.values) {
-    return (
-      <div className="w-full" style={{ height: '500px' }}>
-        <Plotly
-          layout={{
-            title,
-          }}
-          data={[
-            {
-              x: data.values.map((d) => d[xAxis]),
-              y: data.values.map((d) => d[yAxis]),
-              type: 'bar',
-              // name: lineLabel,
-            },
-          ]}
-        />
-      </div>
-    );
-  }
-  const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
-    ['dataCsv', data.url, uniqueId],
-    () => getCsv(data.url as string, bytes ?? 5132288),
-    { enabled: !!data.url }
-  );
-  const { data: parsedData, isLoading: isParsing } = useQuery(
-    ['dataPreview', csvString, uniqueId],
-    () =>
-      parseCsv(
-        data.csv ? (data.csv as string) : (csvString as string),
-        parsingConfig ?? {}
-      ),
-    { enabled: data.csv ? true : !!csvString }
-  );
-  if (isParsing || isDownloadingCSV)
-    <div className="w-full flex justify-center items-center h-[500px]">
-      <LoadingSpinner />
-    </div>;
-  if (parsedData)
-    return (
-      <div className="w-full" style={{ height: '500px' }}>
-        <Plotly
-          layout={{
-            title,
-          }}
-          data={[
-            {
-              x: parsedData.data.map((d: any) => d[xAxis]),
-              y: parsedData.data.map((d: any) => d[yAxis]),
-              type: 'bar',
-              // name: lineLabel, TODO: commented out because this doesn't work
-            },
-          ]}
-        />
-      </div>
-    );
-  return (
-    <div className="w-full flex justify-center items-center h-[500px]">
-      <LoadingSpinner />
-    </div>
-  );
-};

packages/components/src/components/PlotlyLineChart.tsx

@@ -1,155 +0,0 @@
-import { QueryClient, QueryClientProvider, useQuery } from 'react-query';
-import { Plotly } from './Plotly';
-import Papa, { ParseConfig } from 'papaparse';
-import LoadingSpinner from './LoadingSpinner';
-import { Data } from '../types/properties';
-
-const queryClient = new QueryClient();
-
-async function getCsv(url: string, bytes: number) {
-  const response = await fetch(url, {
-    headers: {
-      Range: `bytes=0-${bytes}`,
-    },
-  });
-  const data = await response.text();
-  return data;
-}
-
-async function parseCsv(
-  file: string,
-  parsingConfig: ParseConfig
-): Promise<any> {
-  return new Promise((resolve, reject) => {
-    Papa.parse(file, {
-      ...parsingConfig,
-      header: true,
-      dynamicTyping: true,
-      skipEmptyLines: true,
-      transform: (value: string): string => {
-        return value.trim();
-      },
-      complete: (results: any) => {
-        return resolve(results);
-      },
-      error: (error: any) => {
-        return reject(error);
-      },
-    });
-  });
-}
-
-export interface PlotlyLineChartProps {
-  data: Data;
-  bytes?: number;
-  parsingConfig?: ParseConfig;
-  xAxis: string;
-  yAxis: string;
-  lineLabel?: string;
-  title?: string;
-  uniqueId?: number;
-}
-
-export const PlotlyLineChart: React.FC<PlotlyLineChartProps> = ({
-  data,
-  bytes = 5132288,
-  parsingConfig = {},
-  xAxis,
-  yAxis,
-  lineLabel,
-  title = '',
-  uniqueId,
-}) => {
-  uniqueId = uniqueId ?? Math.random();
-  return (
-    // Provide the client to your App
-    <QueryClientProvider client={queryClient}>
-      <LineChartInner
-        data={data}
-        uniqueId={uniqueId}
-        bytes={bytes}
-        parsingConfig={parsingConfig}
-        xAxis={xAxis}
-        yAxis={yAxis}
-        lineLabel={lineLabel ?? yAxis}
-        title={title}
-      />
-    </QueryClientProvider>
-  );
-};
-
-const LineChartInner: React.FC<PlotlyLineChartProps> = ({
-  data,
-  uniqueId,
-  bytes,
-  parsingConfig,
-  xAxis,
-  yAxis,
-  lineLabel,
-  title,
-}) => {
-  const values = data.values;
-  const url = data.url;
-  const csv = data.csv;
-
-  if (values) {
-    return (
-      <div className="w-full" style={{ height: '500px' }}>
-        <Plotly
-          layout={{
-            title,
-          }}
-          data={[
-            {
-              x: values.map((d) => d[xAxis]),
-              y: values.map((d) => d[yAxis]),
-              mode: 'lines',
-              name: lineLabel,
-            },
-          ]}
-        />
-      </div>
-    );
-  }
-  const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
-    ['dataCsv', url, uniqueId],
-    () => getCsv(url as string, bytes ?? 5132288),
-    { enabled: !!url }
-  );
-  const { data: parsedData, isLoading: isParsing } = useQuery(
-    ['dataPreview', csvString, uniqueId],
-    () =>
-      parseCsv(
-        csv ? (csv as string) : (csvString as string),
-        parsingConfig ?? {}
-      ),
-    { enabled: csv ? true : !!csvString }
-  );
-  if (isParsing || isDownloadingCSV)
-    <div className="w-full flex justify-center items-center h-[500px]">
-      <LoadingSpinner />
-    </div>;
-  if (parsedData)
-    return (
-      <div className="w-full" style={{ height: '500px' }}>
-        <Plotly
-          layout={{
-            title,
-          }}
-          data={[
-            {
-              x: parsedData.data.map((d: any) => d[xAxis]),
-              y: parsedData.data.map((d: any) => d[yAxis]),
-              mode: 'lines',
-              name: lineLabel,
-            },
-          ]}
-        />
-      </div>
-    );
-  return (
-    <div className="w-full flex justify-center items-center h-[500px]">
-      <LoadingSpinner />
-    </div>
-  );
-};

packages/components/src/components/Table.tsx

@@ -6,8 +6,6 @@ import {
   getFilteredRowModel,
   getPaginationRowModel,
   getSortedRowModel,
-  PaginationState,
-  Table as ReactTable,
   useReactTable,
 } from '@tanstack/react-table';
 
@@ -27,19 +25,12 @@ import DebouncedInput from './DebouncedInput';
 import loadData from '../lib/loadData';
 import LoadingSpinner from './LoadingSpinner';
 
-export type TableData = { cols: {key: string, name: string}[]; data: any[]; total: number };
-
 export type TableProps = {
   data?: Array<{ [key: string]: number | string }>;
   cols?: Array<{ [key: string]: string }>;
   csv?: string;
   url?: string;
   fullWidth?: boolean;
-  datastoreConfig?: {
-    dataStoreURI: string;
-    rowsPerPage?: number;
-    dataMapperFn: (data) => Promise<TableData> | TableData;
-  };
 };
 
 export const Table = ({
@@ -48,28 +39,8 @@ export const Table = ({
   csv = '',
   url = '',
   fullWidth = false,
-  datastoreConfig,
 }: TableProps) => {
   const [isLoading, setIsLoading] = useState<boolean>(false);
-  const [pageMap, setPageMap] = useState(new Map<number, boolean>());
-  const {
-    dataMapperFn,
-    dataStoreURI,
-    rowsPerPage = 10,
-  } = datastoreConfig ?? {};
-
-  const [globalFilter, setGlobalFilter] = useState('');
-  const [isLoadingPage, setIsLoadingPage] = useState<boolean>(false);
-  const [totalOfRows, setTotalOfRows] = useState<number>(0);
-
-  const [{ pageIndex, pageSize }, setPagination] = useState<PaginationState>({
-    pageIndex: 0,
-    pageSize: rowsPerPage,
-  });
-
-  const [lastIndex, setLastIndex] = useState(pageSize);
-  const [startIndex, setStartIndex] = useState(0);
-  const [hasSorted, setHasSorted] = useState(false);
 
   if (csv) {
     const out = parseCsv(csv);
@@ -91,56 +62,21 @@ export const Table = ({
     );
   }, [data, cols]);
 
-  let table: ReactTable<unknown>;
+  const [globalFilter, setGlobalFilter] = useState('');
 
-  if (datastoreConfig) {
-    useEffect(() => {
-      setIsLoading(true);
-      fetch(`${dataStoreURI}&limit=${rowsPerPage}&offset=0`)
-        .then((res) => res.json())
-        .then(async (res) => {
-          const { data, cols, total } = await dataMapperFn(res);
-          setData(data);
-          setCols(cols);
-          setTotalOfRows(Math.ceil(total / rowsPerPage));
-          pageMap.set(0, true);
-        })
-        .finally(() => setIsLoading(false));
-    }, [dataStoreURI]);
-
-    table = useReactTable({
-      data,
-      pageCount: totalOfRows,
-      columns: tableCols,
-      getCoreRowModel: getCoreRowModel(),
-      state: {
-        pagination: { pageIndex, pageSize },
-      },
-      getFilteredRowModel: getFilteredRowModel(),
-      manualPagination: true,
-      onPaginationChange: setPagination,
-      getSortedRowModel: getSortedRowModel(),
-    });
-
-    useEffect(() => {
-      if (!hasSorted) return;
-      queryDataByText(globalFilter);
-    }, [table.getState().sorting]);
-  } else {
-    table = useReactTable({
-      data,
-      columns: tableCols,
-      getCoreRowModel: getCoreRowModel(),
-      state: {
-        globalFilter,
-      },
-      globalFilterFn: globalFilterFn,
-      onGlobalFilterChange: setGlobalFilter,
-      getFilteredRowModel: getFilteredRowModel(),
-      getPaginationRowModel: getPaginationRowModel(),
-      getSortedRowModel: getSortedRowModel(),
-    });
-  }
+  const table = useReactTable({
+    data,
+    columns: tableCols,
+    getCoreRowModel: getCoreRowModel(),
+    state: {
+      globalFilter,
+    },
+    globalFilterFn: globalFilterFn,
+    onGlobalFilterChange: setGlobalFilter,
+    getFilteredRowModel: getFilteredRowModel(),
+    getPaginationRowModel: getPaginationRowModel(),
+    getSortedRowModel: getSortedRowModel(),
+  });
 
   useEffect(() => {
     if (url) {
@@ -155,70 +91,6 @@ export const Table = ({
     }
   }, [url]);
 
-  const queryDataByText = (filter) => {
-    setIsLoadingPage(true);
-    const sortedParam = getSortParam();
-    fetch(
-      `${dataStoreURI}&limit=${rowsPerPage}&offset=0&q=${filter}${sortedParam}`
-    )
-      .then((res) => res.json())
-      .then(async (res) => {
-        const { data, total = 0 } = await dataMapperFn(res);
-        setTotalOfRows(Math.ceil(total / rowsPerPage));
-        setData(data);
-        const newMap = new Map();
-        newMap.set(0, true);
-        setPageMap(newMap);
-        table.setPageIndex(0);
-        setStartIndex(0);
-        setLastIndex(pageSize);
-      })
-      .finally(() => setIsLoadingPage(false));
-  };
-
-  const getSortParam = () => {
-    const sort = table.getState().sorting;
-    return sort.length == 0
-      ? ``
-      : '&sort=' +
-          sort
-            .map(
-              (x, i) =>
-                `${x.id}${
-                  i === sort.length - 1 ? (x.desc ? ` desc` : ` asc`) : `,`
-                }`
-            )
-            .reduce((x1, x2) => x1 + x2);
-  };
-
-  const queryPaginatedData = (newPageIndex) => {
-    let newStartIndex = newPageIndex * pageSize;
-    setStartIndex(newStartIndex);
-    setLastIndex(newStartIndex + pageSize);
-
-    if (!pageMap.get(newPageIndex)) pageMap.set(newPageIndex, true);
-    else return;
-
-    const sortedParam = getSortParam();
-
-    setIsLoadingPage(true);
-    fetch(
-      `${dataStoreURI}&limit=${rowsPerPage}&offset=${
-        newStartIndex + pageSize
-      }&q=${globalFilter}${sortedParam}`
-    )
-      .then((res) => res.json())
-      .then(async (res) => {
-        const { data: responseData } = await dataMapperFn(res);
-        responseData.forEach((e) => {
-          data[newStartIndex] = e;
-          newStartIndex++;
-        });
-        setData([...data]);
-      })
-      .finally(() => setIsLoadingPage(false));
-  };
-
   return isLoading ? (
     <div className="w-full h-full min-h-[500px] flex items-center justify-center">
       <LoadingSpinner />
@@ -227,10 +99,7 @@ export const Table = ({
     <div className={`${fullWidth ? 'w-[90vw] ml-[calc(50%-45vw)]' : 'w-full'}`}>
       <DebouncedInput
         value={globalFilter ?? ''}
-        onChange={(value: any) => {
-          if (datastoreConfig) queryDataByText(String(value));
-          setGlobalFilter(String(value));
-        }}
+        onChange={(value: any) => setGlobalFilter(String(value))}
         className="p-2 text-sm shadow border border-block"
         placeholder="Search all columns..."
       />
@@ -245,10 +114,7 @@ export const Table = ({
                       className: h.column.getCanSort()
                         ? 'cursor-pointer select-none'
                         : '',
-                      onClick: (v) => {
-                        setHasSorted(true);
-                        h.column.getToggleSortingHandler()(v);
-                      },
+                      onClick: h.column.getToggleSortingHandler(),
                     }}
                   >
                     {flexRender(h.column.columnDef.header, h.getContext())}
@@ -269,28 +135,15 @@ export const Table = ({
           ))}
         </thead>
         <tbody>
-          {datastoreConfig && isLoadingPage ? (
-            <tr>
-              <td colSpan={cols.length} rowSpan={cols.length}>
-                <div className="w-full h-full flex items-center justify-center pt-6">
-                  <LoadingSpinner />
-                </div>
-              </td>
+          {table.getRowModel().rows.map((r) => (
+            <tr key={r.id} className="border-b border-b-slate-200">
+              {r.getVisibleCells().map((c) => (
+                <td key={c.id} className="py-2">
+                  {flexRender(c.column.columnDef.cell, c.getContext())}
+                </td>
+              ))}
             </tr>
-          ) : (
-            (datastoreConfig
-              ? table.getRowModel().rows.slice(startIndex, lastIndex)
-              : table.getRowModel().rows
-            ).map((r) => (
-              <tr key={r.id} className="border-b border-b-slate-200">
-                {r.getVisibleCells().map((c) => (
-                  <td key={c.id} className="py-2">
-                    {flexRender(c.column.columnDef.cell, c.getContext())}
-                  </td>
-                ))}
-              </tr>
-            ))
-          )}
+          ))}
         </tbody>
       </table>
       <div className="flex gap-2 items-center justify-center mt-10">
@@ -298,10 +151,7 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanPreviousPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => {
-            if (datastoreConfig) queryPaginatedData(0);
-            table.setPageIndex(0);
-          }}
+          onClick={() => table.setPageIndex(0)}
           disabled={!table.getCanPreviousPage()}
         >
           <ChevronDoubleLeftIcon />
@@ -310,12 +160,7 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanPreviousPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => {
-            if (datastoreConfig) {
-              queryPaginatedData(table.getState().pagination.pageIndex - 1);
-            }
-            table.previousPage();
-          }}
+          onClick={() => table.previousPage()}
           disabled={!table.getCanPreviousPage()}
         >
           <ChevronLeftIcon />
@@ -331,11 +176,7 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanNextPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => {
-            if (datastoreConfig)
-              queryPaginatedData(table.getState().pagination.pageIndex + 1);
-            table.nextPage();
-          }}
+          onClick={() => table.nextPage()}
           disabled={!table.getCanNextPage()}
         >
           <ChevronRightIcon />
@@ -344,11 +185,7 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanNextPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => {
-            const pageIndexToNavigate = table.getPageCount() - 1;
-            if (datastoreConfig) queryPaginatedData(pageIndexToNavigate);
-            table.setPageIndex(pageIndexToNavigate);
-          }}
+          onClick={() => table.setPageIndex(table.getPageCount() - 1)}
           disabled={!table.getCanNextPage()}
         >
           <ChevronDoubleRightIcon />

packages/components/src/components/Vega.tsx

@@ -1,7 +1,6 @@
 //  Wrapper for the Vega component
 import { Vega as VegaOg } from "react-vega";
-import { VegaProps } from "react-vega/lib/Vega";
 
-export function Vega(props: VegaProps) {
+export function Vega(props) {
   return <VegaOg {...props} />;
 }

packages/components/src/components/VegaLite.tsx

@@ -1,9 +1,8 @@
 //  Wrapper for the Vega Lite component
-import { VegaLite as VegaLiteOg } from 'react-vega';
-import { VegaLiteProps } from 'react-vega/lib/VegaLite';
-import applyFullWidthDirective from '../lib/applyFullWidthDirective';
+import { VegaLite as VegaLiteOg } from "react-vega";
+import applyFullWidthDirective from "../lib/applyFullWidthDirective";
 
-export function VegaLite(props: VegaLiteProps) {
+export function VegaLite(props) {
   const Component = applyFullWidthDirective({ Component: VegaLiteOg });
 
   return <Component {...props} />;

packages/components/src/index.ts

@@ -1,17 +1,10 @@
+export * from './components/Table';
 export * from './components/Catalog';
 export * from './components/LineChart';
 export * from './components/Vega';
 export * from './components/VegaLite';
 export * from './components/FlatUiTable';
+export * from './components/OpenLayers/OpenLayers';
 export * from './components/Map';
 export * from './components/PdfViewer';
 export * from "./components/Excel";
-export * from "./components/Iframe";
-export * from "./components/Plotly";
-export * from "./components/PlotlyLineChart";
-export * from "./components/PlotlyBarChart";
-// NOTE: components that are hidden for now
-// TODO: deprecate those components?
-// export * from './components/Table';
-// export * from "./components/BucketViewer";
-// export * from './components/OpenLayers/OpenLayers';

packages/components/src/types/properties.ts

@@ -1,18 +0,0 @@
-/*
- * All components should use this interface for
- * its data property.
- * Based on vega.
- *
- */
-
-type URL = string; // Just in case we want to transform it into an object with configurations
-export interface Data {
-  url?: URL;
-  values?: { [key: string]: number | string | null | undefined }[];
-  csv?: string;
-}
-
-export interface GeospatialData {
-  url?: URL;
-  geojson?: GeoJSON.GeoJSON;
-}

packages/components/stories/BucketViewer.bkp.ts

@@ -1,100 +0,0 @@
-// NOTE: this component was renamed with .bkp so that it's hidden
-// from the Storybook app
-
-import { type Meta, type StoryObj } from '@storybook/react';
-
-import {
-  BucketViewer,
-  BucketViewerProps,
-} from '../src/components/BucketViewer';
-import LoadingSpinner from '../src/components/LoadingSpinner';
-
-// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
-const meta: Meta = {
-  title: 'Components/BucketViewer',
-  component: BucketViewer,
-  tags: ['autodocs'],
-  argTypes: {
-    domain: {
-      description: 'Bucket domain URI',
-    },
-    suffix: {
-      description: 'Suffix of bucket domain',
-    },
-    downloadConfig: {
-      description: `Bucket file download configuration`,
-    },
-    filterState: {
-      description: `State with values used to filter the bucket files`,
-    },
-    paginationConfig: {
-      description: `Configuration to show and stylise the pagination on the component`,
-    },
-  },
-};
-
-export default meta;
-
-type Story = StoryObj<BucketViewerProps>;
-
-// More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
-export const Normal: Story = {
-  name: 'Bucket viewer',
-  args: {
-    domain: 'https://ssen-smart-meter.datopian.workers.dev',
-    suffix: '/',
-    dataMapperFn: async (rawData: Response) => {
-      const result = await rawData.json();
-      return result.objects.map((e) => ({
-        downloadFileUri: e.downloadLink,
-        fileName: e.key.replace(/^(\w+\/)/g, ''),
-        dateProps: {
-          date: new Date(e.uploaded),
-          dateFormatter: (date) => date.toLocaleDateString(),
-        },
-      }));
-    },
-  },
-};
-
-export const WithPagination: Story = {
-  name: 'With pagination',
-  args: {
-    domain: 'https://ssen-smart-meter.datopian.workers.dev',
-    suffix: '/',
-    paginationConfig: {
-      itemsPerPage: 3,
-    },
-    dataMapperFn: async (rawData: Response) => {
-      const result = await rawData.json();
-      return result.objects.map((e) => ({
-        downloadFileUri: e.downloadLink,
-        fileName: e.key.replace(/^(\w+\/)/g, ''),
-        dateProps: {
-          date: new Date(e.uploaded),
-          dateFormatter: (date) => date.toLocaleDateString(),
-        },
-      }));
-    },
-  },
-};
-
-export const WithComponentOnHoverOfEachBucketFile: Story = {
-  name: 'With component on hover of each bucket file',
-  args: {
-    domain: 'https://ssen-smart-meter.datopian.workers.dev',
-    suffix: '/',
-    downloadConfig: { hoverOfTheFileComponent: `HOVER COMPONENT` },
-    dataMapperFn: async (rawData: Response) => {
-      const result = await rawData.json();
-      return result.objects.map((e) => ({
-        downloadFileUri: e.downloadLink,
-        fileName: e.key.replace(/^(\w+\/)/g, ''),
-        dateProps: {
-          date: new Date(e.uploaded),
-          dateFormatter: (date) => date.toLocaleDateString(),
-        },
-      }));
-    },
-  },
-};

packages/components/stories/Catalog.stories.ts

@@ -10,14 +10,11 @@ const meta: Meta = {
   argTypes: {
     datasets: {
       description:
-        "Array of items to be displayed on the searchable list. Must have the following properties: \n\n \
-`_id`: item's unique id \n\n \
-`url_path`: href of the item \n\n \
-`metadata`: object with a `title` property, that will be displayed as the title of the item, together with any other custom fields that might or not be faceted.",
+        'Lists of datasets to be displayed in the list, will usually be automatically available',
     },
     facets: {
       description:
-        "Array of strings, which are name of properties in the datasets' `metadata`, which are going to be faceted.",
+        'List of frontmatter fields that should be used as filters, needs to match exactly with the field name',
     },
   },
 };
@@ -34,42 +31,7 @@ export const WithoutFacets: Story = {
       {
         _id: '07026b22d49916754df1dc8ffb9ccd1c31878aae',
         url_path: 'dataset-4',
-        metadata: {
-          title: 'Detecting Abusive Albanian',
-        },
-      },
-      {
-        _id: '42c86cf3c4fbbab11d91c2a7d6dcb8f750bc4e19',
-        url_path: 'dataset-1',
-        metadata: {
-          title: 'AbuseEval v1.0',
-        },
-      },
-      {
-        _id: '80001dd32a752421fdcc64e91fbd237dc31d6bb3',
-        url_path: 'dataset-2',
-        metadata: {
-          title:
-            'Abusive Language Detection on Arabic Social Media (Al Jazeera)',
-        },
-      },
-      {
-        _id: '96649d05d8193f4333b10015af76c6562971bd8c',
-        url_path: 'dataset-3',
-        metadata: {
-          title: 'CoRAL: a Context-aware Croatian Abusive Language Dataset',
-        },
-      },
-    ],
-  },
-};
-export const WithFacets: Story = {
-  name: 'Catalog with facets',
-  args: {
-    datasets: [
-      {
-        _id: '07026b22d49916754df1dc8ffb9ccd1c31878aae',
-        url_path: 'dataset-4',
+        file_path: 'content/dataset-4/index.md',
         metadata: {
           title: 'Detecting Abusive Albanian',
           'link-to-publication': 'https://arxiv.org/abs/2107.13592',
@@ -158,6 +120,107 @@ export const WithFacets: Story = {
         },
       },
     ],
-    facets: ['language', 'platform'],
   },
 };
+;
+
+export const WithFacets: Story = {
+  name: 'Catalog with facets',
+  args: {
+    datasets: [
+      {
+        _id: '07026b22d49916754df1dc8ffb9ccd1c31878aae',
+        url_path: 'dataset-4',
+        file_path: 'content/dataset-4/index.md',
+        metadata: {
+          title: 'Detecting Abusive Albanian',
+          'link-to-publication': 'https://arxiv.org/abs/2107.13592',
+          'link-to-data': 'https://doi.org/10.6084/m9.figshare.19333298.v1',
+          'task-description':
+            'Hierarchical (offensive/not; untargeted/targeted; person/group/other)',
+          'details-of-task':
+            'Detect and categorise abusive language in social media data',
+          'size-of-dataset': 11874,
+          'percentage-abusive': 13.2,
+          language: 'Albanian',
+          'level-of-annotation': ['Posts'],
+          platform: ['Instagram', 'Youtube'],
+          medium: ['Text'],
+          reference:
+            'Nurce, E., Keci, J., Derczynski, L., 2021. Detecting Abusive Albanian. arXiv:2107.13592',
+        },
+      },
+      {
+        _id: '42c86cf3c4fbbab11d91c2a7d6dcb8f750bc4e19',
+        url_path: 'dataset-1',
+        file_path: 'content/dataset-1/index.md',
+        metadata: {
+          title: 'AbuseEval v1.0',
+          'link-to-publication':
+            'http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.760.pdf',
+          'link-to-data': 'https://github.com/tommasoc80/AbuseEval',
+          'task-description':
+            'Explicitness annotation of offensive and abusive content',
+          'details-of-task':
+            'Enriched versions of the OffensEval/OLID dataset with the distinction of explicit/implicit offensive messages and the new dimension for abusive messages. Labels for offensive language: EXPLICIT, IMPLICT, NOT; Labels for abusive language: EXPLICIT, IMPLICT, NOTABU',
+          'size-of-dataset': 14100,
+          'percentage-abusive': 20.75,
+          language: 'English',
+          'level-of-annotation': ['Tweets'],
+          platform: ['Twitter'],
+          medium: ['Text'],
+          reference:
+            'Caselli, T., Basile, V., Jelena, M., Inga, K., and Michael, G. 2020. "I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language". The 12th Language Resources and Evaluation Conference (pp. 6193-6202). European Language Resources Association.',
+        },
+      },
+      {
+        _id: '80001dd32a752421fdcc64e91fbd237dc31d6bb3',
+        url_path: 'dataset-2',
+        file_path: 'content/dataset-2/index.md',
+        metadata: {
+          title:
+            'Abusive Language Detection on Arabic Social Media (Al Jazeera)',
+          'link-to-publication': 'https://www.aclweb.org/anthology/W17-3008',
+          'link-to-data':
+            'http://alt.qcri.org/~hmubarak/offensive/AJCommentsClassification-CF.xlsx',
+          'task-description':
+            'Ternary (Obscene, Offensive but not obscene, Clean)',
+          'details-of-task': 'Incivility',
+          'size-of-dataset': 32000,
+          'percentage-abusive': 0.81,
+          language: 'Arabic',
+          'level-of-annotation': ['Posts'],
+          platform: ['AlJazeera'],
+          medium: ['Text'],
+          reference:
+            'Mubarak, H., Darwish, K. and Magdy, W., 2017. Abusive Language Detection on Arabic Social Media. In: Proceedings of the First Workshop on Abusive Language Online. Vancouver, Canada: Association for Computational Linguistics, pp.52-56.',
+        },
+      },
+      {
+        _id: '96649d05d8193f4333b10015af76c6562971bd8c',
+        url_path: 'dataset-3',
+        file_path: 'content/dataset-3/index.md',
+        metadata: {
+          title: 'CoRAL: a Context-aware Croatian Abusive Language Dataset',
+          'link-to-publication':
+            'https://aclanthology.org/2022.findings-aacl.21/',
+          'link-to-data':
+            'https://github.com/shekharRavi/CoRAL-dataset-Findings-of-the-ACL-AACL-IJCNLP-2022',
+          'task-description':
+            'Multi-class based on context dependency categories (CDC)',
+          'details-of-task': 'Detectioning CDC from abusive comments',
+          'size-of-dataset': 2240,
+          'percentage-abusive': 100,
+          language: 'Croatian',
+          'level-of-annotation': ['Posts'],
+          platform: ['Posts'],
+          medium: ['Newspaper Comments'],
+          reference:
+            'Ravi Shekhar, Mladen Karan and Matthew Purver (2022). CoRAL: a Context-aware Croatian Abusive Language Dataset. Findings of the ACL: AACL-IJCNLP.',
+        },
+      },
+    ],
+    facets: ['language', 'platform']
+  },
+};
+;

packages/components/stories/Excel.stories.ts

@@ -4,13 +4,13 @@ import { Excel, ExcelProps } from '../src/components/Excel';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Tabular/Excel',
+  title: 'Components/Excel',
   component: Excel,
   tags: ['autodocs'],
   argTypes: {
-    data: {
+    url: {
       description:
-        'Object with a `url` property pointing to the Excel file to be displayed, e.g.: `{ url: "https://url.to/data.csv" }`',
+        'Url of the file to be displayed e.g.: "https://url.to/data.csv"',
     },
   },
 };
@@ -22,17 +22,13 @@ type Story = StoryObj<ExcelProps>;
 export const SingleSheet: Story = {
   name: 'Excel file with just one sheet',
   args: {
-    data: {
-      url: 'https://sheetjs.com/pres.xlsx',
-    },
+    url: 'https://sheetjs.com/pres.xlsx',
   },
 };
 
 export const MultipleSheet: Story = {
   name: 'Excel file with multiple sheets',
   args: {
-    data: {
-      url: 'https://storage.portaljs.org/IC-Gantt-Chart-Project-Template-8857.xlsx',
-    },
+    url: 'https://storage.portaljs.org/IC-Gantt-Chart-Project-Template-8857.xlsx',
   },
 };

packages/components/stories/FlatUiTable.stories.ts

@@ -4,31 +4,29 @@ import { FlatUiTable, FlatUiTableProps } from '../src/components/FlatUiTable';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Tabular/FlatUiTable',
+  title: 'Components/FlatUiTable',
   component: FlatUiTable,
   tags: ['autodocs'],
   argTypes: {
     data: {
       description:
-        'Data to be displayed. \n\n \
-Must be an object with one of the following properties: `url`, `values` or `csv` \n\n \
-`url`: URL pointing to a CSV file. \n\n \
-`values`: array of objects. \n\n \
-`csv`: string with valid CSV. \n\n \
-',
+        'Data to be displayed in the table, must be setup as an array of key value pairs',
+    },
+    csv: {
+      description: 'CSV data as string.',
+    },
+    url: {
+      description:
+        'Fetch the data from a CSV file remotely. only the first 5MB of data will be displayed',
     },
     bytes: {
       description:
-        'Fetch the data from a CSV file remotely. Only the first <bytes> of data will be displayed. Defaults to 5MB.',
+        'Fetch the data from a CSV file remotely. only the first <bytes> of data will be displayed',
     },
     parsingConfig: {
       description:
         'Configuration for parsing the CSV data. See https://www.papaparse.com/docs#config for more details',
     },
-    uniqueId: {
-      description:
-        'Provide a unique ID to help with cache revalidation of the fetched data.',
-    },
   },
 };
 
@@ -38,40 +36,34 @@ type Story = StoryObj<FlatUiTableProps>;
 
 // More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
 export const FromColumnsAndData: Story = {
-  name: 'Table from array or objects',
+  name: 'Table data',
   args: {
-    data: {
-      values: [
-        { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
-        { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
-        { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
-        { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
-        { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
-        { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
-        { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
-      ],
-    },
+    data: [
+      { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
+      { id: 2, lastName: 'Lannister', firstName: 'Cersei', age: 42 },
+      { id: 3, lastName: 'Lannister', firstName: 'Jaime', age: 45 },
+      { id: 4, lastName: 'Stark', firstName: 'Arya', age: 16 },
+      { id: 7, lastName: 'Clifford', firstName: 'Ferrara', age: 44 },
+      { id: 8, lastName: 'Frances', firstName: 'Rossini', age: 36 },
+      { id: 9, lastName: 'Roxie', firstName: 'Harvey', age: 65 },
+    ],
   },
 };
 
 export const FromRawCSV: Story = {
-  name: 'Table from inline CSV',
+  name: 'Table from raw CSV',
   args: {
-    data: {
-      csv: `
+    rawCsv: `
     Year,Temp Anomaly
     1850,-0.418
     2020,0.923
     `,
-    },
   },
 };
 
 export const FromURL: Story = {
   name: 'Table from URL',
   args: {
-    data: {
-      url: 'https://storage.openspending.org/alberta-budget/__os_imported__alberta_total.csv',
-    },
+    url: 'https://storage.openspending.org/alberta-budget/__os_imported__alberta_total.csv',
   },
 };

packages/components/stories/Iframe.stories.ts

@@ -1,33 +0,0 @@
-import { type Meta, type StoryObj } from '@storybook/react';
-
-import { Iframe, IframeProps } from '../src/components/Iframe';
-
-const meta: Meta = {
-  title: 'Components/Embedding/Iframe',
-  component: Iframe,
-  tags: ['autodocs'],
-  argTypes: {
-    data: {
-      description:
-        'Object with a `url` property pointing to the page to be embeded.',
-    },
-    style: {
-      description:
-        'Style object of the component. See example at https://react.dev/learn#displaying-data. Defaults to `{ width: "100%", height: "100%" }`',
-    },
-  },
-};
-
-export default meta;
-
-type Story = StoryObj<IframeProps>;
-
-export const Normal: Story = {
-  name: 'Iframe',
-  args: {
-    data: {
-      url: 'https://app.powerbi.com/view?r=eyJrIjoiYzBmN2Q2MzYtYzE3MS00ODkxLWE5OWMtZTQ2MjBlMDljMDk4IiwidCI6Ijk1M2IwZjgzLTFjZTYtNDVjMy04MmM5LTFkODQ3ZTM3MjMzOSIsImMiOjh9',
-    },
-    style: { width: `100%`, height: `600px` },
-  },
-};

packages/components/stories/Introduction.mdx

@@ -4,6 +4,6 @@ import { Meta } from '@storybook/blocks';
 
 # Welcome to the PortalJS components guide
 
-**Official Website:** [portaljs.com](https://portaljs.com)  
-**Docs:** [portaljs.com/opensource](https://portaljs.com/opensource)  
+**Official Website:** [portaljs.org](https://portaljs.org)  
+**Docs:** [portaljs.org/docs](https://portaljs.org/docs)  
 **GitHub:** [github.com/datopian/portaljs](https://github.com/datopian/portaljs)

packages/components/stories/LineChart.stories.ts

@@ -4,40 +4,37 @@ import { LineChart, LineChartProps } from '../src/components/LineChart';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Charts/LineChart',
+  title: 'Components/LineChart',
   component: LineChart,
   tags: ['autodocs'],
   argTypes: {
     data: {
       description:
-        'Data to be displayed. \n\n \
-Must be an object with one of the following properties: `url` or `values` \n\n \
-`url`: URL pointing to a CSV file. \n\n \
-`values`: array of objects \n\n',
+        'Data to be displayed.\n\n E.g.: [["1990", 1], ["1991", 2]] \n\nOR\n\n "https://url.to/data.csv"',
     },
     title: {
-      description: 'Title to display on the chart.',
+      description: 'Title to display on the chart. Optional.',
     },
     xAxis: {
       description:
-        'Name of the column header or object property that represents the X-axis on the data.',
+        'Name of the X axis on the data. Required when the "data" parameter is an URL.',
     },
     xAxisType: {
-      description: 'Type of the X-axis.',
+      description: 'Type of the X axis',
     },
     xAxisTimeUnit: {
-      description: 'Time unit of the X-axis, in case its type is `temporal.`',
+      description: 'Time unit of the X axis (optional)',
     },
     yAxis: {
       description:
-        'Name of the column headers or object properties that represent the Y-axis on the data.',
+        'Name of the Y axis on the data. Required when the "data" parameter is an URL.',
     },
     yAxisType: {
-      description: 'Type of the Y-axis',
+      description: 'Type of the Y axis',
     },
-    symbol: {
+    fullWidth: {
       description:
-        'Name of the column header or object property that represents a series for multiple series.',
+        'Whether the component should be rendered as full bleed or not',
     },
   },
 };
@@ -50,112 +47,22 @@ type Story = StoryObj<LineChartProps>;
 export const FromDataPoints: Story = {
   name: 'Line chart from array of data points',
   args: {
-    data: {
-      values: [
-        { year: '1850', value: -0.41765878 },
-        { year: '1851', value: -0.2333498 },
-        { year: '1852', value: -0.22939907 },
-        { year: '1853', value: -0.27035445 },
-        { year: '1854', value: -0.29163003 },
-      ],
-    },
-    xAxis: 'year',
-    yAxis: 'value',
-  },
-};
-
-export const MultiSeries: Story = {
-  name: 'Line chart with multiple series (specifying symbol)',
-  args: {
-    data: {
-      values: [
-        { year: '1850', value: -0.41765878, z: 'A' },
-        { year: '1851', value: -0.2333498, z: 'A' },
-        { year: '1852', value: -0.22939907, z: 'A' },
-        { year: '1853', value: -0.27035445, z: 'A' },
-        { year: '1854', value: -0.29163003, z: 'A' },
-        { year: '1850', value: -0.42993882, z: 'B' },
-        { year: '1851', value: -0.30365549, z: 'B' },
-        { year: '1852', value: -0.27905189, z: 'B' },
-        { year: '1853', value: -0.22939704, z: 'B' },
-        { year: '1854', value: -0.25688013, z: 'B' },
-        { year: '1850', value: -0.4757164, z: 'C' },
-        { year: '1851', value: -0.41971018, z: 'C' },
-        { year: '1852', value: -0.40724799, z: 'C' },
-        { year: '1853', value: -0.45049156, z: 'C' },
-        { year: '1854', value: -0.41896583, z: 'C' },
-      ],
-    },
-    xAxis: 'year',
-    yAxis: 'value',
-    symbol: 'z',
-  },
-};
-
-export const MultiColumns: Story = {
-  name: 'Line chart with multiple series (with multiple columns)',
-  args: {
-    data: {
-      values: [
-        { year: '1850', A: -0.41765878, B: -0.42993882, C: -0.4757164 },
-        { year: '1851', A: -0.2333498, B: -0.30365549, C: -0.41971018 },
-        { year: '1852', A: -0.22939907, B: -0.27905189, C: -0.40724799 },
-        { year: '1853', A: -0.27035445, B: -0.22939704, C: -0.45049156 },
-        { year: '1854', A: -0.29163003, B: -0.25688013, C: -0.41896583 },
-      ],
-    },
-    xAxis: 'year',
-    yAxis: ['A', 'B', 'C'],
+    data: [
+      ['1850', -0.41765878],
+      ['1851', -0.2333498],
+      ['1852', -0.22939907],
+      ['1853', -0.27035445],
+      ['1854', -0.29163003],
+    ],
   },
 };
 
 export const FromURL: Story = {
   name: 'Line chart from URL',
   args: {
-    data: {
-      url: 'https://raw.githubusercontent.com/datasets/oil-prices/main/data/wti-year.csv',
-    },
     title: 'Oil Price x Year',
+    data: 'https://raw.githubusercontent.com/datasets/oil-prices/main/data/wti-year.csv',
     xAxis: 'Date',
     yAxis: 'Price',
   },
 };
-
-
-// export const FromURLMulti: Story = {
-//   name: 'Line chart from URL Multi Column',
-//   args: {
-//     data: {
-//       url: 'https://raw.githubusercontent.com/datasets/sea-level-rise/refs/heads/main/data/epa-sea-level.csv',
-//     },
-//     title: 'Sea Level Rise (1880-2023)',
-//     xAxis: 'Year',
-//     yAxis: ["CSIRO Adjusted Sea Level", "NOAA Adjusted Sea Level"],
-//     xAxisType: 'temporal',
-//     xAxisTimeUnit: 'year',
-//     yAxisType: 'quantitative'
-//   },
-// };
-
-// export const MultipleSeriesMissingValues: Story = {
-//   name: 'Line chart with missing values',
-//   args: {
-//     data: {
-//       values: [
-//         { year: '2020', seriesA: 10, seriesB: 15 },
-//         { year: '2021', seriesA: 20 },  // seriesB missing
-//         { year: '2022', seriesA: 15 },  // seriesB missing
-//         { year: '2023', seriesB: 30 },  // seriesA missing
-//         { year: '2024', seriesA: 25, seriesB: 35 },
-//         { year: '2024', seriesA: 20, seriesB: 40 },
-//         { year: '2024', seriesB: 45 },
-//       ],
-//     },
-//     title: 'Handling Missing Data Points',
-//     xAxis: 'year',
-//     yAxis: ['seriesA', 'seriesB'],
-//     xAxisType: 'temporal',
-//     xAxisTimeUnit: 'year',
-//     yAxisType: 'quantitative'
-//   },
-// };

packages/components/stories/Map.stories.ts

@@ -4,33 +4,22 @@ import { Map, MapProps } from '../src/components/Map';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Geospatial/Map',
+  title: 'Components/Map',
   component: Map,
   tags: ['autodocs'],
   argTypes: {
     layers: {
       description:
-        'Array of layers to be displayed on the map. Should be an object with: \n\n \
-`data`: object with either a `url` property pointing to a GeoJSON file or a `geojson` property with a GeoJSON object. \n\n \
-`name`: name of the layer. \n\n \
-`colorscale`: object with a `starting` and `ending` colors that will be used to create a gradient and color the map. \n\n \
-`tooltip`: `true` to show all available features on the tooltip, object with a `propNames` property as an array of strings to choose which features to display. \n\n',
+        'Data to be displayed.\n\n GeoJSON Object \n\nOR\n\n URL to GeoJSON Object',
     },
     title: {
-      description: 'Title to display on the map.',
+      description: 'Title to display on the map. Optional.',
     },
     center: {
       description: 'Initial coordinates of the center of the map',
     },
     zoom: {
-      description: 'Initial zoom level',
-    },
-    style: {
-      description: "CSS styles to be applied to the map's container.",
-    },
-    autoZoomConfiguration: {
-      description:
-        "Pass a layer's name to automatically zoom to the bounding area of a layer.",
+      description: 'Zoom level',
     },
   },
 };
@@ -43,15 +32,9 @@ type Story = StoryObj<MapProps>;
 export const GeoJSONPolygons: Story = {
   name: 'GeoJSON polygons map',
   args: {
-    tileLayerName:'MapBox',
-    tileLayerOptions:{
-      accessToken : 'pk.eyJ1Ijoid2lsbHktcGFsbWFyZWpvIiwiYSI6ImNqNzk5NmRpNDFzb2cyeG9sc2luMHNjajUifQ.lkoVRFSI8hOLH4uJeOzwXw',
-    },
     layers: [
       {
-        data: {
-          url: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
-        },
+        data: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
         name: 'Polygons',
         tooltip: { propNames: ['name'] },
         colorScale: {
@@ -71,9 +54,7 @@ export const GeoJSONPoints: Story = {
   args: {
     layers: [
       {
-        data: {
-          url: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
-        },
+        data: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
         name: 'Points',
         tooltip: { propNames: ['Location'] },
       },
@@ -89,16 +70,12 @@ export const GeoJSONMultipleLayers: Story = {
   args: {
     layers: [
       {
-        data: {
-          url: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
-        },
+        data: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
         name: 'Points',
         tooltip: true,
       },
       {
-        data: {
-          url: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
-        },
+        data: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
         name: 'Polygons',
         tooltip: true,
         colorScale: {
@@ -112,35 +89,3 @@ export const GeoJSONMultipleLayers: Story = {
     zoom: 2,
   },
 };
-
-export const GeoJSONMultipleLayersWithAutoZoomInSpecifiedLayer: Story = {
-  name: 'GeoJSON polygons and points map with auto zoom in the points layer',
-  args: {
-    layers: [
-      {
-        data: {
-          url: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
-        },
-        name: 'Points',
-        tooltip: true,
-      },
-      {
-        data: {
-          url: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
-        },
-        name: 'Polygons',
-        tooltip: true,
-        colorScale: {
-          starting: '#ff0000',
-          ending: '#00ff00',
-        },
-      },
-    ],
-    title: 'Polygons and points',
-    center: { latitude: 45, longitude: 0 },
-    zoom: 2,
-    autoZoomConfiguration: {
-      layerName: 'Points',
-    },
-  },
-};

packages/components/stories/OpenLayers.stories.tsx

@@ -1,6 +1,3 @@
-// NOTE: this component was renamed with .bkp so that it's hidden
-// from the Storybook app
-
 import type { Meta, StoryObj } from '@storybook/react';
 import React from 'react';
 import OpenLayers from '../src/components/OpenLayers/OpenLayers';

packages/components/stories/PdfViewer.stories.ts

@@ -3,21 +3,19 @@ import type { Meta, StoryObj } from '@storybook/react';
 import { PdfViewer, PdfViewerProps } from '../src/components/PdfViewer';
 
 const meta: Meta = {
-  title: 'Components/Embedding/PdfViewer',
+  title: 'Components/PdfViewer',
   component: PdfViewer,
   tags: ['autodocs'],
   argTypes: {
-    data: {
-      description:
-        'Object with a `url` property pointing to the PDF file to be displayed, e.g.: `{ url: "https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK" }`.',
+    url: {
+      description: 'URL to PDF file',
     },
     parentClassName: {
-      description:
-        'HTML classes to be applied to the container of the PDF viewer. [Tailwind](https://tailwindcss.com/) classes, such as `h-96` to define the height of the component, can be used on this field.',
+      description: 'Classname for the parent div of the pdf viewer',
     },
-    layout: {
+    layour: {
       description:
-        'Set to `true` if you want to display a layout with zoom level, page count, printing button and other controls.',
+        'Set to true if you want to have a layout with zoom level, page count, printing button etc',
       defaultValue: false,
     },
   },
@@ -27,23 +25,26 @@ export default meta;
 
 type Story = StoryObj<PdfViewerProps>;
 
-export const PdfViewerStoryWithoutControlsLayout: Story = {
-  name: 'PDF Viewer without controls layout',
+export const PdfViewerStory: Story = {
+  name: 'PdfViewer',
   args: {
-    data: {
-      url: 'https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK',
-    },
-    parentClassName: 'h-96',
+    url: 'https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK',
   },
 };
 
-export const PdfViewerStoryWithControlsLayout: Story = {
-  name: 'PdfViewer with controls layout',
+export const PdfViewerStoryWithLayout: Story = {
+  name: 'PdfViewer with the default layout',
   args: {
-    data: {
-      url: 'https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK',
-    },
+    url: 'https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK',
+    layout: true,
+  },
+};
+
+export const PdfViewerStoryWithHeight: Story = {
+  name: 'PdfViewer with a custom height',
+  args: {
+    url: 'https://cdn.filestackcontent.com/wcrjf9qPTCKXV3hMXDwK',
+    parentClassName: 'h-96',
     layout: true,
-    parentClassName: 'h-96',
   },
 };

packages/components/stories/Plotly.stories.ts

@@ -1,49 +0,0 @@
-import type { Meta, StoryObj } from '@storybook/react';
-
-import { Plotly } from '../src/components/Plotly';
-
-// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
-const meta: Meta = {
-  title: 'Components/Charts/Plotly',
-  component: Plotly,
-  tags: ['autodocs'],
-  argTypes: {
-    data: {
-      description:
-        "Plotly's `data` prop. You can find references on how to use these props at https://github.com/plotly/react-plotly.js/#basic-props.",
-    },
-    layout: {
-      description:
-        "Plotly's `layout` prop. You can find references on how to use these props at https://github.com/plotly/react-plotly.js/#basic-props.",
-    },
-  },
-};
-
-export default meta;
-
-type Story = StoryObj<any>;
-
-// More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
-export const Primary: Story = {
-  name: 'Line chart',
-  args: {
-    data: [
-      {
-        x: [1, 2, 3],
-        y: [2, 6, 3],
-        type: 'scatter',
-        mode: 'lines+markers',
-        marker: { color: 'red' },
-      },
-    ],
-    layout: {
-      title: 'Chart built with Plotly',
-      xaxis: {
-        title: 'x Axis',
-      },
-      yaxis: {
-        title: 'y Axis',
-      },
-    },
-  },
-};

packages/components/stories/PlotlyBarChart.stories.ts

@@ -1,102 +0,0 @@
-import type { Meta, StoryObj } from '@storybook/react';
-
-import {
-  PlotlyBarChart,
-  PlotlyBarChartProps,
-} from '../src/components/PlotlyBarChart';
-
-// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
-const meta: Meta = {
-  title: 'Components/Charts/PlotlyBarChart',
-  component: PlotlyBarChart,
-  tags: ['autodocs'],
-  argTypes: {
-    data: {
-      description:
-        'Data to be displayed. \n\n \
-Must be an object with one of the following properties: `url`, `values` or `csv` \n\n \
-`url`: URL pointing to a CSV file. \n\n \
-`values`: array of objects (check out [this example](/?path=/story/components-plotlybarchart--from-data-points)) \n\n \
-`csv`: string with valid CSV (check out [this example](/?path=/story/components-plotlybarchart--from-inline-csv)) \n\n \
-',
-    },
-    bytes: {
-      // TODO: likely this should be an extra option on the data parameter,
-      // specific to URLs
-      description:
-        "How many bytes to read from the url so that the entire file doesn's have to be fetched.",
-    },
-    parsingConfig: {
-      description:
-        'If using URL or CSV, this parsing config will be used to parse the data. Check https://www.papaparse.com/ for more info.',
-    },
-    title: {
-      description: 'Title to display on the chart.',
-    },
-    // TODO: commented out because this doesn't work
-    // lineLabel: {
-    //   description:
-    //     'Label to display on the line, Optional, will use yAxis if not provided',
-    // },
-    xAxis: {
-      description:
-        'Name of the column header or object property that represents the X-axis on the data.',
-    },
-    yAxis: {
-      description:
-        'Name of the column header or object property that represents the Y-axis on the data.',
-    },
-    uniqueId: {
-      description: 'Provide a unique ID to help with cache revalidation of the fetched data.'
-    }
-  },
-};
-
-export default meta;
-
-type Story = StoryObj<PlotlyBarChartProps>;
-
-export const FromDataPoints: Story = {
-  name: 'Bar chart from array of data points',
-  args: {
-    data: {
-      values: [
-        { year: '1850', temperature: -0.41765878 },
-        { year: '1851', temperature: -0.2333498 },
-        { year: '1852', temperature: -0.22939907 },
-        { year: '1853', temperature: -0.27035445 },
-        { year: '1854', temperature: -0.29163003 },
-      ],
-    },
-    xAxis: 'year',
-    yAxis: 'temperature',
-  },
-};
-
-export const FromURL: Story = {
-  name: 'Bar chart from URL',
-  args: {
-    title: 'Apple Stock Prices',
-    data: {
-      url: 'https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv',
-    },
-    xAxis: 'Date',
-    yAxis: 'AAPL.Open',
-  },
-};
-
-export const FromInlineCSV: Story = {
-  name: 'Bar chart from inline CSV',
-  args: {
-    title: 'Apple Stock Prices',
-    data: {
-      csv: `Date,AAPL.Open,AAPL.High,AAPL.Low,AAPL.Close,AAPL.Volume,AAPL.Adjusted,dn,mavg,up,direction
-2015-02-17,127.489998,128.880005,126.919998,127.830002,63152400,122.905254,106.7410523,117.9276669,129.1142814,Increasing
-2015-02-18,127.629997,128.779999,127.449997,128.720001,44891700,123.760965,107.842423,118.9403335,130.0382439,Increasing
-2015-02-19,128.479996,129.029999,128.330002,128.449997,37362400,123.501363,108.8942449,119.8891668,130.8840887,Decreasing
-2015-02-20,128.619995,129.5,128.050003,129.5,48948400,124.510914,109.7854494,120.7635001,131.7415509,Increasing`,
-    },
-    xAxis: 'Date',
-    yAxis: 'AAPL.Open',
-  },
-};

packages/components/stories/PlotlyLineChart.stories.ts

@@ -1,101 +0,0 @@
-import type { Meta, StoryObj } from '@storybook/react';
-
-import {
-  PlotlyLineChart,
-  PlotlyLineChartProps,
-} from '../src/components/PlotlyLineChart';
-
-const meta: Meta = {
-  title: 'Components/Charts/PlotlyLineChart',
-  component: PlotlyLineChart,
-  tags: ['autodocs'],
-  argTypes: {
-    data: {
-      description:
-        'Data to be displayed. \n\n \
-Must be an object with one of the following properties: `url`, `values` or `csv` \n\n \
-`url`: URL pointing to a CSV file. \n\n \
-`values`: array of objects. \n\n \
-`csv`: string with valid CSV. \n\n \
-',
-    },
-    bytes: {
-      // TODO: likely this should be an extra option on the data parameter,
-      // specific to URLs
-      description:
-        "How many bytes to read from the url so that the entire file doesn's have to be fetched.",
-    },
-    parsingConfig: {
-      description:
-        'If using URL or CSV, this parsing config will be used to parse the data. Check https://www.papaparse.com/ for more info',
-    },
-    title: {
-      description: 'Title to display on the chart.',
-    },
-    lineLabel: {
-      description:
-        'Label to display on the line, will use yAxis if not provided',
-    },
-    xAxis: {
-      description:
-        'Name of the column header or object property that represents the X-axis on the data.',
-    },
-    yAxis: {
-      description:
-        'Name of the column header or object property that represents the Y-axis on the data.',
-    },
-    uniqueId: {
-      description:
-        'Provide a unique ID to help with cache revalidation of the fetched data.',
-    },
-  },
-};
-
-export default meta;
-
-type Story = StoryObj<PlotlyLineChartProps>;
-
-export const FromDataPoints: Story = {
-  name: 'Line chart from array of data points',
-  args: {
-    data: {
-      values: [
-        { year: '1850', temperature: -0.41765878 },
-        { year: '1851', temperature: -0.2333498 },
-        { year: '1852', temperature: -0.22939907 },
-        { year: '1853', temperature: -0.27035445 },
-        { year: '1854', temperature: -0.29163003 },
-      ],
-    },
-    xAxis: 'year',
-    yAxis: 'temperature',
-  },
-};
-
-export const FromURL: Story = {
-  name: 'Line chart from URL',
-  args: {
-    title: 'Oil Price x Year',
-    data: {
-      url: 'https://raw.githubusercontent.com/datasets/oil-prices/main/data/wti-year.csv',
-    },
-    xAxis: 'Date',
-    yAxis: 'Price',
-  },
-};
-
-export const FromInlineCSV: Story = {
-  name: 'Bar chart from inline CSV',
-  args: {
-    title: 'Apple Stock Prices',
-    data: {
-      csv: `Date,AAPL.Open,AAPL.High,AAPL.Low,AAPL.Close,AAPL.Volume,AAPL.Adjusted,dn,mavg,up,direction
-2015-02-17,127.489998,128.880005,126.919998,127.830002,63152400,122.905254,106.7410523,117.9276669,129.1142814,Increasing
-2015-02-18,127.629997,128.779999,127.449997,128.720001,44891700,123.760965,107.842423,118.9403335,130.0382439,Increasing
-2015-02-19,128.479996,129.029999,128.330002,128.449997,37362400,123.501363,108.8942449,119.8891668,130.8840887,Decreasing
-2015-02-20,128.619995,129.5,128.050003,129.5,48948400,124.510914,109.7854494,120.7635001,131.7415509,Increasing`,
-    },
-    xAxis: 'Date',
-    yAxis: 'AAPL.Open',
-  },
-};

packages/components/stories/Table.stories.ts

@@ -1,33 +1,25 @@
-// NOTE: this component was renamed with .bkp so that it's hidden
-// from the Storybook app
-
 import type { Meta, StoryObj } from '@storybook/react';
 
 import { Table, TableProps } from '../src/components/Table';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Tabular/Table',
+  title: 'Components/Table',
   component: Table,
   tags: ['autodocs'],
   argTypes: {
     data: {
-      description:
-        'Data to be displayed in the table, must also set "cols" to work.',
+      description: "Data to be displayed in the table, must also set \"cols\" to work."
     },
     cols: {
-      description:
-        'Columns to be displayed in the table, must also set "data" to work.',
+      description: "Columns to be displayed in the table, must also set \"data\" to work."
     },
     csv: {
-      description: 'CSV data as string.',
+      description: "CSV data as string.",
     },
     url: {
-      description: 'Fetch the data from a CSV file remotely.',
-    },
-    datastoreConfig: {
-      description: `Configuration to use CKAN's datastore API extension integrated with the component`,
-    },
+      description: "Fetch the data from a CSV file remotely."
+    }
   },
 };
 
@@ -37,7 +29,7 @@ type Story = StoryObj<TableProps>;
 
 // More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
 export const FromColumnsAndData: Story = {
-  name: 'Table from columns and data',
+  name: "Table from columns and data",
   args: {
     data: [
       { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
@@ -57,40 +49,21 @@ export const FromColumnsAndData: Story = {
   },
 };
 
-export const WithDataStoreIntegration: Story = {
-  name: 'Table with datastore integration',
-  args: {
-    datastoreConfig: {
-      dataStoreURI: `https://www.civicdata.com/api/action/datastore_search?resource_id=46ec0807-31ff-497f-bfa0-f31c796cdee8`,
-      dataMapperFn: ({
-        result,
-      }: {
-        result: { fields: { id }[]; records: []; total: number };
-      }) => {
-        return {
-          data: result.records,
-          cols: result.fields.map((x) => ({ key: x.id, name: x.id })),
-          total: result.total,
-        };
-      },
-    },
-  },
-};
-
 export const FromRawCSV: Story = {
-  name: 'Table from raw CSV',
+  name: "Table from raw CSV",
   args: {
     csv: `
     Year,Temp Anomaly
     1850,-0.418
     2020,0.923
-    `,
-  },
+    `
+  }
 };
 
 export const FromURL: Story = {
-  name: 'Table from URL',
+  name: "Table from URL",
   args: {
-    url: 'https://raw.githubusercontent.com/datasets/finance-vix/main/data/vix-daily.csv',
-  },
+    url: "https://raw.githubusercontent.com/datasets/finance-vix/main/data/vix-daily.csv"
+  }
 };
+

packages/components/stories/Vega.stories.ts

@@ -4,19 +4,9 @@ import { Vega } from '../src/components/Vega';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Charts/Vega',
+  title: 'Components/Vega',
   component: Vega,
   tags: ['autodocs'],
-  argTypes: {
-    data: {
-      description:
-        "Vega's `data` prop. You can find references on how to use this prop at https://vega.github.io/vega/docs/data/",
-    },
-    spec: {
-      description:
-        "Vega's `spec` prop. You can find references on how to use this prop at https://vega.github.io/vega/docs/specification/",
-    },
-  },
 };
 
 export default meta;
@@ -25,7 +15,7 @@ type Story = StoryObj<any>;
 
 // More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
 export const Primary: Story = {
-  name: 'Bar chart',
+  name: 'Chart built with Vega',
   args: {
     data: {
       table: [

packages/components/stories/VegaLite.stories.ts

@@ -4,7 +4,7 @@ import { VegaLite } from '../src/components/VegaLite';
 
 // More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
 const meta: Meta = {
-  title: 'Components/Charts/VegaLite',
+  title: 'Components/VegaLite',
   component: VegaLite,
   tags: ['autodocs'],
   argTypes: {
@@ -25,7 +25,7 @@ type Story = StoryObj<any>;
 
 // More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
 export const Primary: Story = {
-  name: 'Bar chart',
+  name: 'Chart built with Vega Lite',
   args: {
     data: {
       table: [

packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@portaljs/core",
-  "version": "1.0.9",
+  "version": "1.0.8",
   "description": "Core Portal.JS components, configs and utils.",
   "repository": {
     "type": "git",

packages/core/src/ui/Nav/Nav.tsx

@@ -53,7 +53,7 @@ export const Nav: React.FC<Props> = ({
     <nav className="flex justify-between">
       {/* Mobile navigation  */}
       <div className="mr-2 sm:mr-4 flex lg:hidden">
-        <NavMobile {...{title, links, social, search, defaultTheme, themeToggleIcon}}>{children}</NavMobile>
+        <NavMobile links={links}>{children}</NavMobile>
       </div>
       {/* Non-mobile navigation */}
       <div className="flex flex-none items-center">

packages/core/src/ui/Nav/NavMobile.tsx

@@ -4,16 +4,20 @@ import { useRouter } from "next/router.js";
 import { useEffect, useState } from "react";
 import { SearchContext, SearchField } from "../Search";
 import { MenuIcon, CloseIcon } from "../Icons";
-import type { NavConfig, ThemeConfig } from "./Nav";
+import { NavLink, SearchProviderConfig } from "../types";
 
-interface Props extends NavConfig, ThemeConfig, React.PropsWithChildren {}
+interface Props extends React.PropsWithChildren {
+  author?: string;
+  links?: Array<NavLink>;
+  search?: SearchProviderConfig;
+}
 
-// TODO: Search doesn't appear
+// TODO why mobile navigation only accepts author and regular nav accepts different things like title, logo, version
 export const NavMobile: React.FC<Props> = ({
   children,
-  title,
   links,
   search,
+  author,
 }) => {
   const router = useRouter();
   const [isOpen, setIsOpen] = useState(false);
@@ -73,8 +77,8 @@ export const NavMobile: React.FC<Props> = ({
               legacyBehavior
             >
               {/* <Logomark className="h-9 w-9" /> */}
-              <div className="font-extrabold text-primary dark:text-primary-dark text-lg ml-6">
-                {title}
+              <div className="font-extrabold text-primary dark:text-primary-dark text-2xl ml-6">
+                {author}
               </div>
             </Link>
           </div>
@@ -102,7 +106,9 @@ export const NavMobile: React.FC<Props> = ({
               ))}
             </ul>
           )}
-          <div className="pt-6">{children}</div>
+          {/* <div className="pt-6 border border-t-2">
+                        {children}
+                    </div> */}
         </Dialog.Panel>
       </Dialog>
     </>

packages/core/src/ui/SiteToc/SiteToc.tsx

@@ -46,8 +46,8 @@ export const SiteToc: React.FC<Props> = ({ currentPath, nav }) => {
 
   return (
     <nav data-testid="lhs-sidebar" className="flex flex-col space-y-3 text-sm">
-      {sortNavGroupChildren(nav).map((n, index) => (
-        <NavComponent key={index} item={n} isActive={false} />
+      {sortNavGroupChildren(nav).map((n) => (
+        <NavComponent item={n} isActive={false} />
       ))}
     </nav>
   );
@@ -96,8 +96,8 @@ const NavComponent: React.FC<{
             leaveTo="transform scale-95 opacity-0"
           >
             <Disclosure.Panel className="flex flex-col space-y-3 pl-5 mt-3">
-              {sortNavGroupChildren(item.children).map((subItem, index) => (
-                <NavComponent key={index} item={subItem} isActive={false} />
+              {sortNavGroupChildren(item.children).map((subItem) => (
+                <NavComponent item={subItem} isActive={false} />
               ))}
             </Disclosure.Panel>
           </Transition>

packages/core/src/ui/analytics/GoogleAnalytics.tsx

@@ -1,36 +0,0 @@
-import Script from 'next/script.js'
-
-export interface GoogleAnalyticsProps {
-  googleAnalyticsId: string
-}
-
-export const GA = ({ googleAnalyticsId }: GoogleAnalyticsProps) => {
-  return (
-    <>
-      <Script
-        strategy="afterInteractive"
-        src={`https://www.googletagmanager.com/gtag/js?id=${googleAnalyticsId}`}
-      />
-
-      <Script strategy="afterInteractive" id="ga-script">
-        {`
-            window.dataLayer = window.dataLayer || [];
-            function gtag(){dataLayer.push(arguments);}
-            gtag('js', new Date());
-            gtag('config', '${googleAnalyticsId}');
-        `}
-      </Script>
-    </>
-  )
-}
-
-// https://developers.google.com/analytics/devguides/collection/gtagjs/events
-export const logEvent = (action, category, label, value) => {
-  // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-  // @ts-ignore
-  window.gtag?.('event', action, {
-    event_category: category,
-    event_label: label,
-    value: value,
-  })
-}

packages/core/src/ui/analytics/Plausible.tsx

@@ -1,41 +0,0 @@
-import Script from 'next/script.js'
-
-export interface PlausibleProps {
-  plausibleDataDomain: string
-  dataApi?: string
-  src?: string
-}
-
-/**
- * Plausible analytics component.
- * To proxy the requests through your own domain, you can use the dataApi and src attribute.
- * See [Plausible docs](https://plausible.io/docs/proxy/guides/nextjs#step-2-adjust-your-deployed-script)
- * for more information.
- *
- */
-export const Plausible = ({
-  plausibleDataDomain,
-  dataApi = undefined,
-  src = 'https://plausible.io/js/plausible.js',
-}: PlausibleProps) => {
-  return (
-    <>
-      <Script
-        strategy="lazyOnload"
-        data-domain={plausibleDataDomain}
-        data-api={dataApi}
-        src={src}
-      />
-      <Script strategy="lazyOnload" id="plausible-script">
-        {`
-            window.plausible = window.plausible || function() { (window.plausible.q = window.plausible.q || []).push(arguments) }
-        `}
-      </Script>
-    </>
-  )
-}
-
-// https://plausible.io/docs/custom-event-goals
-export const logEvent = (eventName, ...rest) => {
-  return window.plausible?.(eventName, ...rest)
-}

packages/core/src/ui/analytics/Posthog.tsx

@@ -1,25 +0,0 @@
-import Script from 'next/script.js'
-
-export interface PosthogProps {
-  posthogProjectApiKey: string
-  apiHost?: string
-}
-
-/**
- * Posthog analytics component.
- * See [Posthog docs](https://posthog.com/docs/libraries/js#option-1-add-javascript-snippet-to-your-html-badgerecommendedbadge) for more information.
- *
- */
-export const Posthog = ({
-  posthogProjectApiKey,
-  apiHost = 'https://app.posthog.com',
-}: PosthogProps) => {
-  return (
-    <Script strategy="lazyOnload" id="posthog-script">
-      {`
-          !function(t,e){var o,n,p,r;e.__SV||(window.posthog=e,e._i=[],e.init=function(i,s,a){function g(t,e){var o=e.split(".");2==o.length&&(t=t[o[0]],e=o[1]),t[e]=function(){t.push([e].concat(Array.prototype.slice.call(arguments,0)))}}(p=t.createElement("script")).type="text/javascript",p.async=!0,p.src=s.api_host+"/static/array.js",(r=t.getElementsByTagName("script")[0]).parentNode.insertBefore(p,r);var u=e;for(void 0!==a?u=e[a]=[]:a="posthog",u.people=u.people||[],u.toString=function(t){var e="posthog";return"posthog"!==a&&(e+="."+a),t||(e+=" (stub)"),e},u.people.toString=function(){return u.toString(1)+".people (stub)"},o="capture identify alias people.set people.set_once set_config register register_once unregister opt_out_capturing has_opted_out_capturing opt_in_capturing reset isFeatureEnabled onFeatureFlags".split(" "),n=0;n<o.length;n++)g(u,o[n]);e._i.push([i,s,a])},e.__SV=1)}(document,window.posthog||[]);
-          posthog.init('${posthogProjectApiKey}',{api_host:'${apiHost}'})
-      `}
-    </Script>
-  )
-}

packages/core/src/ui/analytics/SimpleAnalytics.tsx

@@ -1,29 +0,0 @@
-import Script from 'next/script.js'
-
-export interface SimpleAnalyticsProps {
-  src?: string
-}
-
-export const SimpleAnalytics = ({
-  src = 'https://scripts.simpleanalyticscdn.com/latest.js',
-}: SimpleAnalyticsProps) => {
-  return (
-    <>
-      <Script strategy="lazyOnload" id="sa-script">
-        {`
-            window.sa_event=window.sa_event||function(){var a=[].slice.call(arguments);window.sa_event.q?window.sa_event.q.push(a):window.sa_event.q=[a]};
-        `}
-      </Script>
-      <Script strategy="lazyOnload" src={src} />
-    </>
-  )
-}
-
-// https://docs.simpleanalytics.com/events
-export const logEvent = (eventName, callback) => {
-  if (callback) {
-    return window.sa_event?.(eventName, callback)
-  } else {
-    return window.sa_event?.(eventName)
-  }
-}

packages/core/src/ui/analytics/Umami.tsx

@@ -1,20 +0,0 @@
-import Script from 'next/script.js'
-
-export interface UmamiProps {
-  umamiWebsiteId: string
-  src?: string
-}
-
-export const Umami = ({
-  umamiWebsiteId,
-  src = 'https://analytics.umami.is/script.js',
-}: UmamiProps) => {
-  return (
-    <Script
-      async
-      defer
-      data-website-id={umamiWebsiteId}
-      src={src} // Replace with your umami instance
-    />
-  )
-}

packages/core/src/ui/analytics/index.tsx

@@ -1,82 +0,0 @@
-/* eslint-disable @typescript-eslint/no-explicit-any */
-import { GA, GoogleAnalyticsProps } from "./GoogleAnalytics";
-import { Plausible, PlausibleProps } from "./Plausible";
-import { SimpleAnalytics, SimpleAnalyticsProps } from "./SimpleAnalytics";
-import { Umami, UmamiProps } from "./Umami";
-import { Posthog, PosthogProps } from "./Posthog";
-
-declare global {
-    interface Window {
-        // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-        // @ts-ignore
-        gtag?: (...args: any[]) => void;
-        plausible?: (...args: any[]) => void;
-        sa_event?: (...args: any[]) => void;
-    }
-}
-
-export interface AnalyticsConfig {
-    googleAnalytics?: GoogleAnalyticsProps;
-    plausibleAnalytics?: PlausibleProps;
-    umamiAnalytics?: UmamiProps;
-    posthogAnalytics?: PosthogProps;
-    simpleAnalytics?: SimpleAnalyticsProps;
-}
-
-/**
- * @example
- * const analytics: AnalyticsConfig = {
- *  plausibleDataDomain: '', // e.g. tailwind-nextjs-starter-blog.vercel.app
- *  simpleAnalytics: false, // true or false
- *  umamiWebsiteId: '', // e.g. 123e4567-e89b-12d3-a456-426614174000
- *  posthogProjectApiKey: '', // e.g. AhnJK8392ndPOav87as450xd
- *  googleAnalyticsId: '', // e.g. UA-000000-2 or G-XXXXXXX
- * }
- */
-export interface AnalyticsProps {
-    analyticsConfig: AnalyticsConfig;
-}
-
-const isProduction = true || process.env["NODE_ENV"] === "production";
-
-/**
- * Supports Plausible, Simple Analytics, Umami, Posthog or Google Analytics.
- * All components default to the hosted service, but can be configured to use a self-hosted
- * or proxied version of the script by providing the `src` / `apiHost` props.
- *
- * Note: If you want to use an analytics provider you have to add it to the
- * content security policy in the `next.config.js` file.
- * @param {AnalyticsProps} { analytics }
- * @return {*}
- */
-export const Analytics = ({ analyticsConfig }: AnalyticsProps) => {
-    return (
-        <>
-            {isProduction && analyticsConfig.plausibleAnalytics && (
-                <Plausible {...analyticsConfig.plausibleAnalytics} />
-            )}
-            {isProduction && analyticsConfig.simpleAnalytics && (
-                <SimpleAnalytics {...analyticsConfig.simpleAnalytics} />
-            )}
-            {isProduction && analyticsConfig.posthogAnalytics && (
-                <Posthog {...analyticsConfig.posthogAnalytics} />
-            )}
-            {isProduction && analyticsConfig.umamiAnalytics && (
-                <Umami {...analyticsConfig.umamiAnalytics} />
-            )}
-            {isProduction && analyticsConfig.googleAnalytics && (
-                <GA {...analyticsConfig.googleAnalytics} />
-            )}
-        </>
-    );
-};
-
-export { GA, Plausible, SimpleAnalytics, Umami, Posthog };
-
-export type {
-    GoogleAnalyticsProps,
-    PlausibleProps,
-    UmamiProps,
-    PosthogProps,
-    SimpleAnalyticsProps,
-};

packages/core/src/ui/index.ts

@@ -21,4 +21,3 @@ export { SiteToc, NavItem, NavGroup } from "./SiteToc";
 export { Comments, CommentsConfig } from "./Comments";
 export { AuthorConfig } from "./types";
 export { Hero } from "./Hero";
-export { Analytics, AnalyticsConfig } from "./analytics";

packages/core/src/utils/gtag.ts

@@ -7,8 +7,6 @@ export const pageview = ({
   analyticsID: string;
 }) => {
   if (typeof window.gtag !== undefined) {
-    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-    // @ts-ignore
     window.gtag("config", analyticsID, {
       page_path: url,
     });
@@ -18,8 +16,6 @@ export const pageview = ({
 // https://developers.google.com/analytics/devguides/collection/gtagjs/events
 export const event = ({ action, category, label, value }) => {
   if (typeof window.gtag !== undefined) {
-    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-    // @ts-ignore
     window.gtag("event", action, {
       event_category: category,
       event_label: label,

packages/remark-wiki-link/CHANGELOG.md

@@ -1,17 +1,5 @@
 # @portaljs/remark-wiki-link
 
-## 1.2.0
-
-### Minor Changes
-
-- [#1084](https://github.com/datopian/datahub/pull/1084) [`57952e08`](https://github.com/datopian/datahub/commit/57952e0817770138881e7492dc9f43e9910b56a8) Thanks [@mohamedsalem401](https://github.com/mohamedsalem401)! - Add image resize feature
-
-## 1.1.2
-
-### Patch Changes
-
-- [#1040](https://github.com/datopian/portaljs/pull/1040) [`85bb6cb9`](https://github.com/datopian/portaljs/commit/85bb6cb98c53bedc2add3d014927570b5dd1bbdf) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Changed regex to permit any symbols other than #
-
 ## 1.1.1
 
 ### Patch Changes

packages/remark-wiki-link/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@portaljs/remark-wiki-link",
-  "version": "1.2.0",
+  "version": "1.1.1",
   "description": "Parse and render wiki-style links in markdown especially Obsidian style links.",
   "repository": {
     "type": "git",

packages/remark-wiki-link/src/lib/fromMarkdown.ts

@@ -1,23 +1,23 @@
-import { isSupportedFileFormat } from './isSupportedFileFormat';
+import { isSupportedFileFormat } from "./isSupportedFileFormat";
 
 const defaultWikiLinkResolver = (target: string) => {
   // for [[#heading]] links
   if (!target) {
     return [];
   }
-  let permalink = target.replace(/\/index$/, '');
+  let permalink = target.replace(/\/index$/, "");
   // TODO what to do with [[index]] link?
   if (permalink.length === 0) {
-    permalink = '/';
+    permalink = "/";
   }
   return [permalink];
 };
 
 export interface FromMarkdownOptions {
   pathFormat?:
-    | 'raw' // default; use for regular relative or absolute paths
-    | 'obsidian-absolute' // use for Obsidian-style absolute paths (with no leading slash)
-    | 'obsidian-short'; // use for Obsidian-style shortened paths (shortest path possible)
+  | "raw" // default; use for regular relative or absolute paths
+  | "obsidian-absolute" // use for Obsidian-style absolute paths (with no leading slash)
+  | "obsidian-short"; // use for Obsidian-style shortened paths (shortest path possible)
   permalinks?: string[]; // list of permalinks to match possible permalinks of a wiki link against
   wikiLinkResolver?: (name: string) => string[]; // function to resolve wiki links to an array of possible permalinks
   newClassName?: string; // class name to add to links that don't have a matching permalink
@@ -25,23 +25,14 @@ export interface FromMarkdownOptions {
   hrefTemplate?: (permalink: string) => string; // function to generate the href attribute of a link
 }
 
-export function getImageSize(size: string) {
-  // eslint-disable-next-line prefer-const
-  let [width, height] = size.split('x');
-
-  if (!height) height = width;
-
-  return { width, height };
-}
-
 // mdas-util-from-markdown extension
 // https://github.com/syntax-tree/mdast-util-from-markdown#extension
 function fromMarkdown(opts: FromMarkdownOptions = {}) {
-  const pathFormat = opts.pathFormat || 'raw';
+  const pathFormat = opts.pathFormat || "raw";
   const permalinks = opts.permalinks || [];
   const wikiLinkResolver = opts.wikiLinkResolver || defaultWikiLinkResolver;
-  const newClassName = opts.newClassName || 'new';
-  const wikiLinkClassName = opts.wikiLinkClassName || 'internal';
+  const newClassName = opts.newClassName || "new";
+  const wikiLinkClassName = opts.wikiLinkClassName || "internal";
   const defaultHrefTemplate = (permalink: string) => permalink;
 
   const hrefTemplate = opts.hrefTemplate || defaultHrefTemplate;
@@ -53,9 +44,9 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
   function enterWikiLink(token) {
     this.enter(
       {
-        type: 'wikiLink',
+        type: "wikiLink",
         data: {
-          isEmbed: token.isType === 'embed',
+          isEmbed: token.isType === "embed",
           target: null, // the target of the link, e.g. "Foo Bar#Heading" in "[[Foo Bar#Heading]]"
           alias: null, // the alias of the link, e.g. "Foo" in "[[Foo Bar|Foo]]"
           permalink: null, // TODO shouldn't this be named just "link"?
@@ -88,19 +79,19 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
       data: { isEmbed, target, alias },
     } = wikiLink;
     // eslint-disable-next-line no-useless-escape
-    const wikiLinkWithHeadingPattern = /^(.*?)(#.*)?$/u;
-    const [, path, heading = ''] = target.match(wikiLinkWithHeadingPattern);
+    const wikiLinkWithHeadingPattern = /([\p{Letter}\d\s\/\.-_]*)(#.*)?/u;
+    const [, path, heading = ""] = target.match(wikiLinkWithHeadingPattern);
 
     const possibleWikiLinkPermalinks = wikiLinkResolver(path);
 
     const matchingPermalink = permalinks.find((e) => {
       return possibleWikiLinkPermalinks.find((p) => {
-        if (pathFormat === 'obsidian-short') {
+        if (pathFormat === "obsidian-short") {
           if (e === p || e.endsWith(p)) {
             return true;
           }
-        } else if (pathFormat === 'obsidian-absolute') {
-          if (e === '/' + p) {
+        } else if (pathFormat === "obsidian-absolute") {
+          if (e === "/" + p) {
             return true;
           }
         } else {
@@ -115,19 +106,20 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
     // TODO this is ugly
     const link =
       matchingPermalink ||
-      (pathFormat === 'obsidian-absolute'
-        ? '/' + possibleWikiLinkPermalinks[0]
+      (pathFormat === "obsidian-absolute"
+        ? "/" + possibleWikiLinkPermalinks[0]
         : possibleWikiLinkPermalinks[0]) ||
-      '';
+      "";
 
     wikiLink.data.exists = !!matchingPermalink;
     wikiLink.data.permalink = link;
+
     // remove leading # if the target is a heading on the same page
-    const displayName = alias || target.replace(/^#/, '');
-    const headingId = heading.replace(/\s+/g, '-').toLowerCase();
+    const displayName = alias || target.replace(/^#/, "");
+    const headingId = heading.replace(/\s+/, "-").toLowerCase();
     let classNames = wikiLinkClassName;
     if (!matchingPermalink) {
-      classNames += ' ' + newClassName;
+      classNames += " " + newClassName;
     }
 
     if (isEmbed) {
@@ -135,55 +127,44 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
       if (!isSupportedFormat) {
         // Temporarily render note transclusion as a regular wiki link
         if (!format) {
-          wikiLink.data.hName = 'a';
+          wikiLink.data.hName = "a";
           wikiLink.data.hProperties = {
-            className: classNames + ' ' + 'transclusion',
+            className: classNames + " " + "transclusion",
             href: hrefTemplate(link) + headingId,
           };
-          wikiLink.data.hChildren = [{ type: 'text', value: displayName }];
+          wikiLink.data.hChildren = [{ type: "text", value: displayName }];
+
         } else {
-          wikiLink.data.hName = 'p';
+          wikiLink.data.hName = "p";
           wikiLink.data.hChildren = [
             {
-              type: 'text',
+              type: "text",
               value: `![[${target}]]`,
             },
           ];
         }
-      } else if (format === 'pdf') {
-        wikiLink.data.hName = 'iframe';
+      } else if (format === "pdf") {
+        wikiLink.data.hName = "iframe";
         wikiLink.data.hProperties = {
           className: classNames,
-          width: '100%',
+          width: "100%",
           src: `${hrefTemplate(link)}#toolbar=0`,
         };
       } else {
-        const hasDimensions = alias && /^\d+(x\d+)?$/.test(alias);
-        // Take the target as alt text except if alt name was provided [[target|alt text]]
-        const altText = hasDimensions || !alias ? target : alias;
-
-        wikiLink.data.hName = 'img';  
-        wikiLink.data.hProperties = {  
-            className: classNames,  
-            src: hrefTemplate(link),  
-            alt: altText
-        };  
-
-        if (hasDimensions) {  
-          const { width, height } = getImageSize(alias as string);  
-          Object.assign(wikiLink.data.hProperties, {  
-            width,  
-            height,  
-          }); 
-        }
+        wikiLink.data.hName = "img";
+        wikiLink.data.hProperties = {
+          className: classNames,
+          src: hrefTemplate(link),
+          alt: displayName,
+        };
       }
     } else {
-      wikiLink.data.hName = 'a';
+      wikiLink.data.hName = "a";
       wikiLink.data.hProperties = {
         className: classNames,
         href: hrefTemplate(link) + headingId,
       };
-      wikiLink.data.hChildren = [{ type: 'text', value: displayName }];
+      wikiLink.data.hChildren = [{ type: "text", value: displayName }];
     }
   }
 

packages/remark-wiki-link/src/lib/html.ts

@@ -1,24 +1,23 @@
-import { getImageSize } from './fromMarkdown';
-import { isSupportedFileFormat } from './isSupportedFileFormat';
+import { isSupportedFileFormat } from "./isSupportedFileFormat";
 
 const defaultWikiLinkResolver = (target: string) => {
   // for [[#heading]] links
   if (!target) {
     return [];
   }
-  let permalink = target.replace(/\/index$/, '');
+  let permalink = target.replace(/\/index$/, "");
   // TODO what to do with [[index]] link?
   if (permalink.length === 0) {
-    permalink = '/';
+    permalink = "/";
   }
   return [permalink];
 };
 
 export interface HtmlOptions {
   pathFormat?:
-  | 'raw' // default; use for regular relative or absolute paths
-  | 'obsidian-absolute' // use for Obsidian-style absolute paths (with no leading slash)
-  | 'obsidian-short'; // use for Obsidian-style shortened paths (shortest path possible)
+  | "raw" // default; use for regular relative or absolute paths
+  | "obsidian-absolute" // use for Obsidian-style absolute paths (with no leading slash)
+  | "obsidian-short"; // use for Obsidian-style shortened paths (shortest path possible)
   permalinks?: string[]; // list of permalinks to match possible permalinks of a wiki link against
   wikiLinkResolver?: (name: string) => string[]; // function to resolve wiki links to an array of possible permalinks
   newClassName?: string; // class name to add to links that don't have a matching permalink
@@ -29,11 +28,11 @@ export interface HtmlOptions {
 // Micromark HtmlExtension
 // https://github.com/micromark/micromark#htmlextension
 function html(opts: HtmlOptions = {}) {
-  const pathFormat = opts.pathFormat || 'raw';
+  const pathFormat = opts.pathFormat || "raw";
   const permalinks = opts.permalinks || [];
   const wikiLinkResolver = opts.wikiLinkResolver || defaultWikiLinkResolver;
-  const newClassName = opts.newClassName || 'new';
-  const wikiLinkClassName = opts.wikiLinkClassName || 'internal';
+  const newClassName = opts.newClassName || "new";
+  const wikiLinkClassName = opts.wikiLinkClassName || "internal";
   const defaultHrefTemplate = (permalink: string) => permalink;
   const hrefTemplate = opts.hrefTemplate || defaultHrefTemplate;
 
@@ -42,21 +41,21 @@ function html(opts: HtmlOptions = {}) {
   }
 
   function enterWikiLink() {
-    let stack = this.getData('wikiLinkStack');
-    if (!stack) this.setData('wikiLinkStack', (stack = []));
+    let stack = this.getData("wikiLinkStack");
+    if (!stack) this.setData("wikiLinkStack", (stack = []));
 
     stack.push({});
   }
 
   function exitWikiLinkTarget(token) {
     const target = this.sliceSerialize(token);
-    const current = top(this.getData('wikiLinkStack'));
+    const current = top(this.getData("wikiLinkStack"));
     current.target = target;
   }
 
   function exitWikiLinkAlias(token) {
     const alias = this.sliceSerialize(token);
-    const current = top(this.getData('wikiLinkStack'));
+    const current = top(this.getData("wikiLinkStack"));
     current.alias = alias;
   }
 
@@ -65,7 +64,7 @@ function html(opts: HtmlOptions = {}) {
     const { target, alias } = wikiLink;
     const isEmbed = token.isType === "embed";
     // eslint-disable-next-line no-useless-escape
-    const wikiLinkWithHeadingPattern = /^(.*?)(#.*)?$/u;
+    const wikiLinkWithHeadingPattern = /([\w\s\/\.-]*)(#.*)?/;
     const [, path, heading = ""] = target.match(wikiLinkWithHeadingPattern);
 
     const possibleWikiLinkPermalinks = wikiLinkResolver(path);
@@ -100,7 +99,7 @@ function html(opts: HtmlOptions = {}) {
     // remove leading # if the target is a heading on the same page
     const displayName = alias || target.replace(/^#/, "");
     // replace spaces with dashes and lowercase headings
-    const headingId = heading.replace(/\s+/g, "-").toLowerCase();
+    const headingId = heading.replace(/\s+/, "-").toLowerCase();
     let classNames = wikiLinkClassName;
     if (!matchingPermalink) {
       classNames += " " + newClassName;
@@ -112,9 +111,7 @@ function html(opts: HtmlOptions = {}) {
         // Temporarily render note transclusion as a regular wiki link
         if (!format) {
           this.tag(
-            `<a href="${hrefTemplate(
-              link + headingId
-            )}" class="${classNames} transclusion">`
+            `<a href="${hrefTemplate(link + headingId)}" class="${classNames} transclusion">`
           );
           this.raw(displayName);
           this.tag("</a>");
@@ -128,18 +125,11 @@ function html(opts: HtmlOptions = {}) {
           )}#toolbar=0" class="${classNames}" />`
         );
       } else {
-        const hasDimensions = alias && /^\d+(x\d+)?$/.test(alias);
-        // Take the target as alt text except if alt name was provided [[target|alt text]]
-        const altText = hasDimensions || !alias ? target : alias;
-        let imgAttributes = `src="${hrefTemplate(
-          link
-        )}" alt="${altText}" class="${classNames}"`;
-
-        if (hasDimensions) {
-          const { width, height } = getImageSize(alias as string);
-          imgAttributes += ` width="${width}" height="${height}"`;
-        }
-        this.tag(`<img ${imgAttributes} />`);
+        this.tag(
+          `<img src="${hrefTemplate(
+            link
+          )}" alt="${displayName}" class="${classNames}" />`
+        );
       }
     } else {
       this.tag(

packages/remark-wiki-link/src/lib/remarkWikiLink.ts

@@ -1,203 +1,42 @@
-import { isSupportedFileFormat } from './isSupportedFileFormat';
+import { toMarkdown } from "mdast-util-wiki-link";
+import { syntax, SyntaxOptions } from "./syntax";
+import { fromMarkdown, FromMarkdownOptions } from "./fromMarkdown";
 
-const defaultWikiLinkResolver = (target: string) => {
-  // for [[#heading]] links
-  if (!target) {
-    return [];
-  }
-  let permalink = target.replace(/\/index$/, '');
-  // TODO what to do with [[index]] link?
-  if (permalink.length === 0) {
-    permalink = '/';
-  }
-  return [permalink];
-};
+let warningIssued = false;
 
-export interface FromMarkdownOptions {
-  pathFormat?:
-    | 'raw' // default; use for regular relative or absolute paths
-    | 'obsidian-absolute' // use for Obsidian-style absolute paths (with no leading slash)
-    | 'obsidian-short'; // use for Obsidian-style shortened paths (shortest path possible)
-  permalinks?: string[]; // list of permalinks to match possible permalinks of a wiki link against
-  wikiLinkResolver?: (name: string) => string[]; // function to resolve wiki links to an array of possible permalinks
-  newClassName?: string; // class name to add to links that don't have a matching permalink
-  wikiLinkClassName?: string; // class name to add to all wiki links
-  hrefTemplate?: (permalink: string) => string; // function to generate the href attribute of a link
-}
+type RemarkWikiLinkOptions = FromMarkdownOptions & SyntaxOptions;
 
-export function getImageSize(size: string) {
-  // eslint-disable-next-line prefer-const
-  let [width, height] = size.split('x');
+function remarkWikiLink(opts: RemarkWikiLinkOptions = {}) {
+  const data = this.data(); // this is a reference to the processor
 
-  if (!height) height = width;
-
-  return { width, height };
-}
-
-// mdas-util-from-markdown extension
-// https://github.com/syntax-tree/mdast-util-from-markdown#extension
-function fromMarkdown(opts: FromMarkdownOptions = {}) {
-  const pathFormat = opts.pathFormat || 'raw';
-  const permalinks = opts.permalinks || [];
-  const wikiLinkResolver = opts.wikiLinkResolver || defaultWikiLinkResolver;
-  const newClassName = opts.newClassName || 'new';
-  const wikiLinkClassName = opts.wikiLinkClassName || 'internal';
-  const defaultHrefTemplate = (permalink: string) => permalink;
-
-  const hrefTemplate = opts.hrefTemplate || defaultHrefTemplate;
-
-  function top(stack) {
-    return stack[stack.length - 1];
+  function add(field, value) {
+    if (data[field]) data[field].push(value);
+    else data[field] = [value];
   }
 
-  function enterWikiLink(token) {
-    this.enter(
-      {
-        type: 'wikiLink',
-        data: {
-          isEmbed: token.isType === 'embed',
-          target: null, // the target of the link, e.g. "Foo Bar#Heading" in "[[Foo Bar#Heading]]"
-          alias: null, // the alias of the link, e.g. "Foo" in "[[Foo Bar|Foo]]"
-          permalink: null, // TODO shouldn't this be named just "link"?
-          exists: null, // TODO is this even needed here?
-          // fields for mdast-util-to-hast (used e.g. by remark-rehype)
-          hName: null,
-          hProperties: null,
-          hChildren: null,
-        },
-      },
-      token
+  if (
+    !warningIssued &&
+    ((this.Parser &&
+      this.Parser.prototype &&
+      this.Parser.prototype.blockTokenizers) ||
+      (this.Compiler &&
+        this.Compiler.prototype &&
+        this.Compiler.prototype.visitors))
+  ) {
+    warningIssued = true;
+    console.warn(
+      "[remark-wiki-link] Warning: please upgrade to remark 13 to use this plugin"
     );
   }
 
-  function exitWikiLinkTarget(token) {
-    const target = this.sliceSerialize(token);
-    const current = top(this.stack);
-    current.data.target = target;
-  }
-
-  function exitWikiLinkAlias(token) {
-    const alias = this.sliceSerialize(token);
-    const current = top(this.stack);
-    current.data.alias = alias;
-  }
-
-  function exitWikiLink(token) {
-    const wikiLink = top(this.stack)
-    const {
-        data: {isEmbed, target, alias},
-    } = wikiLink;
-    this.exit(token);
-    // eslint-disable-next-line no-useless-escape
-    const wikiLinkWithHeadingPattern = /^(.*?)(#.*)?$/u;
-    const [, path, heading = ''] = target.match(wikiLinkWithHeadingPattern);
-
-    const possibleWikiLinkPermalinks = wikiLinkResolver(path);
-
-    const matchingPermalink = permalinks.find((e) => {
-      return possibleWikiLinkPermalinks.find((p) => {
-        if (pathFormat === 'obsidian-short') {
-          if (e === p || e.endsWith(p)) {
-            return true;
-          }
-        } else if (pathFormat === 'obsidian-absolute') {
-          if (e === '/' + p) {
-            return true;
-          }
-        } else {
-          if (e === p) {
-            return true;
-          }
-        }
-        return false;
-      });
-    });
-
-    // TODO this is ugly
-    const link =
-      matchingPermalink ||
-      (pathFormat === 'obsidian-absolute'
-        ? '/' + possibleWikiLinkPermalinks[0]
-        : possibleWikiLinkPermalinks[0]) ||
-      '';
-
-    wikiLink.data.exists = !!matchingPermalink;
-    wikiLink.data.permalink = link;
-    // remove leading # if the target is a heading on the same page
-    const displayName = alias || target.replace(/^#/, '');
-    const headingId = heading.replace(/\s+/g, '-').toLowerCase();
-    let classNames = wikiLinkClassName;
-    if (!matchingPermalink) {
-      classNames += ' ' + newClassName;
-    }
-
-    if (isEmbed) {
-      const [isSupportedFormat, format] = isSupportedFileFormat(target);
-      if (!isSupportedFormat) {
-        // Temporarily render note transclusion as a regular wiki link
-        if (!format) {
-          wikiLink.data.hName = 'a';
-          wikiLink.data.hProperties = {
-            className: classNames + ' ' + 'transclusion',
-            href: hrefTemplate(link) + headingId,
-          };
-          wikiLink.data.hChildren = [{ type: 'text', value: displayName }];
-        } else {
-          wikiLink.data.hName = 'p';
-          wikiLink.data.hChildren = [
-            {
-              type: 'text',
-              value: `![[${target}]]`,
-            },
-          ];
-        }
-      } else if (format === 'pdf') {
-        wikiLink.data.hName = 'iframe';
-        wikiLink.data.hProperties = {
-          className: classNames,
-          width: '100%',
-          src: `${hrefTemplate(link)}#toolbar=0`,
-        };
-      } else {
-        const hasDimensions = alias && /^\d+(x\d+)?$/.test(alias);
-        // Take the target as alt text except if alt name was provided [[target|alt text]]
-        const altText = hasDimensions || !alias ? target : alias;
-
-        wikiLink.data.hName = 'img';
-        wikiLink.data.hProperties = {
-            className: classNames,
-            src: hrefTemplate(link),
-            alt: altText
-        };
-
-        if (hasDimensions) {
-          const { width, height } = getImageSize(alias as string);
-          Object.assign(wikiLink.data.hProperties, {
-            width,
-            height,
-          });
-        }
-      }
-    } else {
-      wikiLink.data.hName = 'a';
-      wikiLink.data.hProperties = {
-        className: classNames,
-        href: hrefTemplate(link) + headingId,
-      };
-      wikiLink.data.hChildren = [{ type: 'text', value: displayName }];
-    }
-  }
-
-  return {
-    enter: {
-      wikiLink: enterWikiLink,
-    },
-    exit: {
-      wikiLinkTarget: exitWikiLinkTarget,
-      wikiLinkAlias: exitWikiLinkAlias,
-      wikiLink: exitWikiLink,
-    },
-  };
+  // add extensions to packages used by remark-parse
+  // micromark extensions
+  add("micromarkExtensions", syntax(opts));
+  // mdast-util-from-markdown extensions
+  add("fromMarkdownExtensions", fromMarkdown(opts));
+  // mdast-util-to-markdown extensions
+  add("toMarkdownExtensions", toMarkdown(opts));
 }
 
-export { fromMarkdown };
+export default remarkWikiLink;
+export { remarkWikiLink };

packages/remark-wiki-link/src/utils/getPermalinks.ts

@@ -38,5 +38,6 @@ const defaultPathToPermalinkFunc = (
     .replace(markdownFolder, "") // make the permalink relative to the markdown folder
     .replace(/\.(mdx|md)/, "")
     .replace(/\\/g, "/") // replace windows backslash with forward slash
+    .replace(/\/index$/, ""); // remove index from the end of the permalink
   return permalink.length > 0 ? permalink : "/"; // for home page
 };

packages/remark-wiki-link/test/getPermalinks.spec.ts

@@ -1,20 +1,23 @@
 import * as path from "path";
+// import * as url from "url";
 import { getPermalinks } from "../src/utils";
 
+// const __dirname = url.fileURLToPath(new URL(".", import.meta.url));
+// const markdownFolder = path.join(__dirname, "/fixtures/content");
 const markdownFolder = path.join(
   ".",
-  "test/fixtures/content"
+  "/packages/remark-wiki-link/test/fixtures/content"
 );
 
 describe("getPermalinks", () => {
   test("should return an array of permalinks", () => {
     const expectedPermalinks = [
-      "/README",
+      "/", // /index.md
       "/abc",
       "/blog/first-post",
       "/blog/Second Post",
       "/blog/third-post",
-      "/blog/README",
+      "/blog", // /blog/index.md
       "/blog/tutorials/first-tutorial",
       "/assets/Pasted Image 123.png",
     ];
@@ -25,4 +28,35 @@ describe("getPermalinks", () => {
       expect(expectedPermalinks).toContain(permalink);
     });
   });
+
+  test("should return an array of permalinks with custom path -> permalink converter function", () => {
+    const expectedPermalinks = [
+      "/", // /index.md
+      "/abc",
+      "/blog/first-post",
+      "/blog/second-post",
+      "/blog/third-post",
+      "/blog", // /blog/index.md
+      "/blog/tutorials/first-tutorial",
+      "/assets/pasted-image-123.png",
+    ];
+
+    const func = (filePath: string, markdownFolder: string) => {
+      const permalink = filePath
+        .replace(markdownFolder, "") // make the permalink relative to the markdown folder
+        .replace(/\.(mdx|md)/, "")
+        .replace(/\\/g, "/") // replace windows backslash with forward slash
+        .replace(/\/index$/, "") // remove index from the end of the permalink
+        .replace(/ /g, "-") // replace spaces with hyphens
+        .toLowerCase(); // convert to lowercase
+
+      return permalink.length > 0 ? permalink : "/"; // for home page
+    };
+
+    const permalinks = getPermalinks(markdownFolder, [/\.DS_Store/], func);
+    expect(permalinks).toHaveLength(expectedPermalinks.length);
+    permalinks.forEach((permalink) => {
+      expect(expectedPermalinks).toContain(permalink);
+    });
+  });
 });

packages/remark-wiki-link/test/micromarkExtensionWikiLink.spec.ts

@@ -48,7 +48,7 @@ describe("micromark-extension-wiki-link", () => {
           html({
             permalinks: ["/some/folder/Wiki Link"],
             pathFormat: "obsidian-short",
-          }) as any, // TODO type fix
+          }) as any // TODO type fix
         ],
       });
       expect(serialized).toBe(
@@ -75,7 +75,7 @@ describe("micromark-extension-wiki-link", () => {
           html({
             permalinks: ["/some/folder/Wiki Link"],
             pathFormat: "obsidian-absolute",
-          }) as any, // TODO type fix
+          }) as any // TODO type fix
         ],
       });
       expect(serialized).toBe(
@@ -97,14 +97,10 @@ describe("micromark-extension-wiki-link", () => {
     });
 
     test("parses a wiki link with heading and alias", () => {
-      const serialized = micromark(
-        "[[Wiki Link#Some Heading|Alias]]",
-        "ascii",
-        {
-          extensions: [syntax()],
-          htmlExtensions: [html() as any], // TODO type fix
-        }
-      );
+      const serialized = micromark("[[Wiki Link#Some Heading|Alias]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html() as any], // TODO type fix
+      });
       // note: lowercased and hyphenated heading
       expect(serialized).toBe(
         '<p><a href="Wiki Link#some-heading" class="internal new">Alias</a></p>'
@@ -138,7 +134,7 @@ describe("micromark-extension-wiki-link", () => {
         extensions: [syntax()],
         htmlExtensions: [html() as any], // TODO type fix
       });
-      expect(serialized).toBe('<p>![[My Image.xyz]]</p>');
+      expect(serialized).toBe("<p>![[My Image.xyz]]</p>");
     });
 
     test("parses and image ambed with a matching permalink", () => {
@@ -151,28 +147,6 @@ describe("micromark-extension-wiki-link", () => {
       );
     });
 
-    // TODO: Fix alt attribute
-    test("Can identify the dimensions of the image if exists", () => {
-      const serialized = micromark("![[My Image.jpg|200]]", "ascii", {
-        extensions: [syntax()],
-        htmlExtensions: [html({ permalinks: ["My Image.jpg"] }) as any], // TODO type fix
-      });
-      expect(serialized).toBe(
-        '<p><img src="My Image.jpg" alt="My Image.jpg" class="internal" width="200" height="200" /></p>'
-      );
-    });
-
-    // TODO: Fix alt attribute
-    test("Can identify the dimensions of the image if exists", () => {
-      const serialized = micromark("![[My Image.jpg|200x200]]", "ascii", {
-        extensions: [syntax()],
-        htmlExtensions: [html({ permalinks: ["My Image.jpg"] }) as any], // TODO type fix
-      });
-      expect(serialized).toBe(
-        '<p><img src="My Image.jpg" alt="My Image.jpg" class="internal" width="200" height="200" /></p>'
-      );
-    });
-
     test("parses an image embed with a matching permalink and Obsidian-style shortedned path", () => {
       const serialized = micromark("![[My Image.jpg]]", {
         extensions: [syntax()],
@@ -180,7 +154,7 @@ describe("micromark-extension-wiki-link", () => {
           html({
             permalinks: ["/assets/My Image.jpg"],
             pathFormat: "obsidian-short",
-          }) as any, // TODO type fix
+          }) as any // TODO type fix
         ],
       });
       expect(serialized).toBe(
@@ -215,7 +189,7 @@ describe("micromark-extension-wiki-link", () => {
         extensions: [syntax()],
         htmlExtensions: [html() as any], // TODO type fix
       });
-      expect(serialized).toBe('<p>[[Wiki Link</p>');
+      expect(serialized).toBe("<p>[[Wiki Link</p>");
     });
 
     test("doesn't parse a wiki link with one missing closing bracket", () => {
@@ -223,7 +197,7 @@ describe("micromark-extension-wiki-link", () => {
         extensions: [syntax()],
         htmlExtensions: [html() as any], // TODO type fix
       });
-      expect(serialized).toBe('<p>[[Wiki Link]</p>');
+      expect(serialized).toBe("<p>[[Wiki Link]</p>");
     });
 
     test("doesn't parse a wiki link with a missing opening bracket", () => {
@@ -231,7 +205,7 @@ describe("micromark-extension-wiki-link", () => {
         extensions: [syntax()],
         htmlExtensions: [html() as any], // TODO type fix
       });
-      expect(serialized).toBe('<p>[Wiki Link]]</p>');
+      expect(serialized).toBe("<p>[Wiki Link]]</p>");
     });
 
     test("doesn't parse a wiki link in single brackets", () => {
@@ -239,7 +213,7 @@ describe("micromark-extension-wiki-link", () => {
         extensions: [syntax()],
         htmlExtensions: [html() as any], // TODO type fix
       });
-      expect(serialized).toBe('<p>[Wiki Link]</p>');
+      expect(serialized).toBe("<p>[Wiki Link]</p>");
     });
   });
 
@@ -251,7 +225,7 @@ describe("micromark-extension-wiki-link", () => {
           html({
             newClassName: "test-new",
             wikiLinkClassName: "test-wiki-link",
-          }) as any, // TODO type fix
+          }) as any // TODO type fix
         ],
       });
       expect(serialized).toBe(
@@ -277,7 +251,7 @@ describe("micromark-extension-wiki-link", () => {
             wikiLinkResolver: (page) => [
               page.replace(/\s+/, "-").toLowerCase(),
             ],
-          }) as any, // TODO type fix
+          }) as any // TODO type fix
         ],
       });
       expect(serialized).toBe(
@@ -286,6 +260,56 @@ describe("micromark-extension-wiki-link", () => {
     });
   });
 
+  test("parses wiki links to index files", () => {
+    const serialized = micromark("[[/some/folder/index]]", "ascii", {
+      extensions: [syntax()],
+      htmlExtensions: [html() as any], // TODO type fix
+    });
+    expect(serialized).toBe(
+      '<p><a href="/some/folder" class="internal new">/some/folder/index</a></p>'
+    );
+  });
+
+  describe("other", () => {
+    test("parses a wiki link to some index page in a folder with no matching permalink", () => {
+      const serialized = micromark("[[/some/folder/index]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html() as any], // TODO type fix
+      });
+      expect(serialized).toBe(
+        '<p><a href="/some/folder" class="internal new">/some/folder/index</a></p>'
+      );
+    });
+
+    test("parses a wiki link to some index page in a folder with a matching permalink", () => {
+      const serialized = micromark("[[/some/folder/index]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html({ permalinks: ["/some/folder"] }) as any], // TODO type fix
+      });
+      expect(serialized).toBe(
+        '<p><a href="/some/folder" class="internal">/some/folder/index</a></p>'
+      );
+    });
+
+    test("parses a wiki link to home index page with no matching permalink", () => {
+      const serialized = micromark("[[/index]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html() as any], // TODO type fix
+      });
+      expect(serialized).toBe(
+        '<p><a href="/" class="internal new">/index</a></p>'
+      );
+    });
+
+    test("parses a wiki link to home index page with a matching permalink", () => {
+      const serialized = micromark("[[/index]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html({ permalinks: ["/"] }) as any], // TODO type fix
+      });
+      expect(serialized).toBe('<p><a href="/" class="internal">/index</a></p>');
+    });
+  });
+
   describe("transclusions", () => {
     test("parsers a transclusion as a regular wiki link", () => {
       const serialized = micromark("![[Some Page]]", "ascii", {
@@ -297,14 +321,4 @@ describe("micromark-extension-wiki-link", () => {
       );
     });
   });
-
-  describe("Links with special characters", () => {
-    test("parses a link with special characters and symbols", () => {
-      const serialized = micromark("[[li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#LI NK-W(i)th-àcèô íã_a(n)d_uNdErlinE!:ª%@'*º$ °~./\\]]", "ascii", {
-        extensions: [syntax()],
-        htmlExtensions: [html() as any],
-      });
-      expect(serialized).toBe(`<p><a href="li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô-íã_a(n)d_underline!:ª%@'*º$-°~./\\" class="internal new">li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#LI NK-W(i)th-àcèô íã_a(n)d_uNdErlinE!:ª%@'*º$ °~./\\</a></p>`);
-    });
-  });
 });

packages/remark-wiki-link/test/remarkWikiLink.spec.ts

@@ -246,28 +246,6 @@ describe("remark-wiki-link", () => {
         expect(node.data?.hName).toEqual("img");
         expect((node.data?.hProperties as any).src).toEqual("My Image.png");
         expect((node.data?.hProperties as any).alt).toEqual("My Image.png");
-        expect((node.data?.hProperties as any).width).toBeUndefined();
-        expect((node.data?.hProperties as any).height).toBeUndefined();
-      });
-    });
-
-    test("Can identify the dimensions of the image if exists", () => {
-      const processor = unified().use(markdown).use(wikiLinkPlugin);
-
-      let ast = processor.parse("![[My Image.png|132x612]]");
-      ast = processor.runSync(ast);
-
-      expect(select("wikiLink", ast)).not.toEqual(null);
-
-      visit(ast, "wikiLink", (node: Node) => {
-        expect(node.data?.isEmbed).toEqual(true);
-        expect(node.data?.target).toEqual("My Image.png");
-        expect(node.data?.permalink).toEqual("My Image.png");
-        expect(node.data?.hName).toEqual("img");
-        expect((node.data?.hProperties as any).src).toEqual("My Image.png");
-        expect((node.data?.hProperties as any).alt).toEqual("My Image.png");
-        expect((node.data?.hProperties as any).width).toBe("132");
-        expect((node.data?.hProperties as any).height).toBe("612");
       });
     });
 
@@ -383,36 +361,6 @@ describe("remark-wiki-link", () => {
     });
   });
 
-  describe("Links with special characters", () => {
-    test("parses a link with special characters and symbols", () => {
-      const processor = unified().use(markdown).use(wikiLinkPlugin);
-
-      let ast = processor.parse(
-        "[[li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô íã_a(n)D_UNDERLINE!:ª%@'*º$ °~./\\]]"
-      );
-      ast = processor.runSync(ast);
-      expect(select("wikiLink", ast)).not.toEqual(null);
-
-      visit(ast, "wikiLink", (node: Node) => {
-        expect(node.data?.exists).toEqual(false);
-        expect(node.data?.permalink).toEqual(
-          "li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\"
-        );
-        expect(node.data?.alias).toEqual(null);
-        expect(node.data?.hName).toEqual("a");
-        expect((node.data?.hProperties as any).className).toEqual(
-          "internal new"
-        );
-        expect((node.data?.hProperties as any).href).toEqual(
-          "li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô-íã_a(n)d_underline!:ª%@'*º$-°~./\\"
-        );
-        expect((node.data?.hChildren as any)[0].value).toEqual(
-          "li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô íã_a(n)D_UNDERLINE!:ª%@'*º$ °~./\\"
-        );
-      });
-    });
-  });
-
   describe("invalid wiki links", () => {
     test("doesn't parse a wiki link with two missing closing brackets", () => {
       const processor = unified().use(markdown).use(wikiLinkPlugin);
@@ -485,6 +433,109 @@ describe("remark-wiki-link", () => {
     });
   });
 
+  test("parses wiki links to index files", () => {
+    const processor = unified().use(markdown).use(wikiLinkPlugin);
+
+    let ast = processor.parse("[[/some/folder/index]]");
+    ast = processor.runSync(ast);
+
+    expect(select("wikiLink", ast)).not.toEqual(null);
+
+    visit(ast, "wikiLink", (node: Node) => {
+      expect(node.data?.exists).toEqual(false);
+      expect(node.data?.permalink).toEqual("/some/folder");
+      expect(node.data?.alias).toEqual(null);
+      expect(node.data?.hName).toEqual("a");
+      expect((node.data?.hProperties as any).className).toEqual("internal new");
+      expect((node.data?.hProperties as any).href).toEqual("/some/folder");
+      expect((node.data?.hChildren as any)[0].value).toEqual(
+        "/some/folder/index"
+      );
+    });
+  });
+
+  describe("other", () => {
+    test("parses a wiki link to some index page in a folder with no matching permalink", () => {
+      const processor = unified().use(markdown).use(wikiLinkPlugin);
+
+      let ast = processor.parse("[[/some/folder/index]]");
+      ast = processor.runSync(ast);
+
+      visit(ast, "wikiLink", (node: Node) => {
+        expect(node.data?.exists).toEqual(false);
+        expect(node.data?.permalink).toEqual("/some/folder");
+        expect(node.data?.alias).toEqual(null);
+        expect(node.data?.hName).toEqual("a");
+        expect((node.data?.hProperties as any).className).toEqual(
+          "internal new"
+        );
+        expect((node.data?.hProperties as any).href).toEqual("/some/folder");
+        expect((node.data?.hChildren as any)[0].value).toEqual(
+          "/some/folder/index"
+        );
+      });
+    });
+
+    test("parses a wiki link to some index page in a folder with a matching permalink", () => {
+      const processor = unified()
+        .use(markdown)
+        .use(wikiLinkPlugin, { permalinks: ["/some/folder"] });
+
+      let ast = processor.parse("[[/some/folder/index]]");
+      ast = processor.runSync(ast);
+
+      visit(ast, "wikiLink", (node: Node) => {
+        expect(node.data?.exists).toEqual(true);
+        expect(node.data?.permalink).toEqual("/some/folder");
+        expect(node.data?.alias).toEqual(null);
+        expect(node.data?.hName).toEqual("a");
+        expect((node.data?.hProperties as any).className).toEqual("internal");
+        expect((node.data?.hProperties as any).href).toEqual("/some/folder");
+        expect((node.data?.hChildren as any)[0].value).toEqual(
+          "/some/folder/index"
+        );
+      });
+    });
+
+    test("parses a wiki link to home index page with no matching permalink", () => {
+      const processor = unified().use(markdown).use(wikiLinkPlugin);
+
+      let ast = processor.parse("[[/index]]");
+      ast = processor.runSync(ast);
+
+      visit(ast, "wikiLink", (node: Node) => {
+        expect(node.data?.exists).toEqual(false);
+        expect(node.data?.permalink).toEqual("/");
+        expect(node.data?.alias).toEqual(null);
+        expect(node.data?.hName).toEqual("a");
+        expect((node.data?.hProperties as any).className).toEqual(
+          "internal new"
+        );
+        expect((node.data?.hProperties as any).href).toEqual("/");
+        expect((node.data?.hChildren as any)[0].value).toEqual("/index");
+      });
+    });
+
+    test("parses a wiki link to home index page with a matching permalink", () => {
+      const processor = unified()
+        .use(markdown)
+        .use(wikiLinkPlugin, { permalinks: ["/"] });
+
+      let ast = processor.parse("[[/index]]");
+      ast = processor.runSync(ast);
+
+      visit(ast, "wikiLink", (node: Node) => {
+        expect(node.data?.exists).toEqual(true);
+        expect(node.data?.permalink).toEqual("/");
+        expect(node.data?.alias).toEqual(null);
+        expect(node.data?.hName).toEqual("a");
+        expect((node.data?.hProperties as any).className).toEqual("internal");
+        expect((node.data?.hProperties as any).href).toEqual("/");
+        expect((node.data?.hChildren as any)[0].value).toEqual("/index");
+      });
+    });
+  });
+
   describe("transclusions", () => {
     test("replaces a transclusion with a regular wiki link", () => {
       const processor = unified().use(markdown).use(wikiLinkPlugin);

site/components/JSONLD.tsx

@@ -12,7 +12,7 @@ export default function JSONLD({
     return <></>;
   }
 
-  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://portaljs.com';
+  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://portaljs.org';
   const pageUrl = `${baseUrl}/${meta.urlPath}`;
 
   const imageMatches = source.match(

site/components/Layout.tsx

@@ -81,6 +81,7 @@ export default function Layout({
     }
     return section.children.findIndex(isActive) > -1;
   }
+
   return (
     <>
       {title && <NextSeo title={title} description={description} />}

site/components/Showcases.tsx

@@ -22,41 +22,11 @@ const items = [
     sourceUrl: 'https://github.com/FCSCOpendata/frontend',
   },
   {
-    title: 'Frictionless Data',
-    href: 'https://datahub.io/core/co2-ppm',
-    repository: 'https://github.com/datopian/datahub/tree/main/examples/dataset-frictionless',
-    image: '/images/showcases/frictionless-capture.png',
-    description: 'Progressive open-source framework for building data infrastructure - data management, data integration, data flows, etc. It includes various data standards and provides software to work with data.',
+    title: 'Datahub Open Data',
+    href: 'https://opendata.datahub.io/',
+    image: '/images/showcases/datahub.webp',
+    description: 'Demo Data Portal by DataHub',
   },
-  {
-    title: "OpenSpending",
-    image: "/images/showcases/openspending.png",
-    href: "https://www.openspending.org",
-    repository: 'https://github.com/datopian/datahub/tree/main/examples/openspending',
-    description: "OpenSpending is a free, open and global platform to search, visualise and analyse fiscal data in the public sphere."
-  },
-  {
-    title: "FiveThirtyEight",
-    image: "/images/showcases/fivethirtyeight.png",
-    href: "https://fivethirtyeight.portaljs.org/",
-    repository: 'https://github.com/datopian/datahub/tree/main/examples/fivethirtyeight',
-    description: "This is a replica of data.fivethirtyeight.com using PortalJS."
-  },
-  {
-    title: "Github Datasets",
-    image: "/images/showcases/github-datasets.png",
-    href: "https://example.portaljs.org/",
-    repository: 'https://github.com/datopian/datahub/tree/main/examples/github-backed-catalog',
-    description: "A simple data catalog that get its data from a list of GitHub repos that serve as datasets."
-  },
-  {
-    title: "Hatespeech Data",
-    image: "/images/showcases/turing.png",
-    href: "https://hatespeechdata.com/",
-    repository: 'https://github.com/datopian/datahub/tree/main/examples/turing',
-    description: "Datasets annotated for hate speech, online abuse, and offensive language which are useful for training a natural language processing system to detect this online abuse."
-  },
-  
 ];
 
 export default function Showcases() {

site/components/ShowcasesItem.tsx

@@ -1,6 +1,10 @@
 export default function ShowcasesItem({ item }) {
   return (
-    <div className="rounded overflow-hidden group relative border-1 shadow-lg">
+    <a
+      className="rounded overflow-hidden group relative border-1 shadow-lg"
+      target="_blank"
+      href={item.href}
+    >
       <div
         className="bg-cover bg-no-repeat bg-top aspect-video w-full group-hover:blur-sm group-hover:scale-105 transition-all duration-200"
         style={{ backgroundImage: `url(${item.image})` }}
@@ -12,48 +16,9 @@ export default function ShowcasesItem({ item }) {
           <div className="text-center text-primary-dark">
             <span className="text-xl font-semibold">{item.title}</span>
             <p className="text-base font-medium">{item.description}</p>
-            <div className="flex justify-center mt-2 gap-2 ">
-              {item.href && (
-                <a
-                  target="_blank"
-                  className=" text-white w-8 h-8 p-1 bg-primary rounded-full hover:scale-110 transition cursor-pointer z-50"
-                  rel="noreferrer"
-                  href={item.href}
-                >
-                  <svg
-                    xmlns="http://www.w3.org/2000/svg"
-                    viewBox="0 0 420 420"
-                    stroke="white"
-                    fill="none"
-                  >
-                    <path stroke-width="26" d="M209,15a195,195 0 1,0 2,0z" />
-                    <path
-                      stroke-width="18"
-                      d="m210,15v390m195-195H15M59,90a260,260 0 0,0 302,0 m0,240 a260,260 0 0,0-302,0M195,20a250,250 0 0,0 0,382 m30,0 a250,250 0 0,0 0-382"
-                    />
-                  </svg>
-                </a>
-              )}
-              {item.repository && (
-                <a
-                  target="_blank"
-                  rel="noreferrer"
-                  className="w-8 h-8 bg-black rounded-full p-1 hover:scale-110 transition cursor-pointer z-50"
-                  href={item.repository}
-                >
-                  <svg
-                    aria-hidden="true"
-                    viewBox="0 0 16 16"
-                    fill="currentColor"
-                  >
-                    <path d="M8 0C3.58 0 0 3.58 0 8C0 11.54 2.29 14.53 5.47 15.59C5.87 15.66 6.02 15.42 6.02 15.21C6.02 15.02 6.01 14.39 6.01 13.72C4 14.09 3.48 13.23 3.32 12.78C3.23 12.55 2.84 11.84 2.5 11.65C2.22 11.5 1.82 11.13 2.49 11.12C3.12 11.11 3.57 11.7 3.72 11.94C4.44 13.15 5.59 12.81 6.05 12.6C6.12 12.08 6.33 11.73 6.56 11.53C4.78 11.33 2.92 10.64 2.92 7.58C2.92 6.71 3.23 5.99 3.74 5.43C3.66 5.23 3.38 4.41 3.82 3.31C3.82 3.31 4.49 3.1 6.02 4.13C6.66 3.95 7.34 3.86 8.02 3.86C8.7 3.86 9.38 3.95 10.02 4.13C11.55 3.09 12.22 3.31 12.22 3.31C12.66 4.41 12.38 5.23 12.3 5.43C12.81 5.99 13.12 6.7 13.12 7.58C13.12 10.65 11.25 11.33 9.47 11.53C9.76 11.78 10.01 12.26 10.01 13.01C10.01 14.08 10 14.94 10 15.21C10 15.42 10.15 15.67 10.55 15.59C13.71 14.53 16 11.53 16 8C16 3.58 12.42 0 8 0Z" />
-                  </svg>
-                </a>
-              )}
-            </div>
           </div>
         </div>
       </div>
-    </div>
+    </a>
   );
 }

site/content/blog/enhancing-geospatial-data-visualization-with-portaljs.md

@@ -1,23 +1,21 @@
 ---
-title: 'Adding Maps to PortalJS: Enhancing Geospatial Data Visualization with PortalJS'
+title: 'Enhancing Geospatial Data Visualization with PortalJS'
 date: 2023-07-18
 authors: ['João Demenech', 'Luccas Mateus', 'Yoana Popova']
 filetype: 'blog'
 ---
 
-This post walks you though adding maps and geospatial visualizations to PortalJS.
+Are you keen on building rich and interactive data portals? Do you find value in the power and flexibility of JavaScript, Nextjs, and React? In that case, allow us to introduce you to [PortalJS](https://portaljs.org/), a state-of-the-art framework leveraging these technologies to help you build amazing data portals.
 
-Are you interested in building rich and interactive data portals? Do you find value in the power and flexibility of JavaScript, Nextjs, and React? If so, [PortalJS](https://portaljs.com/) is for you. It's a state-of-the-art framework leveraging these technologies to help you build rich data portals.
+Perhaps you already understand that the effective data visualization lies in the adept utilization of various data components. Within [PortalJS](https://portaljs.org/), we take data visualization a step further. It's not just about displaying data - it's about telling a captivating story through the strategic orchestration of a diverse array of data components.
 
-Effective data visualization lies in the use  of various data components. Within [PortalJS](https://portaljs.com/), we take data visualization a step further. It's not just about displaying data - it's about telling a story through combining a variety of data components.
-
-In this post we will share our latest enhancement to PortalJS: maps, a powerful tool for visualizing geospatial data. In this post, we will to take you on a tour of our experiments and progress in enhancing map functionalities on PortalJS. The journey is still in its early stages, with new facets being unveiled and refined as we perfect our API.
+We are now eager to share our latest enhancement to [PortalJS](https://portaljs.org/): maps, a powerful tool for visualizing geospatial data. In this post, we will to take you on a tour of our experiments and progress in enhancing map functionalities on [PortalJS](https://portaljs.org/). Our journey into this innovative feature is still in its early stages, with new facets being unveiled and refined as we perfect our API. Still, this exciting development opens a new avenue for visualizing data, enhancing your ability to convey complex geospatial information with clarity and precision.
 
 ## Exploring Map Formats
 
-Maps play a crucial role in geospatial data visualization. Several formats exist for storing and sharing this type of data, with GeoJSON, KML, and shapefiles being among the most popular. As a prominent figure in the field of open-source data portal platforms, [PortalJS](https://portaljs.com/) strives to support as many map formats as possible.
+Maps play a crucial role in geospatial data visualization. Several formats exist for storing and sharing this type of data, with GeoJSON, KML, and shapefiles being among the most popular. As a prominent figure in the field of open-source data portal platforms, [PortalJS](https://portaljs.org/) strives to support as many map formats as possible.
 
-Taking inspiration from the ckanext-geoview extension, we currently support KML and GeoJSON formats in [PortalJS](https://portaljs.com/). This remarkable extension is a plugin for CKAN, the world’s leading open source data management system, that enables users to visualize geospatial data in diverse formats on an interactive map. Apart from KML and GeoJSON formats support, our roadmap entails extending compatibility to encompass all other formats supported by ckanext-geoview. Rest assured, we are committed to empowering users with a wide array of map format options in the future.
+Taking inspiration from the ckanext-geoview extension, we currently support KML and GeoJSON formats in [PortalJS](https://portaljs.org/). This remarkable extension is a plugin for CKAN, the world’s leading open source data management system, that enables users to visualize geospatial data in diverse formats on an interactive map. Apart from KML and GeoJSON formats support, our roadmap entails extending compatibility to encompass all other formats supported by ckanext-geoview. Rest assured, we are committed to empowering users with a wide array of map format options in the future.
 
 So, what makes these formats special?
 
@@ -27,7 +25,7 @@ So, what makes these formats special?
 
 ## Unveiling the Power of Leaflet and OpenLayers
 
-To display maps in [PortalJS](https://portaljs.com/), we utilize two powerful JavaScript libraries for creating interactive maps based on different layers: Leaflet and OpenLayers. Each offers distinct advantages (and disadvantages), inspiring us to integrate both and give users the flexibility to choose.
+To display maps in [PortalJS](https://portaljs.org/), we utilize two powerful JavaScript libraries for creating interactive maps based on different layers: Leaflet and OpenLayers. Each offers distinct advantages (and disadvantages), inspiring us to integrate both and give users the flexibility to choose.
 
 Leaflet is the leading open-source JavaScript library known for its mobile-friendly, interactive maps. With its compact size (just 42 KB of JS), it provides all the map features most developers need. Leaflet is designed with simplicity, performance and usability in mind. It works efficiently across all major desktop and mobile platforms.
 
@@ -59,8 +57,8 @@ Users can also choose a region of focus, which will depend on the data, by setti
 
 Through our ongoing enhancements to the [PortalJS library](https://storybook.portaljs.org/), we aim to empower users to create engaging and informative data portals featuring diverse map formats and data components.
 
-Why not give [PortalJS](https://portaljs.com/) a try today and discover the possibilities for your own data portals? To get started, check out our comprehensive documentation here: [PortalJS Documentation](https://portaljs.com/opensource).
+Why not give [PortalJS](https://portaljs.org/) a try today and discover the possibilities for your own data portals? To get started, check out our comprehensive documentation here: [PortalJS Documentation](https://portaljs.org/docs).
 
-Have questions or comments about using [PortalJS](https://portaljs.com/) for your data portals? Feel free to share your thoughts on our [Discord channel](https://discord.com/invite/EeyfGrGu4U). We're here to help you make the most of your data.
+Have questions or comments about using [PortalJS](https://portaljs.org/) for your data portals? Feel free to share your thoughts on our [Discord channel](https://discord.com/invite/EeyfGrGu4U). We're here to help you make the most of your data.
 
-Stay tuned for more exciting developments as we continue to enhance [PortalJS](https://portaljs.com/)!
+Stay tuned for more exciting developments as we continue to enhance [PortalJS](https://portaljs.org/)!

site/content/blog/example-ckan-2021.md

@@ -4,7 +4,7 @@ authors: ['Luccas Mateus']
 date: 2021-04-20
 ---
 
-We have created a full data portal demo using DataHub PortalJS all backed by a CKAN instance storing data and metadata, you can see below a screenshot of the homepage and of an individual dataset page.
+We have created a full data portal demo using PortalJS all backed by a CKAN instance storing data and metadata, you can see below a screenshot of the homepage and of an individual dataset page.
 
 ![](https://i.imgur.com/ai0VLS4.png)
 ![](https://i.imgur.com/3RhXOW4.png)
@@ -14,7 +14,7 @@ We have created a full data portal demo using DataHub PortalJS all backed by a C
 To create a Portal app, run the following command in your terminal:
 
 ```console
-npx create-next-app -e https://github.com/datopian/datahub/tree/main/examples/ckan
+npx create-next-app -e https://github.com/datopian/portaljs/tree/main/examples/ckan
 ```
 
 > NB: Under the hood, this uses the tool called create-next-app, which bootstraps an app for you based on our CKAN example.

site/content/blog/markdowndb-launch.md

@@ -3,7 +3,6 @@ title: 'Announcing MarkdownDB: an open source tool to create an SQL API to your
 description: MarkdownDB - an open source library to transform markdown content into sql-queryable data. Build rich markdown-powered sites easily and reliably. New dedicated website at markdowndb.com
 date: 2023-10-11
 authors: ['Ola Rubaj']
-filetype: blog
 ---
 
 Hello, dear readers!

site/content/blog/summer-updates-2023.md

@@ -2,7 +2,6 @@
 title: What We Shipped in Jul-Aug 2023
 authors: ['ola-rubaj']
 date: 2023-09-2
-filetype: blog
 ---
 
 Hey everyone! 👋 Summer has been in full swing, and while I've managed to catch some vacation vibes, I've also been deep into code. I'm super excited to share some of the latest updates and features we've rolled out over the past two months. Let's dive in:
@@ -30,12 +29,12 @@ https://github.com/datopian/markdowndb
 
 ## 📚 The Guide
 
-https://portaljs.com/opensource
+https://portaljs.org/guide
 
 I’ve sketched overviews for two upcoming tutorials:
 
-1. **Collaborating with others on your website**: Learn how to make your website projects a team effort. [See it here](https://portaljs.com/guide#tutorial-3-collaborating-with-others-on-your-website-project)
-2. **Customising your website and previewing your changes locally**: Customize and preview your site changes locally, without headaches. [See it here](https://portaljs.com/guide#tutorial-4-customising-your-website-locally-and-previewing-your-changes-locally)
+1. **Collaborating with others on your website**: Learn how to make your website projects a team effort. [See it here](https://portaljs.org/guide#tutorial-3-collaborating-with-others-on-your-website-project)
+2. **Customising your website and previewing your changes locally**: Customize and preview your site changes locally, without headaches. [See it here](https://portaljs.org/guide#tutorial-4-customising-your-website-locally-and-previewing-your-changes-locally)
 
 ## 🌐 LifeItself.org
 

site/content/blog/the-open-spending-revamp-behind-the-scenes.md

@@ -11,7 +11,7 @@ In our last article, we explored [the Open Spending revamp](https://www.datopian
 
 ## The Core: PortalJS
 
-At the core of the revamped OpenSpending website is [PortalJS](https://portaljs.com), a JavaScript library that's a game-changer in building powerful data portals with data visualizations. What makes it so special? Well, it's packed with reusable React components that make our lives - and yours - a whole lot easier. Take, for example, our sleek CSV previews; they're brought to life by PortalJS' [FlatUI Component](https://storybook.portaljs.org/?path=/story/components-flatuitable--from-url). It helps transform raw numbers into visuals that you can easily understand and use. Curious to know more? Check out the [official PortalJS website](https://portaljs.com).
+At the core of the revamped OpenSpending website is [PortalJS](https://portaljs.org), a JavaScript library that's a game-changer in building powerful data portals with data visualizations. What makes it so special? Well, it's packed with reusable React components that make our lives - and yours - a whole lot easier. Take, for example, our sleek CSV previews; they're brought to life by PortalJS' [FlatUI Component](https://storybook.portaljs.org/?path=/story/components-flatuitable--from-url). It helps transform raw numbers into visuals that you can easily understand and use. Curious to know more? Check out the [official PortalJS website](https://portaljs.org).
 
 ![Data visualization](/assets/blog/2023-10-13-the-open-spending-revamp-behind-the-scenes/data-visualization.png)
 

site/content/config.js

@@ -11,18 +11,20 @@ const config = {
   authorUrl: 'https://datopian.com/',
   navbarTitle: {
     // logo: "/images/logo.svg",
-    text: '🌀 DataHub PortalJS',
+    text: '🌀 PortalJS',
     // version: "Alpha",
   },
   navLinks: [
     { name: 'Docs', href: '/docs' },
     // { name: "Components", href: "/docs/components" },
     { name: 'Blog', href: '/blog' },
+    { name: 'Showcases', href: '/#showcases' },
     { name: 'Howtos', href: '/howtos' },
     { name: 'Guide', href: '/guide' },
     {
-      name: 'Showcases',
-      href: '/showcases/'
+      name: 'Examples',
+      href: 'https://github.com/datopian/portaljs/tree/main/examples',
+      target: '_blank',
     },
     {
       name: 'Components',
@@ -44,7 +46,6 @@ const config = {
       { rel: 'icon', href: '/favicon.ico' },
       { rel: 'apple-touch-icon', href: '/icon.png', sizes: '120x120' },
     ],
-    canonical: 'https://portaljs.com/',
     openGraph: {
       type: 'website',
       title:
@@ -68,8 +69,8 @@ const config = {
       cardType: 'summary_large_image',
     },
   },
-  github: 'https://github.com/datopian/datahub',
-  discord: 'https://discord.gg/KrRzMKU',
+  github: 'https://github.com/datopian/portaljs',
+  discord: 'https://discord.gg/EeyfGrGu4U',
   tableOfContents: true,
   analytics: 'G-96GWZHMH57',
   // editLinkShow: true,

site/content/docs/dms/authentication.md

@@ -0,0 +1,249 @@
+# Authentication
+
+## Introduction
+
+The core function of authentication is to **Identify** Users of the Portal (in a federated way) so we can base access on their identity.
+
+There are 3 major conceptual components: Identity, Accounts and Sessions which come together in the following stages:
+
+* **Root Identity Determination:** Determine Identity often via Delegation
+* **Sessions:** Persistence of the identity in the web application in a secure way (without new identity determination on each request! I don't want to have to login via third party service every time)
+* **Account (aka profile):** Storing Related Account/Profile Information in our application (not in third party identity) eg. email, name (other preferences)
+  * This will get auto-created usually at first Identification
+  * In limited case this can be seen as a cache of info from Identity system (e.g. your email)
+  * However often richer info that is app specific that is generated (relevant for personalization)
+
+### Root Identity Determination options :key:
+
+The identity determination can be done in multiple ways. In this article we're considering following 3 options that we believe are widely used:
+
+- Password authentication - traditional username and password pair
+- Single Sign-on (SSO) via protocols such as OAuth, SAML, OpenID Connect
+- One-time password (OTP) via email or SMS (aka passwordless connection)
+
+#### Password authentication
+
+Traditional way of authentication of users. When signing up user provides at least username and password pair which is then stored in a database for future authentication processes. Normally, additional information such as email address, full name etc. is also requested when registering.
+
+Examples of password authentication in popular services:
+
+- GitHub - https://github.com/join
+- GitLab - https://gitlab.com/users/sign_up
+- NPM - https://www.npmjs.com/signup
+
+#### Single Sign-on (SSO)
+
+The way of delegating identity determination process to some third-party service. Normally, popular social network services are used, e.g., Google, Facebook, Twitter etc. SSO implementations can be done using OAuth or SAML protocols. In addition, there is OpenID Connect protocol which is an extension of OAuth2.0.
+
+- OAuth
+  - JWT based
+  - JSON based
+  - 'webby'
+- SAML
+  - XML based
+  - SOAP based
+  - 'enterprisey'
+
+List of OAuth providers:
+
+https://en.wikipedia.org/wiki/List_of_OAuth_providers
+
+Examples of SSO in popular projects:
+
+- https://datahub.io/login
+- https://vercel.com/signup
+
+#### One-time password (OTP)
+
+Also known as dynamic password, OTP also solves limitations of traditional password authentication method. Usually, the one time passwords are received via email or SMS.
+
+### Account (aka profile) 
+
+- Storage of user profile information (email, fullname, gravatar etc.)
+- Retrieving user profile information via API
+- Updating profile
+- Deleting profile
+
+### Sessions
+
+- Log out: DePersisting the Session
+- Invalidating all Sessions: e.g. if a security issue
+- Sessions outside of browsers
+
+## Key Job Stories
+
+When a user signs in, I want to know her/his identity so that I can limit access and editing based on who she/he is.
+
+When a user visits the data portal for the first time, I want to provide him/her a way to register easily/quickly so that more people uses the data portal.
+
+When I visit the data portal for the first time, I want to sign up using my existing social network account so that I don't need to remember yet another credentials.
+
+When I'm using the CLI app (or anything else outside browser), I want to be able to login so that I can work from the terminal (e.g., have write access: editing datasets etc.).
+
+[More job stories](#more-job-stories).
+
+## CKAN 2 (CKAN Classic)
+
+### Basic CKAN authentication
+
+In classic system, we have basic CKAN authentication. Below is how registration page looks like:
+
+![CKAN Classic register page](/static/img/docs/dms/ckan-register.png)
+
+Registration flow in CKAN Classic:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: fill in the form and submit
+  ckan->>ckan: check access (if user can create user)
+  ckan->>ckan: parse params
+  ckan->>ckan: check recaptcha
+  ckan->>ckan: call 'user_create' action
+  ckan->>ckan.model: add a new user into db
+  ckan->>ckan: create an activity
+  ckan->>ckan: log the user
+  ckan->>user: redirect to dashboard
+```
+
+We can extend basic CKAN authentication with:
+
+- LDAP
+  - https://extensions.ckan.org/extension/ldap/
+  - https://github.com/NaturalHistoryMuseum/ckanext-ldap
+- OAuth - see below
+- SAML - https://extensions.ckan.org/extension/saml2/
+
+### CKAN Classic as OAuth client
+
+CKAN Classic can also be used as OAuth client:
+
+- https://github.com/conwetlab/ckanext-oauth2 - this is the only one that's maintained.
+- https://github.com/etalab/ckanext-oauth2 - outdated, the one above is based on this.
+- https://github.com/okfn/ckanext-oauth - last commit 9 years ago.
+- https://github.com/ckan/ckanext-oauth2waad - Windows Azure Active Directory specific and outdated.
+
+How it works:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: request for login via OAuth provider
+  ckan->>ckan.oauth: raise 401 and call `challenge` function
+  ckan.oauth->>user: redirect the user to the 3rd party log in page
+  user->>3rdparty: perform login
+  3rdparty->>ckan.oauth: redirect to /oauth2/callback with token
+  ckan.oauth->>3rdparty: call `authenticate` with token
+  3rdparty->>ckan.oauth: return user info
+  ckan.oauth->>ckan: if doesn't exist save that info in db or update it
+  ckan.oauth->>ckan.oauth: add cookies
+  ckan.oauth->>user: redirect to dashboard
+```
+
+## CKAN 3 (Next Gen)
+
+We have considered some of popular and/or modern solutions for identity management that we can implement in CKAN 3:
+
+https://docs.google.com/spreadsheets/d/1qXZyzAbA2NtpnoSZRJ2K_EbaWJnvxkrKVzQ_2rD5eQw/edit#gid=0
+
+Shortlist based on scores from the spreadsheet above:
+
+- Auth0
+- AuthN
+- Ory/Kratos
+
+Recommendation:
+
+All projects from the shortlist can be considered for a project. It worth to give a try for each of them and find out what works best for your project's needs. Testing out Auth0 should be straightforward and take less than an hour. AuthN and Ory/Kratos would require to build docker images and to run it locally but overall it should not be time consuming.
+
+### Existing work
+
+In datahub.io we have implemented SSO via Google/Github. Below is sequence diagram showing the auth flow with datopian/auth + frontend express app (similar to CKAN 3 frontend):
+
+```mermaid
+sequenceDiagram
+
+  frontend.login->>auth.authenticate: authenticate(jwt=None,next=/success/...)
+  auth.authenticate->>frontend.login: failed + here are urls for logging on 3rd party including success
+  frontend.login->>user: login form with login urls to 3rd party including next url in state
+  user->>3rdparty: login
+  3rdparty->>auth.oauth_response: success
+  auth.oauth_response->>frontend.success: redirect to next url
+  frontend.success->>auth.authenticate: with valid jwt
+  auth.authenticate->>frontend.success: valid + here is profile
+  frontend.success->>frontend.success: decode jwt, check it, then see localstorage
+  frontend.success->>frontend.dashboard: redirect to dashboard
+```
+
+## CKAN 2 to CKAN 3 (aka Next Gen)
+
+How does this conceptual framework map to an evolution of CKAN 2 to CKAN 3?
+
+```mermaid
+graph TD
+
+subgraph "CKAN Classic"
+  Signup["Classic signup, e.g., self-service or by sysadmin"]
+  Login["Classic login if you're using the classic UI"]
+  OAuth["OAuth2(ORY/Hydra)"]
+end
+
+subgraph "Authentication service (ORY/Kratos)"
+  SSO["Social Sign-On: Github, Google, Facebook"]
+  CC["CKAN Classic"]
+  Admins["Sysadmin users"]
+  Curators["Data curators"]
+  Users["Regular users"]
+end
+
+subgraph "Frontend v3"
+  SignupFront["Signup via Kratos"]
+  LoginFront["Login via Kratos"]
+end
+
+SignupFront --"Regular user"--> SSO
+LoginFront --"Regular user"--> SSO
+
+LoginFront --"Data curator"--> CC
+
+CC --> Admins
+CC --> Curators
+SSO --> Users
+
+CC --"Redirect"--> OAuth
+OAuth --> Login
+```
+
+Sequence diagram of login process:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)
+
+From ORY/Kratos:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)
+
+
+Kratos to Hydra in CKAN Classic:
+
+WIP
+
+Questions
+
+* Does CKAN Classic allow us to store arbitrary account information (are there "extras")
+* How would we avoid having to support identity persistence, delegation etc in both NG frontend and Classic Admin UI?
+  * Can we share cookies (e.g. via using subdomains)
+* How is login, identity determination etc done at least for frontend in DataHub.io
+* Should account UI really be in NG frontend vs Classic Admin UI?
+* how can we handle "invite a user" to my org set up ... (it's basically post processing after sign up ...)
+
+## Appendix
+
+### More job stories
+
+When a user visits the data portal, I want to provide multiple options for him/her to sign up so that I have more users registered and using the data portal.
+
+When a user needs to change his/her profile info, I want to make sure it is possible, so that I have the up-to-date information about users.
+
+When my personal info (email etc.) is changed, I want to edit it in my profile so that I provide up-to-date information about me and I receive messages (eg, notifications) properly.
+
+When I decide to stop using the data portal, I want to be able to delete my account, so that my personal details aren't stored in the service that I don't need anymore.

site/content/docs/dms/blob-storage.md

@@ -0,0 +1,215 @@
+# Blob Storage
+
+## Introduction
+
+DMS and data portals often need to *store* data as well as metadata. As such, they require a system for doing this. This page focuses on Blob Storage aka Bulk or Raw storage (see [storage](/docs/dms/storage) page for an overview of all types of storage).
+
+Blob storage is for storing "blobs" of data, that is a raw stream of bytes like files on a filesystem. For blob storage think local filesystem or cloud storage like S3, GCS, etc.
+
+Blob Storage in a DMS can be provided via:
+
+* Local file system: storing on disk or storage directly connected to the instance
+* Cloud storage like S3, Google Cloud Storage, Azure storage etc
+
+Today, cloud storage would be the default in most cases.
+
+### Features
+
+* Storage: Persistent, cost-efficient storage
+* Download: Fast, reliable download (possibly even with support for edge distribution)
+* Upload: reliable and rapid upload 
+  * Direct upload to (cloud) storage by clients i.e. without going via the DMS. Why? Because cloud storage has many features that it would be costly replicate (e.g. multipart, resumable etc), excellent performance and reliability for upload. It also cuts out the middleman of the DMS backend thereby saving bandwidth, reducing load on the DMS backend and improving performance
+  * Upload UI: having an excellent UI for doing upload. NB: this UI is considered part of the [publish feature](/docs/dms/publish)
+* Cloud: integrate with cloud storage
+* Permissions: restricting access to data stored in blob storage based on the permissions of the DMS. For example, if Joe does not have access to a dataset on the DMS he should not be able to access associated blob data in the storage system
+
+## Flows
+
+### Direct to Cloud Upload
+
+Want: Direct upload to cloud storage ... But you need to authorize that ... So give them a token from your app
+
+A sequence diagram illustrating the process for a direct to cloud upload:
+
+```mermaid
+sequenceDiagram
+
+participant Browser as Client (Browser / Code)
+participant Authz as Authz Server
+participant BitStore as Storage Access Token Service
+participant Storage as Cloud Storage
+
+  Browser->>Authz: Give me a BitStore access token
+  Authz->>Browser: Token
+  Browser->>BitStore: Get a signed upload URL (access token, file metdata)
+  BitStore->>Browser: Signed URL
+  Browser->>Storage: Upload file (signed URL)
+  Storage->>Browser: OK (storage metadata)
+```
+
+Here's a more elaborate version showing storage of metadata into the MetaStore afterwards (and skipping the Authz service):
+
+```mermaid
+sequenceDiagram
+
+  participant browser as Client (Browser / Code)
+  participant vfts as MetaStore
+  participant bitstore as Storage Access Token Service
+  participant storage as Cloud Storage
+  
+  browser->>browser: Select files to upload
+  browser->>browser: calculate file hashes (if doing content addressable)
+  browser->bitstore: get signed URLs(file1.csv URL, file2.csv URL, auth info)
+  bitstore->>browser: signed URLs
+  browser->>storage: upload file1.csv
+  storage->>browser: OK
+  browser->>storage: upload file2.csv
+  storage->>browser: OK
+  browser->>browser: Compose datapackage.json
+  browser->>vfts: create dataset(datapackage.json, file1.csv pointer, file2.csv pointer, jwt token, ...)
+  vfts->>browser: OK
+```
+
+## CKAN 2 (Classic)
+
+Blob Storage is known as the FileStore in CKAN v2 and below. The default is local disk storage.
+
+There is support for cloud storage via a variety of extensions the most prominent of which is `ckanext-cloudstorage`: https://github.com/TkTech/ckanext-cloudstorage
+
+There are a variety of issues:
+
+* Cloud storage is not a first class citizen in CKAN: CKAN defaults to local file storage but cloud storage is the default in the world and has much better scalability, performance as well as integratability with cloud deployment
+* The FileStore interface definition has a poor separation of concerns (for example, blob storage file paths is set in the FileStore component not in core CKAN) which makes it hard / hacky to extend and use for key use cases e.g. versioning.
+* `ckanext-cloudstorage` (the default cloud storage extension) is ok but has many issues e.g.
+  * No direct to cloud upload: it uses CKAN backend as a middleman so all data must go via ckan backend
+  * Implements its own (sometimes unreliable) version of multipart upload (which means additional code which isn't as reliable as cloud storage providers interface)
+  * No access to advanced features such as resumability etc
+
+Generally, we at Datopian have seen a lot of issues around multipart / large file upload stability with clients and are still seeing issues when a lot of large files are uploaded via scripts. Fixing and refactoring code related to storage is very costly, and tends to result in client specific "hacks".
+
+## CKAN v3
+
+An approach to blob storage that leverages cloud blob storage directly (i.e. without having to upload and serve all files via the CKAN web server), unlocking the performance characteristics of the storage backend directly. It is designed with a microservice approach and supports direct to cloud uploads and downloads. The key components are listed in the next section. You can read more about the overall design approach in the [design section below](#Design).
+
+It is backwards compatible with CKAN v2 and has been successfully deployed with CKAN v2.8 and v2.9.
+
+**Status: Production.**
+
+### Components
+
+* [ckanext-blob-storage](https://github.com/datopian/ckanext-blob-storage) (formerly known as ckanext-external-storage)
+  * Hooking CKAN to Giftless replacing resource storage
+  * Depends on giftless-client and ckanext-authz-service
+  * Doesn't implement IUploader - completely overrides upload / download routes for resources
+* [Giftless](https://github.com/datopian/giftless) - Git LFS compatible implementation for storage with some extras on top. This hands out access tokens to store data in cloud storage.
+  * Docs at https://giftless.datopian.com
+  * Backends for Azure, Google Cloud Storage and local
+  * Multipart support (on top of standard LFS protocol)
+  * Accepts JWT tokens for authentication and authorization
+* [ckanext-authz-service](https://github.com/datopian/ckanext-authz-service/) - This extension uses CKAN’s built-in authentication and authorization capabilities to: a) Generate JWT tokens and provide them via CKAN’s Web API to clients and b) Validate JWT tokens.
+  * Allows hooking CKAN's authentication and authorization capabilities to generate signed JWT tokens, to integrate with external systems
+  * Not specific for Giftless, but this is what it was built for
+* [ckanext-asset-storage](https://github.com/datopian/ckanext-asset-storage) - this takes care of storing non-data assets e.g. organization images etc.
+  * CKAN IUploader for assets (not resources!)
+  * Pluggable backends - currently local and Azure
+  * Much cleaner than older implementations (ckanext-cloudstorage etc.)
+
+Clients:
+
+* [giftless-client-py](https://github.com/datopian/giftless-client) - Python client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools
+* [giftless-client-js](https://github.com/datopian/giftless-client-js) - Javascript client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools for creating uploaders in the UI
+
+## Design
+
+### Purpose
+
+The goal of this project is to create a more **_flexible_** system for storing **_data files_** (AKA “resources”) for **_CKAN_ and _other implementations_** of a data portal so that CKAN can support versioning, large file upload (and great file upload UX), plug easily into cloud and local file storage backends and, in general, is easy to customize both for storage layer and for CKAN client code of that layer
+
+### Features
+
+* Do one thing and do it well: provide an API to store and retrieve files from storage, in a way that is pluggable into a micro-services based application and to existing CKAN (2.8 / 2.9)
+* Does not force, and in fact is not aware of, a specific file naming logic (i.e. resource file names could be based on a user given name, a content hash, a revision ID or any mixture of these - it is up to the using system to decide)
+* Does not force a specific storage backend; Should support Amazon S3, Azure Storage and local file storage in some way initially but in general backend should be pluggable
+* Does not force a specific authentication scheme; Expects a signed JWT token, does not care who signed it and how the user got authenticated
+* Does not force complex authorization scheme; Leave it to external system to do complex authorization if needed;
+  * By default, the system can work in an “admin party” mode where all authenticated users have full access to all files. This will be “good enough” for many DMS implementations including CKAN.
+  * Potentially, allow plugging in a more complex authorization logic that relies on JWT claims to perform granular authorization checks
+
+### For Data Files (i.e. Blobs)
+
+This system is about storing and providing access to blobs, or streams of bytes; It is not about providing access to the data stored within (i.e. it is not meant to replace CKAN’s datastore).
+
+### For CKAN – whilst not necessarily CKAN Specific
+
+While the system’s design should not be CKAN specific in any way, our current client needs require us to provide a CKAN extension that integrates with this system.
+
+CKAN’s current IUploader interface has been identified to be too narrow to provide the functionality required by complex projects (resource versioning, direct cloud uploads and downloads, large file support and multipart support). While some of these needs could be and have been “hacked” through the IUploader interface, the implementations have been over complex and hard to debug.
+
+Our goal should be to provide a CKAN extension that provides the following functionality directly:
+
+* Uploading and downloading resource files directly from the client if supported by the storage backend
+  *   Multipart upload support if supported by storage backend
+  *   Handling of signed URLs for uploads and private downloads
+  *   Client side code for handling multipart uploads
+  *   TBD: If storage backend does not support direct uploads / downloads, fall back to …
+
+In addition, this extension should provide an API for other extensions to do things like:
+
+*   Set the file naming scheme (We need this for ckanext-versions)
+*   Lower level file access, e.g. move and delete files. We may need this in the future to optimize storage and deduplicate files as proposed for ckanext-versions
+
+In addition, this extension must “play nice” with common CKAN features such as the datastore extension and related datapusher / xloader extensions.
+
+### Usable For other DMS implementations
+
+There should be nothing in this system, except for the CKAN extension described above, that is specific to CKAN. That will allow to re-use and re-integrate this system as a micro-service in other DMS implementations such as ckan-ng and others.
+In fact, the core part of this system should be a generic, abstract storage service with a light authorization layer. This could make it useful in a host of situations where storage micro-service is needed.
+
+### High Level Principles
+
+Common Principles
+
+* Uploads and downloads directly from cloud provides to browser
+* Signed uploads / downloads - for private / authorized only data access
+* Support for AWS, Azure and potentially GCP storage
+* Support for local (non cloud) storage, potentially through a system like [https://min.io/](https://min.io/)
+* Multipart / large file upload support (a few GB in size should be supported for Gates)
+* Not opinionated about file naming / paths; Allow users to set file locations under some pre-defined patchs / buckets
+* Client side support - browser widgets / code for uploading and downloading files / multipart uploads directly to different backends
+* Well-documented flow for using from API (not browser)
+* Provided API for deleting and moving files
+* Provided API for accessing storage-level metadata (e.g. file MD5) (do we need this could be useful for processes that do things like deduplicate storage)
+* Provided API for managing storage-level object level settings (e.g. “Content-disposition” / “Content-type” headers, etc.)
+* Authorization based on some kind of portable scheme (JWT)
+
+CKAN integration specific (implemented as a CKAN extension)
+
+* JWT generation based on current CKAN user permissions
+* Client widgets integration (or CKAN specific widgets) in right places in CKAN templates
+* Hook into resource upload / download / deletion controllers in CKAN
+* API to allow other extensions to control storage level object metadata (headers, path)
+* API to allow other extensions to hook into lifecycle events - upload completion, download request, deletion etc.
+
+
+### Components
+
+The Decoupled Storage solution should be split into several parts, with some parts being independent of others:
+
+* [External] Cloud Storage service (or API similar if local file system) e.g. S3, GCS, Azure Storage, Min.io (for local file system)
+* Cloud Storage Access Service
+* [External] Permissions Service for granting general permission tokens that give access to Cloud Storage Access Service
+  * JWT tokens can be generated by any party that has the right signing key. Thus, we can initially do without this if JWT signing is implemented as part of the CKAN extension
+* Browser based Client for Cloud Storage (compatible with #1 and with different cloud vendors)
+* CKAN extension that wraps the two parts above to provide a storage solution for CKAN
+
+### Questions
+
+* What is file structure in cloud ... i.e. What is the file path for uploaded files? Options:
+  * Client chooses a name/path
+  * Content addressable i.e. the name is given by the content? How? Use a hash.]
+    * Beauty of that: standard way to name things. The same thing has the same name (modulo collisions)
+    * Goes with versioning => same file = same name, diff file = diff name
+* And do you enforce that from your app
+  * Request for token needs to include the destination file path

site/content/docs/dms/ckan-client-guide/index.md

@@ -0,0 +1,503 @@
+# CKAN Client Guide
+
+Guide to interacting with [CKAN](/docs/dms/ckan) for power users such as data scientists, data engineers and data wranglers.
+
+This guide is about adding and managing data in CKAN programmatically and it assumes:
+
+* You are familiar with key concepts like metadata, data, etc.
+* You are working programmatically with a programming language such as Python, JavaScript or R (_coming soon_).
+
+## Frictionless Formats
+
+Clients use [Frictionless formats](https://specs.frictionlessdata.io/) by default for describing dataset and resource objects passed to client methods. Internally, we then use the a *CKAN {'<=>'} Frictionless Mapper* (both [in JavaScript]( https://github.com/datopian/frictionless-ckan-mapper-js ) and [in Python](https://github.com/frictionlessdata/frictionless-ckan-mapper)) to convert objects to CKAN formats before calling the API. **Thus, you can use _Frictionless Formats_ by default with the client**.
+
+>[!tip]As CKAN moves to Frictionless to default this will gradually become unnecessary.
+
+## Quick start
+
+Most of this guide has Python programming language in mind, including its [convention regading using _snake case_ for instances and methods names](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles).
+
+If needed, you can adapt the instructions to JavaScript and R (coming soon) by using _camel case_ instead — for example, if in the Python code we have  `client.push_blob(…)`, in JavaScript it would be `client.pushBlob(…)`.
+
+### Prerequisites
+
+Install the client for your language of choice:
+
+* Python: https://github.com/datopian/ckan-client-py#install
+* JavaScript: https://github.com/datopian/ckan-client-js#install
+* R: _coming soon_
+
+### Create a client
+
+#### Python
+
+```python
+from ckanclient import Client
+
+
+api_key = '771a05ad-af90-4a70-beea-cbb050059e14'
+api_url = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+lfs_url = 'http://localhost:9419'
+
+client = Client(api_url, organization, dataset, lfs_url)
+```
+
+#### JavaScript
+
+```javascript
+const { Client } = require('ckanClient')
+
+apiKey = '771a05ad-af90-4a70-beea-cbb050059e14'
+apiUrl = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+
+const client = Client(apiKey, organization, dataset, apiUrl)
+```
+
+### Upload a resource
+
+That is to say, upload a file, implicitly creating a new dataset.
+
+#### Python
+
+```python
+from frictionless import describe
+
+
+resource = describe('my-data.csv')
+client.push_blob(resource)
+```
+
+### Create a new empty Dataset with metadata
+
+#### Python
+
+```python
+client.create('my-data')
+client.push(resource)
+```
+
+### Adding a resource to an existing Dataset
+
+>[!note]Not implemented yet.
+
+
+```python
+client.create('my-data')
+client.push_resource(resource)
+```
+
+### Edit a Dataset's metadata
+
+>[!note]Not implemented yet.
+
+
+```python
+dataset = client.retrieve('sample-dataset')
+client.update_metadata(
+    dataset,
+    metadata: {'maintainer_email': 'sample@datopian.com'}
+)
+```
+
+For details of metadata see the [metadata reference below](#metadata-reference).
+
+## API - Porcelain
+
+### `Client.create`
+
+Expects as a single argument: a _string_, or a _dict_ (in Python), or an _object_ (in JavaScript). This argument is either a valid dataset name or dictionary with metadata for the dataset in Frictionless format.
+
+### `Client.push`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a dataset metadata in Frictionless format.
+
+### `Client.retrieve`
+
+Expects a single argument: a string with a dataset name or uniquer ID. Returns a Frictionless resource as a _dict_ (in Python) or as an _Promisse .&lt;object&gt;_ (in JavaScript).
+
+### `Client.push_blob`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a Frictionless resource.
+
+## API - Plumbing
+
+### `Client.action`
+
+This method bridges access to the CKAN API _action endpoint_.
+
+#### In Python
+
+Arguments:
+
+| Name                 | Type       | Default    | Description                                                  |
+| -------------------- | ---------- | ---------- | ------------------------------------------------------------ |
+| `name`               | `str`      | (required) | The action name, for example, `site_read`, `package_show`…   |
+| `payload`            | `dict`     | (required) | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `http_get`           | `bool`     | `False`    | Optional, if `True` will make `GET` request, otherwise `POST`. |
+| `transform_payload`  | `function` | `None`     | Function to mutate the `payload` before making the request (useful to convert to and from CKAN and Frictionless formats). |
+| `transform_response` | `function` | `None`     | function to mutate the response data before returning it (useful to convert to and from CKAN and Frictionless formats). |
+
+>[!note]The CKAN API uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you can pass `frictionless_ckan_mapper.frictionless_to_ckan` as `transform_payload`, and `frictionless_ckan_mapper.ckan_to_frictionless` as `transform_response`.
+
+
+#### In JavaScript
+
+Arguments:
+
+| Name         | Type                | Default            | Description                                                  |
+| ------------ | ------------------- | ------------------ | ------------------------------------------------------------ |
+| `actionName` | <code>string</code> | (required)         | The action name, for example, `site_read`, `package_show`…   |
+| `payload`    | <code>object</code> | (required)         | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `useHttpGet` | <code>object</code> | <code>false</code> | Optional, if `True` will make `GET` request, otherwise `POST`. |
+
+>[!note]The JavaScript implementation uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you need to convert from Frictionless to CKAN before calling `action` , and from CKAN to Frictionless after calling `action`.
+
+## Metadata reference
+
+>[!info]Your site may have custom metadata that differs from the example set below.
+
+
+### Profile
+
+**(`string`)** Defaults to _data-resource_.
+
+The profile of this descriptor.
+
+Every Package and Resource descriptor has a profile. The default profile, if none is declared, is `data-package` for Package and `data-resource` for Resource.
+
+#### Examples
+
+- `{"profile":"tabular-data-package"}`
+
+- `{"profile":"http://example.com/my-profiles-json-schema.json"}`
+
+### Name
+
+**(`string`)**
+
+An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+
+This is ideally a url-usable and human-readable name. Name `SHOULD` be invariant, meaning it `SHOULD NOT` change when its parent descriptor is updated.
+
+#### Example
+
+- `{"name":"my-nice-name"}`
+
+### Path
+
+A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+
+The dereferenced value of each referenced data source in `path` `MUST` be commensurate with a native, dereferenced representation of the data the resource describes. For example, in a *Tabular* Data Resource, this means that the dereferenced value of `path` `MUST` be an array.
+
+#### Validation
+
+##### It must satisfy one of these conditions
+
+###### Path
+
+**(`string`)**
+
+A fully qualified URL, or a POSIX file path..
+
+Implementations need to negotiate the type of path provided, and dereference the data accordingly.
+
+**Examples**
+
+- `{"path":"file.csv"}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+**(`array`)**
+
+**Examples**
+
+- `["file.csv"]`
+
+- `["http://example.com/file.csv"]`
+
+#### Examples
+
+- `{"path":["file.csv","file2.csv"]}`
+
+- `{"path":["http://example.com/file.csv","http://example.com/file2.csv"]}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+### Data
+
+Inline data for this resource.
+
+### Schema
+
+**(`object`)**
+
+A schema for this resource.
+
+### Title
+
+**(`string`)**
+
+A human-readable title.
+
+#### Example
+
+- `{"title":"My Package Title"}`
+
+### Description
+
+**(`string`)**
+
+A text description. Markdown is encouraged.
+
+#### Example
+
+- `{"description":"# My Package description\nAll about my package."}`
+
+### Home Page
+
+**(`string`)**
+
+The home on the web that is related to this data package.
+
+#### Example
+
+- `{"homepage":"http://example.com/"}`
+
+### Sources
+
+**(`array`)**
+
+The raw sources for this resource.
+
+#### Example
+
+- `{"sources":[{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]}`
+
+### Licenses
+
+**(`array`)**
+
+The license(s) under which the resource is published.
+
+This property is not legally binding and does not guarantee that the package is licensed under the terms defined herein.
+
+#### Example
+
+- `{"licenses":[{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]}`
+
+### Format
+
+**(`string`)**
+
+The file format of this resource.
+
+`csv`, `xls`, `json` are examples of common formats.
+
+#### Example
+
+- `{"format":"xls"}`
+
+### Media Type
+
+**(`string`)**
+
+The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+
+#### Example
+
+- `{"mediatype":"text/csv"}`
+
+### Encoding
+
+**(`string`)** Defaults to _utf-8_.
+
+The file encoding of this resource.
+
+#### Example
+
+- `{"encoding":"utf-8"}`
+
+### Bytes
+
+**(`integer`)**
+
+The size of this resource in bytes.
+
+#### Example
+
+- `{"bytes":2082}`
+
+### Hash
+
+**(`string`)**
+
+The MD5 hash of this resource. Indicate other hashing algorithms with the {'{algorithm}'}:{'{hash}'} format.
+
+#### Examples
+
+- `{"hash":"d25c9c77f588f5dc32059d2da1136c02"}`
+
+- `{"hash":"SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"}`
+
+## Generating templates
+
+You can use [`jsv`](https://github.com/datopian/jsv) to generate a template  script in Python, JavaScript, and R.
+
+To install it:
+
+```
+$ npm install -g git+https://github.com/datopian/jsv.git
+```
+
+### Python
+
+```
+$ jsv data-resource.json --output py
+```
+
+**Output**
+```python
+dataset_metadata = {
+    "profile": "data-resource",  # The profile of this descriptor.
+    # [example] "profile": "tabular-data-package"
+    # [example] "profile": "http://example.com/my-profiles-json-schema.json"
+    "name": "my-nice-name",  # An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+    "path": ["file.csv","file2.csv"],  # A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+    # [example] "path": ["http://example.com/file.csv","http://example.com/file2.csv"]
+    # [example] "path": "http://example.com/file.csv"
+    "data": None,  # Inline data for this resource.
+    "schema": None,  # A schema for this resource.
+    "title": "My Package Title",  # A human-readable title.
+    "description": "# My Package description\nAll about my package.",  # A text description. Markdown is encouraged.
+    "homepage": "http://example.com/",  # The home on the web that is related to this data package.
+    "sources": [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}],  # The raw sources for this resource.
+    "licenses": [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}],  # The license(s) under which the resource is published.
+    "format": "xls",  # The file format of this resource.
+    "mediatype": "text/csv",  # The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+    "encoding": "utf-8",  # The file encoding of this resource.
+    # [example] "encoding": "utf-8"
+    "bytes": 2082,  # The size of this resource in bytes.
+    "hash": "d25c9c77f588f5dc32059d2da1136c02",  # The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+    # [example] "hash": "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+}
+```
+
+
+### JavaScript
+
+```
+$ jsv data-resource.json --output js
+```
+
+**Output**
+```javascript
+const datasetMetadata = {
+  // The profile of this descriptor.
+  profile: "data-resource",
+  // [example] profile: "tabular-data-package"
+  // [example] profile: "http://example.com/my-profiles-json-schema.json"
+  // An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+  name: "my-nice-name",
+  // A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+  path: ["file.csv", "file2.csv"],
+  // [example] path: ["http://example.com/file.csv","http://example.com/file2.csv"]
+  // [example] path: "http://example.com/file.csv"
+  // Inline data for this resource.
+  data: null,
+  // A schema for this resource.
+  schema: null,
+  // A human-readable title.
+  title: "My Package Title",
+  // A text description. Markdown is encouraged.
+  description: "# My Package description\nAll about my package.",
+  // The home on the web that is related to this data package.
+  homepage: "http://example.com/",
+  // The raw sources for this resource.
+  sources: [
+    {
+      title: "World Bank and OECD",
+      path: "http://data.worldbank.org/indicator/NY.GDP.MKTP.CD",
+    },
+  ],
+  // The license(s) under which the resource is published.
+  licenses: [
+    {
+      name: "odc-pddl-1.0",
+      path: "http://opendatacommons.org/licenses/pddl/",
+      title: "Open Data Commons Public Domain Dedication and License v1.0",
+    },
+  ],
+  // The file format of this resource.
+  format: "xls",
+  // The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+  mediatype: "text/csv",
+  // The file encoding of this resource.
+  encoding: "utf-8",
+  // [example] encoding: "utf-8"
+  // The size of this resource in bytes.
+  bytes: 2082,
+  // The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+  hash: "d25c9c77f588f5dc32059d2da1136c02",
+  // [example] hash: "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+};
+```
+
+### R
+
+```
+$ jsv data-resource.json --output r
+```
+
+**Output** 
+```r
+# The profile of this descriptor.
+profile <- "data-resource"
+# [example] profile <- "tabular-data-package"
+# [example] profile <- "http://example.com/my-profiles-json-schema.json"
+# An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+name <- "my-nice-name"
+# A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+path <- ["file.csv","file2.csv"]
+# [example] path <- ["http://example.com/file.csv","http://example.com/file2.csv"]
+# [example] path <- "http://example.com/file.csv"
+# Inline data for this resource.
+data <- NA
+# A schema for this resource.
+schema <- NA
+# A human-readable title.
+title <- "My Package Title"
+# A text description. Markdown is encouraged.
+description <- "# My Package description\nAll about my package."
+# The home on the web that is related to this data package.
+homepage <- "http://example.com/"
+# The raw sources for this resource.
+sources <- [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]
+# The license(s) under which the resource is published.
+licenses <- [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]
+# The file format of this resource.
+format <- "xls"
+# The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+mediatype <- "text/csv"
+# The file encoding of this resource.
+encoding <- "utf-8"
+# [example] encoding <- "utf-8"
+# The size of this resource in bytes.
+bytes <- 2082L
+# The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+hash <- "d25c9c77f588f5dc32059d2da1136c02"
+# [example] hash <- "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+
+```
+
+
+## Design Principles
+
+The client **should** use Frictionless formats by default for describing dataset and resource objects passed to client methods.
+
+In addition, where more than metadata is needed (e.g., we need to access the data stream, or get the schema) we expect the _Dataset_ and _Resource_ objects to follow the [Frictionless Data Lib pattern](https://github.com/frictionlessdata/project/blob/master/rfcs/0004-frictionless-data-lib-pattern.md).

site/content/docs/dms/ckan-enterprise/index.md

@@ -0,0 +1,108 @@
+# CKAN Enterprise
+
+## Introduction
+
+CKAN Enterprise is our name for what we plan would become our standard "base" distribution for CKAN going forward:
+
+* It is a CKAN standard code base with micro-services.
+* Enterprise grade data catalog and portal targeted at Gov (open data portals) and Enterprise (Data Catalogs +).
+* It is also known as [Datopian DMS](https://www.datopian.com/datopian-dms/).
+
+## Roadmap 2021 and beyond
+
+|                   | Current                                                                                    | CKAN Enterprise                                                 |
+|-------------------|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
+| Raw storage       | Filestore                                                                                  | Giftless                                                        |
+| Data Loader (db)  | DataPusher extension                                                                       | Aircan                                                          |
+| Data Storage (db) | Postgres                                                                                   | Any database engine. By default, Postgres                       |
+| Data API (read)   | Built-in DataStore extension's API including SQL endpoint                                  | GraphQL based standalone micro-service                          |
+| Frontend (public) | Build-in frontend into CKAN Classic python app (some projects are using nodejs app)        | PortalJS or nodejs app                                          |
+| Data Explorer     | ReclineJS (some projects that uses nodejs app for frontend have React based Data Explorer) | GraphQL based Data Explorer                                     |
+| Auth              | Traditional login/password + extendable with CKAN Classic extensions                       | SSO with default Google, Github, Facebook and Microsoft options |
+| Permissions       | CKAN Classic based permissions                                                             | Existing permissions exposed via JWT based authz API            |
+
+## Timeline 2021
+
+To develop a base distribution of CKAN Enterprise, we want to build a demo project with the features from the roadmap. This way we can:
+
+* understand its advantages/limitations;
+* compare against other instances of CKAN;
+* demonstrate for the potential clients.
+
+High level overview of the planned features with ETA:
+
+| Name                          | Description                          | Effort | ETA |
+| ----------------------------- | ------------------------------------ | ------ | --- |
+| [Init](#Init)                 | Select CKAN version and deploy to DX | xs     | Q2  |
+| [Blobstore](#Blobstore)       | Integrate Giftless for raw storage   | s      | Q2  |
+| [Versioning](#Versioning)     | Develop/integrate new versioning sys | l      | Q3  |
+| [DataLoader](#DataLoader)     | Develop/integrate Aircan             | xl     | Q3  |
+| [Data API](#Data-API)         | Integrate new Data API (read)        | m      | Q2  |
+| [Frontend](#Frontend)         | Build a theme using PortalJS         | s      | Q2  |
+| [DataExplorer](#DataExplorer) | Integrate into PortalJS              | s      | Q2  |
+| [Permissions](#Permissions)   | Develop permissions in read frontend | m      | Q4  |
+| [Auth](#Auth)                 | Integrate                            | s      | Q4  |
+
+### Init
+
+Initialize a new project for development of CKAN Enterprise.
+
+Tasks:
+
+* Boot project in Datopian-DX cluster
+* Use CKAN v2.8.x (latest patch) or 2.9.x
+* Don't setup DataPusher
+* Namespace: `ckan-enterprise`
+* Domain: `enterprise.ckan.datopian.com`
+
+### Blobstore
+
+See [blob storage](/docs/dms/blob-storage#ckan-v3)
+
+### Versioning
+
+See [versioning](/docs/dms/versioning#ckan-v3)
+
+### DataLoader
+
+See [DataLoader](/docs/dms/load)
+
+### Data API
+
+* Install new [Data API service](https://github.com/datopian/data-api) in the project
+* Install Hasura service in the project
+* Set it up to work with DB of CKAN Enterprise
+* Read more about Data API [here](/docs/dms/data-api#read-api-3)
+
+Notes:
+
+* We could experiment and use various features of Hasura, eg:
+  * Setting up row/column limits per user role (permissions)
+  * Subscriptions to auto load new data rows
+
+### Frontend
+
+PortalJS for the read frontend of CKAN Enterprise. [Read more](/docs/dms/frontend/#frontend).
+
+### DataExplorer
+
+A new Data Explorer based on GraphQL API: https://github.com/datopian/data-explorer-graphql
+
+### Permissions
+
+See [permissions](/docs/dms/permissions#permissions-authorization).
+
+### Auth
+
+Next generation, Kratos based, authentication (mostly SSO with no Traditional login by default) with following options out of the box:
+
+* GitHub
+* Google
+* Facebook
+* Microsoft
+
+Easy to add:
+
+* Discord
+* GitLab
+* Slack

site/content/docs/dms/ckan-v3/index.md

@@ -0,0 +1,365 @@
+# CKAN v3
+
+## Introduction
+
+This document describes the architectures of CKAN v2 ("CKAN Classic"), CKAN v3 (also known as "CKAN Next Gen" for Next Generation), and CKAN v3 hybrid. The latter is an intermediate approach towards v3, where we still use CKAN v2 and common extensions, and only create microservices for new features.
+
+You will also find out how to do common tasks such as theming or testing, in each of the architectures.
+
+*Note: this blog post has an overview of the more decoupled, microservices approach at the core of v3: https://www.datopian.com/2021/05/17/a-more-decoupled-ckan/*
+
+## CKAN v2, CKAN v3 and Why v3
+
+In yellow, you see one single Python process:
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"]
+  end
+```
+
+When you want to extend core functionality of CKAN v2 (Classic), you write a Python package that must be installed in CKAN. This way, the extension will also run in the same process as the core functionality. This is known as a monolithic architecture.
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+  end
+```
+
+When you start to add multiple features, through extensions, what you get is one single Python process running many non-related functionalities.
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["CKAN Extension 2"]
+    ckancore --> ckanext3["CKAN Extension 3"]
+    ckancore --> ckanext4["CKAN Extension 4"]
+    ckancore --> ckanext5["CKAN Extension 5"]
+  end
+```
+
+This monolithic approach has advantages in terms of simplicity of development and deployment, especially when the system is small. However, as it grows in scale and scope, there are an increasing number of issues.
+
+In this approach, an optional extension has the ability to crash the whole CKAN instance. Every new feature must be written in the same language and framework (e.g. Python, leveraging Flask or Django). And, perhaps most fundamentally, the overall system is highly coupled, making it complex and hard to understand, debug, extend, and evolve.
+
+### Microservices and CKAN v3
+
+The main way to address these problems while gaining extra benefits is to move to a microservices-based architecture.
+
+Thus, we recommend building the next version of CKAN – CKAN v3 – on a microservices approach.
+
+[!tip]CKAN v3 is sometimes also referred to as CKAN Next Gen(eration).
+
+With microservices, each piece of functionality runs in its own service and process. 
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["CKAN frontend"]
+    ckanfrontend1["Frontend"]
+  end
+  
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+### Incremental Evolution – Hybrid v3
+
+One of the other advantages of the microservices approach is that it can also be used to extend and evolve current CKAN v2 solutions in an incremental way. We term these kinds of solutions "Hybrid v3," as they are a mix of v2 and v3 together. 
+
+For example, a Hybrid v3 data portal could use a new microservice written in Node for the frontend, and combine that with CKAN v2 (with v2 extensions).
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["CKAN frontend"]
+    ckanfrontend1["Frontend"]
+  end
+  
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["CKAN Extension 2"]
+  end
+  
+  ckanfrontend1 --> ckancore
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+The hybrid approach means we can evolve CKAN v2 "Classic" to CKAN v3 "Next Gen" incrementally. In particular, it allows people to keep using their existing v2 extensions, and upgrade them to new microservices gradually.
+
+### Comparison of Approaches
+
+|              | CKAN v2 (Classic) | CKAN v3 (Next Gen) | CKAN v3 Hybrid |
+| ------------ | ------------------| -------------------| ---------------|
+| Architecture | Monolithic        | Microservice       | Microservice with v2 core |
+| Language     | Python            | You can write services in any language you like.<br/><br/>Frontend default: JS.<br/>Backend default: Python | Python and any language you like for microservices. |
+| Frontend (and theming) | Python with Python CKAN extension | Flexible. Default is modern JS/NodeJS based | Can use old frontend but default to new JS-based frontend. |
+| Data Packages | Add-on, no integration | Default internal and external format | Data Packages with converter to old CKAN format. |
+| Extension | Extensions are libraries that are added to core runtime. They must therefore be built in python and are loaded into the core process at build time. "Template/inheritance" model where hooks are in core and it is core that loads and calls plugins. This means that if a hook does not exist in core then the extension is stymied. | Extensions are microservices and can be written in any language. They are loaded into the url space via kubernetes routing manager. Extensions hook into "core" via APIs (rather than in code). Follows a "composition" model rather than inheritance model | Can use old style extensions or microservices. |
+| Resource Scaling | You have a single application so scaling is of the core application. | You can scale individual microservices as needed. | Mix of v2 and v3 |
+
+## Why v3: Long Version
+
+What are the problems with CKAN v2's monolithic architecture in relation to microservices v3?
+
+* **Poor Developer Experience (DX), innovability, and scalability due to coupling**. Monolithic means "one big system" => Coupling & Complexity => hard to understand, change and extend. Changes in one area can unexpectedly affect other areas. 
+  * DX to develop a small new API requires wiring into CKAN core via an extension. Extensions can interact in unexpected ways.
+  * The core of people who fully understand CKAN has stayed small for a reason: there's a lot of understand.
+  * https://github.com/ckan/ckan/issues/5333 is an example of a small bug that's hard to track down due to various paths involved.
+  * Harder to make incremental changes due to coupling (e.g. Python 3 upgrade requires *everything* to be fixed at once - can't do rolling releases).
+* **Stability**. One bad extension crashes or slows down the whole system
+* **One language => Less developer flexibility (Poor DX)**. Have to write *everything* in Python, including the frontend. This is an issue especially for the frontend: almost all modern frontend development is heavily Javascript-based and theme is the #1 thing people want to customize in CKAN. At the moment, that requires installing *all* of CKAN core (using Docker) plus some familiarity with Python and Jinja templating. This is a big ask.
+* **Extension stablity and testing**. Testing of extensions is painful (at least without careful factoring in a separate mini library) and are therefore often not tested; they don't have Continuous Integration (CI) or Continuous Deployment (CD). As an example, a highly experienced Python developer at Datopian was still struggling to get extension tests working 6 months into their CKAN work.
+* **DX is poor especially when getting started**. Getting CKAN up and running requires multiple external services (database, Solr, Redis, etc.) making Docker the only viable way for bootstraping a local development environment. This makes getting started with CKAN daunting and painful.
+* **Vertical scalability is poor**. Scaling the system is costly as you have to replicate the whole core process in every machine.
+* **System is highly coupled.** Extensions b/c in process tend to end up with significant coupling to core which makes them brittle (has improved with plugins.toolkit)
+  * Upgrading core to Python 3 requires upgrading *all* extensions because they run in the same process.
+  * Search Index is not a separate API, but in Core. So replacing Solr is hard.
+
+The top 2 customizations of CKAN are slow and painful and require deep knowledge of CKAN:
+
+* Theming a site.
+* Customizing the metadata.
+
+## Architectures
+
+### CKAN v2 (Classic)
+
+This diagram is based on the file `docker-compose.yml` of [github.com/okfn/docker-ckan](https://github.com/okfn/docker-ckan) (`docker-compose.dev.yml` has the same components, but different configuration).
+ 
+A difference from this diagram to the file is that we are not including DataPusher, as it is not a required dependency.
+
+>[!tip]Databases may run as Docker containers, or rely on third-party services such as Amazon Relational Database Service (RDS).
+
+
+
+```mermaid
+graph LR
+
+CKAN[CKAN web app]
+
+CKAN --> DB[(Database)]
+CKAN --> Solr[(Solr)]
+CKAN --> Redis[(Redis)]
+
+subgraph Docker container
+  CKAN
+end
+```
+
+Same setup showing some of the key extensions explicitly:
+
+```mermaid
+graph LR
+  core[CKAN Core] --> DB[(Database)]
+  datastore --> DB2[(Database - DataStore)]
+  core --> Solr[(Solr)]
+  core --> Redis[(Redis)]
+  
+  subgraph Docker container
+    core
+    datastore
+    datapusher
+    imageview
+    ...
+  end
+```
+
+CKAN ships with several core extensions that are built-in. Here, together with the list of main components, we list a couple of them:
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+CKAN | Application (API + Worker) | [Link](https://github.com/ckan/ckan) | Data management system (DMS) for powering data hubs and data portals. It's a monolithical web application that includes several built-in extensions and dependencies, such as a job queue service. In theory, it's possible to run it without any extensions.
+datapusher | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/datapusher) | It could also be called "datapusher-connect." It's a glue code to connect with a separate microservice called DataPusher, which performs actions when new data arrives.
+datastore | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/datastore) | The interface between CKAN and the structure database, the one receiving datasets and resources (CSVs). It includes an API for the database and an administrative UI.
+imageview | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/imageview) | It provides an interface for creating HTML templates for image resources.
+multilingual | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/multilingual) | It provides an interface for translation and localization.
+Database | Database | | People tend to use a single PostgreSQL instance for this. Separated in multiple databases, it's the place where CKAN stores its own information (sometimes referred as "MetaStore" and "HubStore"), rows of resources (StructuredStore or DataStore), and raw datasets and resources ("BlobStore" or "FileStore"). The latter may store data in the local filesystem or cloud providers, via extensions.
+Solr | Database | | It provides indexing and full-text search for CKAN.
+Redis | Database | | Lightweight key-value store, used for caching and job queues.
+
+### CKAN v3 (Next Gen)
+
+CKAN Next Gen is still a DMS, as CKAN Classic; but rather than a monolithical architecture, it follows the microservices approach. CKAN Classic is not a dependency anymore, as we have smaller services providing functionality that we may or many not choose to include. This description is based on [Datopian's Technical Documentation](/docs/dms/ckan-v3/next-gen/#roadmap).
+
+```mermaid
+graph LR
+  subgraph api3["..."]
+    api31["API"]
+  end
+
+  subgraph api2["Administration"]
+    api21["API"]
+  end
+  
+  subgraph api1["Authentication"]
+    api11["API"]
+  end
+
+  subgraph frontend["Frontend"]
+    frontendapi["API"]
+  end
+  
+  subgraph storage["Raw Resources Storage"]
+    storageapi["API"]
+  end
+  
+  storageapi --> cloudstorage[(Cloud Storage)]
+  
+  frontendapi --> storageapi
+  frontendapi --> api11
+  frontendapi --> api21
+  frontendapi --> api31
+```
+
+At this moment, many important features are only available through CKAN extensions, so that brings us to the hybrid approach.
+
+### CKAN Hybrid v3 (Next Gen)
+
+We may sometimes make an explit distinction between CKAN v3 "hybrid" and "pure." The reason is because we want to ensure that we're not there yet – we have many opportunities to extract features out of CKAN and CKAN Extensions.
+
+In this approach, we still rely on CKAN Classic and all its extensions. Many already had many tests and bugs fixed, so we can deliver more if not forced to rewrite everything from scratch.
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["Frontend"]
+    ckanfrontend1["Frontend v2"]
+    theme["[Project-specific theme]"]
+  end
+  
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["[Project-specific extension]"]
+  end
+  
+  ckanfrontend1 --> ckancore
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+Frontend v2 | Application | [Link](https://github.com/datopian/frontend-v2) | Node application for Data Portals. It communicates with a CKAN Classic instance, through its API, to get data and render HTML. It is written to be extensible, such as connecting to other applications and theming.
+[Project-specific theme] | Frontend Theme | e.g., [Link](https://github.com/datopian/frontend-oddk) | Extension to Frontend v2 where you can personalize the interface, create different pages, and connect with other APIs.
+[API 1] | Application | e.g., [Link](https://github.com/datopian/data-subscriptions) | Any application with an API to communicate with the user-facing Frontend v2 or to run tasks in background. Given the current architecture, often, this API is usually designed to work with CKAN interfaces. Over time, we may choose to make it more generic, and even replace CKAN Core with other applications.
+
+## Job Stories
+
+In this spreadsheet, you will find a list of common job stories in CKAN projects. Also, how you can accomplish them in CKAN v2, v3, and Hybrid v3.
+
+https://docs.google.com/spreadsheets/d/1cLK8xylprmVsoQIbdphqz9-ccSpdDABQExvKdvNJqaQ/edit#gid=757361856
+
+## Glossary
+
+### API
+
+An HTTP API, usually following the REST style.
+
+### Application
+
+A Python package, an API, a worker... It may have other applications as dependencies.
+  
+### CKAN Extension
+
+A Python package following specification from [CKAN Extending guide](https://docs.ckan.org/en/2.8/extensions/index.html).
+  
+### Database
+
+An organized collection of data.
+
+### Dataset
+
+A group of resources made to be distributed together.
+  
+### Frontend Theme
+
+A Node project specializing behavior present in [Frontend v2](https://github.com/datopian/frontend-v2).
+  
+### Resource
+
+A data blob. Common formats are CSV, JSON, and PDF.
+  
+### System
+
+A group of applications and databases that work together to accomplish a set of tasks.
+  
+### Worker
+
+An application that runs tasks in background. They may run recurrently according to a given schedule, or as soon as it's requested by another application.
+
+## Appendix
+
+### Architecture - CKAN v2 with DataPusher
+
+```mermaid
+graph TB
+  subgraph DataPusher
+    datapusherapi["DataPusher API"]
+    datapusherworker["CKAN Service Provider"]
+    SQLite[(SQLite)]
+  end
+  
+  subgraph CKAN
+    core
+    datapusher[datapusher ext]
+    datastore
+    ...
+  end
+  
+  core[CKAN Core] --> datastore
+  datastore --> DB[(Database)]
+  datapusherapi --> core
+  datapusher --> datapusherapi
+```
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+DataPusher | System | [Link](https://github.com/ckan/datapusher) | Microservice that parses data files and uploads them to the datastore.
+DataPusher API | API | [Link](https://github.com/ckan/datapusher) | HTTP API written in Flask. It is called from the built-in `datapusher` CKAN extension whenever a resource is created (and has the right type).
+CKAN Service Provider | Worker | [Link](https://github.com/ckan/ckan-service-provider) | Library for making web services that make functions available as synchronous or asynchronous jobs.
+SQLite | Database | | Unknown use. Possibly a worker dependency.
+
+### Old Next Gen Page
+
+Prior to this page, we had one called "Next Gen." It has intersections with this article, although it focuses more on the benefits of microservices. For the time being, the page still exists in [/ckan-v3/next-gen](/docs/dms/ckan-v3/next-gen), although it may get merged with this one in the future.

site/content/docs/dms/ckan-v3/next-gen.md

@@ -0,0 +1,203 @@
+# Next Gen
+
+“Next Gen” (NG) is our name for the evolution of CKAN from its current state as “CKAN Classic”.
+
+Next Gen has a decoupled, microservice architecture in contrast to CKAN Classic's monolithic architecture. It is also built from the ground up on the Frictionless Data principles and specifications which provide a simple, well-defined and widely adopted set of core interfaces and tooling for managing data.
+
+## Classic to Next Gen
+
+CKAN classic: monolithic architecture -- everything is one big python application. Extension is done at code level and "compiled in" at compile/run-time (i.e. you end up with one big docker file).
+
+CKAN Next Gen: decoupled, service-oriented -- services connected by network calls. Extension is done by adding new services,
+
+```mermaid
+graph LR
+
+subgraph "CKAN Classic"
+  plugins
+end
+
+subgraph "CKAN Next Gen"
+  microservices
+end
+
+plugins --> microservices
+```
+
+You can read more about monolithic vs microservice architectures in the [Appendix below](#appendix-monolithic-vs-microservice-architecture).
+
+
+## Next Gen lays the foundation for the future and brings major immediate benefits
+
+Next Gen's new approach is important in several major ways.
+
+### Microservices are the Future
+
+First, decoupled microservices have become *the* way to design and deploy (web) applications after first being pioneered by the likes of Amazon in the early 2000s. And in the last five to ten years have brought microservices "for the masses" with relevant tooling and technology standardized, open-sourced and widely deployed -- not only with containerization such as Docker, Kubernetes but also in programming languages like (server-side) Javascript and Golang.
+
+By adopting a microservice approach CKAN can reap the the benefits of what is becoming a mature and standard way to design and build (web) applications. This includes the immediate advantages of being aligned with the technical paradigm such as tooling and developer familiarity.
+
+### Microservices bring Scalability, Reliability, Extensibility and Flexibility
+
+In addition, and even more importantly, the microservices approach brings major benefits in:
+
+1. **Scalability**: dramatically easier and cheaper to scale up -- and down -- in size *and* complexity. Size-wise this is because you can replicate individual services rather than the whole application. Complexity-wise this is because monolithic architectures tend to become "big" where service-oriented encourages smaller lightweight components with cleaner interfaces. This means you can have a much smaller core making it easier to install, setup and extend. It also means you can use what you need making solutions easier to maintain and upgrade.
+2. **Reliability**: easier (and cheaper) to build highly reliable, high availability solutions because microservices make isolation and replication easier. For example, in a microservice architecture a problem in CKAN's harvester won't impact your main portal because they run in separate containers. Similarly, you can scale the harvester system separately from the web frontend.
+3. **Extensibility**: much easier to create and maintain extensions because they are a decoupled service and interfaces are leaner and cleaner.
+4. **Flexibility** aka "Bring your own tech": services can be written in any language so, for example, you can write your frontend in javascript and your backend in Python. In a monolithic architecture all parts must be written in the same language because everything is compiled together. This flexibility makes it easier to use the best tool for the job. It also makes it much easier for teams to collaborate and cooperate and fewer bottlenecks in development.
+
+ASIDE: decoupled microservices reflect the "unix" way of building networked applications. As with the "unix way" in general, whilst this approach better -- and simpler -- in the long-run, in the short-run it often needs sustantial foundational work (those Unix authors were legends!). It may also be, at least initially, more resource intensive and more complex infrastructurally. Thus, whilst this approach is "better" it was not suprising that it was initially used for for complex and/or high end applications e.g. Amazon. This also explains why it took a while for this approach to get adoption -- it is only in the last few year that we have robust, lightweight, easy to use tooling and patterns for microservices -- "microservices for the masses" if you like.
+
+In summary, the Next Gen approach provides an essential foundation for the continuing growth and evolution of CKAN as a platform for building world-class data portal and data management solutions.
+
+## Evolution not Revolution: Next Gen Components Work with CKAN Classic
+
+*Gradual evolution from CKAN classic (keep what is working, keep your investments, incremental change)*
+
+Next Gen components are specifically designed to work with CKAN "Classic" in its current form. This means existing CKAN users can immediately benefit from Next Gen components and features whilst retaining the value of their existing investment. New (or existing) CKAN-based solutions can adopt a "hybrid" approach using components from both Classic and Next Gen. It also means that the owner of a CKAN-based solution can incrementally evolve from "Classic" to "Next Gen" by replacing one component one at a time, gaining new functionality without sacrificing existing work.
+
+ASIDE: we're fortunate that CKAN Classic itself was ahead of its time in its level of "service-orientation". From the start, it had a very rich and robust API and it has continued to develop this with almost almost all functionality exposed via the API. It is this rich API and well factored design that makes it relatively straightforward to evolve CKAN in its current "Classic" form towards Next Gen.
+
+## New Features plus Existing Functionality Improved
+
+In addition to its architecture, Next Gen provides a variety of improvements and extensions to CKAN Classic's functionality. For example:
+
+* Theming and Frontend Customization: theming and customizing CKAN's frontend has got radically easier and quicker. See [Frontend section »][frontend]
+* DMS + CMS unified: integrate the full power of a modern CMS into your data portal and have one unified interface for data and content. See [Frontend section »][frontend]
+* Data Explorer: the existing CKAN data preview/explorer has been completely rewritten in modern React-based Javascript (ReclineJS is now 7y old!). See [Data Explorer section »][explorer]
+* Dashboards: build rich data-driven dashboards and integrate. See [Dashboards section »][dashboards]
+* Harvesting: simpler, more powerful harvesting built on modern ETL. See [Harvesting section »][harvesting]
+
+And each of these features is easily deployed into an existing CKAN solution!
+
+[frontend]: /docs/dms/frontend
+[explorer]: /docs/dms/data-explorer
+[dashboards]: /docs/dms/dashboards
+[harvesting]: /docs/dms/harvesting
+
+## Roadmap
+
+The journey to Next Gen from Classic can proceed step by step -- it does not need to be a big bang. Like refurbishing and extending a house, we can add a room here or renovate a room there whilst continuing to live happily in the building (and benefitting as our new bathroom comes online, or we get a new conservatory!).
+
+Here's an overview of the journey to Next Gen and current implementation status. More granular information on particular features may sometimes be found on the individual feature page, for example for [Harvesting here](/docs/dms/harvesting#design).
+
+```mermaid
+graph LR
+
+start[Start]
+themefe[Read Frontend]
+authfe[Authentication in FE]
+authzfe[Authorization in FE]
+previews[Previews]
+explorer[Explorer]
+permsserv[Permissions Service]
+orgs[Organizations]
+
+
+subgraph Start
+  start
+end
+
+subgraph Frontend
+  start --> themefe
+  themefe --> authfe
+  authfe --> authzfe
+  themefe --> revisioningfe[Revision UI]
+end
+
+subgraph Harvesting
+  start --> harvestetl[Harvesting ETL + Runner]
+  harvestetl --> harvestui[Harvest UI]
+end
+
+subgraph "Admin UI"
+  managedataset[Manage Dataset]
+  manageorg[Manage Organization]
+  manageuser[Manage Users]
+  manageconfig[Manage Config]
+  
+  start --> managedataset
+  start --> manageorg
+  managedataset --> manageconfig
+end
+
+subgraph "Backend (API)"
+  start --> permsserv
+  start --> revision[Backend Revisioning]
+end
+
+datastore[DataStore]
+
+subgraph DataStore
+  start --> datastore
+  datastore --> dataload[Data Load]
+end
+
+subgraph Explorer
+  themefe --> previews
+  previews --> explorer
+end
+
+subgraph Organizations
+  start --> orgs
+end
+
+subgraph Key
+  done[Done]
+  nearlydone[Nearly Done]
+  inprogress[In Progress]
+  next[Next Up]
+end
+
+classDef done fill:#21bf73,stroke:#333,stroke-width:3px;
+classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:3px;
+classDef inprogress fill:orange,stroke:#333,stroke-width:2px;
+classDef next fill:pink,stroke:#333,stroke-width:1px;
+
+class done,themefe,previews,explorer,harvestetl done;
+class nearlydone,authfe,harvestui nearlydone;
+class inprogress,dataload inprogress;
+class next,permsserv next;
+```
+
+## Appendix: Monolithic vs Microservice architecture
+
+Monolithic: Libraries or modules communicate via function calls (inside one big application)
+
+Microservices: Services communicate over a network
+
+The best introduction and definition of microservices comes from Martin Fowler https://martinfowler.com/microservices/
+
+> Microservice architectures will use libraries, but their primary way of componentizing their own software is by breaking down into services. We define libraries as components that are linked into a program and called using in-memory function calls, while services are out-of-process components who communicate with a mechanism such as a web service request, or remote procedure call. https://martinfowler.com/articles/microservices.html
+
+### Monolithic
+
+```mermaid
+graph TD
+
+subgraph "Monolithic - all inside"
+  a
+  b
+  c
+end
+
+a --in-memory function all--> b
+a --in-memory function all--> c
+```
+
+### Microservice
+
+```mermaid
+graph TD
+subgraph "A Container"
+  a
+end
+subgraph "B Container"
+  b
+end
+subgraph "C Container"
+  c
+end
+a -.network call.-> b
+a -.network call.-> c
+```

site/content/docs/dms/ckan.md

@@ -0,0 +1,23 @@
+---
+sidebar: auto
+---
+
+# CKAN Classic
+
+CKAN (Classic) already has great documentation at: https://docs.ckan.org/
+
+This material is a complement to those docs as well as details of our particular setup.  Here, among other things, you'll learn how to: 
+
+* [Get Started with CKAN for Development -- install and run CKAN on your local machine](/docs/dms/ckan/getting-started)
+* [Play around with a CKAN instance including importing and visualising data](/docs/dms/ckan/play-around)
+* [Install Extensions](/docs/dms/ckan/install-extension)
+* [Create Your Own Extension](/docs/dms/ckan/create-extension)
+* [Client Guide](/docs/dms/ckan-client-guide)
+* [FAQ](/docs/dms/ckan/faq)
+
+[start]: /docs/dms/ckan/getting-started
+[play]: /docs/dms/ckan/play-around
+
+[CKAN]: https://ckan.org/
+[docs]: https://docs.ckan.org/
+

site/content/docs/dms/ckan/create-extension.md

@@ -0,0 +1,162 @@
+---
+sidebar: auto
+---
+
+# Introduction
+A CKAN extension is a Python package that modifies or extends CKAN. Each extension contains one or more plugins that must be added to your CKAN config file to activate the extension’s features.
+
+## Creating and Installing extensions
+1. Boot up your docker compose
+```
+docker-compose -f docker-compose.dev.yml up
+```
+
+
+2. To create an extension template using this docker composition execute:
+
+```
+docker-compose -f docker-compose.dev.yml exec ckan-dev /bin/bash -c "paster --plugin=ckan create -t ckanext ckanext-example_extension -o /srv/app/src_extensions"
+```
+
+This command will create an extension template in your local `./src` folder that is mounted inside the containers in the `/srv/app/src_extension` directory. Any extension cloned on the `src` folder will be installed in the CKAN container when booting up Docker Compose (`docker-compose up`). This includes installing any requirements listed in a `requirements.txt` (or `pip-requirements.txt`) file and running `python setup.py develop`.
+
+
+3. Add the plugin to the `CKAN__PLUGINS` setting in your `.env` file.
+
+```
+CKAN__PLUGINS=stats text_view recline_view example_extension
+```
+
+
+4. Restart your docker-compose:
+
+```
+# Shut down your instance with crtl+c and then run it again with:
+docker-compose -f docker-compose.dev.yml up
+```
+> [!tip]CKAN will be started running on the paster development server with the '--reload' option to watch changes in the extension files.
+
+You should see the following output in the console:
+
+```
+...
+ckan-dev_1    | Installed /srv/app/src_extensions/ckanext-example_extension
+...
+```
+
+## Edit the extension
+
+Let's edit a template to change the way CKAN is displayed to the user!
+
+1. First you will need write permissions to the extension folder since it was created by the user running docker. Replace `your_username` and execute the following command:
+
+> [!tip]You can find out your current username by typing 'echo $USER' in the terminal.
+
+```
+sudo chown -R <your_username>:<your_username> src/ckanext-example_extension
+```
+
+2. The previous comamand creates all the files and folder structure needed for our extension. Open `src/ckanext-example_extension/ckanext/example_extension/plugin.py` to see the main file of our extension that we will edit to add custom functionality:
+
+```python
+import ckan.plugins as plugins
+import ckan.plugins.toolkit as toolkit
+
+
+class Example_ExtensionPlugin(plugins.SingletonPlugin):
+    plugins.implements(plugins.IConfigurer)
+
+    # IConfigurer
+
+    def update_config(self, config_):
+        toolkit.add_template_directory(config_, 'templates')
+        toolkit.add_public_directory(config_, 'public')
+        toolkit.add_resource('fanstatic', 'example_theme')
+```
+
+3. We will create a custom Flask Blueprint to extend our CKAN instance with more endpoints. In order to create a new blueprint and add an endpoint we need to:
+ - Import Blueprint and render_template from the flask module.
+ - Create the functions that will be used as endpoints
+ - Implement the IBlueprint interface in our plugin and add the new endpoint.
+
+4. From flask import Blueprint and render_template,
+
+```python
+import ckan.plugins as plugins
+import ckan.plugins.toolkit as toolkit
+
+from flask import Blueprint, render_template
+
+class Example_ExtensionPlugin(plugins.SingletonPlugin):
+    plugins.implements(plugins.IConfigurer)
+
+    # IConfigurer
+
+    def update_config(self, config_):
+        toolkit.add_template_directory(config_, 'templates')
+        toolkit.add_public_directory(config_, 'public')
+        toolkit.add_resource('fanstatic', 'example_extension')
+```
+
+5. Create a new function: hello_plugin
+```python
+import ckan.plugins as plugins
+import ckan.plugins.toolkit as toolkit
+
+from flask import Blueprint, render_template
+
+def hello_plugin():
+    u'''A simple view function'''
+    return u'Hello World, this is served from an extension'
+
+class Example_ExtensionPlugin(plugins.SingletonPlugin):
+    plugins.implements(plugins.IConfigurer)
+
+    # IConfigurer
+
+    def update_config(self, config_):
+        toolkit.add_template_directory(config_, 'templates')
+        toolkit.add_public_directory(config_, 'public')
+        toolkit.add_resource('fanstatic', 'example_extension')
+```
+6. Implement the IBlueprint interface in our plugin and add the new endpoint.
+
+```python
+import ckan.plugins as plugins
+import ckan.plugins.toolkit as toolkit
+
+from flask import Blueprint, render_template
+
+def hello_plugin():
+    u'''A simple view function'''
+    return u'Hello World, this is served from an extension'
+
+class Example_ExtensionPlugin(plugins.SingletonPlugin):
+    plugins.implements(plugins.IConfigurer)
+    plugins.implements(plugins.IBlueprint)
+
+    # IConfigurer
+
+    def update_config(self, config_):
+        toolkit.add_template_directory(config_, 'templates')
+        toolkit.add_public_directory(config_, 'public')
+        toolkit.add_resource('fanstatic', 'example_extension')
+
+    # IBlueprint
+
+    def get_blueprint(self):
+        u'''Return a Flask Blueprint object to be registered by the app.'''
+        # Create Blueprint for plugin
+        blueprint = Blueprint(self.name, self.__module__)
+        blueprint.template_folder = u'templates'
+        # Add plugin url rules to Blueprint object
+        blueprint.add_url_rule('/hello_plugin', '/hello_plugin', hello_plugin)
+        return blueprint
+
+```
+
+6. Go back to the browser and navigate to http://ckan:5000/hello_plugin. You should see the value returned by our view!
+
+![New Blueprint output](https://i.imgur.com/AZjTDbN.png)
+
+Now that you have added a new view and endpoint to your plugin you are ready for the next step of the tutorial! You can also check the complete code of this plugin in the [ckan repository](https://github.com/ckan/ckan/tree/master/ckanext/example_flask_iblueprint).

site/content/docs/dms/ckan/faq.md

@@ -0,0 +1,110 @@
+---
+sidebar: auto
+---
+
+# FAQ
+
+This page provides answers to some frequently asked questions.
+
+## How to create an extension template in my local machine
+
+You can use the `paster` command in the same way as a source install. To create an extension execute the following command:
+
+```
+docker-compose -f docker-compose.dev.yml exec ckan-dev /bin/bash -c "paster --plugin=ckan create -t ckanext ckanext-myext -o /srv/app/src_extensions"
+```
+
+This will create an extension template inside the container's folder `/srv/app/src_extensions` which is mapped to your local `src/` folder.
+
+Now you can navigate to your local folder `src/` and see the extension created by the previous command and open the project in your favorite IDE.
+
+
+## How to separate that extension in a new git repository so I can have the independence to install it in other instances
+
+Crucial thing is to understand that extensions get their repositories on GitHub (or elsewhere). You can first create a repository for extension and later clone in `src/` or do opposite as following:
+
+* Create the Extension, for example: `ckanext-myext`.
+```
+docker-compose -f docker-compose.dev.yml exec ckan-dev /bin/bash -c "paster --plugin=ckan create -t ckanext ckanext-myext -o /srv/app/src_extensions"
+```
+
+* Init your new git repository into the extension folder `src/ckanext-myext`
+```
+cd src/ckanext-myext
+git init
+```
+* Configure remote/origin
+```
+git remote add origin <remote_repository_url>
+```
+* Add your files and push the first commit
+```
+git add .
+git commit -m 'Initial Commit'
+git push
+```
+
+**Note:** The `src/` folder is gitignored in `okfn/docker-ckan` repository, so initializing new git repositories inside is ok.
+
+## How to quickly refresh the changes in my extension into the dockerized environment so I can have quick feedback of my changes
+
+This docker-compose setup for dev environment is already configured so that it sets `debug=True` inside configuration file and auto reloads on python and templates related changes. You do not have to reload when making changes to HTML, javascript or configuration files - you just need to refresh the page in the browser.
+
+See the CKAN images section of the [repository documentation](https://github.com/okfn/docker-ckan#ckan-images) for more detail
+
+## How to run tests for my extension in the dockerized environment so I can have a quick test-development cycle
+
+We write and store unit tests inside the `ckanext/myext/tests` directory. To run unit tests you need to be running the `ckan-dev` service of this docker-compose setup.
+
+* Once running, in another terminal window run the test command:
+```
+docker-compose -f docker-compose.dev.yml exec ckan-dev nosetests --ckan-dev --nologcapture --reset-db -s -v --with-pylons=/srv/app/src_extensions/ckanext-myext/test.ini /srv/app/src_extensions/ckanext-myext/
+```
+
+You can also pass nosetest arguments to debug
+```
+--ipdb --ipdb-failure
+```
+
+**Note:** Right now all tests will be run, it is not possible to choose a specific file or test.
+
+## How to debug my methods in the dockerized environment so I can have a better understanding of whats going on with my logic
+
+To run a container and be able to add a breakpoint with `pdb`, run the `ckan-dev` container with the `--service-ports` option:
+
+```
+docker-compose -f docker-compose.dev.yml run --service-ports ckan-dev
+```
+
+This will start a new container, displaying the standard output in your terminal. If you add a breakpoint in a source file in the `src` folder (`import pdb; pdb.set_trace()`) you will be able to inspect it in this terminal next time the code is executed.
+
+## How to debug core CKAN code
+
+Currently, this docker-compose setup doesn't allow us to debug core CKAN code since it lives inside the container. However, we can do some hacks so the container uses a local clone of the CKAN core hosted in our machine. To do it:
+
+- Create a new folder called `ckan_src` in this `docker-ckan` folder at the same level of the `src/`
+- Clone ckan and checkout the version you want to debug/edit
+
+```
+git https://github.com/ckan/ckan/ ckan_src
+cd ckan_src
+git checkout ckan-2.8.3
+```
+
+- Edit `docker-compose.dev.yml` and add an entry to ckan-dev's and ckan-worker-dev's volumes. This will allow the docker container to access the CKAN code hosted in our machine.
+
+```
+   - ./ckan_src:/srv/app/ckan_src
+```
+
+- Create a script in `ckan/docker-entrypoint.d/z_install_ckan.sh` to install CKAN inside the container from the cloned repository (instead of the one installed in the Dockerfile)
+
+```
+#!/bin/bash
+echo "*********************************************"
+echo "overriding with ckan installation with ckan_src"
+pip install -e /srv/app/ckan_src
+echo "*********************************************"
+```
+
+That's it. This will install CKAN inside the container in development mode, from the shared folder. Now you can open the `ckan_src/` folder from your favorite IDE and start working on CKAN.

site/content/docs/dms/ckan/getting-started.md

@@ -0,0 +1,77 @@
+# CKAN: Getting Started for Development
+
+## Prerequisites
+
+CKAN has a rich tech stack so we have opted to standardize our instructions with Docker Compose, which will help you spin up every service in a few commands.
+
+If you already have Docker-compose, you are ready to go!
+
+If not, please, follow instructions on [how to install docker-compose](https://docs.docker.com/compose/install/).
+
+On Ubuntu you can run:
+
+```
+sudo apt-get update
+sudo apt-get install docker-compose
+```
+
+## Cloning the repo
+
+```
+git clone https://github.com/okfn/docker-ckan
+# or git clone git@github.com:okfn/docker-ckan.git
+cd docker-ckan
+```
+
+## Booting CKAN
+
+Create a local environment file:
+
+```
+cp .env.example .env
+```
+
+Build and Run the instances:
+
+> [!tip]'docker-compose' must be run with 'sudo'. If you want to change this, you can follow the steps below. NOTE: The 'docker' group grants privileges equivalent to the 'root' user.  
+
+Create the `docker` group: `sudo groupadd docker`  
+
+Add your user to the `docker` group: `sudo usermod -aG docker $USER`  
+
+Change the storage directory ownership from `root` to `ckan` by adding the commads below to the `ckan/Dockerfile.dev`
+
+```
+RUN mkdir -p /var/lib/ckan/storage/uploads
+RUN chown -R ckan:ckan /var/lib/ckan/storage
+```
+
+At this point, you can log out and log back in for these changes to apply. You can also use the command `newgrp docker` to temporarily enable the new group for the current terminal session.
+
+```
+docker-compose -f docker-compose.dev.yml up --build
+```
+
+When you see this log message:
+
+![](https://i.imgur.com/WUIiNRt.png)
+
+You can navigate to `http://localhost:5000`
+
+![CKAN Home Page](https://i.imgur.com/T5LWo8A.png)
+
+and log in with the credentials that docker-compose setup created for you [user: `ckan_admin` password:`test1234`].
+
+>[!tip]To learn key concepts about CKAN, including what it is and how it works, you can read the User Guide.
+[CKAN User Guide](https://docs.ckan.org/en/2.8/user-guide.html).
+
+
+## Next Steps
+
+[Play around with CKAN portal](/docs/dms/ckan/play-around).
+
+## Troubleshooting
+
+Login / Logout button breaks the experience:
+
+- Change the URL from `http://ckan:5000` to `http://localhost:5000`. A complete fix is described in the [Play around with CKAN portal](/docs/dms/ckan/play-around). (Your next step. ;))

site/content/docs/dms/ckan/install-extension.md

@@ -0,0 +1,76 @@
+---
+sidebar: auto
+---
+
+# Installing extensions
+
+A CKAN extension is a Python package that modifies or extends CKAN. Each extension contains one or more plugins that must be added to your CKAN config file to activate the extension’s features.
+
+In this sections we will teach you only how to install existing extensions. See [next steps](/docs/dms/ckan/create-extension) in case you need to create or modify extensions
+
+## Add new extension
+
+Lets install [Hello World](https://github.com/rclark/ckanext-helloworld) on the portal. For that we need to do 2 thing:
+
+1. Install extension when building docker image
+2. Add new extension to CKAN plugins
+
+### Install extension on docker build
+
+For this we need to modify Dockerfile for ckan service. Let's edit it:
+
+```
+vi ckan/Dockerfile.dev
+
+# Add following
+RUN pip install -e git+https://github.com/rclark/ckanext-helloworld.git#egg=ckanext-helloworld
+```
+
+*Note:* In this example we use vi editor, but you can choose any of your choice.
+
+### Add new extension to plugins
+
+We need to modify .env file for that - Search for `CKAN_PLUGINS` and add new extension to the existing list:
+
+```
+vi .env
+
+CKAN__PLUGINS=helloworld envvars image_view text_view recline_view datastore datapusher
+```
+
+## Check extension is installed
+
+After modifying configuration files you will need to restart the portal. If your CKAN protal is up and running bring it down and re-start
+
+```
+docker-compose -f docker-compose.dev.yml stop
+docker-compose -f docker-compose.dev.yml up --build
+```
+
+### Check what extensions you already have:
+
+http://ckan:5000/api/3/action/status_show
+
+Response should include list of all extensions including `helloworld` in it.
+
+```
+"extensions": [
+  "envvars",
+  "helloworld",
+  "image_view",
+  "text_view",
+  "recline_view",
+  "datastore",
+  "datapusher"
+]
+```
+
+### Check the extension is actually working
+
+This extension simply adds new route `/hello/world/name` to the base ckan and says hello
+
+http://ckan:5000/hello/world/John-Doe
+
+## Next steps
+
+[Create your own extension](/docs/dms/ckan/create-extension)

site/content/docs/dms/ckan/play-around.md

@@ -0,0 +1,285 @@
+---
+sidebar: auto
+---
+
+# How to play around with CKAN
+
+In this section, we are going to show some basic functionality of CKAN focused on the API.
+
+## Prerequisites
+
+- We assume you've already completed the [Getting Started Guide](/docs/dms/ckan/getting-started).
+- You have a basic understanding of Key data portal concepts:
+
+CKAN is a tool for making data portals to manage and publish datasets. You can read about the key concepts such as Datasets and Organizations in the User Guide -- or you can just dive in and play around!
+
+https://docs.ckan.org/en/2.9/user-guide.html
+
+>[!tip]
+Install a [JSON formatter plugin for Chrome](https://chrome.google.com/webstore/detail/json-formatter/bcjindcccaagfpapjjmafapmmgkkhgoa?hl=en) or browser of your choice.
+
+If you are familiar with the command line tool `curl`, you can use that.
+
+In this tutorial, we will be using `curl`, but for most of the commands, you can paste a link in your browser. For POST commands, you can use [Postman](https://www.getpostman.com/) or [Google Chrome Plugin](https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop).
+
+
+## First steps
+
+>[!tip]
+By default the portal is accessible on http://localhost:5000. Let's update your `/etc/hosts` to access it on http://ckan:5000:
+
+```
+vi /etc/hosts      # You can use the editor of your choice
+# add following
+127.0.0.1 ckan
+```
+
+
+At this point, you should be able to access the portal on http://ckan:5000.
+
+![CKAN Home Page](https://i.imgur.com/T5LWo8A.png)
+
+Let's add some fixtures to it. For software, a fixture is something used consistently (in this case, data for you to play around with). Run the following from your terminal (do NOT cut the previous docker process as this one depends on the already launched docker, run in another terminal):
+
+```sh
+docker-compose -f docker-compose.dev.yml exec ckan-dev ckan seed basic
+```
+
+Optionally you can `exec` into a running container using 
+
+```sh
+docker exec -it [name of container] sh
+```
+
+and run the `ckan` command there
+```sh
+ckan seed basic
+```
+
+You should be able to see 2 new datasets on home page:
+
+![CKAN with data](https://i.imgur.com/BiSifyb.png)
+
+To get more details on ckan commands please visit [CKAN Commands Reference](https://docs.ckan.org/en/2.9/maintaining/cli.html#ckan-commands-reference).
+
+### Check CKAN API
+
+This tutorial focuses on the CKAN API as that is central to development work and requires more guidance. We also invite you to explore the user interface which you can do directly yourself by visiting http://ckan:5000/.
+
+#### Let's check the portal status
+
+Go to http://ckan:5000/api/3/action/status_show.
+
+You should see something like this:
+
+```json
+{
+  "help": "https://ckan:5000/api/3/action/help_show?name=status_show",
+  "success": true,
+  "result": {
+  "ckan_version": "2.9.x",
+  "site_url": "https://ckan:5000",
+  "site_description": "Testing",
+  "site_title": "CKAN Demo",
+  "error_emails_to": null,
+  "locale_default": "en",
+  "extensions": [
+    "envvars",
+    ...
+    "demo"
+    ]
+  }
+}
+```
+
+This means everything is OK: the CKAN portal is up and running, the API is working as expected. In case you see an internal server error, please check the logs in your terminal.
+
+### A Few useful API endpoints to start with
+
+CKAN's Action API is a powerful, RPC-style API that exposes all of CKAN's core features to API clients. All of a CKAN website's core functionality (everything you can do with the web interface and more) can be used by external code that calls the CKAN API.
+
+#### Get a list of all datasets on the portal
+
+http://ckan:5000/api/3/action/package_list
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=package_list",
+  "success": true,
+  "result": ["annakarenina", "warandpeace"]
+}
+```
+
+#### Search for a dataset
+
+http://ckan:5000/api/3/action/package_search?q=russian
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=package_search",
+  "success": true,
+  "result": {
+    "count": 2,
+    ...
+  }
+}
+```
+
+#### Get dataset details
+
+http://ckan:5000/api/3/action/package_show?id=annakarenina
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=package_show",
+  "success": true,
+  "result": {
+    "license_title": "Other (Open)",
+    ...
+  }
+}
+```
+
+#### Search for a resource
+
+http://ckan:5000/api/3/action/resource_search?query=format:plain%20text
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=resource_search",
+  "success": true,
+  "result": {
+    "count": 1,
+    "results": [
+      {
+        "mimetype": null,
+        ...
+      }
+    ]
+  }
+}
+```
+
+#### Get resource details
+
+http://ckan:5000/api/3/action/resource_show?id=288455e8-c09c-4360-b73a-8b55378c474a
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=resource_show",
+  "success": true,
+  "result": {
+    "mimetype": null,
+    ...
+  }
+}
+```
+
+*Note:* These are only a few examples. You can find a full list of API actions in the [CKAN API guide](https://docs.ckan.org/en/2.9/api/#action-api-reference).
+
+### Create Organizations, Datasets and Resources
+
+There are 4 steps:
+
+- Get an API key;
+- Create an organization;
+- Create dataset inside an organization (you can't create a dataset without a parent organization);
+- And add resources to the dataset.
+
+#### Get a Sysadmin Key
+
+To create your first dataset, you need an API key.
+
+You can see sysadmin credentials in the file `.env`. By default, they should be
+
+- Username: `ckan_admin`
+- Password: `test1234`
+
+1. Navigate to http://ckan:5000/user/login and login.
+2. Click on your username (`ckan_admin`) in the upright corner.
+3. Scroll down until you see `API Key` on the left side of the screen and copy its value. It should look similar to `c7325sd4-7sj3-543a-90df-kfifsdk335`.
+
+#### Create Organization
+
+You can create an organization from the browser easily, but let's use [CKAN API](https://docs.ckan.org/en/2.9/api/#ckan.logic.action.create.organization_create) to do so.
+
+```sh
+curl -X POST http://ckan:5000/api/3/action/organization_create -H "Authorization: 9c04a69d-79f4-4b4b-b4e1-f2ac31ed961c" -d '{
+  "name": "demo-organization",
+  "title": "Demo Organization",
+  "description": "This is my awesome organization"
+}'
+```
+
+Response:
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=organization_create",
+  "success": true,
+  "result": {"users": [
+    {
+    "email_hash":
+    ...
+    }
+  ]}
+}
+```
+
+#### Create Dataset
+
+Now, we are ready to create our first dataset.
+
+```sh
+curl -X POST http://ckan:5000/api/3/action/package_create -H "Authorization: 9c04a69d-79f4-4b4b-b4e1-f2ac31ed961c" -d '{
+ "name": "my-first-dataset",
+ "title": "My First Dataset",
+ "description": "This is my first dataset!",
+ "owner_org": "demo-organization"
+}'
+```
+
+Response:
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=package_create",
+  "success": true,
+  "result": {
+    "license_title": null,
+    ...
+  }
+}
+```
+
+This will create an empty (draft) dataset.
+
+#### Add a resource to it
+
+```sh
+curl -X POST http://ckan:5000/api/3/action/resource_create -H "Authorization: 9c04a69d-79f4-4b4b-b4e1-f2ac31ed961c" -d '{
+  "package_id": "my-first-dataset",
+  "url":  "https://raw.githubusercontent.com/frictionlessdata/test-data/master/files/csv/100kb.csv",
+  "description": "This is the best resource ever!" ,
+  "name": "brand-new-resource"
+}'
+```
+
+Response:
+
+```json
+{
+  "help": "http://ckan:5000/api/3/action/help_show?name=resource_create",
+  "success": true,
+  "result": {
+    "cache_last_updated": null,
+    ...
+  }
+}
+```
+
+That's it! Now you should be able to see your dataset on the portal at http://ckan:5000/dataset/my-first-dataset.
+
+## Next steps
+
+* [Install Extensions](/docs/dms/ckan/install-extension).