[alan-turing][m] - cleanup

## Changes:

- Move the first version of the tutorial aside
- Rewrite the tutorial so that it's more similar to the Next.js one
- Minor fixes on the basic-example

.gitignore

@@ -44,3 +44,7 @@ Thumbs.db
 # Env
 .env
 **/.env
+
+# MarkdownDB
+*.db
+**/*.db

README.md

@@ -48,7 +48,7 @@ https://portaljs.org/docs
 
 # Community
 
-If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/An7Bu5x8).
+If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
 
 # Appendix
 

examples/alan-turing-portal/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+  "extends": "next/core-web-vitals"
+}

examples/alan-turing-portal/.gitignore

@@ -0,0 +1,35 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# generated files
+/public/rss/

examples/alan-turing-portal/LICENSE.md

@@ -0,0 +1,129 @@
+# Tailwind UI License
+
+## Personal License
+
+Tailwind Labs Inc. grants you an on-going, non-exclusive license to use the Components and Templates.
+
+The license grants permission to **one individual** (the Licensee) to access and use the Components and Templates.
+
+You **can**:
+
+- Use the Components and Templates to create unlimited End Products.
+- Modify the Components and Templates to create derivative components and templates. Those components and templates are subject to this license.
+- Use the Components and Templates to create unlimited End Products for unlimited Clients.
+- Use the Components and Templates to create End Products where the End Product is sold to End Users.
+- Use the Components and Templates to create End Products that are open source and freely available to End Users.
+
+You **cannot**:
+
+- Use the Components and Templates to create End Products that are designed to allow an End User to build their own End Products using the Components and Templates or derivatives of the Components and Templates.
+- Re-distribute the Components and Templates or derivatives of the Components and Templates separately from an End Product, neither in code or as design assets.
+- Share your access to the Components and Templates with any other individuals.
+- Use the Components and Templates to produce anything that may be deemed by Tailwind Labs Inc, in their sole and absolute discretion, to be competitive or in conflict with the business of Tailwind Labs Inc.
+
+### Example usage
+
+Examples of usage **allowed** by the license:
+
+- Creating a personal website by yourself.
+- Creating a website or web application for a client that will be owned by that client.
+- Creating a commercial SaaS application (like an invoicing app for example) where end users have to pay a fee to use the application.
+- Creating a commercial self-hosted web application that is sold to end users for a one-time fee.
+- Creating a web application where the primary purpose is clearly not to simply re-distribute the components (like a conference organization app that uses the components for its UI for example) that is free and open source, where the source code is publicly available.
+
+Examples of usage **not allowed** by the license:
+
+- Creating a repository of your favorite Tailwind UI components or templates (or derivatives based on Tailwind UI components or templates) and publishing it publicly.
+- Creating a React or Vue version of Tailwind UI and making it available either for sale or for free.
+- Create a Figma or Sketch UI kit based on the Tailwind UI component designs.
+- Creating a "website builder" project where end users can build their own websites using components or templates included with or derived from Tailwind UI.
+- Creating a theme, template, or project starter kit using the components or templates and making it available either for sale or for free.
+- Creating an admin panel tool (like [Laravel Nova](https://nova.laravel.com/) or [ActiveAdmin](https://activeadmin.info/)) that is made available either for sale or for free.
+
+In simple terms, use Tailwind UI for anything you like as long as it doesn't compete with Tailwind UI.
+
+### Personal License Definitions
+
+Licensee is the individual who has purchased a Personal License.
+
+Components and Templates are the source code and design assets made available to the Licensee after purchasing a Tailwind UI license.
+
+End Product is any artifact produced that incorporates the Components or Templates or derivatives of the Components or Templates.
+
+End User is a user of an End Product.
+
+Client is an individual or entity receiving custom professional services directly from the Licensee, produced specifically for that individual or entity. Customers of software-as-a-service products are not considered clients for the purpose of this document.
+
+## Team License
+
+Tailwind Labs Inc. grants you an on-going, non-exclusive license to use the Components and Templates.
+
+The license grants permission for **up to 25 Employees and Contractors of the Licensee** to access and use the Components and Templates.
+
+You **can**:
+
+- Use the Components and Templates to create unlimited End Products.
+- Modify the Components and Templates to create derivative components and templates. Those components and templates are subject to this license.
+- Use the Components and Templates to create unlimited End Products for unlimited Clients.
+- Use the Components and Templates to create End Products where the End Product is sold to End Users.
+- Use the Components and Templates to create End Products that are open source and freely available to End Users.
+
+You **cannot**:
+
+- Use the Components or Templates to create End Products that are designed to allow an End User to build their own End Products using the Components or Templates or derivatives of the Components or Templates.
+- Re-distribute the Components or Templates or derivatives of the Components or Templates separately from an End Product.
+- Use the Components or Templates to create End Products that are the property of any individual or entity other than the Licensee or Clients of the Licensee.
+- Use the Components or Templates to produce anything that may be deemed by Tailwind Labs Inc, in their sole and absolute discretion, to be competitive or in conflict with the business of Tailwind Labs Inc.
+
+### Example usage
+
+Examples of usage **allowed** by the license:
+
+- Creating a website for your company.
+- Creating a website or web application for a client that will be owned by that client.
+- Creating a commercial SaaS application (like an invoicing app for example) where end users have to pay a fee to use the application.
+- Creating a commercial self-hosted web application that is sold to end users for a one-time fee.
+- Creating a web application where the primary purpose is clearly not to simply re-distribute the components or templates (like a conference organization app that uses the components or a template for its UI for example) that is free and open source, where the source code is publicly available.
+
+Examples of use **not allowed** by the license:
+
+- Creating a repository of your favorite Tailwind UI components or template (or derivatives based on Tailwind UI components or templates) and publishing it publicly.
+- Creating a React or Vue version of Tailwind UI and making it available either for sale or for free.
+- Creating a "website builder" project where end users can build their own websites using components or templates included with or derived from Tailwind UI.
+- Creating a theme or template using the components or templates and making it available either for sale or for free.
+- Creating an admin panel tool (like [Laravel Nova](https://nova.laravel.com/) or [ActiveAdmin](https://activeadmin.info/)) that is made available either for sale or for free.
+- Creating any End Product that is not the sole property of either your company or a client of your company. For example your employees/contractors can't use your company Tailwind UI license to build their own websites or side projects.
+
+### Team License Definitions
+
+Licensee is the business entity who has purchased a Team License.
+
+Components and Templates are the source code and design assets made available to the Licensee after purchasing a Tailwind UI license.
+
+End Product is any artifact produced that incorporates the Components or Templates or derivatives of the Components or Templates.
+
+End User is a user of an End Product.
+
+Employee is a full-time or part-time employee of the Licensee.
+
+Contractor is an individual or business entity contracted to perform services for the Licensee.
+
+Client is an individual or entity receiving custom professional services directly from the Licensee, produced specifically for that individual or entity. Customers of software-as-a-service products are not considered clients for the purpose of this document.
+
+## Enforcement
+
+If you are found to be in violation of the license, access to your Tailwind UI account will be terminated, and a refund may be issued at our discretion. When license violation is blatant and malicious (such as intentionally redistributing the Components or Templates through private warez channels), no refund will be issued.
+
+The copyright of the Components and Templates is owned by Tailwind Labs Inc. You are granted only the permissions described in this license; all other rights are reserved. Tailwind Labs Inc. reserves the right to pursue legal remedies for any unauthorized use of the Components or Templates outside the scope of this license.
+
+## Liability
+
+Tailwind Labs Inc.’s liability to you for costs, damages, or other losses arising from your use of the Components or Templates — including third-party claims against you — is limited to a refund of your license fee. Tailwind Labs Inc. may not be held liable for any consequential damages related to your use of the Components or Templates.
+
+This Agreement is governed by the laws of the Province of Ontario and the applicable laws of Canada. Legal proceedings related to this Agreement may only be brought in the courts of Ontario. You agree to service of process at the e-mail address on your original order.
+
+## Questions?
+
+Unsure which license you need, or unsure if your use case is covered by our licenses?
+
+Email us at [support@tailwindui.com](mailto:support@tailwindui.com) with your questions.

examples/alan-turing-portal/README.md

@@ -0,0 +1,19 @@
+## Getting started
+
+To get started with this template, first install the npm dependencies:
+
+```bash
+npm install
+```
+
+Next, run the development server:
+
+```bash
+npm run dev
+```
+
+Finally, open [http://localhost:3000](http://localhost:3000) in your browser to view the website.
+
+## License
+
+This site template is a commercial product and is licensed under the [Tailwind UI license](https://tailwindui.com/license).

examples/alan-turing-portal/components/Card.jsx

@@ -0,0 +1,94 @@
+import Link from 'next/link'
+import clsx from 'clsx'
+
+function ChevronRightIcon(props) {
+  return (
+    <svg viewBox="0 0 16 16" fill="none" aria-hidden="true" {...props}>
+      <path
+        d="M6.75 5.75 9.25 8l-2.5 2.25"
+        strokeWidth="1.5"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  )
+}
+
+export function Card({ as: Component = 'div', className, children }) {
+  return (
+    <Component
+      className={clsx(className, 'group relative flex flex-col items-start')}
+    >
+      {children}
+    </Component>
+  )
+}
+
+Card.Link = function CardLink({ children, ...props }) {
+  return (
+    <>
+      <div className="absolute -inset-x-4 -inset-y-6 z-0 scale-95 bg-zinc-50 opacity-0 transition group-hover:scale-100 group-hover:opacity-100 dark:bg-zinc-800/50 sm:-inset-x-6 sm:rounded-2xl" />
+      <Link {...props}>
+        <span className="absolute -inset-x-4 -inset-y-6 z-20 sm:-inset-x-6 sm:rounded-2xl" />
+        <span className="relative z-10">{children}</span>
+      </Link>
+    </>
+  )
+}
+
+Card.Title = function CardTitle({ as: Component = 'h2', href, children }) {
+  return (
+    <Component className="text-base font-semibold tracking-tight text-zinc-800 dark:text-zinc-100">
+      {href ? <Card.Link href={href}>{children}</Card.Link> : children}
+    </Component>
+  )
+}
+
+Card.Description = function CardDescription({ children }) {
+  return (
+    <p className="z-10 mt-2 text-sm text-zinc-600 dark:text-zinc-400">
+      {children}
+    </p>
+  )
+}
+
+Card.Cta = function CardCta({ children }) {
+  return (
+    <div
+      aria-hidden="true"
+      className="relative z-10 mt-4 flex items-center text-sm font-medium text-teal-500"
+    >
+      {children}
+      <ChevronRightIcon className="ml-1 h-4 w-4 stroke-current" />
+    </div>
+  )
+}
+
+Card.Eyebrow = function CardEyebrow({
+  as: Component = 'p',
+  decorate = false,
+  className,
+  children,
+  ...props
+}) {
+  return (
+    <Component
+      className={clsx(
+        className,
+        'relative z-10 order-first mb-3 flex items-center text-sm text-zinc-400 dark:text-zinc-500',
+        decorate && 'pl-3.5'
+      )}
+      {...props}
+    >
+      {decorate && (
+        <span
+          className="absolute inset-y-0 left-0 flex items-center"
+          aria-hidden="true"
+        >
+          <span className="h-4 w-0.5 rounded-full bg-zinc-200 dark:bg-zinc-500" />
+        </span>
+      )}
+      {children}
+    </Component>
+  )
+}

examples/alan-turing-portal/components/Container.jsx

@@ -0,0 +1,42 @@
+import { forwardRef } from 'react'
+import clsx from 'clsx'
+
+const OuterContainer = forwardRef(function OuterContainer(
+  { className, children, ...props },
+  ref
+) {
+  return (
+    <div ref={ref} className={clsx('sm:px-8', className)} {...props}>
+      <div className="mx-auto max-w-7xl lg:px-8">{children}</div>
+    </div>
+  )
+})
+
+const InnerContainer = forwardRef(function InnerContainer(
+  { className, children, ...props },
+  ref
+) {
+  return (
+    <div
+      ref={ref}
+      className={clsx('relative px-4 sm:px-8 lg:px-12', className)}
+      {...props}
+    >
+      <div className="mx-auto max-w-2xl lg:max-w-5xl">{children}</div>
+    </div>
+  )
+})
+
+export const Container = forwardRef(function Container(
+  { children, ...props },
+  ref
+) {
+  return (
+    <OuterContainer ref={ref} {...props}>
+      <InnerContainer>{children}</InnerContainer>
+    </OuterContainer>
+  )
+})
+
+Container.Outer = OuterContainer
+Container.Inner = InnerContainer

examples/alan-turing-portal/components/Footer.jsx

@@ -0,0 +1,36 @@
+import Link from 'next/link'
+
+import { Container } from '../components/Container'
+
+function NavLink({ href, children }) {
+  return (
+    <Link
+      href={href}
+      className="transition hover:text-teal-500 dark:hover:text-teal-400"
+    >
+      {children}
+    </Link>
+  )
+}
+
+export function Footer() {
+  return (
+    <footer className="mt-32">
+      <Container.Outer>
+        <div className="border-t border-zinc-100 pb-16 pt-10 dark:border-zinc-700/40">
+          <Container.Inner>
+            <div className="flex flex-col items-center justify-between gap-6 sm:flex-row">
+              <p className="text-sm font-medium text-zinc-800 dark:text-zinc-200">
+                hatespeechdata maintained by <a href='https://github.com/leondz'>leondz</a>
+              </p>
+              <p className="text-sm text-zinc-400 dark:text-zinc-500">
+                &copy; {new Date().getFullYear()} Leon Derczynski. All rights
+                reserved.
+              </p>
+            </div>
+          </Container.Inner>
+        </div>
+      </Container.Outer>
+    </footer>
+  )
+}

examples/alan-turing-portal/components/Header.jsx

@@ -0,0 +1,109 @@
+import { useRef } from 'react'
+import { useRouter } from 'next/router'
+
+import { Container } from '../components/Container'
+
+function SunIcon(props) {
+  return (
+    <svg
+      viewBox="0 0 24 24"
+      strokeWidth="1.5"
+      strokeLinecap="round"
+      strokeLinejoin="round"
+      aria-hidden="true"
+      {...props}
+    >
+      <path d="M8 12.25A4.25 4.25 0 0 1 12.25 8v0a4.25 4.25 0 0 1 4.25 4.25v0a4.25 4.25 0 0 1-4.25 4.25v0A4.25 4.25 0 0 1 8 12.25v0Z" />
+      <path
+        d="M12.25 3v1.5M21.5 12.25H20M18.791 18.791l-1.06-1.06M18.791 5.709l-1.06 1.06M12.25 20v1.5M4.5 12.25H3M6.77 6.77 5.709 5.709M6.77 17.73l-1.061 1.061"
+        fill="none"
+      />
+    </svg>
+  )
+}
+
+function MoonIcon(props) {
+  return (
+    <svg viewBox="0 0 24 24" aria-hidden="true" {...props}>
+      <path
+        d="M17.25 16.22a6.937 6.937 0 0 1-9.47-9.47 7.451 7.451 0 1 0 9.47 9.47ZM12.75 7C17 7 17 2.75 17 2.75S17 7 21.25 7C17 7 17 11.25 17 11.25S17 7 12.75 7Z"
+        strokeWidth="1.5"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  )
+}
+
+function ModeToggle() {
+  function disableTransitionsTemporarily() {
+    document.documentElement.classList.add('[&_*]:!transition-none')
+    window.setTimeout(() => {
+      document.documentElement.classList.remove('[&_*]:!transition-none')
+    }, 0)
+  }
+
+  function toggleMode() {
+    disableTransitionsTemporarily()
+
+    let darkModeMediaQuery = window.matchMedia('(prefers-color-scheme: dark)')
+    let isSystemDarkMode = darkModeMediaQuery.matches
+    let isDarkMode = document.documentElement.classList.toggle('dark')
+
+    if (isDarkMode === isSystemDarkMode) {
+      delete window.localStorage.isDarkMode
+    } else {
+      window.localStorage.isDarkMode = isDarkMode
+    }
+  }
+
+  return (
+    <button
+      type="button"
+      aria-label="Toggle dark mode"
+      className="group rounded-full bg-white/90 px-3 py-2 shadow-lg shadow-zinc-800/5 ring-1 ring-zinc-900/5 backdrop-blur transition dark:bg-zinc-800/90 dark:ring-white/10 dark:hover:ring-white/20"
+      onClick={toggleMode}
+    >
+      <SunIcon className="h-6 w-6 fill-zinc-100 stroke-zinc-500 transition group-hover:fill-zinc-200 group-hover:stroke-zinc-700 dark:hidden [@media(prefers-color-scheme:dark)]:fill-teal-50 [@media(prefers-color-scheme:dark)]:stroke-teal-500 [@media(prefers-color-scheme:dark)]:group-hover:fill-teal-50 [@media(prefers-color-scheme:dark)]:group-hover:stroke-teal-600" />
+      <MoonIcon className="hidden h-6 w-6 fill-zinc-700 stroke-zinc-500 transition dark:block [@media(prefers-color-scheme:dark)]:group-hover:stroke-zinc-400 [@media_not_(prefers-color-scheme:dark)]:fill-teal-400/10 [@media_not_(prefers-color-scheme:dark)]:stroke-teal-500" />
+    </button>
+  )
+}
+
+export function Header() {
+  let isHomePage = useRouter().pathname === '/'
+
+  let headerRef = useRef()
+
+  return (
+    <>
+      <header
+        className="pointer-events-none relative z-50 flex flex-col"
+        style={{
+          height: 'var(--header-height)',
+          marginBottom: 'var(--header-mb)',
+        }}
+      >
+        <div
+          ref={headerRef}
+          className="top-0 z-10 h-16 pt-6"
+          style={{ position: 'var(--header-position)' }}
+        >
+          <Container
+            className="top-[var(--header-top,theme(spacing.6))] w-full"
+            style={{ position: 'var(--header-inner-position)' }}
+          >
+            <div className="relative flex gap-4">
+              <div className="flex justify-end md:flex-1">
+                <div className="pointer-events-auto">
+                  <ModeToggle />
+                </div>
+              </div>
+            </div>
+          </Container>
+        </div>
+      </header>
+      {isHomePage && <div style={{ height: 'var(--content-offset)' }} />}
+    </>
+  )
+}

examples/alan-turing-portal/content/abusive-language-detection-on-arabic-social-media-al-jazeera.md

@@ -0,0 +1,14 @@
+---
+title: "Abusive Language Detection on Arabic Social Media (Al Jazeera)"
+link-to-publication: https://www.aclweb.org/anthology/W17-3008
+link-to-data: http://alt.qcri.org/~hmubarak/offensive/AJCommentsClassification-CF.xlsx
+task-description: Ternary (Obscene, Offensive but not obscene, Clean)
+details-of-task: Incivility
+size-of-dataset:  32000
+percentage-abusive: 0.81
+language: Arabic
+level-of-annotation: ["Posts"]
+platform: ["AlJazeera"]
+medium: ["Text"]
+reference: "Mubarak, H., Darwish, K. and Magdy, W., 2017. Abusive Language Detection on Arabic Social Media. In: Proceedings of the First Workshop on Abusive Language Online. Vancouver, Canada: Association for Computational Linguistics, pp.52-56."
+---

examples/alan-turing-portal/content/detecting-abusive-albanian.md

@@ -0,0 +1,14 @@
+---
+title: Detecting Abusive Albanian
+link-to-publication: https://arxiv.org/abs/2107.13592
+link-to-data: https://doi.org/10.6084/m9.figshare.19333298.v1
+task-description: Hierarchical (offensive/not; untargeted/targeted; person/group/other)
+details-of-task: Detect and categorise abusive language in social media data
+size-of-dataset: 11874
+percentage-abusive: 13.2
+language: Albanian
+level-of-annotation: ["Posts"]
+platform: ["Instagram", "Youtube"]
+medium: ["Text"]
+reference: "Nurce, E., Keci, J., Derczynski, L., 2021. Detecting Abusive Albanian. arXiv:2107.13592"
+---

examples/alan-turing-portal/content/hate-speech-detection-in-the-bengali-language-a-dataset-and-its-baseline-evaluation.md

@@ -0,0 +1,14 @@
+---
+title: "Hate Speech Detection in the Bengali language: A Dataset and its Baseline Evaluation"
+link-to-publication: https://arxiv.org/pdf/2012.09686.pdf
+link-to-data: https://www.kaggle.com/naurosromim/bengali-hate-speech-dataset
+task-description: Binary (hateful, not)
+details-of-task: "Several categories: sports, entertainment, crime, religion, politics, celebrity and meme"
+size-of-dataset: 30000
+percentage-abusive: 0.33
+language: Bengali
+level-of-annotation: ["Posts"]
+platform: ["Youtube", "Facebook"]
+medium: ["Text"]
+reference: "Romim, N., Ahmed, M., Talukder, H., & Islam, M. S. (2021). Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computational Intelligence (pp. 457-468). Springer, Singapore."
+---

examples/alan-turing-portal/content/index.md

@@ -0,0 +1,9 @@
+This page catalogues datasets annotated for hate speech, online abuse, and offensive language. They may be useful for e.g. training a natural language processing system to detect this language.
+
+The list is maintained by Leon Derczynski, Bertie Vidgen, Hannah Rose Kirk, Pica Johansson, Yi-Ling Chung, Mads Guldborg Kjeldgaard Kongsbak, Laila Sprejer, and Philine Zeinert.
+
+We provide a list of datasets and keywords. If you would like to contribute to our catalogue or add your dataset, please see the instructions for contributing.
+
+If you use these resources, please cite (and read!) our paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out. And if you would like to find other resources for researching online hate, visit The Alan Turing Institute’s Online Hate Research Hub or read The Alan Turing Institute’s Reading List on Online Hate and Abuse Research.
+
+If you’re looking for a good paper on online hate training datasets (beyond our paper, of course!) then have a look at ‘Resources and benchmark corpora for hate speech detection: a systematic review’ by Poletto et al. in Language Resources and Evaluation.

examples/alan-turing-portal/content/let-mi-an-arabic-levantine-twitter-dataset-for-mysogynistic-language.md

@@ -0,0 +1,14 @@
+---
+title: "Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language"
+link-to-publication: https://arxiv.org/abs/2103.10195
+link-to-data: https://drive.google.com/file/d/1mM2vnjsy7QfUmdVUpKqHRJjZyQobhTrW/view
+task-description: Binary (misogyny/none) and Multi-class (none, discredit, derailing, dominance, stereotyping & objectification, threat of violence, sexual harassment, damning)
+details-of-task: Introducing an Arabic Levantine Twitter dataset for Misogynistic language
+size-of-dataset: 6603
+percentage-abusive: 48.76
+language: Arabic
+level-of-annotation: ["Posts"]
+platform: ["Twitter"]
+medium: ["Text", "Images"]
+reference: "Hala Mulki and Bilal Ghanem. 2021. Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 154–163, Kyiv, Ukraine (Virtual). Association for Computational Linguistics"
+---

examples/alan-turing-portal/jsconfig.json

@@ -0,0 +1,11 @@
+{
+  "compilerOptions": {
+    "baseUrl": ".",
+    "paths": {
+      "@/components/*": ["components/*"],
+      "@/pages/*": ["pages/*"],
+      "@/lib/*": ["lib/*"]
+    }
+  }
+}
+

examples/alan-turing-portal/lib/formatDate.js

@@ -0,0 +1,8 @@
+export function formatDate(dateString) {
+  return new Date(`${dateString}T00:00:00Z`).toLocaleDateString('en-US', {
+    day: 'numeric',
+    month: 'long',
+    year: 'numeric',
+    timeZone: 'UTC',
+  })
+}

examples/alan-turing-portal/lib/mddb.ts

@@ -0,0 +1,14 @@
+import { MarkdownDB } from "@flowershow/markdowndb";
+
+const dbPath = "markdown.db";
+
+const client = new MarkdownDB({
+  client: "sqlite3",
+  connection: {
+    filename: dbPath,
+  },
+});
+
+const clientPromise = client.init();
+
+export default clientPromise;

examples/alan-turing-portal/next.config.js

@@ -0,0 +1,3 @@
+module.exports = {
+  swcMinify: true,
+};

examples/alan-turing-portal/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "tailwindui-template",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint",
+    "prebuild": "npm run mddb",
+    "mddb": "mddb ./content"
+  },
+  "browserslist": "defaults, not ie <= 11",
+  "dependencies": {
+    "@flowershow/markdowndb": "^0.1.1",
+    "@headlessui/react": "^1.7.13",
+    "@mapbox/rehype-prism": "^0.8.0",
+    "@mdx-js/loader": "^2.1.5",
+    "@mdx-js/react": "^2.1.5",
+    "@next/mdx": "^13.0.2",
+    "@tailwindcss/forms": "^0.5.3",
+    "@tailwindcss/typography": "^0.5.4",
+    "autoprefixer": "^10.4.12",
+    "clsx": "^1.2.1",
+    "fast-glob": "^3.2.11",
+    "feed": "^4.2.2",
+    "flexsearch": "^0.7.31",
+    "focus-visible": "^5.2.0",
+    "next": "13.3.0",
+    "next-router-mock": "^0.9.3",
+    "next-superjson-plugin": "^0.5.7",
+    "postcss-focus-visible": "^6.0.4",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-hook-form": "^7.43.9",
+    "react-markdown": "^8.0.7",
+    "remark-gfm": "^3.0.1",
+    "superjson": "^1.12.3",
+    "tailwindcss": "^3.3.0"
+  },
+  "devDependencies": {
+    "eslint": "8.26.0",
+    "eslint-config-next": "13.0.2",
+    "prettier": "^2.8.7",
+    "prettier-plugin-tailwindcss": "^0.2.6"
+  }
+}

examples/alan-turing-portal/pages/_app.jsx

@@ -0,0 +1,38 @@
+import { useEffect, useRef } from 'react'
+
+import { Footer } from '../components/Footer'
+import { Header } from '../components/Header'
+
+import '../styles/tailwind.css'
+import 'focus-visible'
+
+function usePrevious(value) {
+  let ref = useRef()
+
+  useEffect(() => {
+    ref.current = value
+  }, [value])
+
+  return ref.current
+}
+
+export default function App({ Component, pageProps, router }) {
+  let previousPathname = usePrevious(router.pathname)
+
+  return (
+    <>
+      <div className="fixed inset-0 flex justify-center sm:px-8">
+        <div className="flex w-full max-w-7xl lg:px-8">
+          <div className="w-full bg-white ring-1 ring-zinc-100 dark:bg-zinc-900 dark:ring-zinc-300/20" />
+        </div>
+      </div>
+      <div className="relative">
+        <Header />
+        <main>
+          <Component previousPathname={previousPathname} {...pageProps} />
+        </main>
+        <Footer />
+      </div>
+    </>
+  )
+}

examples/alan-turing-portal/pages/_document.jsx

@@ -0,0 +1,60 @@
+import { Head, Html, Main, NextScript } from 'next/document'
+
+const modeScript = `
+  let darkModeMediaQuery = window.matchMedia('(prefers-color-scheme: dark)')
+
+  updateMode()
+  darkModeMediaQuery.addEventListener('change', updateModeWithoutTransitions)
+  window.addEventListener('storage', updateModeWithoutTransitions)
+
+  function updateMode() {
+    let isSystemDarkMode = darkModeMediaQuery.matches
+    let isDarkMode = window.localStorage.isDarkMode === 'true' || (!('isDarkMode' in window.localStorage) && isSystemDarkMode)
+
+    if (isDarkMode) {
+      document.documentElement.classList.add('dark')
+    } else {
+      document.documentElement.classList.remove('dark')
+    }
+
+    if (isDarkMode === isSystemDarkMode) {
+      delete window.localStorage.isDarkMode
+    }
+  }
+
+  function disableTransitionsTemporarily() {
+    document.documentElement.classList.add('[&_*]:!transition-none')
+    window.setTimeout(() => {
+      document.documentElement.classList.remove('[&_*]:!transition-none')
+    }, 0)
+  }
+
+  function updateModeWithoutTransitions() {
+    disableTransitionsTemporarily()
+    updateMode()
+  }
+`
+
+export default function Document() {
+  return (
+    <Html className="h-full antialiased" lang="en">
+      <Head>
+        <script dangerouslySetInnerHTML={{ __html: modeScript }} />
+        <link
+          rel="alternate"
+          type="application/rss+xml"
+          href={`${process.env.NEXT_PUBLIC_SITE_URL}/rss/feed.xml`}
+        />
+        <link
+          rel="alternate"
+          type="application/feed+json"
+          href={`${process.env.NEXT_PUBLIC_SITE_URL}/rss/feed.json`}
+        />
+      </Head>
+      <body className="flex h-full flex-col bg-zinc-50 dark:bg-black">
+        <Main />
+        <NextScript />
+      </body>
+    </Html>
+  )
+}

examples/alan-turing-portal/pages/index.jsx

@@ -0,0 +1,182 @@
+import Head from 'next/head'
+import fs from 'fs'
+
+import { Card } from '../components/Card'
+import { Container } from '../components/Container'
+import clientPromise from '@/lib/mddb'
+import ReactMarkdown from 'react-markdown'
+import { Index } from 'flexsearch'
+import { useForm } from 'react-hook-form'
+
+function DatasetCard({ dataset }) {
+  return (
+    <Card as="article">
+      <Card.Title>{dataset.title}</Card.Title>
+      <Card.Description>
+        <span className="font-semibold">Link to publication: </span>{' '}
+        <a
+          className="underline transition hover:text-teal-400 dark:hover:text-teal-900 text-ellipsis"
+          href={dataset['link-to-publication']}
+        >
+          {dataset['link-to-publication']}
+        </a>
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Link to data: </span>
+        <a
+          className="underline transition hover:text-teal-600 dark:hover:text-teal-900 text-ellipsis"
+          href={dataset['link-to-data']}
+        >
+          {dataset['link-to-data']}
+        </a>
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Task Description: </span>
+        {dataset['task-description']}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Details of Task: </span>{' '}
+        {dataset['details-of-task']}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Size of Dataset: </span>{' '}
+        {dataset['size-of-dataset']}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Percentage Abusive: </span>
+        {dataset['percentage-abusive']}%
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Language: </span>
+        {dataset['language']}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Level of Annotation: </span>
+        {dataset['level-of-annotation'].join(', ')}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Platform: </span>
+        {dataset['platform'].join(', ')}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Medium: </span>
+        {dataset['medium'].join(', ')}
+      </Card.Description>
+      <Card.Description>
+        <span className="font-semibold">Reference: </span>
+        {dataset['reference']}
+      </Card.Description>
+    </Card>
+  )
+}
+export default function Home({ datasets, indexText, availableLanguages, availablePlatforms }) {
+  const index = new Index()
+  datasets.forEach((dataset) => index.add(dataset.id, `${dataset.title} ${dataset['task-description']} ${dataset['details-of-task']} ${dataset['reference']}`))
+  const { register, watch } = useForm({ defaultValues: {
+    searchTerm: '',
+    lang: '',
+    platform: ''
+  }})
+  return (
+    <>
+      <Head>
+        <title>Hate Speech Dataset Catalogue</title>
+        <meta
+          name="description"
+          content="Catalog of abusive language data (PLoS 2020)"
+        />
+      </Head>
+      <Container className="mt-9">
+        <div className="max-w-2xl">
+          <h1 className="text-4xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
+            Hate Speech Dataset Catalogue
+          </h1>
+          <article className="mt-6 flex flex-col gap-y-2 text-base text-zinc-600 dark:text-zinc-400">
+            <ReactMarkdown>{indexText}</ReactMarkdown>
+          </article>
+        </div>
+      </Container>
+      <Container className="mt-24 md:mt-28">
+        <div className="mx-auto grid max-w-xl grid-cols-1 gap-y-8 lg:max-w-none">
+          <form className="rounded-2xl border border-zinc-100 px-4 py-6 sm:p-6 dark:border-zinc-700/40">
+            <p className="mt-2 text-lg font-semibold text-zinc-600 dark:text-zinc-100">
+              Search for datasets
+            </p>
+            <div className="mt-6 flex flex-col sm:flex-row gap-3">
+              <input
+                placeholder="Search here"
+                aria-label="Hate speech on Twitter"
+                required
+                {...register('searchTerm')}
+                className="min-w-0 flex-auto appearance-none rounded-md border border-zinc-900/10 bg-white px-3 py-[calc(theme(spacing.2)-1px)] shadow-md shadow-zinc-800/5 placeholder:text-zinc-600 focus:border-teal-500 focus:outline-none focus:ring-4 focus:ring-teal-500/10 dark:border-zinc-700 dark:bg-zinc-700/[0.15] dark:text-zinc-200 dark:placeholder:text-zinc-200 dark:focus:border-teal-400 dark:focus:ring-teal-400/10 sm:text-sm"
+              />
+              <select
+                placeholder="Language"
+                defaultValue=""
+                className="min-w-0 flex-auto text-zinc-600 appearance-none rounded-md border border-zinc-900/10 bg-white px-3 py-[calc(theme(spacing.2)-1px)] shadow-md shadow-zinc-800/5 placeholder:text-zinc-400 focus:border-teal-500 focus:outline-none focus:ring-4 focus:ring-teal-500/10 dark:border-zinc-700 dark:bg-zinc-700/[0.15] dark:text-zinc-200 dark:placeholder:text-zinc-500 dark:focus:border-teal-400 dark:focus:ring-teal-400/10 sm:text-sm"
+                {...register('lang')}
+              >
+                <option value="" disabled hidden>Filter by language</option>
+                {availableLanguages.map((lang) => (
+                  <option key={lang} className='dark:bg-white dark:text-black' value={lang}>{lang}</option>
+                ))}
+              </select>
+              <select
+                placeholder="Platform"
+                defaultValue=""
+                className="min-w-0 flex-auto text-zinc-600 appearance-none rounded-md border border-zinc-900/10 bg-white px-3 py-[calc(theme(spacing.2)-1px)] shadow-md shadow-zinc-800/5 placeholder:text-zinc-400 focus:border-teal-500 focus:outline-none focus:ring-4 focus:ring-teal-500/10 dark:border-zinc-700 dark:bg-zinc-700/[0.15] dark:text-zinc-200 dark:placeholder:text-zinc-500 dark:focus:border-teal-400 dark:focus:ring-teal-400/10 sm:text-sm"
+                {...register('platform')}
+              >
+                <option value="" disabled hidden>Filter by platform</option>
+                {availablePlatforms.map((platform) => (
+                  <option key={platform} className='dark:bg-white dark:text-black' value={platform}>{platform}</option>
+                ))}
+              </select>
+            </div>
+          </form>
+          <div className="flex flex-col gap-16">
+            {datasets
+              .filter((dataset) =>
+                watch().searchTerm && watch().searchTerm !== ''
+                  ? index.search(watch().searchTerm).includes(dataset.id)
+                  : true
+              )
+              .filter((dataset) =>
+                watch().lang && watch().lang !== ''
+                  ? dataset.language === watch().lang
+                  : true
+              )
+              .filter((dataset) =>
+                watch().platform && watch().platform !== ''
+                  ? dataset.platform.includes(watch().platform)
+                  : true
+              )
+              .map((dataset) => (
+                <DatasetCard key={dataset.title} dataset={dataset} />
+              ))}
+          </div>
+        </div>
+      </Container>
+    </>
+  )
+}
+
+export async function getStaticProps() {
+  const mddb = await clientPromise
+  const allPages = await mddb.getFiles({ extensions: ['md', 'mdx'] })
+  const datasets = allPages
+    .filter((page) => page.url_path !== '/')
+    .map((page) => ({ ...page.metadata, id: page._id }))
+  const index = allPages.filter((page) => page.url_path === '/')[0]
+  const source = fs.readFileSync(index.file_path, { encoding: 'utf-8' })
+  const availableLanguages = [... new Set(datasets.map((dataset) => dataset.language))]
+  const availablePlatforms = [... new Set(datasets.map((dataset) => dataset.platform).flat())]
+  return {
+    props: {
+      indexText: source,
+      datasets,
+      availableLanguages,
+      availablePlatforms,
+    },
+  }
+}

examples/alan-turing-portal/postcss.config.js

@@ -0,0 +1,9 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    'postcss-focus-visible': {
+      replaceWith: '[data-focus-visible-added]',
+    },
+    autoprefixer: {},
+  },
+}

examples/alan-turing-portal/prettier.config.js

@@ -0,0 +1,5 @@
+module.exports = {
+  singleQuote: true,
+  semi: false,
+  plugins: [require('prettier-plugin-tailwindcss')],
+}

examples/alan-turing-portal/styles/prism.css

@@ -0,0 +1,47 @@
+pre[class*='language-'] {
+  color: theme('colors.zinc.100');
+}
+
+.token.tag,
+.token.class-name,
+.token.selector,
+.token.selector .class,
+.token.selector.class,
+.token.function {
+  color: theme('colors.pink.400');
+}
+
+.token.attr-name,
+.token.keyword,
+.token.rule,
+.token.pseudo-class,
+.token.important {
+  color: theme('colors.zinc.300');
+}
+
+.token.module {
+  color: theme('colors.pink.400');
+}
+
+.token.attr-value,
+.token.class,
+.token.string,
+.token.property {
+  color: theme('colors.teal.300');
+}
+
+.token.punctuation,
+.token.attr-equals {
+  color: theme('colors.zinc.500');
+}
+
+.token.unit,
+.language-css .token.function {
+  color: theme('colors.sky.200');
+}
+
+.token.comment,
+.token.operator,
+.token.combinator {
+  color: theme('colors.zinc.400');
+}

examples/alan-turing-portal/styles/tailwind.css

@@ -0,0 +1,4 @@
+@import 'tailwindcss/base';
+@import 'tailwindcss/components';
+@import './prism.css';
+@import 'tailwindcss/utilities';

examples/alan-turing-portal/tailwind.config.js

@@ -0,0 +1,307 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    "./app/**/*.{js,ts,jsx,tsx,mdx}",
+    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
+    "./components/**/*.{js,ts,jsx,tsx,mdx}",
+  ],
+  darkMode: 'class',
+  plugins: [require('@tailwindcss/typography'), require('@tailwindcss/forms')],
+  theme: {
+    fontSize: {
+      xs: ['0.8125rem', { lineHeight: '1.5rem' }],
+      sm: ['0.875rem', { lineHeight: '1.5rem' }],
+      base: ['1rem', { lineHeight: '1.75rem' }],
+      lg: ['1.125rem', { lineHeight: '1.75rem' }],
+      xl: ['1.25rem', { lineHeight: '2rem' }],
+      '2xl': ['1.5rem', { lineHeight: '2rem' }],
+      '3xl': ['1.875rem', { lineHeight: '2.25rem' }],
+      '4xl': ['2rem', { lineHeight: '2.5rem' }],
+      '5xl': ['3rem', { lineHeight: '3.5rem' }],
+      '6xl': ['3.75rem', { lineHeight: '1' }],
+      '7xl': ['4.5rem', { lineHeight: '1' }],
+      '8xl': ['6rem', { lineHeight: '1' }],
+      '9xl': ['8rem', { lineHeight: '1' }],
+    },
+    typography: (theme) => ({
+      invert: {
+        css: {
+          '--tw-prose-body': 'var(--tw-prose-invert-body)',
+          '--tw-prose-headings': 'var(--tw-prose-invert-headings)',
+          '--tw-prose-links': 'var(--tw-prose-invert-links)',
+          '--tw-prose-links-hover': 'var(--tw-prose-invert-links-hover)',
+          '--tw-prose-underline': 'var(--tw-prose-invert-underline)',
+          '--tw-prose-underline-hover':
+            'var(--tw-prose-invert-underline-hover)',
+          '--tw-prose-bold': 'var(--tw-prose-invert-bold)',
+          '--tw-prose-counters': 'var(--tw-prose-invert-counters)',
+          '--tw-prose-bullets': 'var(--tw-prose-invert-bullets)',
+          '--tw-prose-hr': 'var(--tw-prose-invert-hr)',
+          '--tw-prose-quote-borders': 'var(--tw-prose-invert-quote-borders)',
+          '--tw-prose-captions': 'var(--tw-prose-invert-captions)',
+          '--tw-prose-code': 'var(--tw-prose-invert-code)',
+          '--tw-prose-code-bg': 'var(--tw-prose-invert-code-bg)',
+          '--tw-prose-pre-code': 'var(--tw-prose-invert-pre-code)',
+          '--tw-prose-pre-bg': 'var(--tw-prose-invert-pre-bg)',
+          '--tw-prose-pre-border': 'var(--tw-prose-invert-pre-border)',
+          '--tw-prose-th-borders': 'var(--tw-prose-invert-th-borders)',
+          '--tw-prose-td-borders': 'var(--tw-prose-invert-td-borders)',
+        },
+      },
+      DEFAULT: {
+        css: {
+          '--tw-prose-body': theme('colors.zinc.600'),
+          '--tw-prose-headings': theme('colors.zinc.900'),
+          '--tw-prose-links': theme('colors.teal.500'),
+          '--tw-prose-links-hover': theme('colors.teal.600'),
+          '--tw-prose-underline': theme('colors.teal.500 / 0.2'),
+          '--tw-prose-underline-hover': theme('colors.teal.500'),
+          '--tw-prose-bold': theme('colors.zinc.900'),
+          '--tw-prose-counters': theme('colors.zinc.900'),
+          '--tw-prose-bullets': theme('colors.zinc.900'),
+          '--tw-prose-hr': theme('colors.zinc.100'),
+          '--tw-prose-quote-borders': theme('colors.zinc.200'),
+          '--tw-prose-captions': theme('colors.zinc.400'),
+          '--tw-prose-code': theme('colors.zinc.700'),
+          '--tw-prose-code-bg': theme('colors.zinc.300 / 0.2'),
+          '--tw-prose-pre-code': theme('colors.zinc.100'),
+          '--tw-prose-pre-bg': theme('colors.zinc.900'),
+          '--tw-prose-pre-border': 'transparent',
+          '--tw-prose-th-borders': theme('colors.zinc.200'),
+          '--tw-prose-td-borders': theme('colors.zinc.100'),
+
+          '--tw-prose-invert-body': theme('colors.zinc.400'),
+          '--tw-prose-invert-headings': theme('colors.zinc.200'),
+          '--tw-prose-invert-links': theme('colors.teal.400'),
+          '--tw-prose-invert-links-hover': theme('colors.teal.400'),
+          '--tw-prose-invert-underline': theme('colors.teal.400 / 0.3'),
+          '--tw-prose-invert-underline-hover': theme('colors.teal.400'),
+          '--tw-prose-invert-bold': theme('colors.zinc.200'),
+          '--tw-prose-invert-counters': theme('colors.zinc.200'),
+          '--tw-prose-invert-bullets': theme('colors.zinc.200'),
+          '--tw-prose-invert-hr': theme('colors.zinc.700 / 0.4'),
+          '--tw-prose-invert-quote-borders': theme('colors.zinc.500'),
+          '--tw-prose-invert-captions': theme('colors.zinc.500'),
+          '--tw-prose-invert-code': theme('colors.zinc.300'),
+          '--tw-prose-invert-code-bg': theme('colors.zinc.200 / 0.05'),
+          '--tw-prose-invert-pre-code': theme('colors.zinc.100'),
+          '--tw-prose-invert-pre-bg': 'rgb(0 0 0 / 0.4)',
+          '--tw-prose-invert-pre-border': theme('colors.zinc.200 / 0.1'),
+          '--tw-prose-invert-th-borders': theme('colors.zinc.700'),
+          '--tw-prose-invert-td-borders': theme('colors.zinc.800'),
+
+          // Base
+          color: 'var(--tw-prose-body)',
+          lineHeight: theme('lineHeight.7'),
+          '> *': {
+            marginTop: theme('spacing.10'),
+            marginBottom: theme('spacing.10'),
+          },
+          p: {
+            marginTop: theme('spacing.7'),
+            marginBottom: theme('spacing.7'),
+          },
+
+          // Headings
+          'h2, h3': {
+            color: 'var(--tw-prose-headings)',
+            fontWeight: theme('fontWeight.semibold'),
+          },
+          h2: {
+            fontSize: theme('fontSize.xl')[0],
+            lineHeight: theme('lineHeight.7'),
+            marginTop: theme('spacing.20'),
+            marginBottom: theme('spacing.4'),
+          },
+          h3: {
+            fontSize: theme('fontSize.base')[0],
+            lineHeight: theme('lineHeight.7'),
+            marginTop: theme('spacing.16'),
+            marginBottom: theme('spacing.4'),
+          },
+          ':is(h2, h3) + *': {
+            marginTop: 0,
+          },
+
+          // Images
+          img: {
+            borderRadius: theme('borderRadius.3xl'),
+          },
+
+          // Inline elements
+          a: {
+            color: 'var(--tw-prose-links)',
+            fontWeight: theme('fontWeight.semibold'),
+            textDecoration: 'underline',
+            textDecorationColor: 'var(--tw-prose-underline)',
+            transitionProperty: 'color, text-decoration-color',
+            transitionDuration: theme('transitionDuration.150'),
+            transitionTimingFunction: theme('transitionTimingFunction.in-out'),
+          },
+          'a:hover': {
+            color: 'var(--tw-prose-links-hover)',
+            textDecorationColor: 'var(--tw-prose-underline-hover)',
+          },
+          strong: {
+            color: 'var(--tw-prose-bold)',
+            fontWeight: theme('fontWeight.semibold'),
+          },
+          code: {
+            display: 'inline-block',
+            color: 'var(--tw-prose-code)',
+            fontSize: theme('fontSize.sm')[0],
+            fontWeight: theme('fontWeight.semibold'),
+            backgroundColor: 'var(--tw-prose-code-bg)',
+            borderRadius: theme('borderRadius.lg'),
+            paddingLeft: theme('spacing.1'),
+            paddingRight: theme('spacing.1'),
+          },
+          'a code': {
+            color: 'inherit',
+          },
+          ':is(h2, h3) code': {
+            fontWeight: theme('fontWeight.bold'),
+          },
+
+          // Quotes
+          blockquote: {
+            paddingLeft: theme('spacing.6'),
+            borderLeftWidth: theme('borderWidth.2'),
+            borderLeftColor: 'var(--tw-prose-quote-borders)',
+            fontStyle: 'italic',
+          },
+
+          // Figures
+          figcaption: {
+            color: 'var(--tw-prose-captions)',
+            fontSize: theme('fontSize.sm')[0],
+            lineHeight: theme('lineHeight.6'),
+            marginTop: theme('spacing.3'),
+          },
+          'figcaption > p': {
+            margin: 0,
+          },
+
+          // Lists
+          ul: {
+            listStyleType: 'disc',
+          },
+          ol: {
+            listStyleType: 'decimal',
+          },
+          'ul, ol': {
+            paddingLeft: theme('spacing.6'),
+          },
+          li: {
+            marginTop: theme('spacing.6'),
+            marginBottom: theme('spacing.6'),
+            paddingLeft: theme('spacing[3.5]'),
+          },
+          'li::marker': {
+            fontSize: theme('fontSize.sm')[0],
+            fontWeight: theme('fontWeight.semibold'),
+          },
+          'ol > li::marker': {
+            color: 'var(--tw-prose-counters)',
+          },
+          'ul > li::marker': {
+            color: 'var(--tw-prose-bullets)',
+          },
+          'li :is(ol, ul)': {
+            marginTop: theme('spacing.4'),
+            marginBottom: theme('spacing.4'),
+          },
+          'li :is(li, p)': {
+            marginTop: theme('spacing.3'),
+            marginBottom: theme('spacing.3'),
+          },
+
+          // Code blocks
+          pre: {
+            color: 'var(--tw-prose-pre-code)',
+            fontSize: theme('fontSize.sm')[0],
+            fontWeight: theme('fontWeight.medium'),
+            backgroundColor: 'var(--tw-prose-pre-bg)',
+            borderRadius: theme('borderRadius.3xl'),
+            padding: theme('spacing.8'),
+            overflowX: 'auto',
+            border: '1px solid',
+            borderColor: 'var(--tw-prose-pre-border)',
+          },
+          'pre code': {
+            display: 'inline',
+            color: 'inherit',
+            fontSize: 'inherit',
+            fontWeight: 'inherit',
+            backgroundColor: 'transparent',
+            borderRadius: 0,
+            padding: 0,
+          },
+
+          // Horizontal rules
+          hr: {
+            marginTop: theme('spacing.20'),
+            marginBottom: theme('spacing.20'),
+            borderTopWidth: '1px',
+            borderColor: 'var(--tw-prose-hr)',
+            '@screen lg': {
+              marginLeft: `calc(${theme('spacing.12')} * -1)`,
+              marginRight: `calc(${theme('spacing.12')} * -1)`,
+            },
+          },
+
+          // Tables
+          table: {
+            width: '100%',
+            tableLayout: 'auto',
+            textAlign: 'left',
+            fontSize: theme('fontSize.sm')[0],
+          },
+          thead: {
+            borderBottomWidth: '1px',
+            borderBottomColor: 'var(--tw-prose-th-borders)',
+          },
+          'thead th': {
+            color: 'var(--tw-prose-headings)',
+            fontWeight: theme('fontWeight.semibold'),
+            verticalAlign: 'bottom',
+            paddingBottom: theme('spacing.2'),
+          },
+          'thead th:not(:first-child)': {
+            paddingLeft: theme('spacing.2'),
+          },
+          'thead th:not(:last-child)': {
+            paddingRight: theme('spacing.2'),
+          },
+          'tbody tr': {
+            borderBottomWidth: '1px',
+            borderBottomColor: 'var(--tw-prose-td-borders)',
+          },
+          'tbody tr:last-child': {
+            borderBottomWidth: 0,
+          },
+          'tbody td': {
+            verticalAlign: 'baseline',
+          },
+          tfoot: {
+            borderTopWidth: '1px',
+            borderTopColor: 'var(--tw-prose-th-borders)',
+          },
+          'tfoot td': {
+            verticalAlign: 'top',
+          },
+          ':is(tbody, tfoot) td': {
+            paddingTop: theme('spacing.2'),
+            paddingBottom: theme('spacing.2'),
+          },
+          ':is(tbody, tfoot) td:not(:first-child)': {
+            paddingLeft: theme('spacing.2'),
+          },
+          ':is(tbody, tfoot) td:not(:last-child)': {
+            paddingRight: theme('spacing.2'),
+          },
+        },
+      },
+    }),
+  },
+}

examples/ckan-example/.eslintrc.json

@@ -1,6 +1,5 @@
 {
   "extends": [
-    "plugin:@nrwl/nx/react-typescript",
     "next",
     "next/core-web-vitals",
     "../../.eslintrc.json"

examples/ckan-example/README.md

@@ -1,17 +1,46 @@
 This is a repo intended to serve as an example of a data catalog that get its data from a CKAN Instance.
 
-- Creating a new file inside o `examples` with `create-next-app` like so:
 ```
-npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/ --example-path examples/ckan-example
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/ckan-example
+cd <app-name>
 ```
-- Inside `<app-name>` go to the `project.json` file and replace all instances of `ckan-example` with `<app-name>`
-- Set the `DMS` env variable to the Url of the CKAN Instance Ex: `export DMS=https://demo.dev.datopian.com`
+
+- This project uses CKAN as a backend, so you need to point the project to the CKAN Url desired, you can do so by setting up the `DMS` env variable in your terminal or adding a `.env` file with the following content:
+
+```
+DMS=<ckan url>
+```
+
 - Run the app using:
+
 ```
-nx serve <app-name>
+npm run dev
 ```
+
 Congratulations, you now have something similar to this running on `http://localhost:4200`
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252297726865408/image.png?width=853&height=461)
 If yo go to any one of those pages by clicking on `More info` you will see something similar to this
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252298074988595/image.png?width=853&height=461)
 
+## Deployment
+
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fckan-example&env=DMS&envDescription=URL%20For%20the%20CKAN%20Backend%20Ex%3A%20https%3A%2F%2Fdemo.dev.datopian.com)
+
+By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own github/gitlab/bitbucket account and automatically deploy everything.
+
+
+
+## Extra commands
+
+You can also build the project for production with
+
+```
+npm run build
+```
+
+And run using the production build like so:
+
+```
+npm run start
+```
+

examples/ckan-example/next.config.js

@@ -1,9 +1,3 @@
-// eslint-disable-next-line @typescript-eslint/no-var-requires
-const { withNx } = require('@nrwl/next/plugins/with-nx');
-
-/**
- * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
- **/
 const nextConfig = {
   publicRuntimeConfig: {
     DMS: process.env.DMS ? process.env.DMS : '',
@@ -18,11 +12,6 @@ const nextConfig = {
       ],
     };
   },
-  nx: {
-    // Set this to true if you would like to use SVGR
-    // See: https://github.com/gregberge/svgr
-    svgr: false,
-  },
 };
 
-module.exports = withNx(nextConfig);
+module.exports = nextConfig;

examples/ckan-example/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "my-app",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@heroicons/react": "^2.0.17",
+    "@types/node": "18.16.0",
+    "@types/react": "18.0.38",
+    "@types/react-dom": "18.0.11",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "next": "13.3.1",
+    "next-seo": "^6.0.0",
+    "octokit": "^2.0.14",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-markdown": "^8.0.7",
+    "remark-gfm": "^3.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
+}

examples/ckan-example/postcss.config.js

@@ -1,15 +1,6 @@
-const { join } = require('path');
-
-// Note: If you use library-specific PostCSS/Tailwind configuration then you should remove the `postcssConfig` build
-// option from your application's configuration (i.e. project.json).
-//
-// See: https://nx.dev/guides/using-tailwind-css-in-react#step-4:-applying-configuration-to-libraries
-
 module.exports = {
   plugins: {
-    tailwindcss: {
-      config: join(__dirname, 'tailwind.config.js'),
-    },
+    tailwindcss: {},
     autoprefixer: {},
   },
-};
+}

examples/ckan-example/tailwind.config.js

@@ -1,17 +1,15 @@
-const { createGlobPatternsForDependencies } = require('@nrwl/react/tailwind');
-const { join } = require('path');
-
 /** @type {import('tailwindcss').Config} */
 module.exports = {
   content: [
-    join(
-      __dirname,
-      '{src,pages,components}/**/*!(*.stories|*.spec).{ts,tsx,html}'
-    ),
-    ...createGlobPatternsForDependencies(__dirname),
+    "./app/**/*.{js,ts,jsx,tsx,mdx}",
+    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
+    "./components/**/*.{js,ts,jsx,tsx,mdx}",
   ],
   theme: {
     extend: {},
   },
-  plugins: [],
-};
+  plugins: [
+    require('@tailwindcss/typography')
+  ],
+}
+

examples/ckan-example/tsconfig.json

@@ -1,23 +1,20 @@
 {
-  "extends": "../../tsconfig.base.json",
   "compilerOptions": {
-    "jsx": "preserve",
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
     "allowJs": true,
-    "esModuleInterop": true,
-    "allowSyntheticDefaultImports": true,
+    "skipLibCheck": true,
     "strict": false,
     "forceConsistentCasingInFileNames": true,
     "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
     "resolveJsonModule": true,
     "isolatedModules": true,
-    "incremental": true,
-    "types": ["jest", "node"]
+    "jsx": "preserve",
+    "incremental": true
   },
-  "include": ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "next-env.d.ts"],
-  "exclude": [
-    "node_modules",
-    "jest.config.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.ts"
-  ]
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
+  "exclude": ["node_modules"]
 }

examples/learn-example/.gitignore

@@ -0,0 +1,36 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

examples/learn-example/README.md

@@ -0,0 +1 @@
+PortalJS Learn Example - https://portaljs.org/docs

examples/learn-example/components/DRD.tsx

@@ -0,0 +1,21 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@flowershow/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('./Table')),
+  mermaid: Mermaid,
+  // Excel: dynamic(() => import('../components/Excel')),
+  // TODO: try and make these dynamic ...
+  Vega: dynamic(() => import('./Vega')),
+  VegaLite: dynamic(() => import('./VegaLite')),
+  LineChart: dynamic(() => import('./LineChart')),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/learn-example/components/DebouncedInput.tsx

@@ -0,0 +1,32 @@
+import { useEffect, useState } from "react";
+
+const DebouncedInput = ({
+  value: initialValue,
+  onChange,
+  debounce = 500,
+  ...props
+}) => {
+  const [value, setValue] = useState(initialValue);
+
+  useEffect(() => {
+    setValue(initialValue);
+  }, [initialValue]);
+
+  useEffect(() => {
+    const timeout = setTimeout(() => {
+      onChange(value);
+    }, debounce);
+
+    return () => clearTimeout(timeout);
+  }, [value]);
+
+  return (
+    <input
+      {...props}
+      value={value}
+      onChange={(e) => setValue(e.target.value)}
+    />
+  );
+};
+
+export default DebouncedInput;

examples/learn-example/components/LineChart.tsx

@@ -0,0 +1,55 @@
+import VegaLite from "./VegaLite";
+
+export default function LineChart({
+  data = [],
+  fullWidth = false,
+  title = "",
+  xAxis = "x",
+  yAxis = "y",
+}) {
+  var tmp = data;
+  if (Array.isArray(data)) {
+    tmp = data.map((r, i) => {
+      return { x: r[0], y: r[1] };
+    });
+  }
+  const vegaData = { table: tmp };
+  const spec = {
+    $schema: "https://vega.github.io/schema/vega-lite/v5.json",
+    title,
+    width: "container",
+    height: 300,
+    mark: {
+      type: "line",
+      color: "black",
+      strokeWidth: 1,
+      tooltip: true,
+    },
+    data: {
+      name: "table",
+    },
+    selection: {
+      grid: {
+        type: "interval",
+        bind: "scales",
+      },
+    },
+    encoding: {
+      x: {
+        field: xAxis,
+        timeUnit: "year",
+        type: "temporal",
+      },
+      y: {
+        field: yAxis,
+        type: "quantitative",
+      },
+    },
+  };
+  if (typeof data === 'string') {
+    spec.data = { "url": data } as any
+    return <VegaLite fullWidth={fullWidth} spec={spec} />;
+  }
+
+  return <VegaLite fullWidth={fullWidth} data={vegaData} spec={spec} />;
+}

examples/learn-example/components/Table.tsx

@@ -0,0 +1,189 @@
+import {
+  createColumnHelper,
+  FilterFn,
+  flexRender,
+  getCoreRowModel,
+  getFilteredRowModel,
+  getPaginationRowModel,
+  getSortedRowModel,
+  useReactTable,
+} from "@tanstack/react-table";
+
+import {
+  ArrowDownIcon,
+  ArrowUpIcon,
+  ChevronDoubleLeftIcon,
+  ChevronDoubleRightIcon,
+  ChevronLeftIcon,
+  ChevronRightIcon,
+} from "@heroicons/react/24/solid";
+
+import React, { useEffect, useMemo, useState } from "react";
+
+import parseCsv from "../lib/parseCsv";
+import DebouncedInput from "./DebouncedInput";
+import loadData from "../lib/loadData";
+
+const Table = ({
+  data: ogData = [],
+  cols: ogCols = [],
+  csv = "",
+  url = "",
+  fullWidth = false,
+}) => {
+  if (csv) {
+    const out = parseCsv(csv);
+    ogData = out.rows;
+    ogCols = out.fields;
+  }
+
+  const [data, setData] = React.useState(ogData);
+  const [cols, setCols] = React.useState(ogCols);
+  const [error, setError] = React.useState(""); //  TODO: add error handling
+
+  const tableCols = useMemo(() => {
+    const columnHelper = createColumnHelper();
+    return cols.map((c) =>
+      columnHelper.accessor(c.key, {
+        header: () => c.name,
+        cell: (info) => info.getValue(),
+      })
+    );
+  }, [data, cols]);
+
+  const [globalFilter, setGlobalFilter] = useState("");
+
+  const table = useReactTable({
+    data,
+    columns: tableCols,
+    getCoreRowModel: getCoreRowModel(),
+    state: {
+      globalFilter,
+    },
+    globalFilterFn: globalFilterFn,
+    onGlobalFilterChange: setGlobalFilter,
+    getFilteredRowModel: getFilteredRowModel(),
+    getPaginationRowModel: getPaginationRowModel(),
+    getSortedRowModel: getSortedRowModel(),
+  });
+
+  useEffect(() => {
+    if (url) {
+      loadData(url).then((data) => {
+        const { rows, fields } = parseCsv(data);
+        setData(rows);
+        setCols(fields);
+      });
+    }
+  }, [url]);
+
+  return (
+    <div className={`${fullWidth ? "w-[90vw] ml-[calc(50%-45vw)]" : "w-full"}`}>
+      <DebouncedInput
+        value={globalFilter ?? ""}
+        onChange={(value) => setGlobalFilter(String(value))}
+        className="p-2 text-sm shadow border border-block"
+        placeholder="Search all columns..."
+      />
+      <table>
+        <thead>
+          {table.getHeaderGroups().map((hg) => (
+            <tr key={hg.id}>
+              {hg.headers.map((h) => (
+                <th key={h.id}>
+                  <div
+                    {...{
+                      className: h.column.getCanSort()
+                        ? "cursor-pointer select-none"
+                        : "",
+                      onClick: h.column.getToggleSortingHandler(),
+                    }}
+                  >
+                    {flexRender(h.column.columnDef.header, h.getContext())}
+                    {{
+                      asc: (
+                        <ArrowUpIcon className="inline-block ml-2 h-4 w-4" />
+                      ),
+                      desc: (
+                        <ArrowDownIcon className="inline-block ml-2 h-4 w-4" />
+                      ),
+                    }[h.column.getIsSorted() as string] ?? (
+                      <div className="inline-block ml-2 h-4 w-4" />
+                    )}
+                  </div>
+                </th>
+              ))}
+            </tr>
+          ))}
+        </thead>
+        <tbody>
+          {table.getRowModel().rows.map((r) => (
+            <tr key={r.id}>
+              {r.getVisibleCells().map((c) => (
+                <td key={c.id}>
+                  {flexRender(c.column.columnDef.cell, c.getContext())}
+                </td>
+              ))}
+            </tr>
+          ))}
+        </tbody>
+      </table>
+      <div className="flex gap-2 items-center justify-center">
+        <button
+          className={`w-6 h-6 ${
+            !table.getCanPreviousPage() ? "opacity-25" : "opacity-100"
+          }`}
+          onClick={() => table.setPageIndex(0)}
+          disabled={!table.getCanPreviousPage()}
+        >
+          <ChevronDoubleLeftIcon />
+        </button>
+        <button
+          className={`w-6 h-6 ${
+            !table.getCanPreviousPage() ? "opacity-25" : "opacity-100"
+          }`}
+          onClick={() => table.previousPage()}
+          disabled={!table.getCanPreviousPage()}
+        >
+          <ChevronLeftIcon />
+        </button>
+        <span className="flex items-center gap-1">
+          <div>Page</div>
+          <strong>
+            {table.getState().pagination.pageIndex + 1} of{" "}
+            {table.getPageCount()}
+          </strong>
+        </span>
+        <button
+          className={`w-6 h-6 ${
+            !table.getCanNextPage() ? "opacity-25" : "opacity-100"
+          }`}
+          onClick={() => table.nextPage()}
+          disabled={!table.getCanNextPage()}
+        >
+          <ChevronRightIcon />
+        </button>
+        <button
+          className={`w-6 h-6 ${
+            !table.getCanNextPage() ? "opacity-25" : "opacity-100"
+          }`}
+          onClick={() => table.setPageIndex(table.getPageCount() - 1)}
+          disabled={!table.getCanNextPage()}
+        >
+          <ChevronDoubleRightIcon />
+        </button>
+      </div>
+    </div>
+  );
+};
+
+const globalFilterFn: FilterFn<any> = (row, columnId, filterValue: string) => {
+  const search = filterValue.toLowerCase();
+
+  let value = row.getValue(columnId) as string;
+  if (typeof value === "number") value = String(value);
+
+  return value?.toLowerCase().includes(search);
+};
+
+export default Table;

examples/learn-example/components/Vega.tsx

@@ -0,0 +1,6 @@
+//  Wrapper for the Vega component
+import { Vega as VegaOg } from "react-vega";
+
+export default function Vega(props) {
+  return <VegaOg {...props} />;
+}

examples/learn-example/components/VegaLite.tsx

@@ -0,0 +1,9 @@
+//  Wrapper for the Vega Lite component
+import { VegaLite as VegaLiteOg } from "react-vega";
+import applyFullWidthDirective from "../lib/applyFullWidthDirective";
+
+export default function VegaLite(props) {
+  const Component = applyFullWidthDirective({ Component: VegaLiteOg });
+
+  return <Component {...props} />;
+}

examples/learn-example/content/index.md

@@ -0,0 +1,7 @@
+# My Dataset
+
+Built with PortalJS 
+
+## Table 
+
+<Table url="data.csv" />

examples/learn-example/lib/applyFullWidthDirective.tsx

@@ -0,0 +1,21 @@
+export default function applyFullWidthDirective({
+  Component,
+  defaultWFull = true,
+}) {
+  return (props) => {
+    const newProps = { ...props };
+
+    let newClassName = newProps.className || "";
+    if (newProps.fullWidth === true) {
+      newClassName += " w-[90vw] ml-[calc(50%-45vw)] max-w-none";
+    } else if (defaultWFull) {
+      //  So that charts and tables will have the
+      //  same width as the text content, but images
+      //  can have its width set using the width prop
+      newClassName += " w-full";
+    }
+    newProps.className = newClassName;
+
+    return <Component {...newProps} />;
+  };
+}

examples/learn-example/lib/loadData.tsx

@@ -0,0 +1,5 @@
+export default async function loadData(url: string) {
+  const response = await fetch(url)
+  const data = await response.text()
+  return data
+}

examples/learn-example/lib/markdown.js

@@ -0,0 +1,105 @@
+import matter from "gray-matter";
+import mdxmermaid from "mdx-mermaid";
+import { h } from "hastscript";
+import remarkCallouts from "@flowershow/remark-callouts";
+import remarkEmbed from "@flowershow/remark-embed";
+import remarkGfm from "remark-gfm";
+import remarkMath from "remark-math";
+import remarkSmartypants from "remark-smartypants";
+import remarkToc from "remark-toc";
+import remarkWikiLink from "@flowershow/remark-wiki-link";
+import rehypeAutolinkHeadings from "rehype-autolink-headings";
+import rehypeKatex from "rehype-katex";
+import rehypeSlug from "rehype-slug";
+import rehypePrismPlus from "rehype-prism-plus";
+
+import { serialize } from "next-mdx-remote/serialize";
+
+/**
+ * Parse a markdown or MDX file to an MDX source form + front matter data
+ *
+ * @source: the contents of a markdown or mdx file
+ * @format: used to indicate to next-mdx-remote which format to use (md or mdx)
+ * @returns: { mdxSource: mdxSource, frontMatter: ...}
+ */
+const parse = async function (source, format) {
+  const { content, data, excerpt } = matter(source, {
+    excerpt: (file, options) => {
+      // Generate an excerpt for the file
+      file.excerpt = file.content.split("\n\n")[0];
+    },
+  });
+
+  const mdxSource = await serialize(
+    { value: content, path: format },
+    {
+      // Optionally pass remark/rehype plugins
+      mdxOptions: {
+        remarkPlugins: [
+          remarkEmbed,
+          remarkGfm,
+          [remarkSmartypants, { quotes: false, dashes: "oldschool" }],
+          remarkMath,
+          remarkCallouts,
+          remarkWikiLink,
+          [
+            remarkToc,
+            {
+              heading: "Table of contents",
+              tight: true,
+            },
+          ],
+          [mdxmermaid, {}],
+        ],
+        rehypePlugins: [
+          rehypeSlug,
+          [
+            rehypeAutolinkHeadings,
+            {
+              properties: { className: 'heading-link' },
+              test(element) {
+                return (
+                  ["h2", "h3", "h4", "h5", "h6"].includes(element.tagName) &&
+                  element.properties?.id !== "table-of-contents" &&
+                  element.properties?.className !== "blockquote-heading"
+                );
+              },
+              content() {
+                return [
+                  h(
+                    "svg",
+                    {
+                      xmlns: "http:www.w3.org/2000/svg",
+                      fill: "#ab2b65",
+                      viewBox: "0 0 20 20",
+                      className: "w-5 h-5",
+                    },
+                    [
+                      h("path", {
+                        fillRule: "evenodd",
+                        clipRule: "evenodd",
+                        d: "M9.493 2.853a.75.75 0 00-1.486-.205L7.545 6H4.198a.75.75 0 000 1.5h3.14l-.69 5H3.302a.75.75 0 000 1.5h3.14l-.435 3.148a.75.75 0 001.486.205L7.955 14h2.986l-.434 3.148a.75.75 0 001.486.205L12.456 14h3.346a.75.75 0 000-1.5h-3.14l.69-5h3.346a.75.75 0 000-1.5h-3.14l.435-3.147a.75.75 0 00-1.486-.205L12.045 6H9.059l.434-3.147zM8.852 7.5l-.69 5h2.986l.69-5H8.852z",
+                      }),
+                    ]
+                  ),
+                ];
+              },
+            },
+          ],
+          [rehypeKatex, { output: "mathml" }],
+          [rehypePrismPlus, { ignoreMissing: true }],
+        ],
+        format,
+      },
+      scope: data,
+    }
+  );
+
+  return {
+    mdxSource: mdxSource,
+    frontMatter: data,
+    excerpt,
+  };
+};
+
+export default parse;

examples/learn-example/lib/parseCsv.ts

@@ -0,0 +1,16 @@
+import papa from "papaparse";
+
+const parseCsv = (csv) => {
+  csv = csv.trim();
+  const rawdata = papa.parse(csv, { header: true });
+  const cols = rawdata.meta.fields.map((r, i) => {
+    return { key: r, name: r };
+  });
+
+  return {
+    rows: rawdata.data,
+    fields: cols,
+  };
+};
+
+export default parseCsv;

examples/learn-example/package.json

@@ -0,0 +1,49 @@
+{
+  "name": "basic-example",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@flowershow/core": "^0.4.10",
+    "@flowershow/remark-callouts": "^1.0.0",
+    "@flowershow/remark-embed": "^1.0.0",
+    "@flowershow/remark-wiki-link": "^1.1.2",
+    "@heroicons/react": "^2.0.17",
+    "@opentelemetry/api": "^1.4.0",
+    "@tanstack/react-table": "^8.8.5",
+    "@types/node": "18.16.0",
+    "@types/react": "18.2.0",
+    "@types/react-dom": "18.2.0",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "gray-matter": "^4.0.3",
+    "hastscript": "^7.2.0",
+    "mdx-mermaid": "2.0.0-rc7",
+    "next": "13.2.1",
+    "next-mdx-remote": "^4.4.1",
+    "papaparse": "^5.4.1",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-vega": "^7.6.0",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
+    "remark-gfm": "^3.0.1",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
+}

examples/learn-example/pages/[[...path]].tsx

@@ -0,0 +1,46 @@
+import { promises as fs } from 'fs';
+import path from 'path';
+import parse from '../lib/markdown';
+import DRD from '../components/DRD';
+
+export const getServerSideProps = async (context) => {
+  let pathToFile = 'index.md';
+  if (context.params.path) {
+    pathToFile = context.params.path.join('/') + '/index.md';
+  }
+
+  const indexFile = path.join(process.cwd(), '/content/' + pathToFile);
+  const readme = await fs.readFile(indexFile, 'utf8');
+  let { mdxSource, frontMatter } = await parse(readme, '.mdx');
+  return {
+    props: {
+      mdxSource,
+      frontMatter,
+    },
+  };
+};
+
+export default function DatasetPage({ mdxSource, frontMatter }) {
+  return (
+    <div className="prose mx-auto">
+      <header>
+        <div className="mb-6">
+          <>
+            <h1>{frontMatter.title}</h1>
+            {frontMatter.author && (
+              <div className="-mt-6">
+                <p className="opacity-60 pl-1">{frontMatter.author}</p>
+              </div>
+            )}
+            {frontMatter.description && (
+              <p className="description">{frontMatter.description}</p>
+            )}
+          </>
+        </div>
+      </header>
+      <main>
+        <DRD source={mdxSource} />
+      </main>
+    </div>
+  );
+}

examples/learn-example/pages/_app.tsx

@@ -0,0 +1,6 @@
+import '../styles/globals.css'
+import type { AppProps } from 'next/app'
+
+export default function App({ Component, pageProps }: AppProps) {
+  return <Component {...pageProps} />
+}

examples/learn-example/pages/_document.tsx

@@ -0,0 +1,19 @@
+import Document, { Html, Main, Head, NextScript } from 'next/document';
+
+class MyDocument extends Document {
+  render() {
+    return (
+      <Html>
+        <Head>
+          <link rel="icon" href="/favicon.png" />
+        </Head>
+        <body>
+          <Main />
+          <NextScript />
+        </body>
+      </Html>
+    );
+  }
+}
+
+export default MyDocument;

examples/learn-example/postcss.config.js

@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}

examples/learn-example/public/data.csv

@@ -0,0 +1,6 @@
+Year,Rating
+2008,86
+2009,96
+2010,100
+2011,100
+2012,97

examples/learn-example/styles/globals.css

@@ -0,0 +1,105 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+@import "@flowershow/remark-callouts/styles.css";
+
+.w-5 {
+  width: 1.25rem
+}
+
+.h-5 {
+  height: 1.25rem
+}
+
+/* mathjax */
+.math-inline > mjx-container > svg {
+  display: inline;
+  align-items: center;
+}
+
+/* smooth scrolling in modern browsers */
+html {
+  scroll-behavior: smooth !important;
+}
+
+/* tooltip fade-out clip */
+.tooltip-body::after {
+  content: "";
+  position: absolute;
+  right: 0;
+  top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
+  height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
+  width: 10rem;
+  background: linear-gradient(
+    to right,
+    rgba(255, 255, 255, 0),
+    rgba(255, 255, 255, 1) 100%
+  );
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
+  margin-left: -2rem !important;
+  padding-left: 2rem !important;
+  scroll-margin-top: 4.5rem;
+  position: relative;
+}
+
+.heading-link {
+  padding: 1px;
+  position: absolute;
+  left: 0;
+  top: 50%;
+  transform: translateY(-50%);
+  margin: auto 0;
+  border-radius: 5px;
+  background: #1e293b;
+  opacity: 0;
+  transition: opacity 0.2s;
+}
+
+.light .heading-link {
+  /* border: 1px solid #ab2b65; */
+  /* background: none; */
+  background: #e2e8f0;
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
+  opacity: 100;
+}
+
+.heading-link svg {
+  transform: scale(0.75);
+}
+
+@media screen and (max-width: 640px) {
+  .heading-link {
+    visibility: hidden;
+  }
+}
+
+html,
+body {
+  padding: 0;
+  margin: 0;
+  font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen,
+    Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif;
+}
+
+a {
+  color: inherit;
+  text-decoration: none;
+}
+
+* {
+  box-sizing: border-box;
+}
+
+@media (prefers-color-scheme: dark) {
+  html {
+    color-scheme: dark;
+  }
+  body {
+    color: white;
+    background: black;
+  }
+}

examples/learn-example/tailwind.config.js

@@ -0,0 +1,12 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    './app/**/*.{js,ts,jsx,tsx,mdx}',
+    './pages/**/*.{js,ts,jsx,tsx,mdx}',
+    './components/**/*.{js,ts,jsx,tsx,mdx}',
+  ],
+  theme: {
+    extend: {},
+  },
+  plugins: [require('@tailwindcss/typography')],
+};

examples/learn-example/tsconfig.json

@@ -0,0 +1,20 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": false,
+    "forceConsistentCasingInFileNames": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "preserve",
+    "incremental": true
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", "middleware.ts"],
+  "exclude": ["node_modules"]
+}

examples/simple-example/.eslintrc.json

@@ -1,9 +1,7 @@
 {
   "extends": [
-    "plugin:@nrwl/nx/react-typescript",
     "next",
-    "next/core-web-vitals",
-    "../../.eslintrc.json"
+    "next/core-web-vitals"
   ],
   "ignorePatterns": ["!**/*", ".next/**/*"],
   "overrides": [

examples/simple-example/README.md

@@ -1,25 +1,13 @@
 This is a repo intended to serve as a simple example of a data catalog that get its data from a series of github repos, you can init an example just like this one by.
 
-- Cloning the PortalJS repo on your machine
+- Creating a new project with `create-next-app` like so:
 
 ```
-git clone https://github.com/datopian/portaljs.git
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/simple-example
+cd <app-name>
 ```
 
-- Creating a new file inside the `examples` folder with `create-next-app` like so:
-
-```
-npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/ --example-path examples/simple-example
-```
-
-- Inside `<app-name>` go to the `project.json` file and replace all instances of `simple-example` with `<app-name>`
-- Create a `.env` file with the following content
-
-```
-PROJECT_NAME=<app-name>
-```
-
-- This project uses the github api, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to your .env file like so
+- This project uses the github api, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
 
 ```
 GITHUB_PAT=<github token>
@@ -29,14 +17,20 @@ GITHUB_PAT=<github token>
 - Run the app using:
 
 ```
-nx serve <app-name>
+npm run dev
 ```
 
-Congratulations, you now have something similar to this running on `http://localhost:4200`
+Congratulations, you now have something similar to this running on `http://localhost:3000`
 ![](https://i.imgur.com/jAljJ9C.png)
 If yo go to any one of those pages by clicking on `More info` you will see something similar to this
 ![](https://i.imgur.com/AoJd4O0.png)
 
+## Deployment
+
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fsimple-example)
+
+By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own github/gitlab/bitbucket account and automatically deploy everything.
+
 
 ## Structure of `datasets.json`
 
@@ -64,3 +58,18 @@ You can also add
 
 - A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
 - A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
+
+## Extra commands
+
+You can also build the project for production with
+
+```
+npm run build
+```
+
+And run using the production build like so:
+
+```
+npm run start
+```
+

examples/simple-example/jest.config.ts

@@ -1,11 +0,0 @@
-/* eslint-disable */
-export default {
-  displayName: 'simple-example',
-  preset: '../../jest.preset.js',
-  transform: {
-    '^(?!.*\\.(js|jsx|ts|tsx|css|json)$)': '@nrwl/react/plugins/jest',
-    '^.+\\.[tj]sx?$': ['babel-jest', { presets: ['@nrwl/next/babel'] }],
-  },
-  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx'],
-  coverageDirectory: '../../coverage/examples/simple-example',
-};

examples/simple-example/next.config.js

@@ -1,9 +1,3 @@
-// eslint-disable-next-line @typescript-eslint/no-var-requires
-const { withNx } = require('@nrwl/next/plugins/with-nx');
-
-/**
- * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
- **/
 const nextConfig = {
   async rewrites() {
     return {
@@ -17,13 +11,7 @@ const nextConfig = {
   },
   serverRuntimeConfig: {
     github_pat: process.env.GITHUB_PAT ? process.env.GITHUB_PAT : null,
-    project_name: process.env.PROJECT_NAME ? process.env.PROJECT_NAME : 'simple-example'  
-  }, 
-  nx: {
-    // Set this to true if you would like to use SVGR
-    // See: https://github.com/gregberge/svgr
-    svgr: true,
   },
 };
 
-module.exports = withNx(nextConfig);
+module.exports = nextConfig;

examples/simple-example/package.json

@@ -1,7 +1,32 @@
 {
-  "name": "simple-example",
-  "version": "1.0.0",
-  "description": "",
-  "author": "",
-  "license": "ISC"
+  "name": "my-app",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@types/node": "18.16.0",
+    "@types/react": "18.0.38",
+    "@types/react-dom": "18.0.11",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "next": "13.3.1",
+    "next-seo": "^6.0.0",
+    "octokit": "^2.0.14",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-markdown": "^8.0.7",
+    "remark-gfm": "^3.0.1",
+    "typescript": "5.0.4"
+  },
+  "devDependencies": {
+    "@tailwindcss/typography": "^0.5.9",
+    "autoprefixer": "^10.4.14",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1"
+  }
 }

examples/simple-example/pages/@org/[org]/[...path].tsx

@@ -61,10 +61,9 @@ export default function ProjectPage({ project }) {
 
 // Generates `/posts/1` and `/posts/2`
 export async function getStaticPaths() {
-  const project_name = getConfig().serverRuntimeConfig.project_name;
   const jsonDirectory = path.join(
     process.cwd(),
-    `/examples/${project_name}/datasets.json`
+    'datasets.json'
   );
   const repos = await fs.readFile(jsonDirectory, 'utf8');
 
@@ -89,10 +88,9 @@ export async function getStaticPaths() {
 }
 
 export async function getStaticProps({ params }) {
-  const project_name = getConfig().serverRuntimeConfig.project_name;
   const jsonDirectory = path.join(
     process.cwd(),
-    `/examples/${project_name}/datasets.json`
+    'datasets.json'
   );
   const reposFile = await fs.readFile(jsonDirectory, 'utf8');
   const repos: GithubProject[] = JSON.parse(reposFile);

examples/simple-example/pages/index.tsx

@@ -4,10 +4,9 @@ import { getProject } from '../lib/octokit';
 import getConfig from 'next/config';
 
 export async function getStaticProps() {
-  const project_name = getConfig().serverRuntimeConfig.project_name;
   const jsonDirectory = path.join(
     process.cwd(),
-    `/examples/${project_name}/datasets.json`
+    '/datasets.json'
   );
   const repos = await fs.readFile(jsonDirectory, 'utf8');
   const github_pat = getConfig().serverRuntimeConfig.github_pat;
@@ -83,7 +82,7 @@ export function Datasets({ projects }) {
                 </thead>
                 <tbody className="divide-y divide-gray-200">
                   {projects.map((project) => (
-                    <tr>
+                    <tr key={project.id}>
                       <td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
                         {project.repo_config.name
                           ? project.repo_config.name

examples/simple-example/postcss.config.js

@@ -1,15 +1,6 @@
-const { join } = require('path');
-
-// Note: If you use library-specific PostCSS/Tailwind configuration then you should remove the `postcssConfig` build
-// option from your application's configuration (i.e. project.json).
-//
-// See: https://nx.dev/guides/using-tailwind-css-in-react#step-4:-applying-configuration-to-libraries
-
 module.exports = {
   plugins: {
-    tailwindcss: {
-      config: join(__dirname, 'tailwind.config.js'),
-    },
+    tailwindcss: {},
     autoprefixer: {},
   },
-};
+}

examples/simple-example/project.json

@@ -1,69 +0,0 @@
-{
-  "name": "simple-example",
-  "$schema": "../../node_modules/nx/schemas/project-schema.json",
-  "sourceRoot": "examples/simple-example",
-  "projectType": "application",
-  "targets": {
-    "build": {
-      "executor": "@nrwl/next:build",
-      "outputs": ["{options.outputPath}"],
-      "defaultConfiguration": "production",
-      "options": {
-        "root": "examples/simple-example",
-        "outputPath": "dist/examples/simple-example"
-      },
-      "configurations": {
-        "development": {
-          "outputPath": "examples/simple-example"
-        },
-        "production": {}
-      }
-    },
-    "serve": {
-      "executor": "@nrwl/next:server",
-      "defaultConfiguration": "development",
-      "options": {
-        "buildTarget": "simple-example:build",
-        "dev": true
-      },
-      "configurations": {
-        "development": {
-          "buildTarget": "simple-example:build:development",
-          "dev": true
-        },
-        "production": {
-          "buildTarget": "simple-example:build:production",
-          "dev": false
-        }
-      }
-    },
-    "export": {
-      "executor": "@nrwl/next:export",
-      "options": {
-        "buildTarget": "simple-example:build:production"
-      }
-    },
-    "test": {
-      "executor": "@nrwl/jest:jest",
-      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
-      "options": {
-        "jestConfig": "examples/simple-example/jest.config.ts",
-        "passWithNoTests": true
-      },
-      "configurations": {
-        "ci": {
-          "ci": true,
-          "codeCoverage": true
-        }
-      }
-    },
-    "lint": {
-      "executor": "@nrwl/linter:eslint",
-      "outputs": ["{options.outputFile}"],
-      "options": {
-        "lintFilePatterns": ["examples/simple-example/**/*.{ts,tsx,js,jsx}"]
-      }
-    }
-  },
-  "tags": []
-}

examples/simple-example/specs/index.spec.tsx

@@ -1,11 +0,0 @@
-import React from 'react';
-import { render } from '@testing-library/react';
-
-import Index from '../pages/index';
-
-describe('Index', () => {
-  it('should render successfully', () => {
-    const { baseElement } = render(<Index />);
-    expect(baseElement).toBeTruthy();
-  });
-});

examples/simple-example/tailwind.config.js

@@ -1,21 +1,15 @@
-const { createGlobPatternsForDependencies } = require('@nrwl/react/tailwind');
-const { join } = require('path');
-
 /** @type {import('tailwindcss').Config} */
 module.exports = {
   content: [
-    "node_modules/@flowershow/core/dist/*.js",
-    "node_modules/@flowershow/core/*.js",
-    join(
-      __dirname,
-      '{src,pages,components}/**/*!(*.stories|*.spec).{ts,tsx,html}'
-    ),
-    ...createGlobPatternsForDependencies(__dirname),
+    "./app/**/*.{js,ts,jsx,tsx,mdx}",
+    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
+    "./components/**/*.{js,ts,jsx,tsx,mdx}",
   ],
   theme: {
     extend: {},
   },
   plugins: [
-    require('@tailwindcss/typography'),
+    require('@tailwindcss/typography')
   ],
-};
+}
+

examples/simple-example/tsconfig.json

@@ -1,50 +1,20 @@
 {
-  "extends": "../../tsconfig.base.json",
   "compilerOptions": {
-    "jsx": "preserve",
-    "allowJs": true,
-    "esModuleInterop": true,
-    "allowSyntheticDefaultImports": true,
-    "strict": false,
-    "forceConsistentCasingInFileNames": true,
-    "noEmit": true,
-    "resolveJsonModule": true,
-    "isolatedModules": true,
-    "incremental": true,
-    "types": [
-      "jest",
-      "node"
-    ]
-  },
-  "target": "es2020",
-  "lib": [
-    "dom",
-    "dom.iterable",
-    "esnext"
-  ],
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
     "allowJs": true,
     "skipLibCheck": true,
     "strict": false,
     "forceConsistentCasingInFileNames": true,
     "noEmit": true,
-  "incremental": true,
     "esModuleInterop": true,
     "module": "esnext",
     "moduleResolution": "node",
     "resolveJsonModule": true,
     "isolatedModules": true,
     "jsx": "preserve",
-  "include": [
-    "**/*.ts",
-    "**/*.tsx",
-    "**/*.js",
-    "**/*.jsx",
-    "next-env.d.ts"
-  ],
-  "exclude": [
-    "node_modules",
-    "jest.config.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.ts"
-  ]
+    "incremental": true
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
+  "exclude": ["node_modules"]
 }

examples/simple-example/tsconfig.spec.json

@@ -1,24 +0,0 @@
-{
-  "extends": "./tsconfig.json",
-  "compilerOptions": {
-    "outDir": "../../dist/out-tsc",
-    "module": "commonjs",
-    "types": ["jest", "node"],
-    "jsx": "react"
-  },
-  "paths": {
-      "@/*": ["./*"]
-  },
-  "include": [
-    "jest.config.ts",
-    "src/**/*.test.ts",
-    "src/**/*.spec.ts",
-    "src/**/*.test.tsx",
-    "src/**/*.spec.tsx",
-    "src/**/*.test.js",
-    "src/**/*.spec.js",
-    "src/**/*.test.jsx",
-    "src/**/*.spec.jsx",
-    "src/**/*.d.ts"
-  ]
-}

package.json

@@ -4,63 +4,7 @@
   "license": "MIT",
   "scripts": {},
   "private": true,
-  "dependencies": {
-    "@apollo/client": "^3.7.11",
-    "@apollo/react-hooks": "^4.0.0",
-    "@emotion/react": "^11.10.6",
-    "@emotion/styled": "^11.10.6",
-    "@flowershow/core": "^0.4.9",
-    "@flowershow/markdowndb": "^0.1.0",
-    "@flowershow/remark-callouts": "^1.0.0",
-    "@flowershow/remark-embed": "^1.0.0",
-    "@flowershow/remark-wiki-link": "^1.0.1",
-    "@headlessui/react": "^1.7.13",
-    "@heroicons/react": "^2.0.17",
-    "@mui/icons-material": "^5.11.16",
-    "@mui/material": "^5.11.16",
-    "@mui/x-data-grid": "^6.1.0",
-    "@opentelemetry/api": "^1.4.0",
-    "@tailwindcss/typography": "^0.5.9",
-    "@tanstack/react-table": "^8.8.5",
-    "apollo-cache-inmemory": "^1.6.6",
-    "apollo-link": "^1.2.14",
-    "apollo-link-rest": "^0.9.0",
-    "filesize": "^10.0.7",
-    "gray-matter": "^4.0.3",
-    "html-react-parser": "^3.0.15",
-    "markdown-it": "^13.0.1",
-    "next": "^13.2.1",
-    "next-mdx-remote": "^4.4.1",
-    "next-seo": "^6.0.0",
-    "next-translate": "^2.0.5",
-    "nock": "^13.3.0",
-    "octokit": "^2.0.14",
-    "papaparse": "^5.4.1",
-    "plotly.js-basic-dist": "^2.20.0",
-    "prop-types": "^15.8.1",
-    "react": "18.2.0",
-    "react-dom": "18.2.0",
-    "react-markdown": "^8.0.7",
-    "react-next-github-btn": "^1.2.1",
-    "react-plotly.js": "^2.6.0",
-    "react-plotlyjs": "^0.4.4",
-    "react-vega": "^7.6.0",
-    "rehype-autolink-headings": "^6.1.1",
-    "rehype-katex": "^6.0.2",
-    "rehype-prism-plus": "^1.5.1",
-    "rehype-slug": "^5.1.0",
-    "remark-footnotes": "^4.0.1",
-    "remark-gfm": "^3.0.1",
-    "remark-math": "^5.1.1",
-    "remark-slug": "^7.0.1",
-    "remark-smartypants": "^2.0.0",
-    "remark-toc": "^8.0.1",
-    "slugify": "^1.6.6",
-    "timeago.js": "^4.0.2",
-    "tslib": "^2.3.0",
-    "vega": "^5.24.0",
-    "xlsx": "^0.18.5"
-  },
+  "dependencies": {},
   "devDependencies": {
     "@babel/preset-react": "^7.14.5",
     "@nrwl/cypress": "15.9.2",
@@ -84,7 +28,6 @@
     "@types/react-dom": "18.0.11",
     "@typescript-eslint/eslint-plugin": "^5.36.1",
     "@typescript-eslint/parser": "^5.36.1",
-    "autoprefixer": "10.4.13",
     "babel-jest": "^29.4.1",
     "cypress": "^12.2.0",
     "eslint": "~8.15.0",
@@ -98,11 +41,9 @@
     "jest": "^29.4.1",
     "jest-environment-jsdom": "^29.4.1",
     "nx": "15.9.2",
-    "postcss": "8.4.21",
     "prettier": "^2.6.2",
     "react-test-renderer": "18.2.0",
     "swc-loader": "0.1.15",
-    "tailwindcss": "3.2.7",
     "ts-jest": "^29.0.5",
     "ts-node": "10.9.1",
     "typescript": "~4.9.5"

site/components/Gallery.tsx

@@ -19,6 +19,7 @@ const items = [
     href: 'https://opendata.fcsc.gov.ae/',
     image: '/images/showcases/uae.png',
     description: 'Government Open Data Portal',
+    sourceUrl: 'https://github.com/FCSCOpendata/frontend',
   },
   {
     title: 'Brazil Open Data',
@@ -37,12 +38,18 @@ const items = [
     href: 'https://example.portaljs.org/',
     image: '/images/showcases/example-simple-catalog.png',
     description: 'Simple data catalog',
+    sourceUrl:
+      'https://github.com/datopian/portaljs/tree/main/examples/simple-example',
+    docsUrl: '/docs/example-data-catalog',
   },
   {
     title: 'Example: Portal with CKAN',
     href: 'https://ckan-example.portaljs.org/',
     image: '/images/showcases/example-ckan.png',
     description: 'Simple portal with data coming from CKAN',
+    sourceUrl:
+      'https://github.com/datopian/portaljs/tree/main/examples/ckan-example',
+    docsUrl: '/docs/example-ckan',
   },
 ];
 

site/components/GalleryItem.tsx

@@ -1,3 +1,65 @@
+const IconBeaker = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    stroke-width="1.5"
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      d="M9.75 3.104v5.714a2.25 2.25 0 01-.659 1.591L5 14.5M9.75 3.104c-.251.023-.501.05-.75.082m.75-.082a24.301 24.301 0 014.5 0m0 0v5.714c0 .597.237 1.17.659 1.591L19.8 15.3M14.25 3.104c.251.023.501.05.75.082M19.8 15.3l-1.57.393A9.065 9.065 0 0112 15a9.065 9.065 0 00-6.23-.693L5 14.5m14.8.8l1.402 1.402c1.232 1.232.65 3.318-1.067 3.611A48.309 48.309 0 0112 21c-2.773 0-5.491-.235-8.135-.687-1.718-.293-2.3-2.379-1.067-3.61L5 14.5"
+    />
+  </svg>
+);
+
+const IconDocs = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    strokeWidth={1.5}
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      strokeLinecap="round"
+      strokeLinejoin="round"
+      d="M12 6.042A8.967 8.967 0 006 3.75c-1.052 0-2.062.18-3 .512v14.25A8.987 8.987 0 016 18c2.305 0 4.408.867 6 2.292m0-14.25a8.966 8.966 0 016-2.292c1.052 0 2.062.18 3 .512v14.25A8.987 8.987 0 0018 18a8.967 8.967 0 00-6 2.292m0-14.25v14.25"
+    />
+  </svg>
+);
+
+const IconCode = () => (
+  <svg
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 24 24"
+    strokeWidth={1.5}
+    stroke="currentColor"
+    className="w-6 h-6"
+  >
+    <path
+      strokeLinecap="round"
+      strokeLinejoin="round"
+      d="M17.25 6.75L22.5 12l-5.25 5.25m-10.5 0L1.5 12l5.25-5.25m7.5-3l-4.5 16.5"
+    />
+  </svg>
+);
+
+const ActionButton = ({ title, Icon, href, className = '' }) => (
+  <a
+    title={title}
+    target="_blank"
+    href={href}
+    className={`rounded-full p-2 hover:bg-secondary transition-all duration-250 ${className}`}
+  >
+    <Icon />
+  </a>
+);
+
 export default function GalleryItem({ item }) {
   return (
     <a
@@ -16,6 +78,26 @@ export default function GalleryItem({ item }) {
           <div className="text-center text-primary-dark">
             <span className="text-xl font-semibold">{item.title}</span>
             <p className="text-base font-medium">{item.description}</p>
+            <div className="flex justify-center mt-2">
+              <ActionButton Icon={IconBeaker} title="Demo" href={item.href} />
+              {item.docsUrl && (
+                <ActionButton
+                  Icon={IconDocs}
+                  title="Documentation"
+                  href={item.docsUrl}
+                  className="mx-5"
+                />
+              )}
+              {item.sourceUrl && (
+                <ActionButton
+                  Icon={IconCode}
+                  title="Source code"
+                  href={item.sourceUrl}
+                />
+              )}
+
+              {/* Maybe: Blog post */}
+            </div>
           </div>
         </div>
       </div>

site/content/blog/example-ckan.md

@@ -1,27 +0,0 @@
----
-title: "Example: data catalog with data coming from CKAN"
-authors: ['Luccas Mateus']
-date: 2023-04-20
----
-
-PortalJS is an open source project that aims to simplify the creation of web-based data portals, making it easy for users to create and share data-driven applications. 
-
-The ckan-example added to PortalJS is intended to provide users with an easy way to set up a data catalog that can be used to display and share data stores behind a CKAN Backend. With this example, users can quickly set up a web-based portal that allows them to showcase their data and make it accessible to others, all this being done just by adding a simple env variable pointing to a CKAN Deployment.
-
-To get a feel of the project, users can check the [live deployment](https://ckan-example.portaljs.org).
-
-Below are some screenshots:
-
-### Front page
-
-![](https://i.imgur.com/NlTAIAg.png)
-
-### Individual dataset page
-
-![](https://i.imgur.com/RRoIlGf.png)
-
-## Links
-
-- [Documentation](/docs/example-ckan)  
-- [Repo](https://github.com/datopian/portaljs/tree/main/examples/ckan-example)  
-- [Live Demo](https://ckan-example.portaljs.org)  

site/content/blog/example-data-catalog.md

@@ -1,28 +0,0 @@
----
-title: "Example: simple data catalog"
-authors: ['Luccas Mateus']
-date: 2023-04-20
----
-
-PortalJS is an open source project that aims to simplify the creation of web-based data portals, making it easy for users to create and share data-driven applications. 
-
-The simple-example added to PortalJS is intended to provide users with an easy way to set up a data catalog that can be used to display and share data stores stored in GitHub repositories. With this example, users can quickly set up a web-based portal that allows them to showcase their data and make it accessible to others, all this being done thru the configuration of a simple `datasets.json` file.
-
-To get a feel of the project, users can check the [live deployment](https://example.portaljs.org).
-
-Below are some screenshots:
-
-### Front page
-
-![](https://i.imgur.com/jAljJ9C.png)
-
-### Individual dataset page
-
-![](https://i.imgur.com/AoJd4O0.png)
-
-
-## Links
-
-- [Documentation](/docs/example-data-catalog)  
-- [Repo](https://github.com/datopian/portaljs/tree/main/examples/simple-example)  
-- [Live Demo](https://example.portaljs.org)  

site/content/config.js

@@ -50,7 +50,7 @@ const config = {
     },
   },
   github: "https://github.com/datopian/portaljs",
-  discord: "https://discord.gg/An7Bu5x8",
+  discord: "https://discord.gg/EeyfGrGu4U",
   tableOfContents: true,
   // analytics: "xxxxxx",
   // editLinkShow: true,

site/content/docs/dms/authentication.md

@@ -0,0 +1,249 @@
+# Authentication
+
+## Introduction
+
+The core function of authentication is to **Identify** Users of the Portal (in a federated way) so we can base access on their identity.
+
+There are 3 major conceptual components: Identity, Accounts and Sessions which come together in the following stages:
+
+* **Root Identity Determination:** Determine Identity often via Delegation
+* **Sessions:** Persistence of the identity in the web application in a secure way (without new identity determination on each request! I don't want to have to login via third party service every time)
+* **Account (aka profile):** Storing Related Account/Profile Information in our application (not in third party identity) eg. email, name (other preferences)
+  * This will get auto-created usually at first Identification
+  * In limited case this can be seen as a cache of info from Identity system (e.g. your email)
+  * However often richer info that is app specific that is generated (relevant for personalization)
+
+### Root Identity Determination options :key:
+
+The identity determination can be done in multiple ways. In this article we're considering following 3 options that we believe are widely used:
+
+- Password authentication - traditional username and password pair
+- Single Sign-on (SSO) via protocols such as OAuth, SAML, OpenID Connect
+- One-time password (OTP) via email or SMS (aka passwordless connection)
+
+#### Password authentication
+
+Traditional way of authentication of users. When signing up user provides at least username and password pair which is then stored in a database for future authentication processes. Normally, additional information such as email address, full name etc. is also requested when registering.
+
+Examples of password authentication in popular services:
+
+- GitHub - https://github.com/join
+- GitLab - https://gitlab.com/users/sign_up
+- NPM - https://www.npmjs.com/signup
+
+#### Single Sign-on (SSO)
+
+The way of delegating identity determination process to some third-party service. Normally, popular social network services are used, e.g., Google, Facebook, Twitter etc. SSO implementations can be done using OAuth or SAML protocols. In addition, there is OpenID Connect protocol which is an extension of OAuth2.0.
+
+- OAuth
+  - JWT based
+  - JSON based
+  - 'webby'
+- SAML
+  - XML based
+  - SOAP based
+  - 'enterprisey'
+
+List of OAuth providers:
+
+https://en.wikipedia.org/wiki/List_of_OAuth_providers
+
+Examples of SSO in popular projects:
+
+- https://datahub.io/login
+- https://vercel.com/signup
+
+#### One-time password (OTP)
+
+Also known as dynamic password, OTP also solves limitations of traditional password authentication method. Usually, the one time passwords are received via email or SMS.
+
+### Account (aka profile) 
+
+- Storage of user profile information (email, fullname, gravatar etc.)
+- Retrieving user profile information via API
+- Updating profile
+- Deleting profile
+
+### Sessions
+
+- Log out: DePersisting the Session
+- Invalidating all Sessions: e.g. if a security issue
+- Sessions outside of browsers
+
+## Key Job Stories
+
+When a user signs in, I want to know her/his identity so that I can limit access and editing based on who she/he is.
+
+When a user visits the data portal for the first time, I want to provide him/her a way to register easily/quickly so that more people uses the data portal.
+
+When I visit the data portal for the first time, I want to sign up using my existing social network account so that I don't need to remember yet another credentials.
+
+When I'm using the CLI app (or anything else outside browser), I want to be able to login so that I can work from the terminal (e.g., have write access: editing datasets etc.).
+
+[More job stories](#more-job-stories).
+
+## CKAN 2 (CKAN Classic)
+
+### Basic CKAN authentication
+
+In classic system, we have basic CKAN authentication. Below is how registration page looks like:
+
+![CKAN Classic register page](/static/img/docs/dms/ckan-register.png)
+
+Registration flow in CKAN Classic:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: fill in the form and submit
+  ckan->>ckan: check access (if user can create user)
+  ckan->>ckan: parse params
+  ckan->>ckan: check recaptcha
+  ckan->>ckan: call 'user_create' action
+  ckan->>ckan.model: add a new user into db
+  ckan->>ckan: create an activity
+  ckan->>ckan: log the user
+  ckan->>user: redirect to dashboard
+```
+
+We can extend basic CKAN authentication with:
+
+- LDAP
+  - https://extensions.ckan.org/extension/ldap/
+  - https://github.com/NaturalHistoryMuseum/ckanext-ldap
+- OAuth - see below
+- SAML - https://extensions.ckan.org/extension/saml2/
+
+### CKAN Classic as OAuth client
+
+CKAN Classic can also be used as OAuth client:
+
+- https://github.com/conwetlab/ckanext-oauth2 - this is the only one that's maintained.
+- https://github.com/etalab/ckanext-oauth2 - outdated, the one above is based on this.
+- https://github.com/okfn/ckanext-oauth - last commit 9 years ago.
+- https://github.com/ckan/ckanext-oauth2waad - Windows Azure Active Directory specific and outdated.
+
+How it works:
+
+```mermaid
+sequenceDiagram
+
+  user->>ckan: request for login via OAuth provider
+  ckan->>ckan.oauth: raise 401 and call `challenge` function
+  ckan.oauth->>user: redirect the user to the 3rd party log in page
+  user->>3rdparty: perform login
+  3rdparty->>ckan.oauth: redirect to /oauth2/callback with token
+  ckan.oauth->>3rdparty: call `authenticate` with token
+  3rdparty->>ckan.oauth: return user info
+  ckan.oauth->>ckan: if doesn't exist save that info in db or update it
+  ckan.oauth->>ckan.oauth: add cookies
+  ckan.oauth->>user: redirect to dashboard
+```
+
+## CKAN 3 (Next Gen)
+
+We have considered some of popular and/or modern solutions for identity management that we can implement in CKAN 3:
+
+https://docs.google.com/spreadsheets/d/1qXZyzAbA2NtpnoSZRJ2K_EbaWJnvxkrKVzQ_2rD5eQw/edit#gid=0
+
+Shortlist based on scores from the spreadsheet above:
+
+- Auth0
+- AuthN
+- Ory/Kratos
+
+Recommendation:
+
+All projects from the shortlist can be considered for a project. It worth to give a try for each of them and find out what works best for your project's needs. Testing out Auth0 should be straightforward and take less than an hour. AuthN and Ory/Kratos would require to build docker images and to run it locally but overall it should not be time consuming.
+
+### Existing work
+
+In datahub.io we have implemented SSO via Google/Github. Below is sequence diagram showing the auth flow with datopian/auth + frontend express app (similar to CKAN 3 frontend):
+
+```mermaid
+sequenceDiagram
+
+  frontend.login->>auth.authenticate: authenticate(jwt=None,next=/success/...)
+  auth.authenticate->>frontend.login: failed + here are urls for logging on 3rd party including success
+  frontend.login->>user: login form with login urls to 3rd party including next url in state
+  user->>3rdparty: login
+  3rdparty->>auth.oauth_response: success
+  auth.oauth_response->>frontend.success: redirect to next url
+  frontend.success->>auth.authenticate: with valid jwt
+  auth.authenticate->>frontend.success: valid + here is profile
+  frontend.success->>frontend.success: decode jwt, check it, then see localstorage
+  frontend.success->>frontend.dashboard: redirect to dashboard
+```
+
+## CKAN 2 to CKAN 3 (aka Next Gen)
+
+How does this conceptual framework map to an evolution of CKAN 2 to CKAN 3?
+
+```mermaid
+graph TD
+
+subgraph "CKAN Classic"
+  Signup["Classic signup, e.g., self-service or by sysadmin"]
+  Login["Classic login if you're using the classic UI"]
+  OAuth["OAuth2(ORY/Hydra)"]
+end
+
+subgraph "Authentication service (ORY/Kratos)"
+  SSO["Social Sign-On: Github, Google, Facebook"]
+  CC["CKAN Classic"]
+  Admins["Sysadmin users"]
+  Curators["Data curators"]
+  Users["Regular users"]
+end
+
+subgraph "Frontend v3"
+  SignupFront["Signup via Kratos"]
+  LoginFront["Login via Kratos"]
+end
+
+SignupFront --"Regular user"--> SSO
+LoginFront --"Regular user"--> SSO
+
+LoginFront --"Data curator"--> CC
+
+CC --> Admins
+CC --> Curators
+SSO --> Users
+
+CC --"Redirect"--> OAuth
+OAuth --> Login
+```
+
+Sequence diagram of login process:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG5cdEJyb3dzZXItPj5Gcm9udGVuZDogUmVxdWVzdCB0byBgL2F1dGgvbG9naW5gXG4gIEZyb250ZW5kLT4-S3JhdG9zOiBBdXRoIHJlcXVlc3RcbiAgS3JhdG9zLT4-QnJvd3NlcjogUmVkaXJlY3QgdG8gYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWAgcGFyYW1cbiAgQnJvd3Nlci0-PkZyb250ZW5kOiBHZXQgYC9hdXRoL2xvZ2luP3JlcXVlc3Q9e2lkfWBcbiAgRnJvbnRlbmQtPj5LcmF0b3M6IEZldGNoIGRhdGEgZm9yIHJlbmRlcmluZyB0aGUgZm9ybVxuICBLcmF0b3MtPj5Gcm9udGVuZDogTG9naW4gb3B0aW9uc1xuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlbmRlciB0aGUgbG9naW4gZm9ybSB3aXRoIGF2YWlsYWJsZSBvcHRpb25zXG4gIEJyb3dzZXItPj5Gcm9udGVuZDogU3VwcGx5IGZvcm0gZGF0YVxuICBGcm9udGVuZC0-PktyYXRvczogVmFsaWRhdGUgYW5kIGxvZ2luXG4gIEtyYXRvcy0-PkZyb250ZW5kOiBTZXQgc2Vzc2lvblxuICBGcm9udGVuZC0-PkJyb3dzZXI6IFJlZGlyZWN0IHRvIC9kYXNoYm9hcmRcblxuXG5cdFx0XHRcdFx0IiwibWVybWFpZCI6eyJ0aGVtZSI6ImRlZmF1bHQifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)
+
+From ORY/Kratos:
+
+[![](https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gIHBhcnRpY2lwYW50IEIgYXMgQnJvd3NlclxuICBwYXJ0aWNpcGFudCBLIGFzIE9SWSBLcmF0b3NcbiAgcGFydGljaXBhbnQgQSBhcyBZb3VyIEFwcGxpY2F0aW9uXG5cblxuICBCLT4-SzogSW5pdGlhdGUgTG9naW5cbiAgSy0-PkI6IFJlZGlyZWN0cyB0byB5b3VyIEFwcGxpY2F0aW9uJ3MgL2xvZ2luIGVuZHBvaW50XG4gIEItPj5BOiBDYWxscyAvbG9naW5cbiAgQS0tPj5LOiBGZXRjaGVzIGRhdGEgdG8gcmVuZGVyIGZvcm1zIGV0Y1xuICBCLS0-PkE6IEZpbGxzIG91dCBmb3JtcywgY2xpY2tzIGUuZy4gXCJTdWJtaXQgTG9naW5cIlxuICBCLT4-SzogUE9TVHMgZGF0YSB0b1xuICBLLS0-Pks6IFByb2Nlc3NlcyBMb2dpbiBJbmZvXG5cbiAgYWx0IExvZ2luIGRhdGEgdmFsaWRcbiAgICBLLS0-PkI6IFNldHMgc2Vzc2lvbiBjb29raWVcbiAgICBLLT4-QjogUmVkaXJlY3RzIHRvIGUuZy4gRGFzaGJvYXJkXG4gIGVsc2UgTG9naW4gZGF0YSBpbnZhbGlkXG4gICAgSy0tPj5COiBSZWRpcmVjdHMgdG8geW91ciBBcHBsaWNhaXRvbidzIC9sb2dpbiBlbmRwb2ludFxuICAgIEItPj5BOiBDYWxscyAvbG9naW5cbiAgICBBLS0-Pks6IEZldGNoZXMgZGF0YSB0byByZW5kZXIgZm9ybSBmaWVsZHMgYW5kIGVycm9yc1xuICAgIEItLT4-QTogRmlsbHMgb3V0IGZvcm1zIGFnYWluLCBjb3JyZWN0cyBlcnJvcnNcbiAgICBCLT4-SzogUE9TVHMgZGF0YSBhZ2FpbiAtIGFuZCBzbyBvbi4uLlxuICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJuZXV0cmFsIiwic2VxdWVuY2VEaWFncmFtIjp7ImRpYWdyYW1NYXJnaW5YIjoxNSwiZGlhZ3JhbU1hcmdpblkiOjE1LCJib3hUZXh0TWFyZ2luIjowLCJub3RlTWFyZ2luIjoxNSwibWVzc2FnZU1hcmdpbiI6NDUsIm1pcnJvckFjdG9ycyI6dHJ1ZX19fQ)
+
+
+Kratos to Hydra in CKAN Classic:
+
+WIP
+
+Questions
+
+* Does CKAN Classic allow us to store arbitrary account information (are there "extras")
+* How would we avoid having to support identity persistence, delegation etc in both NG frontend and Classic Admin UI?
+  * Can we share cookies (e.g. via using subdomains)
+* How is login, identity determination etc done at least for frontend in DataHub.io
+* Should account UI really be in NG frontend vs Classic Admin UI?
+* how can we handle "invite a user" to my org set up ... (it's basically post processing after sign up ...)
+
+## Appendix
+
+### More job stories
+
+When a user visits the data portal, I want to provide multiple options for him/her to sign up so that I have more users registered and using the data portal.
+
+When a user needs to change his/her profile info, I want to make sure it is possible, so that I have the up-to-date information about users.
+
+When my personal info (email etc.) is changed, I want to edit it in my profile so that I provide up-to-date information about me and I receive messages (eg, notifications) properly.
+
+When I decide to stop using the data portal, I want to be able to delete my account, so that my personal details aren't stored in the service that I don't need anymore.

site/content/docs/dms/blob-storage.md

@@ -0,0 +1,215 @@
+# Blob Storage
+
+## Introduction
+
+DMS and data portals often need to *store* data as well as metadata. As such, they require a system for doing this. This page focuses on Blob Storage aka Bulk or Raw storage (see [storage](/docs/dms/storage) page for an overview of all types of storage).
+
+Blob storage is for storing "blobs" of data, that is a raw stream of bytes like files on a filesystem. For blob storage think local filesystem or cloud storage like S3, GCS, etc.
+
+Blob Storage in a DMS can be provided via:
+
+* Local file system: storing on disk or storage directly connected to the instance
+* Cloud storage like S3, Google Cloud Storage, Azure storage etc
+
+Today, cloud storage would be the default in most cases.
+
+### Features
+
+* Storage: Persistent, cost-efficient storage
+* Download: Fast, reliable download (possibly even with support for edge distribution)
+* Upload: reliable and rapid upload 
+  * Direct upload to (cloud) storage by clients i.e. without going via the DMS. Why? Because cloud storage has many features that it would be costly replicate (e.g. multipart, resumable etc), excellent performance and reliability for upload. It also cuts out the middleman of the DMS backend thereby saving bandwidth, reducing load on the DMS backend and improving performance
+  * Upload UI: having an excellent UI for doing upload. NB: this UI is considered part of the [publish feature](/docs/dms/publish)
+* Cloud: integrate with cloud storage
+* Permissions: restricting access to data stored in blob storage based on the permissions of the DMS. For example, if Joe does not have access to a dataset on the DMS he should not be able to access associated blob data in the storage system
+
+## Flows
+
+### Direct to Cloud Upload
+
+Want: Direct upload to cloud storage ... But you need to authorize that ... So give them a token from your app
+
+A sequence diagram illustrating the process for a direct to cloud upload:
+
+```mermaid
+sequenceDiagram
+
+participant Browser as Client (Browser / Code)
+participant Authz as Authz Server
+participant BitStore as Storage Access Token Service
+participant Storage as Cloud Storage
+
+  Browser->>Authz: Give me a BitStore access token
+  Authz->>Browser: Token
+  Browser->>BitStore: Get a signed upload URL (access token, file metdata)
+  BitStore->>Browser: Signed URL
+  Browser->>Storage: Upload file (signed URL)
+  Storage->>Browser: OK (storage metadata)
+```
+
+Here's a more elaborate version showing storage of metadata into the MetaStore afterwards (and skipping the Authz service):
+
+```mermaid
+sequenceDiagram
+
+  participant browser as Client (Browser / Code)
+  participant vfts as MetaStore
+  participant bitstore as Storage Access Token Service
+  participant storage as Cloud Storage
+  
+  browser->>browser: Select files to upload
+  browser->>browser: calculate file hashes (if doing content addressable)
+  browser->bitstore: get signed URLs(file1.csv URL, file2.csv URL, auth info)
+  bitstore->>browser: signed URLs
+  browser->>storage: upload file1.csv
+  storage->>browser: OK
+  browser->>storage: upload file2.csv
+  storage->>browser: OK
+  browser->>browser: Compose datapackage.json
+  browser->>vfts: create dataset(datapackage.json, file1.csv pointer, file2.csv pointer, jwt token, ...)
+  vfts->>browser: OK
+```
+
+## CKAN 2 (Classic)
+
+Blob Storage is known as the FileStore in CKAN v2 and below. The default is local disk storage.
+
+There is support for cloud storage via a variety of extensions the most prominent of which is `ckanext-cloudstorage`: https://github.com/TkTech/ckanext-cloudstorage
+
+There are a variety of issues:
+
+* Cloud storage is not a first class citizen in CKAN: CKAN defaults to local file storage but cloud storage is the default in the world and has much better scalability, performance as well as integratability with cloud deployment
+* The FileStore interface definition has a poor separation of concerns (for example, blob storage file paths is set in the FileStore component not in core CKAN) which makes it hard / hacky to extend and use for key use cases e.g. versioning.
+* `ckanext-cloudstorage` (the default cloud storage extension) is ok but has many issues e.g.
+  * No direct to cloud upload: it uses CKAN backend as a middleman so all data must go via ckan backend
+  * Implements its own (sometimes unreliable) version of multipart upload (which means additional code which isn't as reliable as cloud storage providers interface)
+  * No access to advanced features such as resumability etc
+
+Generally, we at Datopian have seen a lot of issues around multipart / large file upload stability with clients and are still seeing issues when a lot of large files are uploaded via scripts. Fixing and refactoring code related to storage is very costly, and tends to result in client specific "hacks".
+
+## CKAN v3
+
+An approach to blob storage that leverages cloud blob storage directly (i.e. without having to upload and serve all files via the CKAN web server), unlocking the performance characteristics of the storage backend directly. It is designed with a microservice approach and supports direct to cloud uploads and downloads. The key components are listed in the next section. You can read more about the overall design approach in the [design section below](#Design).
+
+It is backwards compatible with CKAN v2 and has been successfully deployed with CKAN v2.8 and v2.9.
+
+**Status: Production.**
+
+### Components
+
+* [ckanext-blob-storage](https://github.com/datopian/ckanext-blob-storage) (formerly known as ckanext-external-storage)
+  * Hooking CKAN to Giftless replacing resource storage
+  * Depends on giftless-client and ckanext-authz-service
+  * Doesn't implement IUploader - completely overrides upload / download routes for resources
+* [Giftless](https://github.com/datopian/giftless) - Git LFS compatible implementation for storage with some extras on top. This hands out access tokens to store data in cloud storage.
+  * Docs at https://giftless.datopian.com
+  * Backends for Azure, Google Cloud Storage and local
+  * Multipart support (on top of standard LFS protocol)
+  * Accepts JWT tokens for authentication and authorization
+* [ckanext-authz-service](https://github.com/datopian/ckanext-authz-service/) - This extension uses CKAN’s built-in authentication and authorization capabilities to: a) Generate JWT tokens and provide them via CKAN’s Web API to clients and b) Validate JWT tokens.
+  * Allows hooking CKAN's authentication and authorization capabilities to generate signed JWT tokens, to integrate with external systems
+  * Not specific for Giftless, but this is what it was built for
+* [ckanext-asset-storage](https://github.com/datopian/ckanext-asset-storage) - this takes care of storing non-data assets e.g. organization images etc.
+  * CKAN IUploader for assets (not resources!)
+  * Pluggable backends - currently local and Azure
+  * Much cleaner than older implementations (ckanext-cloudstorage etc.)
+
+Clients:
+
+* [giftless-client-py](https://github.com/datopian/giftless-client) - Python client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools
+* [giftless-client-js](https://github.com/datopian/giftless-client-js) - Javascript client for Git LFS and Giftless-specific features
+  * Used by ckanext-blob-storage and other tools for creating uploaders in the UI
+
+## Design
+
+### Purpose
+
+The goal of this project is to create a more **_flexible_** system for storing **_data files_** (AKA “resources”) for **_CKAN_ and _other implementations_** of a data portal so that CKAN can support versioning, large file upload (and great file upload UX), plug easily into cloud and local file storage backends and, in general, is easy to customize both for storage layer and for CKAN client code of that layer
+
+### Features
+
+* Do one thing and do it well: provide an API to store and retrieve files from storage, in a way that is pluggable into a micro-services based application and to existing CKAN (2.8 / 2.9)
+* Does not force, and in fact is not aware of, a specific file naming logic (i.e. resource file names could be based on a user given name, a content hash, a revision ID or any mixture of these - it is up to the using system to decide)
+* Does not force a specific storage backend; Should support Amazon S3, Azure Storage and local file storage in some way initially but in general backend should be pluggable
+* Does not force a specific authentication scheme; Expects a signed JWT token, does not care who signed it and how the user got authenticated
+* Does not force complex authorization scheme; Leave it to external system to do complex authorization if needed;
+  * By default, the system can work in an “admin party” mode where all authenticated users have full access to all files. This will be “good enough” for many DMS implementations including CKAN.
+  * Potentially, allow plugging in a more complex authorization logic that relies on JWT claims to perform granular authorization checks
+
+### For Data Files (i.e. Blobs)
+
+This system is about storing and providing access to blobs, or streams of bytes; It is not about providing access to the data stored within (i.e. it is not meant to replace CKAN’s datastore).
+
+### For CKAN – whilst not necessarily CKAN Specific
+
+While the system’s design should not be CKAN specific in any way, our current client needs require us to provide a CKAN extension that integrates with this system.
+
+CKAN’s current IUploader interface has been identified to be too narrow to provide the functionality required by complex projects (resource versioning, direct cloud uploads and downloads, large file support and multipart support). While some of these needs could be and have been “hacked” through the IUploader interface, the implementations have been over complex and hard to debug.
+
+Our goal should be to provide a CKAN extension that provides the following functionality directly:
+
+* Uploading and downloading resource files directly from the client if supported by the storage backend
+  *   Multipart upload support if supported by storage backend
+  *   Handling of signed URLs for uploads and private downloads
+  *   Client side code for handling multipart uploads
+  *   TBD: If storage backend does not support direct uploads / downloads, fall back to …
+
+In addition, this extension should provide an API for other extensions to do things like:
+
+*   Set the file naming scheme (We need this for ckanext-versions)
+*   Lower level file access, e.g. move and delete files. We may need this in the future to optimize storage and deduplicate files as proposed for ckanext-versions
+
+In addition, this extension must “play nice” with common CKAN features such as the datastore extension and related datapusher / xloader extensions.
+
+### Usable For other DMS implementations
+
+There should be nothing in this system, except for the CKAN extension described above, that is specific to CKAN. That will allow to re-use and re-integrate this system as a micro-service in other DMS implementations such as ckan-ng and others.
+In fact, the core part of this system should be a generic, abstract storage service with a light authorization layer. This could make it useful in a host of situations where storage micro-service is needed.
+
+### High Level Principles
+
+Common Principles
+
+* Uploads and downloads directly from cloud provides to browser
+* Signed uploads / downloads - for private / authorized only data access
+* Support for AWS, Azure and potentially GCP storage
+* Support for local (non cloud) storage, potentially through a system like [https://min.io/](https://min.io/)
+* Multipart / large file upload support (a few GB in size should be supported for Gates)
+* Not opinionated about file naming / paths; Allow users to set file locations under some pre-defined patchs / buckets
+* Client side support - browser widgets / code for uploading and downloading files / multipart uploads directly to different backends
+* Well-documented flow for using from API (not browser)
+* Provided API for deleting and moving files
+* Provided API for accessing storage-level metadata (e.g. file MD5) (do we need this could be useful for processes that do things like deduplicate storage)
+* Provided API for managing storage-level object level settings (e.g. “Content-disposition” / “Content-type” headers, etc.)
+* Authorization based on some kind of portable scheme (JWT)
+
+CKAN integration specific (implemented as a CKAN extension)
+
+* JWT generation based on current CKAN user permissions
+* Client widgets integration (or CKAN specific widgets) in right places in CKAN templates
+* Hook into resource upload / download / deletion controllers in CKAN
+* API to allow other extensions to control storage level object metadata (headers, path)
+* API to allow other extensions to hook into lifecycle events - upload completion, download request, deletion etc.
+
+
+### Components
+
+The Decoupled Storage solution should be split into several parts, with some parts being independent of others:
+
+* [External] Cloud Storage service (or API similar if local file system) e.g. S3, GCS, Azure Storage, Min.io (for local file system)
+* Cloud Storage Access Service
+* [External] Permissions Service for granting general permission tokens that give access to Cloud Storage Access Service
+  * JWT tokens can be generated by any party that has the right signing key. Thus, we can initially do without this if JWT signing is implemented as part of the CKAN extension
+* Browser based Client for Cloud Storage (compatible with #1 and with different cloud vendors)
+* CKAN extension that wraps the two parts above to provide a storage solution for CKAN
+
+### Questions
+
+* What is file structure in cloud ... i.e. What is the file path for uploaded files? Options:
+  * Client chooses a name/path
+  * Content addressable i.e. the name is given by the content? How? Use a hash.]
+    * Beauty of that: standard way to name things. The same thing has the same name (modulo collisions)
+    * Goes with versioning => same file = same name, diff file = diff name
+* And do you enforce that from your app
+  * Request for token needs to include the destination file path

site/content/docs/dms/ckan-client-guide/index.md

@@ -0,0 +1,503 @@
+# CKAN Client Guide
+
+Guide to interacting with [CKAN](/docs/dms/ckan) for power users such as data scientists, data engineers and data wranglers.
+
+This guide is about adding and managing data in CKAN programmatically and it assumes:
+
+* You are familiar with key concepts like metadata, data, etc.
+* You are working programmatically with a programming language such as Python, JavaScript or R (_coming soon_).
+
+## Frictionless Formats
+
+Clients use [Frictionless formats](https://specs.frictionlessdata.io/) by default for describing dataset and resource objects passed to client methods. Internally, we then use the a *CKAN {'<=>'} Frictionless Mapper* (both [in JavaScript]( https://github.com/datopian/frictionless-ckan-mapper-js ) and [in Python](https://github.com/frictionlessdata/frictionless-ckan-mapper)) to convert objects to CKAN formats before calling the API. **Thus, you can use _Frictionless Formats_ by default with the client**.
+
+>[!tip]As CKAN moves to Frictionless to default this will gradually become unnecessary.
+
+## Quick start
+
+Most of this guide has Python programming language in mind, including its [convention regading using _snake case_ for instances and methods names](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles).
+
+If needed, you can adapt the instructions to JavaScript and R (coming soon) by using _camel case_ instead — for example, if in the Python code we have  `client.push_blob(…)`, in JavaScript it would be `client.pushBlob(…)`.
+
+### Prerequisites
+
+Install the client for your language of choice:
+
+* Python: https://github.com/datopian/ckan-client-py#install
+* JavaScript: https://github.com/datopian/ckan-client-js#install
+* R: _coming soon_
+
+### Create a client
+
+#### Python
+
+```python
+from ckanclient import Client
+
+
+api_key = '771a05ad-af90-4a70-beea-cbb050059e14'
+api_url = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+lfs_url = 'http://localhost:9419'
+
+client = Client(api_url, organization, dataset, lfs_url)
+```
+
+#### JavaScript
+
+```javascript
+const { Client } = require('ckanClient')
+
+apiKey = '771a05ad-af90-4a70-beea-cbb050059e14'
+apiUrl = 'http://localhost:5000'
+organization = 'datopian'
+dataset = 'dailyprices'
+
+const client = Client(apiKey, organization, dataset, apiUrl)
+```
+
+### Upload a resource
+
+That is to say, upload a file, implicitly creating a new dataset.
+
+#### Python
+
+```python
+from frictionless import describe
+
+
+resource = describe('my-data.csv')
+client.push_blob(resource)
+```
+
+### Create a new empty Dataset with metadata
+
+#### Python
+
+```python
+client.create('my-data')
+client.push(resource)
+```
+
+### Adding a resource to an existing Dataset
+
+>[!note]Not implemented yet.
+
+
+```python
+client.create('my-data')
+client.push_resource(resource)
+```
+
+### Edit a Dataset's metadata
+
+>[!note]Not implemented yet.
+
+
+```python
+dataset = client.retrieve('sample-dataset')
+client.update_metadata(
+    dataset,
+    metadata: {'maintainer_email': 'sample@datopian.com'}
+)
+```
+
+For details of metadata see the [metadata reference below](#metadata-reference).
+
+## API - Porcelain
+
+### `Client.create`
+
+Expects as a single argument: a _string_, or a _dict_ (in Python), or an _object_ (in JavaScript). This argument is either a valid dataset name or dictionary with metadata for the dataset in Frictionless format.
+
+### `Client.push`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a dataset metadata in Frictionless format.
+
+### `Client.retrieve`
+
+Expects a single argument: a string with a dataset name or uniquer ID. Returns a Frictionless resource as a _dict_ (in Python) or as an _Promisse .&lt;object&gt;_ (in JavaScript).
+
+### `Client.push_blob`
+
+Expects a single argument: a _dict_ (in Python) or an _object_ (in JavaScript) with a Frictionless resource.
+
+## API - Plumbing
+
+### `Client.action`
+
+This method bridges access to the CKAN API _action endpoint_.
+
+#### In Python
+
+Arguments:
+
+| Name                 | Type       | Default    | Description                                                  |
+| -------------------- | ---------- | ---------- | ------------------------------------------------------------ |
+| `name`               | `str`      | (required) | The action name, for example, `site_read`, `package_show`…   |
+| `payload`            | `dict`     | (required) | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `http_get`           | `bool`     | `False`    | Optional, if `True` will make `GET` request, otherwise `POST`. |
+| `transform_payload`  | `function` | `None`     | Function to mutate the `payload` before making the request (useful to convert to and from CKAN and Frictionless formats). |
+| `transform_response` | `function` | `None`     | function to mutate the response data before returning it (useful to convert to and from CKAN and Frictionless formats). |
+
+>[!note]The CKAN API uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you can pass `frictionless_ckan_mapper.frictionless_to_ckan` as `transform_payload`, and `frictionless_ckan_mapper.ckan_to_frictionless` as `transform_response`.
+
+
+#### In JavaScript
+
+Arguments:
+
+| Name         | Type                | Default            | Description                                                  |
+| ------------ | ------------------- | ------------------ | ------------------------------------------------------------ |
+| `actionName` | <code>string</code> | (required)         | The action name, for example, `site_read`, `package_show`…   |
+| `payload`    | <code>object</code> | (required)         | The payload being sent to CKAN. When a payload is provided to a GET request, it will be converted to URL parameters and each key will be converted to snake case. |
+| `useHttpGet` | <code>object</code> | <code>false</code> | Optional, if `True` will make `GET` request, otherwise `POST`. |
+
+>[!note]The JavaScript implementation uses the CKAN dataset and resource formats (rather than Frictionless formats).
+In other words, to stick to Frictionless formats, you need to convert from Frictionless to CKAN before calling `action` , and from CKAN to Frictionless after calling `action`.
+
+## Metadata reference
+
+>[!info]Your site may have custom metadata that differs from the example set below.
+
+
+### Profile
+
+**(`string`)** Defaults to _data-resource_.
+
+The profile of this descriptor.
+
+Every Package and Resource descriptor has a profile. The default profile, if none is declared, is `data-package` for Package and `data-resource` for Resource.
+
+#### Examples
+
+- `{"profile":"tabular-data-package"}`
+
+- `{"profile":"http://example.com/my-profiles-json-schema.json"}`
+
+### Name
+
+**(`string`)**
+
+An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+
+This is ideally a url-usable and human-readable name. Name `SHOULD` be invariant, meaning it `SHOULD NOT` change when its parent descriptor is updated.
+
+#### Example
+
+- `{"name":"my-nice-name"}`
+
+### Path
+
+A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+
+The dereferenced value of each referenced data source in `path` `MUST` be commensurate with a native, dereferenced representation of the data the resource describes. For example, in a *Tabular* Data Resource, this means that the dereferenced value of `path` `MUST` be an array.
+
+#### Validation
+
+##### It must satisfy one of these conditions
+
+###### Path
+
+**(`string`)**
+
+A fully qualified URL, or a POSIX file path..
+
+Implementations need to negotiate the type of path provided, and dereference the data accordingly.
+
+**Examples**
+
+- `{"path":"file.csv"}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+**(`array`)**
+
+**Examples**
+
+- `["file.csv"]`
+
+- `["http://example.com/file.csv"]`
+
+#### Examples
+
+- `{"path":["file.csv","file2.csv"]}`
+
+- `{"path":["http://example.com/file.csv","http://example.com/file2.csv"]}`
+
+- `{"path":"http://example.com/file.csv"}`
+
+### Data
+
+Inline data for this resource.
+
+### Schema
+
+**(`object`)**
+
+A schema for this resource.
+
+### Title
+
+**(`string`)**
+
+A human-readable title.
+
+#### Example
+
+- `{"title":"My Package Title"}`
+
+### Description
+
+**(`string`)**
+
+A text description. Markdown is encouraged.
+
+#### Example
+
+- `{"description":"# My Package description\nAll about my package."}`
+
+### Home Page
+
+**(`string`)**
+
+The home on the web that is related to this data package.
+
+#### Example
+
+- `{"homepage":"http://example.com/"}`
+
+### Sources
+
+**(`array`)**
+
+The raw sources for this resource.
+
+#### Example
+
+- `{"sources":[{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]}`
+
+### Licenses
+
+**(`array`)**
+
+The license(s) under which the resource is published.
+
+This property is not legally binding and does not guarantee that the package is licensed under the terms defined herein.
+
+#### Example
+
+- `{"licenses":[{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]}`
+
+### Format
+
+**(`string`)**
+
+The file format of this resource.
+
+`csv`, `xls`, `json` are examples of common formats.
+
+#### Example
+
+- `{"format":"xls"}`
+
+### Media Type
+
+**(`string`)**
+
+The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+
+#### Example
+
+- `{"mediatype":"text/csv"}`
+
+### Encoding
+
+**(`string`)** Defaults to _utf-8_.
+
+The file encoding of this resource.
+
+#### Example
+
+- `{"encoding":"utf-8"}`
+
+### Bytes
+
+**(`integer`)**
+
+The size of this resource in bytes.
+
+#### Example
+
+- `{"bytes":2082}`
+
+### Hash
+
+**(`string`)**
+
+The MD5 hash of this resource. Indicate other hashing algorithms with the {'{algorithm}'}:{'{hash}'} format.
+
+#### Examples
+
+- `{"hash":"d25c9c77f588f5dc32059d2da1136c02"}`
+
+- `{"hash":"SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"}`
+
+## Generating templates
+
+You can use [`jsv`](https://github.com/datopian/jsv) to generate a template  script in Python, JavaScript, and R.
+
+To install it:
+
+```
+$ npm install -g git+https://github.com/datopian/jsv.git
+```
+
+### Python
+
+```
+$ jsv data-resource.json --output py
+```
+
+**Output**
+```python
+dataset_metadata = {
+    "profile": "data-resource",  # The profile of this descriptor.
+    # [example] "profile": "tabular-data-package"
+    # [example] "profile": "http://example.com/my-profiles-json-schema.json"
+    "name": "my-nice-name",  # An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+    "path": ["file.csv","file2.csv"],  # A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+    # [example] "path": ["http://example.com/file.csv","http://example.com/file2.csv"]
+    # [example] "path": "http://example.com/file.csv"
+    "data": None,  # Inline data for this resource.
+    "schema": None,  # A schema for this resource.
+    "title": "My Package Title",  # A human-readable title.
+    "description": "# My Package description\nAll about my package.",  # A text description. Markdown is encouraged.
+    "homepage": "http://example.com/",  # The home on the web that is related to this data package.
+    "sources": [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}],  # The raw sources for this resource.
+    "licenses": [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}],  # The license(s) under which the resource is published.
+    "format": "xls",  # The file format of this resource.
+    "mediatype": "text/csv",  # The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+    "encoding": "utf-8",  # The file encoding of this resource.
+    # [example] "encoding": "utf-8"
+    "bytes": 2082,  # The size of this resource in bytes.
+    "hash": "d25c9c77f588f5dc32059d2da1136c02",  # The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+    # [example] "hash": "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+}
+```
+
+
+### JavaScript
+
+```
+$ jsv data-resource.json --output js
+```
+
+**Output**
+```javascript
+const datasetMetadata = {
+  // The profile of this descriptor.
+  profile: "data-resource",
+  // [example] profile: "tabular-data-package"
+  // [example] profile: "http://example.com/my-profiles-json-schema.json"
+  // An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+  name: "my-nice-name",
+  // A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+  path: ["file.csv", "file2.csv"],
+  // [example] path: ["http://example.com/file.csv","http://example.com/file2.csv"]
+  // [example] path: "http://example.com/file.csv"
+  // Inline data for this resource.
+  data: null,
+  // A schema for this resource.
+  schema: null,
+  // A human-readable title.
+  title: "My Package Title",
+  // A text description. Markdown is encouraged.
+  description: "# My Package description\nAll about my package.",
+  // The home on the web that is related to this data package.
+  homepage: "http://example.com/",
+  // The raw sources for this resource.
+  sources: [
+    {
+      title: "World Bank and OECD",
+      path: "http://data.worldbank.org/indicator/NY.GDP.MKTP.CD",
+    },
+  ],
+  // The license(s) under which the resource is published.
+  licenses: [
+    {
+      name: "odc-pddl-1.0",
+      path: "http://opendatacommons.org/licenses/pddl/",
+      title: "Open Data Commons Public Domain Dedication and License v1.0",
+    },
+  ],
+  // The file format of this resource.
+  format: "xls",
+  // The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+  mediatype: "text/csv",
+  // The file encoding of this resource.
+  encoding: "utf-8",
+  // [example] encoding: "utf-8"
+  // The size of this resource in bytes.
+  bytes: 2082,
+  // The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+  hash: "d25c9c77f588f5dc32059d2da1136c02",
+  // [example] hash: "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+};
+```
+
+### R
+
+```
+$ jsv data-resource.json --output r
+```
+
+**Output** 
+```r
+# The profile of this descriptor.
+profile <- "data-resource"
+# [example] profile <- "tabular-data-package"
+# [example] profile <- "http://example.com/my-profiles-json-schema.json"
+# An identifier string. Lower case characters with `.`, `_`, `-` and `/` are allowed.
+name <- "my-nice-name"
+# A reference to the data for this resource, as either a path as a string, or an array of paths as strings. of valid URIs.
+path <- ["file.csv","file2.csv"]
+# [example] path <- ["http://example.com/file.csv","http://example.com/file2.csv"]
+# [example] path <- "http://example.com/file.csv"
+# Inline data for this resource.
+data <- NA
+# A schema for this resource.
+schema <- NA
+# A human-readable title.
+title <- "My Package Title"
+# A text description. Markdown is encouraged.
+description <- "# My Package description\nAll about my package."
+# The home on the web that is related to this data package.
+homepage <- "http://example.com/"
+# The raw sources for this resource.
+sources <- [{"title":"World Bank and OECD","path":"http://data.worldbank.org/indicator/NY.GDP.MKTP.CD"}]
+# The license(s) under which the resource is published.
+licenses <- [{"name":"odc-pddl-1.0","path":"http://opendatacommons.org/licenses/pddl/","title":"Open Data Commons Public Domain Dedication and License v1.0"}]
+# The file format of this resource.
+format <- "xls"
+# The media type of this resource. Can be any valid media type listed with [IANA](https://www.iana.org/assignments/media-types/media-types.xhtml).
+mediatype <- "text/csv"
+# The file encoding of this resource.
+encoding <- "utf-8"
+# [example] encoding <- "utf-8"
+# The size of this resource in bytes.
+bytes <- 2082L
+# The MD5 hash of this resource. Indicate other hashing algorithms with the {algorithm}:{hash} format.
+hash <- "d25c9c77f588f5dc32059d2da1136c02"
+# [example] hash <- "SHA256:5262f12512590031bbcc9a430452bfd75c2791ad6771320bb4b5728bfb78c4d0"
+
+```
+
+
+## Design Principles
+
+The client **should** use Frictionless formats by default for describing dataset and resource objects passed to client methods.
+
+In addition, where more than metadata is needed (e.g., we need to access the data stream, or get the schema) we expect the _Dataset_ and _Resource_ objects to follow the [Frictionless Data Lib pattern](https://github.com/frictionlessdata/project/blob/master/rfcs/0004-frictionless-data-lib-pattern.md).

site/content/docs/dms/ckan-enterprise/index.md

@@ -0,0 +1,108 @@
+# CKAN Enterprise
+
+## Introduction
+
+CKAN Enterprise is our name for what we plan would become our standard "base" distribution for CKAN going forward:
+
+* It is a CKAN standard code base with micro-services.
+* Enterprise grade data catalog and portal targeted at Gov (open data portals) and Enterprise (Data Catalogs +).
+* It is also known as [Datopian DMS](https://www.datopian.com/datopian-dms/).
+
+## Roadmap 2021 and beyond
+
+|                   | Current                                                                                    | CKAN Enterprise                                                 |
+|-------------------|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
+| Raw storage       | Filestore                                                                                  | Giftless                                                        |
+| Data Loader (db)  | DataPusher extension                                                                       | Aircan                                                          |
+| Data Storage (db) | Postgres                                                                                   | Any database engine. By default, Postgres                       |
+| Data API (read)   | Built-in DataStore extension's API including SQL endpoint                                  | GraphQL based standalone micro-service                          |
+| Frontend (public) | Build-in frontend into CKAN Classic python app (some projects are using nodejs app)        | PortalJS or nodejs app                                          |
+| Data Explorer     | ReclineJS (some projects that uses nodejs app for frontend have React based Data Explorer) | GraphQL based Data Explorer                                     |
+| Auth              | Traditional login/password + extendable with CKAN Classic extensions                       | SSO with default Google, Github, Facebook and Microsoft options |
+| Permissions       | CKAN Classic based permissions                                                             | Existing permissions exposed via JWT based authz API            |
+
+## Timeline 2021
+
+To develop a base distribution of CKAN Enterprise, we want to build a demo project with the features from the roadmap. This way we can:
+
+* understand its advantages/limitations;
+* compare against other instances of CKAN;
+* demonstrate for the potential clients.
+
+High level overview of the planned features with ETA:
+
+| Name                          | Description                          | Effort | ETA |
+| ----------------------------- | ------------------------------------ | ------ | --- |
+| [Init](#Init)                 | Select CKAN version and deploy to DX | xs     | Q2  |
+| [Blobstore](#Blobstore)       | Integrate Giftless for raw storage   | s      | Q2  |
+| [Versioning](#Versioning)     | Develop/integrate new versioning sys | l      | Q3  |
+| [DataLoader](#DataLoader)     | Develop/integrate Aircan             | xl     | Q3  |
+| [Data API](#Data-API)         | Integrate new Data API (read)        | m      | Q2  |
+| [Frontend](#Frontend)         | Build a theme using PortalJS         | s      | Q2  |
+| [DataExplorer](#DataExplorer) | Integrate into PortalJS              | s      | Q2  |
+| [Permissions](#Permissions)   | Develop permissions in read frontend | m      | Q4  |
+| [Auth](#Auth)                 | Integrate                            | s      | Q4  |
+
+### Init
+
+Initialize a new project for development of CKAN Enterprise.
+
+Tasks:
+
+* Boot project in Datopian-DX cluster
+* Use CKAN v2.8.x (latest patch) or 2.9.x
+* Don't setup DataPusher
+* Namespace: `ckan-enterprise`
+* Domain: `enterprise.ckan.datopian.com`
+
+### Blobstore
+
+See [blob storage](/docs/dms/blob-storage#ckan-v3)
+
+### Versioning
+
+See [versioning](/docs/dms/versioning#ckan-v3)
+
+### DataLoader
+
+See [DataLoader](/docs/dms/load)
+
+### Data API
+
+* Install new [Data API service](https://github.com/datopian/data-api) in the project
+* Install Hasura service in the project
+* Set it up to work with DB of CKAN Enterprise
+* Read more about Data API [here](/docs/dms/data-api#read-api-3)
+
+Notes:
+
+* We could experiment and use various features of Hasura, eg:
+  * Setting up row/column limits per user role (permissions)
+  * Subscriptions to auto load new data rows
+
+### Frontend
+
+PortalJS for the read frontend of CKAN Enterprise. [Read more](/docs/dms/frontend/#frontend).
+
+### DataExplorer
+
+A new Data Explorer based on GraphQL API: https://github.com/datopian/data-explorer-graphql
+
+### Permissions
+
+See [permissions](/docs/dms/permissions#permissions-authorization).
+
+### Auth
+
+Next generation, Kratos based, authentication (mostly SSO with no Traditional login by default) with following options out of the box:
+
+* GitHub
+* Google
+* Facebook
+* Microsoft
+
+Easy to add:
+
+* Discord
+* GitLab
+* Slack

site/content/docs/dms/ckan-v3/index.md

@@ -0,0 +1,365 @@
+# CKAN v3
+
+## Introduction
+
+This document describes the architectures of CKAN v2 ("CKAN Classic"), CKAN v3 (also known as "CKAN Next Gen" for Next Generation), and CKAN v3 hybrid. The latter is an intermediate approach towards v3, where we still use CKAN v2 and common extensions, and only create microservices for new features.
+
+You will also find out how to do common tasks such as theming or testing, in each of the architectures.
+
+*Note: this blog post has an overview of the more decoupled, microservices approach at the core of v3: https://www.datopian.com/2021/05/17/a-more-decoupled-ckan/*
+
+## CKAN v2, CKAN v3 and Why v3
+
+In yellow, you see one single Python process:
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"]
+  end
+```
+
+When you want to extend core functionality of CKAN v2 (Classic), you write a Python package that must be installed in CKAN. This way, the extension will also run in the same process as the core functionality. This is known as a monolithic architecture.
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+  end
+```
+
+When you start to add multiple features, through extensions, what you get is one single Python process running many non-related functionalities.
+
+```mermaid
+graph TB
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["CKAN Extension 2"]
+    ckancore --> ckanext3["CKAN Extension 3"]
+    ckancore --> ckanext4["CKAN Extension 4"]
+    ckancore --> ckanext5["CKAN Extension 5"]
+  end
+```
+
+This monolithic approach has advantages in terms of simplicity of development and deployment, especially when the system is small. However, as it grows in scale and scope, there are an increasing number of issues.
+
+In this approach, an optional extension has the ability to crash the whole CKAN instance. Every new feature must be written in the same language and framework (e.g. Python, leveraging Flask or Django). And, perhaps most fundamentally, the overall system is highly coupled, making it complex and hard to understand, debug, extend, and evolve.
+
+### Microservices and CKAN v3
+
+The main way to address these problems while gaining extra benefits is to move to a microservices-based architecture.
+
+Thus, we recommend building the next version of CKAN – CKAN v3 – on a microservices approach.
+
+[!tip]CKAN v3 is sometimes also referred to as CKAN Next Gen(eration).
+
+With microservices, each piece of functionality runs in its own service and process. 
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["CKAN frontend"]
+    ckanfrontend1["Frontend"]
+  end
+  
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+### Incremental Evolution – Hybrid v3
+
+One of the other advantages of the microservices approach is that it can also be used to extend and evolve current CKAN v2 solutions in an incremental way. We term these kinds of solutions "Hybrid v3," as they are a mix of v2 and v3 together. 
+
+For example, a Hybrid v3 data portal could use a new microservice written in Node for the frontend, and combine that with CKAN v2 (with v2 extensions).
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["CKAN frontend"]
+    ckanfrontend1["Frontend"]
+  end
+  
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["CKAN Extension 2"]
+  end
+  
+  ckanfrontend1 --> ckancore
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+The hybrid approach means we can evolve CKAN v2 "Classic" to CKAN v3 "Next Gen" incrementally. In particular, it allows people to keep using their existing v2 extensions, and upgrade them to new microservices gradually.
+
+### Comparison of Approaches
+
+|              | CKAN v2 (Classic) | CKAN v3 (Next Gen) | CKAN v3 Hybrid |
+| ------------ | ------------------| -------------------| ---------------|
+| Architecture | Monolithic        | Microservice       | Microservice with v2 core |
+| Language     | Python            | You can write services in any language you like.<br/><br/>Frontend default: JS.<br/>Backend default: Python | Python and any language you like for microservices. |
+| Frontend (and theming) | Python with Python CKAN extension | Flexible. Default is modern JS/NodeJS based | Can use old frontend but default to new JS-based frontend. |
+| Data Packages | Add-on, no integration | Default internal and external format | Data Packages with converter to old CKAN format. |
+| Extension | Extensions are libraries that are added to core runtime. They must therefore be built in python and are loaded into the core process at build time. "Template/inheritance" model where hooks are in core and it is core that loads and calls plugins. This means that if a hook does not exist in core then the extension is stymied. | Extensions are microservices and can be written in any language. They are loaded into the url space via kubernetes routing manager. Extensions hook into "core" via APIs (rather than in code). Follows a "composition" model rather than inheritance model | Can use old style extensions or microservices. |
+| Resource Scaling | You have a single application so scaling is of the core application. | You can scale individual microservices as needed. | Mix of v2 and v3 |
+
+## Why v3: Long Version
+
+What are the problems with CKAN v2's monolithic architecture in relation to microservices v3?
+
+* **Poor Developer Experience (DX), innovability, and scalability due to coupling**. Monolithic means "one big system" => Coupling & Complexity => hard to understand, change and extend. Changes in one area can unexpectedly affect other areas. 
+  * DX to develop a small new API requires wiring into CKAN core via an extension. Extensions can interact in unexpected ways.
+  * The core of people who fully understand CKAN has stayed small for a reason: there's a lot of understand.
+  * https://github.com/ckan/ckan/issues/5333 is an example of a small bug that's hard to track down due to various paths involved.
+  * Harder to make incremental changes due to coupling (e.g. Python 3 upgrade requires *everything* to be fixed at once - can't do rolling releases).
+* **Stability**. One bad extension crashes or slows down the whole system
+* **One language => Less developer flexibility (Poor DX)**. Have to write *everything* in Python, including the frontend. This is an issue especially for the frontend: almost all modern frontend development is heavily Javascript-based and theme is the #1 thing people want to customize in CKAN. At the moment, that requires installing *all* of CKAN core (using Docker) plus some familiarity with Python and Jinja templating. This is a big ask.
+* **Extension stablity and testing**. Testing of extensions is painful (at least without careful factoring in a separate mini library) and are therefore often not tested; they don't have Continuous Integration (CI) or Continuous Deployment (CD). As an example, a highly experienced Python developer at Datopian was still struggling to get extension tests working 6 months into their CKAN work.
+* **DX is poor especially when getting started**. Getting CKAN up and running requires multiple external services (database, Solr, Redis, etc.) making Docker the only viable way for bootstraping a local development environment. This makes getting started with CKAN daunting and painful.
+* **Vertical scalability is poor**. Scaling the system is costly as you have to replicate the whole core process in every machine.
+* **System is highly coupled.** Extensions b/c in process tend to end up with significant coupling to core which makes them brittle (has improved with plugins.toolkit)
+  * Upgrading core to Python 3 requires upgrading *all* extensions because they run in the same process.
+  * Search Index is not a separate API, but in Core. So replacing Solr is hard.
+
+The top 2 customizations of CKAN are slow and painful and require deep knowledge of CKAN:
+
+* Theming a site.
+* Customizing the metadata.
+
+## Architectures
+
+### CKAN v2 (Classic)
+
+This diagram is based on the file `docker-compose.yml` of [github.com/okfn/docker-ckan](https://github.com/okfn/docker-ckan) (`docker-compose.dev.yml` has the same components, but different configuration).
+ 
+A difference from this diagram to the file is that we are not including DataPusher, as it is not a required dependency.
+
+>[!tip]Databases may run as Docker containers, or rely on third-party services such as Amazon Relational Database Service (RDS).
+
+
+
+```mermaid
+graph LR
+
+CKAN[CKAN web app]
+
+CKAN --> DB[(Database)]
+CKAN --> Solr[(Solr)]
+CKAN --> Redis[(Redis)]
+
+subgraph Docker container
+  CKAN
+end
+```
+
+Same setup showing some of the key extensions explicitly:
+
+```mermaid
+graph LR
+  core[CKAN Core] --> DB[(Database)]
+  datastore --> DB2[(Database - DataStore)]
+  core --> Solr[(Solr)]
+  core --> Redis[(Redis)]
+  
+  subgraph Docker container
+    core
+    datastore
+    datapusher
+    imageview
+    ...
+  end
+```
+
+CKAN ships with several core extensions that are built-in. Here, together with the list of main components, we list a couple of them:
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+CKAN | Application (API + Worker) | [Link](https://github.com/ckan/ckan) | Data management system (DMS) for powering data hubs and data portals. It's a monolithical web application that includes several built-in extensions and dependencies, such as a job queue service. In theory, it's possible to run it without any extensions.
+datapusher | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/datapusher) | It could also be called "datapusher-connect." It's a glue code to connect with a separate microservice called DataPusher, which performs actions when new data arrives.
+datastore | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/datastore) | The interface between CKAN and the structure database, the one receiving datasets and resources (CSVs). It includes an API for the database and an administrative UI.
+imageview | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/imageview) | It provides an interface for creating HTML templates for image resources.
+multilingual | CKAN Extension | [Link](https://github.com/ckan/ckan/tree/master/ckanext/multilingual) | It provides an interface for translation and localization.
+Database | Database | | People tend to use a single PostgreSQL instance for this. Separated in multiple databases, it's the place where CKAN stores its own information (sometimes referred as "MetaStore" and "HubStore"), rows of resources (StructuredStore or DataStore), and raw datasets and resources ("BlobStore" or "FileStore"). The latter may store data in the local filesystem or cloud providers, via extensions.
+Solr | Database | | It provides indexing and full-text search for CKAN.
+Redis | Database | | Lightweight key-value store, used for caching and job queues.
+
+### CKAN v3 (Next Gen)
+
+CKAN Next Gen is still a DMS, as CKAN Classic; but rather than a monolithical architecture, it follows the microservices approach. CKAN Classic is not a dependency anymore, as we have smaller services providing functionality that we may or many not choose to include. This description is based on [Datopian's Technical Documentation](/docs/dms/ckan-v3/next-gen/#roadmap).
+
+```mermaid
+graph LR
+  subgraph api3["..."]
+    api31["API"]
+  end
+
+  subgraph api2["Administration"]
+    api21["API"]
+  end
+  
+  subgraph api1["Authentication"]
+    api11["API"]
+  end
+
+  subgraph frontend["Frontend"]
+    frontendapi["API"]
+  end
+  
+  subgraph storage["Raw Resources Storage"]
+    storageapi["API"]
+  end
+  
+  storageapi --> cloudstorage[(Cloud Storage)]
+  
+  frontendapi --> storageapi
+  frontendapi --> api11
+  frontendapi --> api21
+  frontendapi --> api31
+```
+
+At this moment, many important features are only available through CKAN extensions, so that brings us to the hybrid approach.
+
+### CKAN Hybrid v3 (Next Gen)
+
+We may sometimes make an explit distinction between CKAN v3 "hybrid" and "pure." The reason is because we want to ensure that we're not there yet – we have many opportunities to extract features out of CKAN and CKAN Extensions.
+
+In this approach, we still rely on CKAN Classic and all its extensions. Many already had many tests and bugs fixed, so we can deliver more if not forced to rewrite everything from scratch.
+
+```mermaid
+graph TB
+  subgraph ckanapi3["CKAN API 3"]
+    ckanapi31["API 3"]
+  end
+
+  subgraph ckanapi2["CKAN API 2"]
+    ckanapi21["API 2"]
+  end
+  
+  subgraph ckanapi1["CKAN API 1"]
+    ckanapi11["API 1"]
+  end
+
+  subgraph ckanfrontend["Frontend"]
+    ckanfrontend1["Frontend v2"]
+    theme["[Project-specific theme]"]
+  end
+  
+  subgraph ckanclassic["CKAN Classic"]
+    ckancore["Core"] --> ckanext["CKAN Extension 1"]
+    ckancore --> ckanext2["[Project-specific extension]"]
+  end
+  
+  ckanfrontend1 --> ckancore
+  ckanfrontend1 --> ckanapi11
+  ckanfrontend1 --> ckanapi21
+  ckanfrontend1 --> ckanapi31
+```
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+Frontend v2 | Application | [Link](https://github.com/datopian/frontend-v2) | Node application for Data Portals. It communicates with a CKAN Classic instance, through its API, to get data and render HTML. It is written to be extensible, such as connecting to other applications and theming.
+[Project-specific theme] | Frontend Theme | e.g., [Link](https://github.com/datopian/frontend-oddk) | Extension to Frontend v2 where you can personalize the interface, create different pages, and connect with other APIs.
+[API 1] | Application | e.g., [Link](https://github.com/datopian/data-subscriptions) | Any application with an API to communicate with the user-facing Frontend v2 or to run tasks in background. Given the current architecture, often, this API is usually designed to work with CKAN interfaces. Over time, we may choose to make it more generic, and even replace CKAN Core with other applications.
+
+## Job Stories
+
+In this spreadsheet, you will find a list of common job stories in CKAN projects. Also, how you can accomplish them in CKAN v2, v3, and Hybrid v3.
+
+https://docs.google.com/spreadsheets/d/1cLK8xylprmVsoQIbdphqz9-ccSpdDABQExvKdvNJqaQ/edit#gid=757361856
+
+## Glossary
+
+### API
+
+An HTTP API, usually following the REST style.
+
+### Application
+
+A Python package, an API, a worker... It may have other applications as dependencies.
+  
+### CKAN Extension
+
+A Python package following specification from [CKAN Extending guide](https://docs.ckan.org/en/2.8/extensions/index.html).
+  
+### Database
+
+An organized collection of data.
+
+### Dataset
+
+A group of resources made to be distributed together.
+  
+### Frontend Theme
+
+A Node project specializing behavior present in [Frontend v2](https://github.com/datopian/frontend-v2).
+  
+### Resource
+
+A data blob. Common formats are CSV, JSON, and PDF.
+  
+### System
+
+A group of applications and databases that work together to accomplish a set of tasks.
+  
+### Worker
+
+An application that runs tasks in background. They may run recurrently according to a given schedule, or as soon as it's requested by another application.
+
+## Appendix
+
+### Architecture - CKAN v2 with DataPusher
+
+```mermaid
+graph TB
+  subgraph DataPusher
+    datapusherapi["DataPusher API"]
+    datapusherworker["CKAN Service Provider"]
+    SQLite[(SQLite)]
+  end
+  
+  subgraph CKAN
+    core
+    datapusher[datapusher ext]
+    datastore
+    ...
+  end
+  
+  core[CKAN Core] --> datastore
+  datastore --> DB[(Database)]
+  datapusherapi --> core
+  datapusher --> datapusherapi
+```
+
+Name | Type | Repository | Description
+-----|------|------------|------------
+DataPusher | System | [Link](https://github.com/ckan/datapusher) | Microservice that parses data files and uploads them to the datastore.
+DataPusher API | API | [Link](https://github.com/ckan/datapusher) | HTTP API written in Flask. It is called from the built-in `datapusher` CKAN extension whenever a resource is created (and has the right type).
+CKAN Service Provider | Worker | [Link](https://github.com/ckan/ckan-service-provider) | Library for making web services that make functions available as synchronous or asynchronous jobs.
+SQLite | Database | | Unknown use. Possibly a worker dependency.
+
+### Old Next Gen Page
+
+Prior to this page, we had one called "Next Gen." It has intersections with this article, although it focuses more on the benefits of microservices. For the time being, the page still exists in [/ckan-v3/next-gen](/docs/dms/ckan-v3/next-gen), although it may get merged with this one in the future.

site/content/docs/dms/ckan-v3/next-gen.md

@@ -0,0 +1,203 @@
+# Next Gen
+
+“Next Gen” (NG) is our name for the evolution of CKAN from its current state as “CKAN Classic”.
+
+Next Gen has a decoupled, microservice architecture in contrast to CKAN Classic's monolithic architecture. It is also built from the ground up on the Frictionless Data principles and specifications which provide a simple, well-defined and widely adopted set of core interfaces and tooling for managing data.
+
+## Classic to Next Gen
+
+CKAN classic: monolithic architecture -- everything is one big python application. Extension is done at code level and "compiled in" at compile/run-time (i.e. you end up with one big docker file).
+
+CKAN Next Gen: decoupled, service-oriented -- services connected by network calls. Extension is done by adding new services,
+
+```mermaid
+graph LR
+
+subgraph "CKAN Classic"
+  plugins
+end
+
+subgraph "CKAN Next Gen"
+  microservices
+end
+
+plugins --> microservices
+```
+
+You can read more about monolithic vs microservice architectures in the [Appendix below](#appendix-monolithic-vs-microservice-architecture).
+
+
+## Next Gen lays the foundation for the future and brings major immediate benefits
+
+Next Gen's new approach is important in several major ways.
+
+### Microservices are the Future
+
+First, decoupled microservices have become *the* way to design and deploy (web) applications after first being pioneered by the likes of Amazon in the early 2000s. And in the last five to ten years have brought microservices "for the masses" with relevant tooling and technology standardized, open-sourced and widely deployed -- not only with containerization such as Docker, Kubernetes but also in programming languages like (server-side) Javascript and Golang.
+
+By adopting a microservice approach CKAN can reap the the benefits of what is becoming a mature and standard way to design and build (web) applications. This includes the immediate advantages of being aligned with the technical paradigm such as tooling and developer familiarity.
+
+### Microservices bring Scalability, Reliability, Extensibility and Flexibility
+
+In addition, and even more importantly, the microservices approach brings major benefits in:
+
+1. **Scalability**: dramatically easier and cheaper to scale up -- and down -- in size *and* complexity. Size-wise this is because you can replicate individual services rather than the whole application. Complexity-wise this is because monolithic architectures tend to become "big" where service-oriented encourages smaller lightweight components with cleaner interfaces. This means you can have a much smaller core making it easier to install, setup and extend. It also means you can use what you need making solutions easier to maintain and upgrade.
+2. **Reliability**: easier (and cheaper) to build highly reliable, high availability solutions because microservices make isolation and replication easier. For example, in a microservice architecture a problem in CKAN's harvester won't impact your main portal because they run in separate containers. Similarly, you can scale the harvester system separately from the web frontend.
+3. **Extensibility**: much easier to create and maintain extensions because they are a decoupled service and interfaces are leaner and cleaner.
+4. **Flexibility** aka "Bring your own tech": services can be written in any language so, for example, you can write your frontend in javascript and your backend in Python. In a monolithic architecture all parts must be written in the same language because everything is compiled together. This flexibility makes it easier to use the best tool for the job. It also makes it much easier for teams to collaborate and cooperate and fewer bottlenecks in development.
+
+ASIDE: decoupled microservices reflect the "unix" way of building networked applications. As with the "unix way" in general, whilst this approach better -- and simpler -- in the long-run, in the short-run it often needs sustantial foundational work (those Unix authors were legends!). It may also be, at least initially, more resource intensive and more complex infrastructurally. Thus, whilst this approach is "better" it was not suprising that it was initially used for for complex and/or high end applications e.g. Amazon. This also explains why it took a while for this approach to get adoption -- it is only in the last few year that we have robust, lightweight, easy to use tooling and patterns for microservices -- "microservices for the masses" if you like.
+
+In summary, the Next Gen approach provides an essential foundation for the continuing growth and evolution of CKAN as a platform for building world-class data portal and data management solutions.
+
+## Evolution not Revolution: Next Gen Components Work with CKAN Classic
+
+*Gradual evolution from CKAN classic (keep what is working, keep your investments, incremental change)*
+
+Next Gen components are specifically designed to work with CKAN "Classic" in its current form. This means existing CKAN users can immediately benefit from Next Gen components and features whilst retaining the value of their existing investment. New (or existing) CKAN-based solutions can adopt a "hybrid" approach using components from both Classic and Next Gen. It also means that the owner of a CKAN-based solution can incrementally evolve from "Classic" to "Next Gen" by replacing one component one at a time, gaining new functionality without sacrificing existing work.
+
+ASIDE: we're fortunate that CKAN Classic itself was ahead of its time in its level of "service-orientation". From the start, it had a very rich and robust API and it has continued to develop this with almost almost all functionality exposed via the API. It is this rich API and well factored design that makes it relatively straightforward to evolve CKAN in its current "Classic" form towards Next Gen.
+
+## New Features plus Existing Functionality Improved
+
+In addition to its architecture, Next Gen provides a variety of improvements and extensions to CKAN Classic's functionality. For example:
+
+* Theming and Frontend Customization: theming and customizing CKAN's frontend has got radically easier and quicker. See [Frontend section »][frontend]
+* DMS + CMS unified: integrate the full power of a modern CMS into your data portal and have one unified interface for data and content. See [Frontend section »][frontend]
+* Data Explorer: the existing CKAN data preview/explorer has been completely rewritten in modern React-based Javascript (ReclineJS is now 7y old!). See [Data Explorer section »][explorer]
+* Dashboards: build rich data-driven dashboards and integrate. See [Dashboards section »][dashboards]
+* Harvesting: simpler, more powerful harvesting built on modern ETL. See [Harvesting section »][harvesting]
+
+And each of these features is easily deployed into an existing CKAN solution!
+
+[frontend]: /docs/dms/frontend
+[explorer]: /docs/dms/data-explorer
+[dashboards]: /docs/dms/dashboards
+[harvesting]: /docs/dms/harvesting
+
+## Roadmap
+
+The journey to Next Gen from Classic can proceed step by step -- it does not need to be a big bang. Like refurbishing and extending a house, we can add a room here or renovate a room there whilst continuing to live happily in the building (and benefitting as our new bathroom comes online, or we get a new conservatory!).
+
+Here's an overview of the journey to Next Gen and current implementation status. More granular information on particular features may sometimes be found on the individual feature page, for example for [Harvesting here](/docs/dms/harvesting#design).
+
+```mermaid
+graph LR
+
+start[Start]
+themefe[Read Frontend]
+authfe[Authentication in FE]
+authzfe[Authorization in FE]
+previews[Previews]
+explorer[Explorer]
+permsserv[Permissions Service]
+orgs[Organizations]
+
+
+subgraph Start
+  start
+end
+
+subgraph Frontend
+  start --> themefe
+  themefe --> authfe
+  authfe --> authzfe
+  themefe --> revisioningfe[Revision UI]
+end
+
+subgraph Harvesting
+  start --> harvestetl[Harvesting ETL + Runner]
+  harvestetl --> harvestui[Harvest UI]
+end
+
+subgraph "Admin UI"
+  managedataset[Manage Dataset]
+  manageorg[Manage Organization]
+  manageuser[Manage Users]
+  manageconfig[Manage Config]
+  
+  start --> managedataset
+  start --> manageorg
+  managedataset --> manageconfig
+end
+
+subgraph "Backend (API)"
+  start --> permsserv
+  start --> revision[Backend Revisioning]
+end
+
+datastore[DataStore]
+
+subgraph DataStore
+  start --> datastore
+  datastore --> dataload[Data Load]
+end
+
+subgraph Explorer
+  themefe --> previews
+  previews --> explorer
+end
+
+subgraph Organizations
+  start --> orgs
+end
+
+subgraph Key
+  done[Done]
+  nearlydone[Nearly Done]
+  inprogress[In Progress]
+  next[Next Up]
+end
+
+classDef done fill:#21bf73,stroke:#333,stroke-width:3px;
+classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:3px;
+classDef inprogress fill:orange,stroke:#333,stroke-width:2px;
+classDef next fill:pink,stroke:#333,stroke-width:1px;
+
+class done,themefe,previews,explorer,harvestetl done;
+class nearlydone,authfe,harvestui nearlydone;
+class inprogress,dataload inprogress;
+class next,permsserv next;
+```
+
+## Appendix: Monolithic vs Microservice architecture
+
+Monolithic: Libraries or modules communicate via function calls (inside one big application)
+
+Microservices: Services communicate over a network
+
+The best introduction and definition of microservices comes from Martin Fowler https://martinfowler.com/microservices/
+
+> Microservice architectures will use libraries, but their primary way of componentizing their own software is by breaking down into services. We define libraries as components that are linked into a program and called using in-memory function calls, while services are out-of-process components who communicate with a mechanism such as a web service request, or remote procedure call. https://martinfowler.com/articles/microservices.html
+
+### Monolithic
+
+```mermaid
+graph TD
+
+subgraph "Monolithic - all inside"
+  a
+  b
+  c
+end
+
+a --in-memory function all--> b
+a --in-memory function all--> c
+```
+
+### Microservice
+
+```mermaid
+graph TD
+subgraph "A Container"
+  a
+end
+subgraph "B Container"
+  b
+end
+subgraph "C Container"
+  c
+end
+a -.network call.-> b
+a -.network call.-> c
+```