Version Packages (#949)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/config.json

@@ -0,0 +1,14 @@
+{
+  "$schema": "https://unpkg.com/@changesets/config@2.3.0/schema.json",
+  "changelog": [
+    "@changesets/changelog-github",
+    { "repo": "datopian/portaljs" }
+  ],
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "restricted",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

.github/workflows/release.yml

@@ -0,0 +1,39 @@
+name: Release
+
+on:
+  push:
+    branches:
+      - main
+
+concurrency: release-${{ github.ref }}
+
+jobs:
+  release:
+    name: Release
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v3
+
+      - name: Setup Node.js 16.x
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16.x
+
+      - name: Install Dependencies
+        run: npm ci
+
+      - name: Create Release Pull Request or Publish to npm
+        id: changesets
+        uses: changesets/action@v1
+        with:
+          publish: npm run release
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+      # - name: Send a Discord notification if a publish happens
+      #   if: steps.changesets.outputs.published == 'true'
+      #   uses: Ilshidur/action-discord@0.3.2
+      #   with:
+      #     args: 'The project {{ EVENT_PAYLOAD.repository.full_name }} has been deployed.'

.gitignore

@@ -4,6 +4,7 @@
 dist
 tmp
 /out-tsc
+**/*.tgz
 
 # dependencies
 node_modules
@@ -16,6 +17,7 @@ node_modules
 *.launch
 .settings/
 *.sublime-workspace
+.obsidian
 
 # IDE - VSCode
 .vscode/*

CONTRIBUTING.md

@@ -13,7 +13,7 @@ Structure:
   - **dataset-frictionless**: Example utilizing a frictionless dataset as an example
 - **site**: the website for the project, with a landing page and the docs
 - **packages**:
-    - **portaljs-components**: the library of components for creating a data portal
+  - **portaljs-components**: the library of components for creating a data portal
 
 ## How to contribute
 
@@ -26,10 +26,11 @@ If you'd like to work on one of the issues you can:
 3. Clone the forked repository to your machine.
 4. Create a feature branch (e.g. `50-update-readme`, where `50` is the number of the related issue).
 5. Commit your changes to the feature branch.
-6. Push the feature branch to your forked repository.
+6. Add changeset file describing the changes. (See section below)
-7. Create a Pull Request against the original repository.
+7. Push the feature branch to your forked repository.
+8. Create a Pull Request against the original repository.
    - add a short description of the changes included in the PR
-8. Address review comments if requested by our demanding reviewers 😜.
+9. Address review comments if requested by our demanding reviewers 😜.
 
 If you have an idea for improvement, and it doesn't have a corresponding issue yet, simply submit a new one.
 
@@ -62,6 +63,7 @@ or you can use just:
 nx <target> <project>
 # e.g. npx nx serve ckan
 ```
+
 if you have the `nx` binary installed globally in your machine
 
 #### Running multiple tasks
@@ -109,7 +111,7 @@ To check code formatting in selected projects:
 
 ```sh
 npx nx format:check --projects=<array of projects>
-# npx nx format:check --projects=ckan,dataset-frictionless 
+# npx nx format:check --projects=ckan,dataset-frictionless
 ```
 
 To check code formatting in all projects:
@@ -124,7 +126,7 @@ To fix code formatting in selected projects:
 
 ```sh
 npx nx format:write --projects=<array projects>
-# npx nx format:write --projects=ckan,dataset-frictionless 
+# npx nx format:write --projects=ckan,dataset-frictionless
 ```
 
 To fix formatting in all projects:
@@ -174,3 +176,23 @@ To learn more see this [offical docs page](https://nx.dev/reference/nx-json).
 Each project also has it's own configuration file - `project.json`, where you can define and configure it's targets (and more).
 
 To learn more see this [offical docs page](https://nx.dev/reference/project-configuration).
+
+## Changesets and publishing packages
+
+> This monorepo is set up with changesets versioning tool. See their [github repository](https://github.com/changesets/changesets) to learn more.
+
+### What are Changesets?
+
+Changesets are files that describe the intention of a contributor to bump a version of the package according to their changes. Changeset file holds two key bits of information: a version type (following semver), and change information to be added to a changelog.
+
+### Adding changesets
+
+In the root directory of the repo, run:
+
+```
+npx changeset
+```
+
+Select the package that has been changed, the semver version that should be bumped with it and a description of your changes. Please make sure to add the most accurate but also concise information.
+
+To learn about semantic versioning standards see [this semver doc page](https://semver.org/).

examples/alan-turing-portal/README.md

@@ -1,19 +0,0 @@
-## Getting started
-
-To get started with this template, first install the npm dependencies:
-
-```bash
-npm install
-```
-
-Next, run the development server:
-
-```bash
-npm run dev
-```
-
-Finally, open [http://localhost:3000](http://localhost:3000) in your browser to view the website.
-
-## License
-
-This site template is a commercial product and is licensed under the [Tailwind UI license](https://tailwindui.com/license).

examples/alan-turing-portal/components/Header.jsx

@@ -1,109 +0,0 @@
-import { useRef } from 'react'
-import { useRouter } from 'next/router'
-
-import { Container } from '../components/Container'
-
-function SunIcon(props) {
-  return (
-    <svg
-      viewBox="0 0 24 24"
-      strokeWidth="1.5"
-      strokeLinecap="round"
-      strokeLinejoin="round"
-      aria-hidden="true"
-      {...props}
-    >
-      <path d="M8 12.25A4.25 4.25 0 0 1 12.25 8v0a4.25 4.25 0 0 1 4.25 4.25v0a4.25 4.25 0 0 1-4.25 4.25v0A4.25 4.25 0 0 1 8 12.25v0Z" />
-      <path
-        d="M12.25 3v1.5M21.5 12.25H20M18.791 18.791l-1.06-1.06M18.791 5.709l-1.06 1.06M12.25 20v1.5M4.5 12.25H3M6.77 6.77 5.709 5.709M6.77 17.73l-1.061 1.061"
-        fill="none"
-      />
-    </svg>
-  )
-}
-
-function MoonIcon(props) {
-  return (
-    <svg viewBox="0 0 24 24" aria-hidden="true" {...props}>
-      <path
-        d="M17.25 16.22a6.937 6.937 0 0 1-9.47-9.47 7.451 7.451 0 1 0 9.47 9.47ZM12.75 7C17 7 17 2.75 17 2.75S17 7 21.25 7C17 7 17 11.25 17 11.25S17 7 12.75 7Z"
-        strokeWidth="1.5"
-        strokeLinecap="round"
-        strokeLinejoin="round"
-      />
-    </svg>
-  )
-}
-
-function ModeToggle() {
-  function disableTransitionsTemporarily() {
-    document.documentElement.classList.add('[&_*]:!transition-none')
-    window.setTimeout(() => {
-      document.documentElement.classList.remove('[&_*]:!transition-none')
-    }, 0)
-  }
-
-  function toggleMode() {
-    disableTransitionsTemporarily()
-
-    let darkModeMediaQuery = window.matchMedia('(prefers-color-scheme: dark)')
-    let isSystemDarkMode = darkModeMediaQuery.matches
-    let isDarkMode = document.documentElement.classList.toggle('dark')
-
-    if (isDarkMode === isSystemDarkMode) {
-      delete window.localStorage.isDarkMode
-    } else {
-      window.localStorage.isDarkMode = isDarkMode
-    }
-  }
-
-  return (
-    <button
-      type="button"
-      aria-label="Toggle dark mode"
-      className="group rounded-full bg-white/90 px-3 py-2 shadow-lg shadow-zinc-800/5 ring-1 ring-zinc-900/5 backdrop-blur transition dark:bg-zinc-800/90 dark:ring-white/10 dark:hover:ring-white/20"
-      onClick={toggleMode}
-    >
-      <SunIcon className="h-6 w-6 fill-zinc-100 stroke-zinc-500 transition group-hover:fill-zinc-200 group-hover:stroke-zinc-700 dark:hidden [@media(prefers-color-scheme:dark)]:fill-teal-50 [@media(prefers-color-scheme:dark)]:stroke-teal-500 [@media(prefers-color-scheme:dark)]:group-hover:fill-teal-50 [@media(prefers-color-scheme:dark)]:group-hover:stroke-teal-600" />
-      <MoonIcon className="hidden h-6 w-6 fill-zinc-700 stroke-zinc-500 transition dark:block [@media(prefers-color-scheme:dark)]:group-hover:stroke-zinc-400 [@media_not_(prefers-color-scheme:dark)]:fill-teal-400/10 [@media_not_(prefers-color-scheme:dark)]:stroke-teal-500" />
-    </button>
-  )
-}
-
-export function Header() {
-  let isHomePage = useRouter().pathname === '/'
-
-  let headerRef = useRef()
-
-  return (
-    <>
-      <header
-        className="pointer-events-none relative z-50 flex flex-col"
-        style={{
-          height: 'var(--header-height)',
-          marginBottom: 'var(--header-mb)',
-        }}
-      >
-        <div
-          ref={headerRef}
-          className="top-0 z-10 h-16 pt-6"
-          style={{ position: 'var(--header-position)' }}
-        >
-          <Container
-            className="top-[var(--header-top,theme(spacing.6))] w-full"
-            style={{ position: 'var(--header-inner-position)' }}
-          >
-            <div className="relative flex gap-4">
-              <div className="flex justify-end md:flex-1">
-                <div className="pointer-events-auto">
-                  <ModeToggle />
-                </div>
-              </div>
-            </div>
-          </Container>
-        </div>
-      </header>
-      {isHomePage && <div style={{ height: 'var(--content-offset)' }} />}
-    </>
-  )
-}

examples/alan-turing-portal/content/contributing.md

@@ -1,22 +0,0 @@
----
-title: Contributing
----
-
-We accept entries to our catalogue based on pull requests to the content folder. The dataset must be avaliable for download to be included in the list. If you want to add an entry, follow these steps!
-
-Please send just one dataset addition/edit at a time - edit it in, then save. This will make everyone’s life easier (including yours!)
-
-- Go to the repo url file and click the "Add file" dropdown and then click on "Create new file".
-![](https://i.imgur.com/2PR0ZgL.png)
-   
-- In the following page type `content/datasets/<name-of-the-file>.md`. if you want to add an entry to the datasets catalog or `content/keywords/<name-of-the-file>.md` if you want to add an entry to the lists of abusive keywords.
-![](https://i.imgur.com/rr3uSYu.png)
-   
-- Copy the contents of `templates/dataset.md` or `templates/keywords.md` respectively to the camp below, filling out the fields with the correct data format
-![](https://i.imgur.com/x6JIjhz.png)
-   
-- Click on "Commit changes", on the popup make sure you give some brief detail on the proposed change.  and then click on Propose changes
-![](https://i.imgur.com/BxuxKEJ.png)
-   
-- Submit the pull request on the next page when prompted.
-

examples/alan-turing-portal/content/index.md

@@ -1,13 +0,0 @@
----
-title: Hate Speech Dataset Catalogue
----
-
-This page catalogues datasets annotated for hate speech, online abuse, and offensive language. They may be useful for e.g. training a natural language processing system to detect this language.
-
-The list is maintained by Leon Derczynski, Bertie Vidgen, Hannah Rose Kirk, Pica Johansson, Yi-Ling Chung, Mads Guldborg Kjeldgaard Kongsbak, Laila Sprejer, and Philine Zeinert.
-
-We provide a list of datasets and keywords. If you would like to contribute to our catalogue or add your dataset, please see the instructions for contributing.
-
-If you use these resources, please cite (and read!) our paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out. And if you would like to find other resources for researching online hate, visit The Alan Turing Institute’s Online Hate Research Hub or read The Alan Turing Institute’s Reading List on Online Hate and Abuse Research.
-
-If you’re looking for a good paper on online hate training datasets (beyond our paper, of course!) then have a look at ‘Resources and benchmark corpora for hate speech detection: a systematic review’ by Poletto et al. in Language Resources and Evaluation.

examples/ckan-example/.env

@@ -1 +0,0 @@
-DMS=https://demo.dev.datopian.com

examples/ckan-ssg/package.json

@@ -10,24 +10,25 @@
   },
   "dependencies": {
     "@heroicons/react": "^2.0.17",
-    "@types/node": "18.16.0",
+    "@portaljs/ckan": "^0.0.2",
-    "@types/react": "18.0.38",
-    "@types/react-dom": "18.0.11",
-    "eslint": "8.39.0",
-    "eslint-config-next": "13.3.1",
     "next": "13.3.1",
     "next-seo": "^6.0.0",
     "octokit": "^2.0.14",
     "react": "18.2.0",
     "react-dom": "18.2.0",
     "react-markdown": "^8.0.7",
-    "remark-gfm": "^3.0.1",
+    "remark-gfm": "^3.0.1"
-    "typescript": "5.0.4"
   },
   "devDependencies": {
     "@tailwindcss/typography": "^0.5.9",
+    "@types/node": "18.16.0",
+    "@types/react": "18.0.38",
+    "@types/react-dom": "18.0.11",
     "autoprefixer": "^10.4.14",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
     "postcss": "^8.4.23",
-    "tailwindcss": "^3.3.1"
+    "tailwindcss": "^3.3.1",
+    "typescript": "5.0.4"
   }
 }

examples/ckan-ssg/pages/@org/[org]/[dataset]/index.tsx

@@ -11,8 +11,9 @@ import {
   ServerIcon,
   UserIcon,
 } from '@heroicons/react/20/solid';
+import { CKAN } from '@portaljs/ckan';
 
-const dms = getConfig().publicRuntimeConfig.DMS;
+const backend_url = getConfig().publicRuntimeConfig.DMS;
 
 const formatter = new Intl.DateTimeFormat('en-US', {
   year: 'numeric',
@@ -25,14 +26,12 @@ const formatter = new Intl.DateTimeFormat('en-US', {
 });
 
 export const getServerSideProps: GetServerSideProps = async (context) => {
+  const ckan = new CKAN(backend_url)
   const { dataset } = context.query;
-  const response = await fetch(
+  const _dataset = await ckan.getDatasetDetails(dataset as string)
-    `${dms}/api/3/action/package_show?id=${dataset}`
-  );
-  const _dataset = await response.json();
   return {
     props: {
-      dataset: _dataset.result,
+      dataset: _dataset,
     },
   };
 };

examples/ckan-ssg/pages/index.tsx

@@ -1,7 +1,8 @@
 import getConfig from 'next/config';
 import styles from './index.module.css';
+import { CKAN } from '@portaljs/ckan';
 
-const dms = getConfig().publicRuntimeConfig.DMS
+const backend_url = getConfig().publicRuntimeConfig.DMS
 
 const formatter = new Intl.DateTimeFormat('en-US', {
   year: 'numeric',
@@ -15,12 +16,11 @@ const formatter = new Intl.DateTimeFormat('en-US', {
 
 
 export async function getServerSideProps() {
-  const response = await fetch(`${dms}/api/3/action/package_search`)
+  const ckan = new CKAN(backend_url)
-  const datasets = await response.json()
+  const { datasets } = await ckan.packageSearch({ limit: 1000, offset: 0, groups:[], orgs: [], tags: []})
-  const datasetsWithDetails = await Promise.all(datasets.result.results.map(async (dataset) => {
+  const datasetsWithDetails = await Promise.all(datasets.map(async (dataset) => {
-    const response = await fetch(`${dms}/api/3/action/package_show?id=` + dataset.name)
+    const _dataset = await ckan.getDatasetDetails(dataset.name)
-    const json = await response.json()
+    return _dataset
-    return json.result
   }))
 
   return {
@@ -79,7 +79,7 @@ export function Index({ datasets }) {
                 </thead>
                 <tbody className="divide-y divide-gray-200">
                   {datasets.map((dataset) => (
-                    <tr>
+                    <tr key={dataset.name}>
                       <td className="px-3 py-4 text-sm text-gray-500">
                           {dataset.title}
                       </td>

examples/ckan/.gitignore

@@ -0,0 +1,35 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

examples/ckan/README.md

@@ -0,0 +1,38 @@
+This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
+
+[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
+
+The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
+
+This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

examples/ckan/components/DataRichDocument.tsx

@@ -7,13 +7,13 @@ import { Mermaid } from '@flowershow/core';
 // to handle import statements. Instead, you must include components in scope
 // here.
 const components = {
-  Table: dynamic(() => import('./Table')),
+  Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
+  Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
+  FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
   mermaid: Mermaid,
-  // Excel: dynamic(() => import('../components/Excel')),
+  Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
-  // TODO: try and make these dynamic ...
+  VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),
-  Vega: dynamic(() => import('./Vega')),
+  LineChart: dynamic(() => import('@portaljs/components').then(mod => mod.LineChart)),
-  VegaLite: dynamic(() => import('./VegaLite')),
-  LineChart: dynamic(() => import('./LineChart')),
 } as any;
 
 export default function DRD({ source }: { source: any }) {

examples/ckan/content/test.md

@@ -0,0 +1,3 @@
+# Test
+
+Test Data Rich Stories

examples/ckan/lib/markdown.js

@@ -22,7 +22,7 @@ import { serialize } from "next-mdx-remote/serialize";
  * @format: used to indicate to next-mdx-remote which format to use (md or mdx)
  * @returns: { mdxSource: mdxSource, frontMatter: ...}
  */
-const parse = async function (source, format) {
+const parse = async function (source, format, scope) {
   const { content, data, excerpt } = matter(source, {
     excerpt: (file, options) => {
       // Generate an excerpt for the file
@@ -91,7 +91,7 @@ const parse = async function (source, format) {
         ],
         format,
       },
-      scope: data,
+      scope,
     }
   );
 

examples/ckan/next.config.js

@@ -0,0 +1,11 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  reactStrictMode: true,
+  publicRuntimeConfig: {
+    DMS: process.env.DMS
+      ? process.env.DMS.replace(/\/?$/, '')
+      : 'https://demo.dev.datopian.com/',
+  },
+};
+
+module.exports = nextConfig;

examples/ckan/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "ckan",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "prebuild": "npm run mddb",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint",
+    "mddb": "mddb ./content"
+  },
+  "dependencies": {
+    "@flowershow/core": "^0.4.13",
+    "@flowershow/markdowndb": "^0.1.5",
+    "@flowershow/remark-callouts": "^1.0.0",
+    "@flowershow/remark-embed": "^1.0.0",
+    "@githubocto/flat-ui": "^0.14.1",
+    "@heroicons/react": "^2.0.18",
+    "@portaljs/ckan": "^0.0.2",
+    "@portaljs/components": "0.1.6",
+    "@tailwindcss/typography": "^0.5.9",
+    "@types/node": "20.2.3",
+    "@types/react": "18.2.6",
+    "@types/react-dom": "18.2.4",
+    "autoprefixer": "10.4.14",
+    "eslint": "8.41.0",
+    "eslint-config-next": "13.4.3",
+    "isomorphic-unfetch": "^4.0.2",
+    "next": "13.4.3",
+    "next-mdx-remote": "^4.4.1",
+    "papaparse": "^5.4.1",
+    "postcss": "8.4.23",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-query": "^3.39.3",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
+    "tailwindcss": "3.3.2",
+    "typescript": "5.0.4"
+  }
+}

examples/ckan/pages/[org]/[dataset]/index.tsx

@@ -0,0 +1,179 @@
+import Head from "next/head";
+import { CKAN, Dataset } from "@portaljs/ckan";
+import {
+  ChevronRightIcon,
+  HomeIcon,
+  PaperClipIcon,
+} from "@heroicons/react/20/solid";
+import Link from "next/link";
+import getConfig from "next/config";
+
+const backend_url = getConfig().publicRuntimeConfig.DMS
+
+export const getServerSideProps = async (context: any) => {
+  try {
+    const datasetName = context.params?.dataset;
+    if (!datasetName) {
+      return {
+        notFound: true,
+      };
+    }
+    const ckan = new CKAN(backend_url);
+    const dataset = await ckan.getDatasetDetails(datasetName as string);
+    if (!dataset) {
+      return {
+        notFound: true,
+      };
+    }
+    return {
+      props: { dataset },
+    };
+  } catch {
+    return {
+      notFound: true,
+    };
+  }
+};
+
+export default function DatasetPage({
+  dataset,
+}: {
+  dataset: Dataset;
+}): JSX.Element {
+  return (
+    <>
+      <Head>
+        <title>{`${dataset.title || dataset.name} - Dataset`}</title>
+        <meta name="description" content="Generated by create next app" />
+        <link rel="icon" href="/favicon.ico" />
+      </Head>
+      <main className="flex min-h-screen flex-col items-center justify-between p-24 bg-zinc-900">
+        <div className="bg-white p-8 my-4 rounded-lg">
+          <nav className="flex px-4 py-8" aria-label="Breadcrumb">
+            <ol role="list" className="flex items-center space-x-4">
+              <li>
+                <div>
+                  <Link href="/" className="text-gray-400 hover:text-gray-500">
+                    <HomeIcon
+                      className="h-5 w-5 flex-shrink-0"
+                      aria-hidden="true"
+                    />
+                    <span className="sr-only">Home</span>
+                  </Link>
+                </div>
+              </li>
+              <li>
+                <div className="flex items-center">
+                  <ChevronRightIcon
+                    className="h-5 w-5 flex-shrink-0 text-gray-400"
+                    aria-hidden="true"
+                  />
+                  <span
+                    className="ml-4 text-sm font-medium text-gray-500 hover:text-gray-700"
+                    aria-current={"page"}
+                  >
+                    {dataset.name}
+                  </span>
+                </div>
+              </li>
+            </ol>
+          </nav>
+          {dataset && (
+            <div>
+              <div className="px-4 sm:px-0">
+                <h3 className="text-base font-semibold leading-7 text-gray-900">
+                  {dataset.title || dataset.name}
+                </h3>
+                <p className="mt-1 max-w-2xl text-sm leading-6 text-gray-500">
+                  Dataset details
+                </p>
+              </div>
+              <div className="mt-6 border-t border-gray-100">
+                <dl className="divide-y divide-gray-100">
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    <dt className="text-sm font-medium leading-6 text-gray-900">
+                      Title
+                    </dt>
+                    <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                      {dataset.title}
+                    </dd>
+                  </div>
+                  {dataset.tags && dataset.tags.length > 0 && (
+                    <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                      <dt className="text-sm font-medium leading-6 text-gray-900">
+                        Tags
+                      </dt>
+                      <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                        {dataset.tags.map((tag) => tag.display_name).join(", ")}
+                      </dd>
+                    </div>
+                  )}
+                  {dataset.tags && dataset.tags.length > 0 && (
+                    <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                      <dt className="text-sm font-medium leading-6 text-gray-900">
+                        URL
+                      </dt>
+                      <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                        {dataset.url}
+                      </dd>
+                    </div>
+                  )}
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    {dataset.notes && (
+                      <>
+                        <dt className="text-sm font-medium leading-6 text-gray-900">
+                          Description
+                        </dt>
+                        <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                          {dataset.notes}
+                        </dd>
+                      </>
+                    )}
+                  </div>
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    <dt className="text-sm font-medium leading-6 text-gray-900">
+                      Files
+                    </dt>
+                    <dd className="mt-2 text-sm text-gray-900 sm:col-span-2 sm:mt-0">
+                      <ul
+                        role="list"
+                        className="divide-y divide-gray-100 rounded-md border border-gray-200"
+                      >
+                        {dataset.resources.map((resource) => (
+                          <li key={resource.id} className="flex items-center justify-between py-4 pl-4 pr-5 text-sm leading-6">
+                            <div className="flex w-0 flex-1 items-center">
+                              <PaperClipIcon
+                                className="h-5 w-5 flex-shrink-0 text-gray-400"
+                                aria-hidden="true"
+                              />
+                              <div className="ml-4 flex min-w-0 flex-1 gap-2">
+                                <span className="truncate font-medium">
+                                  {resource.name || resource.id}
+                                </span>
+                                <span className="flex-shrink-0 text-gray-400">
+                                  {resource.size}
+                                </span>
+                              </div>
+                            </div>
+                            <div className="ml-4 flex-shrink-0">
+                              <a
+                                href={resource.url}
+                                className="font-medium hover:text-indigo-500 mr-4"
+                              >
+                                Download
+                              </a>
+                            </div>
+                          </li>
+                        ))}
+                      </ul>
+                    </dd>
+                  </div>
+                </dl>
+              </div>
+            </div>
+          )}
+        </div>
+      </main>
+    </>
+  );
+}

examples/ckan/pages/_app.tsx

@@ -0,0 +1,7 @@
+import '@/styles/globals.css'
+import '@portaljs/ckan/styles.css'
+import type { AppProps } from 'next/app'
+
+export default function App({ Component, pageProps }: AppProps) {
+  return <Component {...pageProps} />
+}

examples/ckan/pages/_document.tsx

@@ -0,0 +1,13 @@
+import { Html, Head, Main, NextScript } from 'next/document'
+
+export default function Document() {
+  return (
+    <Html lang="en">
+      <Head />
+      <body>
+        <Main />
+        <NextScript />
+      </body>
+    </Html>
+  )
+}

examples/ckan/pages/api/cors.ts

@@ -0,0 +1,18 @@
+import fetch from 'isomorphic-unfetch';
+
+const Cors = async (req: any, res: any) => {
+  const { url } = req.query;
+  try {
+    const resProxy = await fetch(url, {
+      headers: {
+        Range: 'bytes=0-5132288',
+      },
+    });
+    const data = await resProxy.text();
+    return res.status(200).send(data);
+  } catch (error: any) {
+    res.status(400).send(error.toString());
+  }
+};
+
+export default Cors;

examples/ckan/pages/index.tsx

@@ -0,0 +1,55 @@
+import {
+  CKAN,
+  DatasetSearchForm,
+  ListOfDatasets,
+  PackageSearchOptions,
+  Organization,
+  Group,
+} from '@portaljs/ckan';
+import getConfig from 'next/config';
+import { useState } from 'react';
+
+const backend_url = getConfig().publicRuntimeConfig.DMS;
+
+export async function getServerSideProps() {
+  const ckan = new CKAN(backend_url);
+  const groups = await ckan.getGroupsWithDetails();
+  const orgs = await ckan.getOrgsWithDetails();
+  return {
+    props: {
+      groups,
+      orgs,
+    },
+  };
+}
+
+export default function Home({
+  orgs,
+  groups,
+}: {
+  orgs: Organization[];
+  groups: Group[];
+}) {
+  const ckan = new CKAN(backend_url);
+  const [options, setOptions] = useState<PackageSearchOptions>({
+    offset: 0,
+    limit: 5,
+    tags: [],
+    groups: [],
+    orgs: [],
+  });
+
+  return (
+    <main className="flex min-h-screen flex-col items-center p-24 bg-zinc-900">
+      <DatasetSearchForm
+        options={options}
+        setOptions={setOptions}
+        groups={groups}
+        orgs={orgs}
+      />
+      <div className="bg-white p-8 my-4 rounded-lg">
+        <ListOfDatasets options={options} setOptions={setOptions} ckan={ckan} />{' '}
+      </div>
+    </main>
+  );
+}

examples/ckan/pages/stories/[[...path]].tsx

@@ -0,0 +1,129 @@
+import { existsSync, promises as fs } from 'fs';
+import path from 'path';
+import parse from '../../lib/markdown';
+
+import DataRichDocument from '../../components/DataRichDocument';
+import clientPromise from '../../lib/mddb';
+import getConfig from 'next/config';
+import { CKAN } from '@portaljs/ckan';
+
+export const getStaticPaths = async () => {
+  const contentDir = path.join(process.cwd(), '/content/');
+  const contentFolders = await fs.readdir(contentDir, 'utf8');
+  const paths = contentFolders.map((folder: string) => ({
+    params: { path: [folder.split('.')[0]] },
+  }));
+  return {
+    paths,
+    fallback: false,
+  };
+};
+
+const backend_url = getConfig().publicRuntimeConfig.DMS;
+
+export const getStaticProps = async (context) => {
+  const mddb = await clientPromise;
+  const storyFile = await mddb.getFileByUrl(context.params.path);
+  const md = await fs.readFile(
+    `${process.cwd()}/${storyFile.file_path}`,
+    'utf8'
+  );
+
+  const ckan = new CKAN(backend_url);
+  const datasets = storyFile.metadata.datasets ? await Promise.all(
+    storyFile.metadata.datasets.map(
+      async (datasetName: string) => await ckan.getDatasetDetails(datasetName)
+    )
+  ) : [];
+  const orgs = storyFile.metadata.orgs ? await Promise.all(
+    storyFile.metadata.orgs.map(
+      async (orgName: string) => await ckan.getOrgDetails(orgName)
+    )
+  ) : [];
+
+  let { mdxSource, frontMatter } = await parse(md, '.mdx', { datasets, orgs });
+
+  return {
+    props: {
+      mdxSource,
+      frontMatter: JSON.stringify(frontMatter),
+    },
+  };
+};
+
+export default function DatasetPage({ mdxSource, frontMatter }) {
+  frontMatter = JSON.parse(frontMatter);
+  return (
+    <main className="flex min-h-screen flex-col justify-between p-16 bg-zinc-900">
+      <div className="bg-white p-8 my-4 rounded-lg">
+        <div className="prose mx-auto py-8">
+          <header>
+            <div className="mb-6">
+              <>
+                <h1 className="mb-2">{frontMatter.title}</h1>
+                {frontMatter.author && (
+                  <p className="my-0">
+                    <span className="font-semibold">Author: </span>
+                    <span className="my-0">{frontMatter.author}</span>
+                  </p>
+                )}
+                {frontMatter.description && (
+                  <p className="my-0">
+                    <span className="font-semibold">Description: </span>
+                    <span className="description my-0">
+                      {frontMatter.description}
+                    </span>
+                  </p>
+                )}
+                {frontMatter.modified && (
+                  <p className="my-0">
+                    <span className="font-semibold">Modified: </span>
+                    <span className="description my-0">
+                      {new Date(frontMatter.modified).toLocaleDateString()}
+                    </span>
+                  </p>
+                )}
+                {frontMatter.files && (
+                  <section className="py-6">
+                    <h2 className="mt-0">Data files</h2>
+                    <table className="table-auto">
+                      <thead>
+                        <tr>
+                          <th>File</th>
+                          <th>Format</th>
+                        </tr>
+                      </thead>
+                      <tbody>
+                        {frontMatter.files.map((f) => {
+                          const fileName = f.split('/').slice(-1);
+                          return (
+                            <tr key={`resources-list-${f}`}>
+                              <td>
+                                <a target="_blank" href={f}>
+                                  {fileName}
+                                </a>
+                              </td>
+                              <td>
+                                {fileName[0]
+                                  .split('.')
+                                  .slice(-1)[0]
+                                  .toUpperCase()}
+                              </td>
+                            </tr>
+                          );
+                        })}
+                      </tbody>
+                    </table>
+                  </section>
+                )}
+              </>
+            </div>
+          </header>
+          <main>
+            <DataRichDocument source={mdxSource} />
+          </main>
+        </div>
+      </div>
+    </main>
+  );
+}

examples/ckan/public/next.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>

examples/ckan/public/vercel.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 283 64"><path fill="black" d="M141 16c-11 0-19 7-19 18s9 18 20 18c7 0 13-3 16-7l-7-5c-2 3-6 4-9 4-5 0-9-3-10-7h28v-3c0-11-8-18-19-18zm-9 15c1-4 4-7 9-7s8 3 9 7h-18zm117-15c-11 0-19 7-19 18s9 18 20 18c6 0 12-3 16-7l-8-5c-2 3-5 4-8 4-5 0-9-3-11-7h28l1-3c0-11-8-18-19-18zm-10 15c2-4 5-7 10-7s8 3 9 7h-19zm-39 3c0 6 4 10 10 10 4 0 7-2 9-5l8 5c-3 5-9 8-17 8-11 0-19-7-19-18s8-18 19-18c8 0 14 3 17 8l-8 5c-2-3-5-5-9-5-6 0-10 4-10 10zm83-29v46h-9V5h9zM37 0l37 64H0L37 0zm92 5-27 48L74 5h10l18 30 17-30h10zm59 12v10l-3-1c-6 0-10 4-10 10v15h-9V17h9v9c0-5 6-9 13-9z"/></svg>

examples/ckan/styles/globals.css

@@ -0,0 +1,71 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+
+@import "@flowershow/remark-callouts/styles.css";
+
+/* mathjax */
+.math-inline > mjx-container > svg {
+  display: inline;
+  align-items: center;
+}
+
+/* smooth scrolling in modern browsers */
+html {
+  scroll-behavior: smooth !important;
+}
+
+/* tooltip fade-out clip */
+.tooltip-body::after {
+  content: "";
+  position: absolute;
+  right: 0;
+  top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
+  height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
+  width: 10rem;
+  background: linear-gradient(
+    to right,
+    rgba(255, 255, 255, 0),
+    rgba(255, 255, 255, 1) 100%
+  );
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
+  margin-left: -2rem !important;
+  padding-left: 2rem !important;
+  scroll-margin-top: 4.5rem;
+  position: relative;
+}
+
+.heading-link {
+  padding: 1px;
+  position: absolute;
+  left: 0;
+  top: 50%;
+  transform: translateY(-50%);
+  margin: auto 0;
+  border-radius: 5px;
+  background: #1e293b;
+  opacity: 0;
+  transition: opacity 0.2s;
+}
+
+.light .heading-link {
+  /* border: 1px solid #ab2b65; */
+  /* background: none; */
+  background: #e2e8f0;
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
+  opacity: 100;
+}
+
+.heading-link svg {
+  transform: scale(0.75);
+}
+
+@media screen and (max-width: 640px) {
+  .heading-link {
+    visibility: hidden;
+  }
+}

examples/ckan/tailwind.config.js

@@ -0,0 +1,18 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    './pages/**/*.{js,ts,jsx,tsx,mdx}',
+    './components/**/*.{js,ts,jsx,tsx,mdx}',
+    './app/**/*.{js,ts,jsx,tsx,mdx}',
+  ],
+  theme: {
+    extend: {
+      backgroundImage: {
+        'gradient-radial': 'radial-gradient(var(--tw-gradient-stops))',
+        'gradient-conic':
+          'conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))',
+      },
+    },
+  },
+  plugins: [require('@tailwindcss/typography')],
+}

examples/ckan/tsconfig.json

@@ -0,0 +1,23 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": false,
+    "forceConsistentCasingInFileNames": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "preserve",
+    "incremental": true,
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
+  "exclude": ["node_modules"]
+}

examples/fivethirtyeight/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+  "extends": "next/core-web-vitals"
+}

examples/fivethirtyeight/.gitignore

@@ -0,0 +1,35 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

examples/fivethirtyeight/README.md

@@ -0,0 +1,44 @@
+This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
+
+You might be asking why we did that, there are three main reasons:
+
+- The website has a great UI, with multiple datasets being displayed elegantly and with simplicity.
+- PortalJS allows us to add more functionality to it e.g dataset previews and search functionality.
+- The project follows our same principles of open sourcing and free data, with every dataset being publicly available on Github.
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
+
+[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
+
+The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
+
+This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

examples/fivethirtyeight/components/Breadcrumbs.tsx

@@ -0,0 +1,23 @@
+import Link from "next/link";
+
+function HomeIcon({ className = "" }) {
+  return <div className={`inline-block w-4 ${className}`}><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">    <path d="M 12 2 A 1 1 0 0 0 11.289062 2.296875 L 1.203125 11.097656 A 0.5 0.5 0 0 0 1 11.5 A 0.5 0.5 0 0 0 1.5 12 L 4 12 L 4 20 C 4 20.552 4.448 21 5 21 L 9 21 C 9.552 21 10 20.552 10 20 L 10 14 L 14 14 L 14 20 C 14 20.552 14.448 21 15 21 L 19 21 C 19.552 21 20 20.552 20 20 L 20 12 L 22.5 12 A 0.5 0.5 0 0 0 23 11.5 A 0.5 0.5 0 0 0 22.796875 11.097656 L 12.716797 2.3027344 A 1 1 0 0 0 12.710938 2.296875 A 1 1 0 0 0 12 2 z"/></svg></div>
+}
+
+export default function Breadcrumbs({ links }: { links: { title: string, href?: string, target?: string }[] }) {
+  const current = links.at(-1);
+
+  return <div className="flex items-center uppercase font-black text-xs">
+    <Link className="flex items-center" href='/'><HomeIcon /></Link>
+
+    {/* {links.length > 1 && links.slice(0, -1).map((link) => {
+      return <>
+        <span className="mx-4">/</span>
+        <Link href={link.href}>{link.title}</Link>
+      </>
+    })} */}
+
+    <span className="mx-4">/</span>
+    <span>{current?.title}</span>
+  </div >
+}

examples/fivethirtyeight/components/Layout.tsx

@@ -0,0 +1,99 @@
+import { useState } from 'react';
+import { XMarkIcon } from '@heroicons/react/20/solid';
+import { Transition } from '@headlessui/react';
+
+export default function Layout({ children }: { children: React.ReactNode }) {
+  const [isShowing, setShow] = useState(true);
+  return (
+    <>
+      <Transition
+        show={isShowing}
+        enter="transition-opacity duration-75"
+        enterFrom="opacity-0"
+        enterTo="opacity-100"
+        leave="transition-opacity duration-150"
+        leaveFrom="opacity-100"
+        leaveTo="opacity-0"
+      >
+        <div className="flex items-center gap-x-6 bg-[#3c3c3c] px-6 py-2.5 sm:px-3.5 sm:before:flex-1">
+          <p className="text-sm leading-6 text-white">
+            This is a replica to the awesome{' '}
+            <a
+              className="hover:underline font-bold"
+              href="https://data.fivethirtyeight.com"
+            >
+              data.fivethirtyeight.com
+            </a>{' '}
+            website.{' '}
+            <a
+              className="hover:underline font-bold"
+              href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight#readme"
+            >
+              Read more here
+            </a>{' '}
+          </p>
+          <div className="flex flex-1 justify-end">
+            <button
+              type="button"
+              onClick={() => setShow(false)}
+              className="-m-3 p-3 focus-visible:outline-offset-[-4px]"
+            >
+              <span className="sr-only">Dismiss</span>
+              <XMarkIcon className="h-5 w-5 text-white" aria-hidden="true" />
+            </button>
+          </div>
+        </div>
+      </Transition>
+      <header className="max-w-5xl mx-auto mt-8 w-full">
+        <div className="border-b-2 pb-2.5 mx-2 border-zinc-800 flex justify-between">
+          <h1 className="flex gap-x-1 items-end">
+            <span className="sr-only">FiveThirtyEight</span>
+            <img
+              width="197"
+              height="25"
+              alt="FiveThirtyEight"
+              src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
+            />{' '}
+            <span className="-mb-0.5 text-[#3c3c3c]">replica</span>
+          </h1>
+          <div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
+            <a
+              className="hover:opacity-75 transition"
+              href="https://portaljs.org"
+            >
+              Built with 🌀PortalJS
+            </a>
+            <hr className="h-[80%] border border-[#3c3c3c] opacity-75 my-2"></hr>
+            <a
+              className="hover:opacity-75 transition"
+              href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
+            >
+              Github
+            </a>
+          </div>
+        </div>
+        <div className="mx-2 py-1.5 text-[14px] text-[#3c3c3c] md:hidden">
+          <ul className="flex gap-x-4">
+            <li>
+              <a
+                className="hover:opacity-75 transition"
+                href="https://portaljs.org"
+              >
+                PortalJS
+              </a>
+            </li>
+            <li>
+              <a
+                className="hover:opacity-75 transition"
+                href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
+              >
+                View on Github
+              </a>
+            </li>
+          </ul>
+        </div>
+      </header>
+      {children}
+    </>
+  );
+}

examples/fivethirtyeight/lib/octokit.ts

@@ -0,0 +1,38 @@
+import { Octokit } from 'octokit';
+
+export interface GithubProject {
+  owner: string;
+  repo: string;
+  branch: string;
+  files: string[];
+  readme: string;
+  description?: string;
+  name?: string;
+}
+
+export async function getProjectReadme(
+  owner: string,
+  repo: string,
+  branch: string,
+  readme: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.getContent({
+      owner,
+      repo,
+      path: readme,
+      ref: branch,
+    });
+    const data = response.data as { content?: string };
+    const fileContent = data.content ? data.content : '';
+    if (fileContent === '') {
+      return null;
+    }
+    const decodedContent = Buffer.from(fileContent, 'base64').toString();
+    return decodedContent;
+  } catch (error) {
+    return null;
+  }
+}

examples/fivethirtyeight/next.config.js

@@ -0,0 +1,9 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  reactStrictMode: true,
+  serverRuntimeConfig: {
+    github_pat: process.env.GITHUB_PAT ? process.env.GITHUB_PAT : null,
+  },
+}
+
+module.exports = nextConfig

examples/fivethirtyeight/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "fiverthirtyeight-example",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "@flowershow/core": "^0.4.13",
+    "@headlessui/react": "^1.7.14",
+    "@heroicons/react": "^2.0.18",
+    "@portaljs/components": "^0.1.8",
+    "@tailwindcss/typography": "^0.5.9",
+    "@types/node": "20.1.1",
+    "@types/react": "18.2.6",
+    "@types/react-dom": "18.2.4",
+    "autoprefixer": "10.4.14",
+    "eslint": "8.40.0",
+    "eslint-config-next": "13.4.1",
+    "flexsearch": "^0.7.31",
+    "next": "13.4.1",
+    "next-mdx-remote": "^4.4.1",
+    "next-seo": "^6.0.0",
+    "octokit": "^2.0.14",
+    "postcss": "8.4.23",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-markdown": "^8.0.7",
+    "remark": "^14.0.3",
+    "remark-code-frontmatter": "^1.0.0",
+    "remark-excerpt": "^1.0.0-beta.1",
+    "remark-extract-frontmatter": "^3.2.0",
+    "remark-frontmatter": "^4.0.1",
+    "remark-gfm": "^3.0.1",
+    "tailwindcss": "3.3.2",
+    "timeago.js": "^4.0.2",
+    "to-vfile": "^7.2.4",
+    "typescript": "5.0.4"
+  }
+}

examples/fivethirtyeight/pages/_app.tsx

@@ -0,0 +1,49 @@
+import '@/styles/globals.css';
+import '@portaljs/components/styles.css';
+import { useEffect } from 'react';
+import { pageview } from '@flowershow/core';
+import Script from 'next/script';
+import Head from 'next/head';
+import { useRouter } from 'next/router';
+
+import type { AppProps } from 'next/app';
+
+export default function App({ Component, pageProps }: AppProps) {
+  const router = useRouter();
+
+  useEffect(() => {
+    const handleRouteChange = (url: any) => {
+      pageview(url);
+    };
+    router.events.on('routeChangeComplete', handleRouteChange);
+    return () => {
+      router.events.off('routeChangeComplete', handleRouteChange);
+    };
+  }, [router.events]);
+  return (
+    <>
+      <Head>
+        <link rel="shortcut icon" href="/squared_logo.png" />
+      </Head>
+      <Script
+        strategy="afterInteractive"
+        src="https://www.googletagmanager.com/gtag/js?id=G-3N9SXTC7GS"
+      />
+      <Script
+        id="gtag-init"
+        strategy="afterInteractive"
+        dangerouslySetInnerHTML={{
+          __html: `
+          window.dataLayer = window.dataLayer || [];
+          function gtag(){dataLayer.push(arguments);}
+          gtag('js', new Date());
+          gtag('config', 'G-3N9SXTC7GS', {
+            page_path: window.location.pathname,
+          });
+        `,
+        }}
+      />
+      <Component {...pageProps} />
+    </>
+  );
+}

examples/fivethirtyeight/pages/_document.tsx

@@ -0,0 +1,27 @@
+import { Html, Head, Main, NextScript } from 'next/document';
+
+export default function Document() {
+  return (
+    <Html lang="en">
+      <Head>
+        <link
+          rel="icon"
+          type="image/x-icon"
+          href="https://projects.fivethirtyeight.com/shared/favicon.ico"
+        />
+        <meta
+          property="og:image"
+          content="https://portaljs-fivethirtyeight.vercel.app/share_image.png"
+        />
+        <meta
+          property="twitter:image"
+          content="https://portaljs-fivethirtyeight.vercel.app/share_image.png"
+        />
+      </Head>
+      <body>
+        <Main />
+      </body>
+      <NextScript />
+    </Html>
+  );
+}

examples/fivethirtyeight/pages/datasets/[datasetName].tsx

@@ -0,0 +1,224 @@
+import { NextSeo } from 'next-seo';
+import { promises as fs } from 'fs';
+import path from 'path';
+import getConfig from 'next/config';
+import { getProjectReadme, GithubProject } from '@/lib/octokit';
+import remarkGfm from 'remark-gfm';
+import extract from 'remark-extract-frontmatter';
+import { Dataset } from '..';
+import { GetStaticProps } from 'next';
+import { FlatUiTable } from '@portaljs/components';
+import Breadcrumbs from '@/components/Breadcrumbs';
+import { ReactMarkdown } from 'react-markdown/lib/react-markdown';
+import remarkFrontmatter from 'remark-frontmatter';
+import Layout from '@/components/Layout';
+import { format } from 'timeago.js';
+
+// Request a weekday along with a long date
+const options = {
+  year: 'numeric',
+  month: 'long',
+  day: 'numeric',
+} as const;
+
+export default function DatasetPage({
+  dataset,
+}: {
+  dataset: Dataset & {
+    readme: string | null;
+  };
+}) {
+  return (
+    <>
+      <NextSeo title={`${dataset.name} page`} />
+      <Layout>
+        <main className="max-w-5xl px-2 prose mx-auto my-8 pb-8 prose-thead:border-b-4 prose-table:max-w-5xl prose-table:overflow-scroll prose-thead:overflow-scroll prose-tbody:overflow-scroll prose-thead:pb-2 prose-thead:border-zinc-900 prose-th:uppercase prose-th:text-left prose-th:font-light prose-th:text-xs prose-a:no-underline">
+          <Breadcrumbs links={[{ title: dataset.name, href: '' }]} />
+          <h1 className="uppercase mb-0 mt-16">{dataset.name}</h1>
+          <table className="w-full my-10 mb-8 hidden md:table">
+            <thead className="border-b-4 pb-2 border-zinc-900">
+              <tr>
+                <th className="uppercase text-left font-normal text-xs pb-3">
+                  related content
+                </th>
+                <th className="uppercase text-left font-normal text-xs pb-3">
+                  last updated
+                </th>
+              </tr>
+            </thead>
+            <tbody>
+              <DesktopItem key={dataset.name} dataset={dataset} />
+            </tbody>
+          </table>
+
+          {dataset.readme && (
+            <>
+              {dataset.readme && (
+                <ReactMarkdown
+                  remarkPlugins={[
+                    remarkFrontmatter,
+                    remarkGfm,
+                    [extract, { remove: true }],
+                  ]}
+                >
+                  {dataset.readme}
+                </ReactMarkdown>
+              )}
+            </>
+          )}
+
+          <h2 className="mb-0 mt-10">Files</h2>
+          <div className="inline-block min-w-full py-2 align-middle">
+            <table className="min-w-full divide-y divide-gray-300">
+              <thead className="border-b-4 pb-2 border-zinc-900">
+                <tr>
+                  <th
+                    className="uppercase text-left font-light text-xs pb-3"
+                    scope="col"
+                  >
+                    Name
+                  </th>
+                  <th
+                    className="uppercase text-left font-light text-xs pb-3"
+                    scope="col"
+                  >
+                    Download
+                  </th>
+                </tr>
+              </thead>
+              <tbody className="divide-y divide-gray-200">
+                {dataset.files?.map((file) => (
+                  <tr key={file}>
+                    <td className="whitespace-nowrap text-left py-4 text-sm text-gray-500">
+                      <a href={`#${file.split('/').slice(-1)}`}>
+                        {file.split('/').slice(-1)}
+                      </a>
+                    </td>
+                    <td className="whitespace-nowrap py-4 text-sm text-gray-500">
+                      <a href={file}>
+                        <svg
+                          xmlns="http://www.w3.org/2000/svg"
+                          viewBox="0 0 24 24"
+                          fill="currentColor"
+                          className="w-8 h-8 text-blue-400 hover:text-blue-300 transition mt-1 ml-3"
+                        >
+                          <path
+                            fillRule="evenodd"
+                            d="M12 2.25c-5.385 0-9.75 4.365-9.75 9.75s4.365 9.75 9.75 9.75 9.75-4.365 9.75-9.75S17.385 2.25 12 2.25zm-.53 14.03a.75.75 0 001.06 0l3-3a.75.75 0 10-1.06-1.06l-1.72 1.72V8.25a.75.75 0 00-1.5 0v5.69l-1.72-1.72a.75.75 0 00-1.06 1.06l3 3z"
+                            clipRule="evenodd"
+                          />
+                        </svg>
+                      </a>
+                    </td>
+                  </tr>
+                ))}
+              </tbody>
+            </table>
+          </div>
+          {dataset.files && dataset.files.length > 0 && (
+            <>
+              <h2 className="mb-0 mt-8">Data Previews</h2>
+              {dataset.files?.map((file) => (
+                <div
+                  key={file}
+                  id={file.split('/').slice(-1).join('')}
+                  className="preview-table my-8"
+                >
+                  <h3>{file.split('/').slice(-1)}</h3>
+                  <FlatUiTable url={file} />
+                </div>
+              ))}
+            </>
+          )}
+        </main>
+      </Layout>
+    </>
+  );
+}
+
+export function DesktopItem({ dataset }: { dataset: Dataset }) {
+  return (
+    <>
+      {dataset.articles.map((article, index) => (
+        <tr
+          key={article.url}
+          className={`${
+            index === dataset.articles.length - 1 ? 'border-b' : ''
+          } border-zinc-400`}
+        >
+          <td>
+            <a
+              className="py-8 font-bold hover:underline pr-2"
+              href={article.url}
+            >
+              {article.title}
+            </a>
+          </td>
+          <td className="py-8 font-light text-[14px] min-w-[138px] font-mono text-[#999]">
+            {format(article.date).includes('years')
+              ? new Date(article.date).toLocaleString('en-US', options)
+              : format(article.date)}
+          </td>
+          <td className="py-8 text-end">
+            {index === 0 && (
+              <a
+                className="ml-auto border border-zinc-900 font-light px-[25px] py-2.5 text-sm transition hover:bg-zinc-900 hover:text-white"
+                href={dataset.url}
+              >
+                info
+              </a>
+            )}
+          </td>
+        </tr>
+      ))}
+    </>
+  );
+}
+
+export async function getStaticPaths() {
+  const datasetsFile = path.join(process.cwd(), 'datasets.json');
+  const datasets = await fs.readFile(datasetsFile, 'utf8');
+
+  return {
+    paths: JSON.parse(datasets).map((dataset: Dataset) => {
+      return {
+        params: { datasetName: dataset.name },
+      };
+    }),
+    fallback: false, // can also be true or 'blocking'
+  };
+}
+// change href base check datahub-next
+
+export const getStaticProps: GetStaticProps = async ({ params }) => {
+  const datasetsFile = path.join(process.cwd(), 'datasets.json');
+  const datasetsString = await fs.readFile(datasetsFile, 'utf8');
+  const datasets: Dataset[] = JSON.parse(datasetsString);
+  const dataset: Dataset | undefined = datasets.find(
+    (_dataset) => _dataset.name === params?.datasetName
+  );
+  const github_pat = getConfig().serverRuntimeConfig.github_pat;
+  const readmes = await Promise.all(
+    ['/README.md', '/readme.md', '/Readme.md'].map(
+      async (readme) =>
+        await getProjectReadme(
+          'fivethirtyeight',
+          'data',
+          'master',
+          dataset?.name + readme,
+          github_pat
+        )
+    )
+  );
+  const readme = readmes.find((item) => item !== null);
+  if (!readme) console.log('Readme not found for ' + dataset?.name);
+  return {
+    props: {
+      dataset: {
+        ...dataset,
+        readme,
+        files: dataset && dataset.files ? dataset.files : null,
+      },
+    },
+  };
+};

examples/fivethirtyeight/pages/index.tsx

@@ -0,0 +1,198 @@
+import Image from 'next/image';
+import { Inter } from 'next/font/google';
+import { format } from 'timeago.js';
+import { promises as fs } from 'fs';
+import path from 'path';
+import { NextSeo } from 'next-seo';
+import Layout from '@/components/Layout';
+
+const inter = Inter({ subsets: ['latin'] });
+
+export interface Article {
+  date: string;
+  title: string;
+  url: string;
+}
+
+export interface Dataset {
+  url: string;
+  name: string;
+  displayName: string;
+  articles: Article[];
+  files?: string[];
+}
+
+// Request a weekday along with a long date
+const options = {
+  year: 'numeric',
+  month: 'long',
+  day: 'numeric',
+} as const;
+
+export function MobileItem({ dataset }: { dataset: Dataset }) {
+  return (
+    <div className="flex gap-x-2 pb-2 py-4 items-center justify-between border-b border-zinc-600">
+      <div className="flex flex-col">
+        <span className="font-mono font-light">{dataset.name}</span>
+        {dataset.articles.map((article) => (
+          <div key={article.title} className="py-1 flex flex-col">
+            <span className="font-bold hover:underline">{article.title}</span>
+            <span className="font-light text-base">
+              {format(article.date).includes('years')
+                ? new Date(article.date).toLocaleString('en-US', options)
+                : format(article.date)}
+            </span>{' '}
+          </div>
+        ))}
+      </div>
+      <div className="flex flex-col justify-start">
+        <a
+          className="ml-2 border border-zinc-900 font-light px-4 py-1 text-sm transition hover:bg-zinc-900 hover:text-white"
+          href={dataset.url}
+        >
+          info
+        </a>
+        <a
+          className="ml-2 border border-[#3c3c3c] px-[25px] py-2.5 text-sm transition bg-[#3c3c3c] text-white hover:bg-zinc-900"
+          href={`/datasets/${dataset.name}`}
+        >
+          explore
+        </a>
+      </div>
+    </div>
+  );
+}
+
+export function DesktopItem({ dataset }: { dataset: Dataset }) {
+  return (
+    <>
+      {dataset.articles.map((article, index) => (
+        <tr
+          key={article.url}
+          className={`${
+            index === dataset.articles.length - 1 ? 'border-b' : ''
+          } border-zinc-400`}
+        >
+          <td className="py-8 font-light font-mono text-[13px] text-zinc-700">
+            {index === 0 ? dataset.name : ''}
+          </td>
+          <td>
+            <a
+              className="py-8 font-bold hover:underline pr-2"
+              href={article.url}
+            >
+              {article.title}
+            </a>
+          </td>
+          <td className="py-8 font-light text-[14px] min-w-[138px] font-mono text-[#999]">
+            {format(article.date).includes('years')
+              ? new Date(article.date).toLocaleString('en-US', options)
+              : format(article.date)}
+          </td>
+          <td>
+            {index === 0 && (
+            <a
+              className="ml-2 border border-[#3c3c3c] px-[25px] py-2.5 text-sm transition bg-[#3c3c3c] text-white hover:bg-zinc-900"
+              href={`/datasets/${dataset.name}`}
+            >
+              explore
+            </a>
+            )}
+          </td>
+          <td className="py-8">
+            {index === 0 && (
+              <a
+                className="ml-2 border border-zinc-900 font-light px-[25px] py-2.5 text-sm transition hover:bg-zinc-900 hover:text-white"
+                href={dataset.url}
+              >
+                info
+              </a>
+            )}
+          </td>
+        </tr>
+      ))}
+    </>
+  );
+}
+
+export async function getStaticProps() {
+  const jsonDirectory = path.join(process.cwd(), '/datasets.json');
+  const datasetString = await fs.readFile(jsonDirectory, 'utf8');
+  const datasets = JSON.parse(datasetString);
+  return {
+    props: { datasets },
+  };
+}
+
+export default function Home({ datasets }: { datasets: Dataset[] }) {
+  return (
+    <>
+      <NextSeo title="FiveThirtyEight tribute by PortalJS" />
+      <Layout>
+      <main
+        className={`flex min-h-screen flex-col items-center max-w-5xl mx-auto pt-20 px-2.5 ${inter.className}`}
+      >
+        <div>
+          <h1 className="text-[40px] font-bold text-zinc-800 text-center">
+            Our Data
+          </h1>
+          <p className="max-w-[600px] text-[17px] text-center text-[#6d6f71]">
+            We’re sharing the data and code behind some of our articles and
+            graphics. We hope you’ll use it to check our work and to create
+            stories and visualizations of&nbsp;your&nbsp;own.
+          </p>
+        </div>
+        <article className="w-full px-2 md:hidden py-4">
+          {datasets.map((dataset) => (
+            <MobileItem key={dataset.name} dataset={dataset} />
+          ))}
+        </article>
+        <table className="w-full mt-10 mb-4 hidden md:table">
+          <thead className="border-b-4 pb-2 border-zinc-900">
+            <tr>
+              <th className="uppercase text-left font-normal text-xs pb-3">
+                data set
+              </th>
+              <th className="uppercase text-left font-normal text-xs pb-3">
+                related content
+              </th>
+              <th className="uppercase text-left font-normal text-xs pb-3">
+                last updated
+              </th>
+            </tr>
+          </thead>
+          <tbody>
+            {datasets.map((dataset) => (
+              <DesktopItem key={dataset.name} dataset={dataset} />
+            ))}
+          </tbody>
+        </table>
+        <p className="text-[13px] py-8">
+          Unless otherwise noted, our data sets are available under the{' '}
+          <a
+            className="text-blue-400 hover:underline"
+            href="http://creativecommons.org/licenses/by/4.0/"
+          >
+            Creative Commons Attribution 4.0 International license
+          </a>
+          , and the code is available under the{' '}
+          <a
+            className="text-blue-400 hover:underline"
+            href="http://opensource.org/licenses/MIT"
+          >
+            MIT license
+          </a>
+          . If you find this information useful, please{' '}
+          <a
+            className="text-blue-400 hover:underline"
+            href="mailto:data@fivethirtyeight.com"
+          >
+            let us know
+          </a>
+          .
+        </p>
+      </main>
+      </Layout>
+    </>
+  );
+}

examples/fivethirtyeight/public/next.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>

examples/fivethirtyeight/public/vercel.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 283 64"><path fill="black" d="M141 16c-11 0-19 7-19 18s9 18 20 18c7 0 13-3 16-7l-7-5c-2 3-6 4-9 4-5 0-9-3-10-7h28v-3c0-11-8-18-19-18zm-9 15c1-4 4-7 9-7s8 3 9 7h-18zm117-15c-11 0-19 7-19 18s9 18 20 18c6 0 12-3 16-7l-8-5c-2 3-5 4-8 4-5 0-9-3-11-7h28l1-3c0-11-8-18-19-18zm-10 15c2-4 5-7 10-7s8 3 9 7h-19zm-39 3c0 6 4 10 10 10 4 0 7-2 9-5l8 5c-3 5-9 8-17 8-11 0-19-7-19-18s8-18 19-18c8 0 14 3 17 8l-8 5c-2-3-5-5-9-5-6 0-10 4-10 10zm83-29v46h-9V5h9zM37 0l37 64H0L37 0zm92 5-27 48L74 5h10l18 30 17-30h10zm59 12v10l-3-1c-6 0-10 4-10 10v15h-9V17h9v9c0-5 6-9 13-9z"/></svg>

examples/fivethirtyeight/styles/globals.css

@@ -0,0 +1,11 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+
+.preview-table > div {
+  overflow-y: hidden;
+}
+
+.prose h1 {
+  font-size: 1.5em !important;
+}

examples/fivethirtyeight/tailwind.config.js

@@ -0,0 +1,18 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    './pages/**/*.{js,ts,jsx,tsx,mdx}',
+    './components/**/*.{js,ts,jsx,tsx,mdx}',
+    './app/**/*.{js,ts,jsx,tsx,mdx}',
+  ],
+  theme: {
+    extend: {
+      backgroundImage: {
+        'gradient-radial': 'radial-gradient(var(--tw-gradient-stops))',
+        'gradient-conic':
+          'conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))',
+      },
+    },
+  },
+  plugins: [require('@tailwindcss/typography')],
+};

examples/fivethirtyeight/tsconfig.json

@@ -0,0 +1,23 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "forceConsistentCasingInFileNames": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "preserve",
+    "incremental": true,
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
+  "exclude": ["node_modules"]
+}

examples/github-backed-catalog/.eslintrc.json

@@ -0,0 +1,29 @@
+{
+  "extends": ["next", "next/core-web-vitals"],
+  "ignorePatterns": ["!**/*", ".next/**/*"],
+  "overrides": [
+    {
+      "files": ["*.ts", "*.tsx", "*.js", "*.jsx"],
+      "rules": {
+        "@next/next/no-html-link-for-pages": [
+          "error",
+          "examples/simple-example/pages"
+        ]
+      }
+    },
+    {
+      "files": ["*.ts", "*.tsx"],
+      "rules": {}
+    },
+    {
+      "files": ["*.js", "*.jsx"],
+      "rules": {}
+    }
+  ],
+  "rules": {
+    "@next/next/no-html-link-for-pages": "off"
+  },
+  "env": {
+    "jest": true
+  }
+}

examples/github-backed-catalog/.prettierignore

@@ -0,0 +1,7 @@
+node_modules
+**/.next/**
+**/_next/**
+**/dist/**
+**/__tmp__/**
+lerna.json
+.github

examples/github-backed-catalog/.prettierrc.json

@@ -0,0 +1 @@
+{}

examples/github-backed-catalog/README.md

@@ -0,0 +1,101 @@
+# A data catalog with data on GitHub
+
+This example showcases a simple data catalog that get its data from a list of GitHub repos that serve as datasets.
+
+A `datasets.json` file is used to specify which datasets are going to be part of the data catalog.
+
+The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
+
+You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
+
+## Demo
+
+https://example.portaljs.org/
+
+## Deploy your own
+
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fgithub-backed-catalog)
+
+By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own GitHub/GitLab/Bitbucket account and automatically deploy everything.
+
+## How to use
+
+### Install
+
+Execute `create-next-app` to bootstrap the example:
+
+```
+npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/github-backed-catalog
+cd <app-name>
+```
+
+### Set environment variables
+
+This project uses the GitHub API, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
+
+```
+GITHUB_PAT=<github token>
+```
+
+### Change datasets
+
+You can change the datasets that will be displayed in the data catalog by editing the file `datasets.json`. Some examples can be found inside [this repo](https://github.com/datasets).
+
+### Run in development mode
+
+Run the app using:
+
+```
+npm run dev
+```
+
+Open http://localhost:3000 from your browser. You should see something similar to this:
+
+![](https://i.imgur.com/jAljJ9C.png)
+
+If click on the `info` button for a dataset you will see a page similar to this:
+
+![](https://i.imgur.com/AoJd4O0.png)
+
+## Notes
+
+### Structure of `datasets.json`
+
+The `datasets.json` file is simply a list of datasets, below you can see a minimal example of a dataset:
+
+```json
+{
+  "owner": "fivethirtyeight",
+  "repo": "data",
+  "branch": "master",
+  "files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
+  "readme": "nba-raptor/README.md"
+}
+```
+
+It has:
+
+- A `owner` which is going to be the github repo owner
+- A `repo` which is going to be the github repo name
+- A `branch` which is going to be the branch to which we need to get the files and the readme
+- A list of `files` which is going to be a list of paths with files that you want to show to the world
+- A `readme` which is going to be the path to your data description, it can also be a subpath eg: `example/README.md`
+
+You can also add:
+
+- A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
+- A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
+
+### Extra commands
+
+You can also build the project for production with:
+
+```
+npm run build
+```
+
+And run the production build with:
+
+```
+npm run start
+```

examples/github-backed-catalog/components/DataRichDocument.tsx

@@ -0,0 +1,21 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@flowershow/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
+  Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
+  FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
+  mermaid: Mermaid,
+  Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
+  VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),
+  LineChart: dynamic(() => import('@portaljs/components').then(mod => mod.LineChart)),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/github-backed-catalog/components/_shared/Breadcrumbs.tsx

@@ -0,0 +1,28 @@
+import Link from "next/link";
+import HomeIcon from "../icons/HomeIcon";
+
+export default function Breadcrumbs({
+  links,
+}: {
+  links: { title: string; href?: string; target?: string }[];
+}) {
+  const current = links.at(-1);
+
+  return (
+    <div className="flex items-center uppercase font-black text-xs">
+      <Link className="flex items-center" href="/">
+        <HomeIcon />
+      </Link>
+
+      {/* {links.length > 1 && links.slice(0, -1).map((link) => {
+      return <>
+        <span className="mx-4">/</span>
+        <Link href={link.href}>{link.title}</Link>
+      </>
+    })} */}
+
+      <span className="mx-4">/</span>
+      <span>{current.title}</span>
+    </div>
+  );
+}

examples/github-backed-catalog/components/icons/ExternalLinkIcon.tsx

@@ -0,0 +1,13 @@
+export default function ExternalLinkIcon({ className = "" }) {
+  return (
+    <div className={`inline-block w-4 ${className}`}>
+      <svg
+        xmlns="http://www.w3.org/2000/svg"
+        viewBox="0 0 64 64"
+        fill="currentColor"
+      >
+        <path d="M 40 10 C 38.896 10 38 10.896 38 12 C 38 13.104 38.896 14 40 14 L 47.171875 14 L 30.585938 30.585938 C 29.804938 31.366938 29.804938 32.633063 30.585938 33.414062 C 30.976938 33.805063 31.488 34 32 34 C 32.512 34 33.023063 33.805062 33.414062 33.414062 L 50 16.828125 L 50 24 C 50 25.104 50.896 26 52 26 C 53.104 26 54 25.104 54 24 L 54 12 C 54 10.896 53.104 10 52 10 L 40 10 z M 18 12 C 14.691 12 12 14.691 12 18 L 12 46 C 12 49.309 14.691 52 18 52 L 46 52 C 49.309 52 52 49.309 52 46 L 52 34 C 52 32.896 51.104 32 50 32 C 48.896 32 48 32.896 48 34 L 48 46 C 48 47.103 47.103 48 46 48 L 18 48 C 16.897 48 16 47.103 16 46 L 16 18 C 16 16.897 16.897 16 18 16 L 30 16 C 31.104 16 32 15.104 32 14 C 32 12.896 31.104 12 30 12 L 18 12 z" />
+      </svg>
+    </div>
+  );
+}

examples/github-backed-catalog/components/icons/HomeIcon.tsx

@@ -0,0 +1,10 @@
+export default function HomeIcon({ className = "" }) {
+  return (
+    <div className={`inline-block w-4 ${className}`}>
+      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
+        {" "}
+        <path d="M 12 2 A 1 1 0 0 0 11.289062 2.296875 L 1.203125 11.097656 A 0.5 0.5 0 0 0 1 11.5 A 0.5 0.5 0 0 0 1.5 12 L 4 12 L 4 20 C 4 20.552 4.448 21 5 21 L 9 21 C 9.552 21 10 20.552 10 20 L 10 14 L 14 14 L 14 20 C 14 20.552 14.448 21 15 21 L 19 21 C 19.552 21 20 20.552 20 20 L 20 12 L 22.5 12 A 0.5 0.5 0 0 0 23 11.5 A 0.5 0.5 0 0 0 22.796875 11.097656 L 12.716797 2.3027344 A 1 1 0 0 0 12.710938 2.296875 A 1 1 0 0 0 12 2 z" />
+      </svg>
+    </div>
+  );
+}

examples/github-backed-catalog/datasets.json

@@ -15,14 +15,18 @@
     ],
     "readme": "README.md"
   },
+  {
+    "owner": "luccasmmg",
+    "branch": "main",
+    "repo": "test-data-repo-1",
+    "files": ["data_1.csv", "data_2.csv"],
+    "readme": "README.md"
+  },
   {
     "owner": "datasets",
     "branch": "main",
     "repo": "investor-flow-of-funds-us",
-    "files": [
+    "files": ["data/monthly.csv", "data/weekly.csv"],
-      "data/monthly.csv",
-      "data/weekly.csv"
-    ],
     "readme": "README.md"
   },
   {
@@ -38,7 +42,10 @@
     "owner": "fivethirtyeight",
     "repo": "data",
     "branch": "master",
-    "files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
+    "files": [
+      "nba-raptor/historical_RAPTOR_by_player.csv",
+      "nba-raptor/historical_RAPTOR_by_team.csv"
+    ],
     "readme": "nba-raptor/README.md"
   }
 ]

examples/github-backed-catalog/index.d.ts

@@ -0,0 +1,6 @@
+/* eslint-disable @typescript-eslint/no-explicit-any */
+declare module "*.svg" {
+  const content: any;
+  export const ReactComponent: any;
+  export default content;
+}

examples/github-backed-catalog/lib/markdown.js

@@ -22,7 +22,7 @@ import { serialize } from "next-mdx-remote/serialize";
  * @format: used to indicate to next-mdx-remote which format to use (md or mdx)
  * @returns: { mdxSource: mdxSource, frontMatter: ...}
  */
-const parse = async function (source, format) {
+const parse = async function (source, format, scope) {
   const { content, data, excerpt } = matter(source, {
     excerpt: (file, options) => {
       // Generate an excerpt for the file
@@ -91,7 +91,7 @@ const parse = async function (source, format) {
         ],
         format,
       },
-      scope: data,
+      scope,
     }
   );
 

examples/github-backed-catalog/lib/mddb.ts

@@ -0,0 +1,14 @@
+import { MarkdownDB } from "@flowershow/markdowndb";
+
+const dbPath = "markdown.db";
+
+const client = new MarkdownDB({
+  client: "sqlite3",
+  connection: {
+    filename: dbPath,
+  },
+});
+
+const clientPromise = client.init();
+
+export default clientPromise;

examples/github-backed-catalog/lib/octokit.ts

@@ -0,0 +1,207 @@
+import { Octokit } from "octokit";
+
+export interface GithubProject {
+  owner: string;
+  repo: string;
+  branch: string;
+  files: string[];
+  readme: string;
+  description?: string;
+  name?: string;
+}
+
+export async function getProjectReadme(
+  owner: string,
+  repo: string,
+  branch: string,
+  readme: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.getContent({
+      owner,
+      repo,
+      path: readme,
+      ref: branch,
+    });
+    const data = response.data as { content?: string };
+    const fileContent = data.content ? data.content : "";
+    if (fileContent === "") {
+      return null;
+    }
+    const decodedContent = Buffer.from(fileContent, "base64").toString();
+    return decodedContent;
+  } catch (error) {
+    throw new Error(
+      "Couldn't get project readme please make sure that you are pointing to a valid repo and that the repo in question contains a README.md"
+    );
+  }
+}
+
+export async function getProjectDatapackage(
+  owner: string,
+  repo: string,
+  branch: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.getContent({
+      owner,
+      repo,
+      path: "datapackage.json",
+      ref: branch,
+    });
+    const data = response.data as { content?: string };
+    const fileContent = data.content ? data.content : "";
+    if (fileContent === "") {
+      return null;
+    }
+    const decodedContent = Buffer.from(fileContent, "base64").toString();
+    return JSON.parse(decodedContent);
+  } catch (error) {
+    return null
+  }
+}
+
+export async function getLastUpdated(
+  owner: string,
+  repo: string,
+  branch: string,
+  readme: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.listCommits({
+      owner,
+      repo,
+      ref: branch,
+    });
+    return response.data[0].commit.committer.date;
+  } catch (error) {
+    throw new Error(
+      "Couldn't get project list of commits please make sure that you are pointing to a valid repo"
+    );
+  }
+}
+export async function getProjectMetadata(
+  owner: string,
+  repo: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.get({
+      owner,
+      repo,
+    });
+    return response.data;
+  } catch (error) {
+    throw new Error(
+      "Couldn't get project metadata please make sure that you are pointing to a valid repo"
+    );
+  }
+}
+
+export async function getRepoContents(
+  owner: string,
+  repo: string,
+  branch: string,
+  files: string[],
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const contents = [];
+    for (const path of files) {
+      const response = await octokit.rest.repos.getContent({
+        owner,
+        repo,
+        ref: branch,
+        path: path,
+      });
+      const data = response.data as {
+        download_url?: string;
+        name: string;
+        size: number;
+      };
+      contents.push({
+        download_url: data.download_url,
+        name: data.name,
+        size: data.size,
+      });
+    }
+    return contents;
+  } catch (error) {
+    if (
+      error.message ===
+      'This endpoint can only return blobs smaller than 100 MB in size. The requested blob is too large to fetch via the API, but you can always clone the repository via Git to obtain it.: {"resource":"Blob","field":"data","code":"too_large"}'
+    ) {
+      throw new Error(
+        `The requested files ${files.join(
+          ", "
+        )} are too big making it impossible to fetch via Github API`
+      );
+    }
+    throw new Error(
+      "Couldn't get project contents please make sure that you are pointing to a valid repo"
+    );
+  }
+}
+
+export async function getProject(project: GithubProject, github_pat?: string) {
+  const projectMetadata = await getProjectMetadata(
+    project.owner,
+    project.repo,
+    github_pat
+  );
+  if (!projectMetadata) {
+    return null;
+  }
+  const projectReadme = await getProjectReadme(
+    project.owner,
+    project.repo,
+    project.branch,
+    project.readme,
+    github_pat
+  );
+  let projectData = [];
+  if (project.files) {
+    projectData = await getRepoContents(
+      project.owner,
+      project.repo,
+      project.branch,
+      project.files,
+      github_pat
+    );
+  }
+  const projectBase =
+    project.readme && project.readme.split("/").length > 1
+      ? project.readme.split("/").slice(0, -1).join("/")
+      : "/";
+  const last_updated = await getLastUpdated(
+    project.owner,
+    project.repo,
+    project.branch,
+    projectBase,
+    github_pat
+  );
+
+  const projectDatapackage = await getProjectDatapackage(
+    project.owner,
+    project.repo,
+    project.branch,
+    github_pat
+  );
+
+  return {
+    ...projectMetadata,
+    files: projectData,
+    readmeContent: projectReadme,
+    last_updated,
+    base_path: projectBase,
+    datapackage: projectDatapackage
+  };
+}

examples/github-backed-catalog/next.config.js

@@ -0,0 +1,17 @@
+const nextConfig = {
+  async rewrites() {
+    return {
+      beforeFiles: [
+        {
+          source: "/@:org/:project*",
+          destination: "/@org/:org/:project*",
+        },
+      ],
+    };
+  },
+  serverRuntimeConfig: {
+    github_pat: process.env.GITHUB_PAT ? process.env.GITHUB_PAT : null,
+  },
+};
+
+module.exports = nextConfig;

examples/github-backed-catalog/package.json

@@ -6,27 +6,43 @@
     "dev": "next dev",
     "build": "next build",
     "start": "next start",
-    "lint": "next lint"
+    "lint": "next lint",
+    "prettier": "prettier --write ."
   },
   "dependencies": {
+    "@flowershow/core": "^0.4.13",
+    "@flowershow/markdowndb": "^0.1.5",
+    "@flowershow/remark-callouts": "^1.0.0",
+    "@flowershow/remark-embed": "^1.0.0",
+    "@portaljs/components": "^0.1.6",
     "@types/node": "18.16.0",
     "@types/react": "18.0.38",
     "@types/react-dom": "18.0.11",
     "eslint": "8.39.0",
     "eslint-config-next": "13.3.1",
-    "next": "13.3.1",
+    "next": "13.4.3",
+    "next-mdx-remote": "^4.4.1",
     "next-seo": "^6.0.0",
     "octokit": "^2.0.14",
     "react": "18.2.0",
     "react-dom": "18.2.0",
     "react-markdown": "^8.0.7",
+    "react-timeago": "^7.1.0",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
     "remark-gfm": "^3.0.1",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
     "typescript": "5.0.4"
   },
   "devDependencies": {
     "@tailwindcss/typography": "^0.5.9",
     "autoprefixer": "^10.4.14",
     "postcss": "^8.4.23",
+    "prettier": "2.8.8",
     "tailwindcss": "^3.3.1"
   }
 }

examples/github-backed-catalog/pages/@org/[org]/[...path].tsx

@@ -1,22 +1,35 @@
-import Head from 'next/head';
+import { NextSeo } from "next-seo";
-import { useRouter } from 'next/router';
+import { promises as fs } from "fs";
-
+import path from "path";
-import { NextSeo } from 'next-seo';
+import getConfig from "next/config";
-import { promises as fs } from 'fs';
+import { getProject, GithubProject } from "../../../lib/octokit";
-import path from 'path';
+import ReactMarkdown from "react-markdown";
-import getConfig from 'next/config';
+import remarkGfm from "remark-gfm";
-import { getProject, GithubProject } from '../../../lib/octokit';
+import Breadcrumbs from "../../../components/_shared/Breadcrumbs";
-import ReactMarkdown from 'react-markdown';
+import parse from '../../../lib/markdown';
-import remarkGfm from 'remark-gfm';
+import DataRichDocument from '../../../components/DataRichDocument'
-import Link from 'next/link';
 
 export default function ProjectPage({ project }) {
+  const repoId = `@${project.repo_config.owner}/${project.repo_config.repo}`;
+
   return (
     <>
-      <NextSeo title={`PortalJS - @${project.repo_config.owner}/${project.repo_config.repo}${project.base_path !== '/' ? '/' + project.base_path : ''}`} />
+      <NextSeo
+        title={`${repoId}${
+          project.base_path !== "/" ? "/" + project.base_path : ""
+        } - GitHub Datasets`}
+      />
       <main className="prose mx-auto my-8">
-        <Link href='/'>Back to homepage</Link>
+        <Breadcrumbs links={[{ title: repoId, href: "" }]} />
-        <h1 className="mb-0">Data</h1>
+        <h1 className="mb-0 mt-16">{project.repo_config.name || repoId}</h1>
+        <p className="mb-8">
+          <span className="font-semibold">Repository:</span>{" "}
+          <a target="_blank" href={project.html_url}>
+            {project.html_url}
+          </a>
+        </p>
+
+        <h2 className="mb-0 mt-10">Files</h2>
         <div className="inline-block min-w-full py-2 align-middle">
           <table className="min-w-full divide-y divide-gray-300">
             <thead>
@@ -50,10 +63,10 @@ export default function ProjectPage({ project }) {
           </table>
         </div>
 
-        <h1>Readme</h1>
+        <hr />
-        <ReactMarkdown remarkPlugins={[remarkGfm]}>
+
-          {project.readmeContent}
+        <h2 className="uppercase font-black">Readme</h2>
-        </ReactMarkdown>
+        <DataRichDocument source={project.mdxSource} />
       </main>
     </>
   );
@@ -61,17 +74,14 @@ export default function ProjectPage({ project }) {
 
 // Generates `/posts/1` and `/posts/2`
 export async function getStaticPaths() {
-  const jsonDirectory = path.join(
+  const jsonDirectory = path.join(process.cwd(), "datasets.json");
-    process.cwd(),
+  const repos = await fs.readFile(jsonDirectory, "utf8");
-    'datasets.json'
-  );
-  const repos = await fs.readFile(jsonDirectory, 'utf8');
 
   return {
     paths: JSON.parse(repos).map((repo) => {
       const projectPath =
-        repo.readme.split('/').length > 1
+        repo.readme && repo.readme.split("/").length > 1
-          ? repo.readme.split('/').slice(0, -1)
+          ? repo.readme.split("/").slice(0, -1)
           : null;
       let path = [repo.repo];
       if (projectPath) {
@@ -88,16 +98,13 @@ export async function getStaticPaths() {
 }
 
 export async function getStaticProps({ params }) {
-  const jsonDirectory = path.join(
+  const jsonDirectory = path.join(process.cwd(), "datasets.json");
-    process.cwd(),
+  const reposFile = await fs.readFile(jsonDirectory, "utf8");
-    'datasets.json'
-  );
-  const reposFile = await fs.readFile(jsonDirectory, 'utf8');
   const repos: GithubProject[] = JSON.parse(reposFile);
   const repo = repos.find((_repo) => {
     const projectPath =
-      _repo.readme.split('/').length > 1
+      _repo.readme && _repo.readme.split("/").length > 1
-        ? _repo.readme.split('/').slice(0, -1)
+        ? _repo.readme.split("/").slice(0, -1)
         : null;
     let path = [_repo.repo];
     if (projectPath) {
@@ -112,9 +119,10 @@ export async function getStaticProps({ params }) {
   });
   const github_pat = getConfig().serverRuntimeConfig.github_pat;
   const project = await getProject(repo, github_pat);
+  let { mdxSource, frontMatter } = await parse(project.readmeContent, '.mdx', { project });
   return {
     props: {
-      project: { ...project, repo_config: repo },
+      project: { ...project, repo_config: repo, mdxSource },
     },
   };
 }

examples/github-backed-catalog/pages/_app.tsx

@@ -1,12 +1,12 @@
-import { AppProps } from 'next/app';
+import { AppProps } from "next/app";
-import Head from 'next/head';
+import Head from "next/head";
-import './styles.css';
+import "./styles.css";
 
 function CustomApp({ Component, pageProps }: AppProps) {
   return (
     <>
       <Head>
-        <title>Welcome to simple-example!</title>
+        <title>GitHub Datasets</title>
       </Head>
       <main className="app">
         <Component {...pageProps} />

examples/github-backed-catalog/pages/index.tsx

@@ -0,0 +1,139 @@
+import { promises as fs } from "fs";
+import path from "path";
+import { getProject } from "../lib/octokit";
+import getConfig from "next/config";
+import ExternalLinkIcon from "../components/icons/ExternalLinkIcon";
+import TimeAgo from "react-timeago";
+import Link from "next/link";
+import { NextSeo } from "next-seo";
+
+export async function getStaticProps() {
+  const jsonDirectory = path.join(process.cwd(), "/datasets.json");
+  const repos = await fs.readFile(jsonDirectory, "utf8");
+  const github_pat = getConfig().serverRuntimeConfig.github_pat;
+
+  const projects = await Promise.all(
+    JSON.parse(repos).map(async (repo) => {
+      const project = await getProject(repo, github_pat);
+      return { ...project, repo_config: repo };
+    })
+  );
+  return {
+    props: {
+      projects,
+    },
+  };
+}
+
+export function Datasets({ projects }) {
+  return (
+    <>
+      <NextSeo title="GitHub Datasets" />
+      <div className="bg-white min-h-screen">
+        <div className="mx-auto max-w-7xl px-6 py-16 sm:py-24 lg:px-8">
+          <div className="text-center">
+            <h2 className="text-3xl font-bold leading-10 tracking-tight">
+              GitHub Datasets
+            </h2>
+            <p className="mt-3 mx-auto max-w-2xl text-base leading-7 text-gray-500">
+              Data catalog with datasets hosted on GitHub by{" "}
+              <Link
+                target="_blank"
+                className="underline"
+                href="https://portaljs.org/"
+              >
+                🌀 PortalJS
+              </Link>
+            </p>
+          </div>
+          <div className="mt-20">
+            <div className="-mx-4 -my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
+              <div className="inline-block min-w-full py-2 align-middle sm:px-6 lg:px-8">
+                <table className="min-w-full divide-y divide-gray-300">
+                  <thead>
+                    <tr>
+                      <th
+                        scope="col"
+                        className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                      >
+                        Name
+                      </th>
+                      <th
+                        scope="col"
+                        className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                      >
+                        Repository
+                      </th>
+                      <th
+                        scope="col"
+                        className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                      >
+                        Description
+                      </th>
+                      <th
+                        scope="col"
+                        className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
+                      >
+                        Last updated
+                      </th>
+                      <th
+                        scope="col"
+                        className="relative py-3.5 pl-3 pr-4 sm:pr-0"
+                      ></th>
+                    </tr>
+                  </thead>
+                  <tbody className="divide-y divide-gray-200">
+                    {projects.map((project) => (
+                      <tr key={project.id}>
+                        <td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
+                          {project.repo_config.name
+                            ? project.repo_config.name
+                            : project.full_name +
+                              (project.base_path === "/"
+                                ? ""
+                                : "/" + project.base_path)}
+                        </td>
+                        <td className="whitespace-nowrap px-3 py-6 text-sm group text-gray-500 hover:text-gray-900 transition-all duration-250">
+                          <a
+                            href={project.html_url}
+                            target="_blank"
+                            className="flex items-center"
+                          >
+                            @{project.full_name}{" "}
+                            <ExternalLinkIcon className="ml-1" />
+                          </a>
+                        </td>
+                        <td className="px-3 py-4 text-sm text-gray-500">
+                          {project.repo_config.description
+                            ? project.repo_config.description
+                            : project.description}
+                        </td>
+                        <td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
+                          <TimeAgo date={new Date(project.last_updated)} />
+                        </td>
+                        <td className="relative whitespace-nowrap py-6 pl-3 pr-4 text-right text-sm font-medium sm:pr-0">
+                          <a
+                            href={`/@${project.repo_config.owner}/${
+                              project.repo_config.repo
+                            }/${
+                              project.base_path === "/" ? "" : project.base_path
+                            }`}
+                            className="border border-gray-900 text-gray-900 px-4 py-2 transition-all hover:bg-gray-900 hover:text-white"
+                          >
+                            info
+                          </a>
+                        </td>
+                      </tr>
+                    ))}
+                  </tbody>
+                </table>
+              </div>
+            </div>
+          </div>
+        </div>
+      </div>
+    </>
+  );
+}
+
+export default Datasets;

examples/github-backed-catalog/pages/styles.css

@@ -0,0 +1,149 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+html {
+  -webkit-text-size-adjust: 100%;
+  font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont,
+    Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif,
+    Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;
+  line-height: 1.5;
+  tab-size: 4;
+  scroll-behavior: smooth;
+}
+body {
+  font-family: inherit;
+  line-height: inherit;
+  margin: 0;
+}
+h1,
+h2,
+p,
+pre {
+  margin: 0;
+}
+*,
+::before,
+::after {
+  box-sizing: border-box;
+  border-width: 0;
+  border-style: solid;
+  border-color: currentColor;
+}
+h1,
+h2 {
+  font-size: inherit;
+  font-weight: inherit;
+}
+a {
+  color: inherit;
+  text-decoration: inherit;
+}
+pre {
+  font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
+    Liberation Mono, Courier New, monospace;
+}
+svg {
+  display: block;
+  vertical-align: middle;
+  shape-rendering: auto;
+  text-rendering: optimizeLegibility;
+}
+pre {
+  background-color: rgba(55, 65, 81, 1);
+  border-radius: 0.25rem;
+  color: rgba(229, 231, 235, 1);
+  font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
+    Liberation Mono, Courier New, monospace;
+  overflow: scroll;
+  padding: 0.5rem 0.75rem;
+}
+
+.shadow {
+  box-shadow: 0 0 #0000, 0 0 #0000, 0 10px 15px -3px rgba(0, 0, 0, 0.1),
+    0 4px 6px -2px rgba(0, 0, 0, 0.05);
+}
+.rounded {
+  border-radius: 1.5rem;
+}
+.wrapper {
+  width: 100%;
+}
+.container {
+  margin-left: auto;
+  margin-right: auto;
+  max-width: 768px;
+  padding-bottom: 3rem;
+  padding-left: 1rem;
+  padding-right: 1rem;
+  color: rgba(55, 65, 81, 1);
+  width: 100%;
+}
+
+@import "@flowershow/remark-callouts/styles.css";
+
+/* mathjax */
+.math-inline > mjx-container > svg {
+  display: inline;
+  align-items: center;
+}
+
+/* smooth scrolling in modern browsers */
+html {
+  scroll-behavior: smooth !important;
+}
+
+/* tooltip fade-out clip */
+.tooltip-body::after {
+  content: "";
+  position: absolute;
+  right: 0;
+  top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
+  height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
+  width: 10rem;
+  background: linear-gradient(
+    to right,
+    rgba(255, 255, 255, 0),
+    rgba(255, 255, 255, 1) 100%
+  );
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
+  margin-left: -2rem !important;
+  padding-left: 2rem !important;
+  scroll-margin-top: 4.5rem;
+  position: relative;
+}
+
+.heading-link {
+  padding: 1px;
+  position: absolute;
+  left: 0;
+  top: 50%;
+  transform: translateY(-50%);
+  margin: auto 0;
+  border-radius: 5px;
+  background: #1e293b;
+  opacity: 0;
+  transition: opacity 0.2s;
+}
+
+.light .heading-link {
+  /* border: 1px solid #ab2b65; */
+  /* background: none; */
+  background: #e2e8f0;
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
+  opacity: 100;
+}
+
+.heading-link svg {
+  transform: scale(0.75);
+}
+
+@media screen and (max-width: 640px) {
+  .heading-link {
+    visibility: hidden;
+  }
+}
+

examples/github-backed-catalog/postcss.config.js

@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+};

examples/github-backed-catalog/tailwind.config.js

@@ -8,8 +8,5 @@ module.exports = {
   theme: {
     extend: {},
   },
-  plugins: [
+  plugins: [require("@tailwindcss/typography")],
-    require('@tailwindcss/typography')
+};
-  ],
-}
-