Update packages/remark-wiki-link/src/lib/remarkWikiLink.ts

[md][datahub] updated links from portaljs.org to portaljs.com

.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/config.json

@@ -0,0 +1,14 @@
+{
+  "$schema": "https://unpkg.com/@changesets/config@2.3.0/schema.json",
+  "changelog": [
+    "@changesets/changelog-github",
+    { "repo": "datopian/portaljs" }
+  ],
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "restricted",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

.github/workflows/release.yml

@@ -0,0 +1,39 @@
+name: Release
+
+on:
+  push:
+    branches:
+      - main
+
+concurrency: release-${{ github.ref }}
+
+jobs:
+  release:
+    name: Release
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v3
+
+      - name: Setup Node.js 16.x
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16.x
+
+      - name: Install Dependencies
+        run: npm ci
+
+      - name: Create Release Pull Request or Publish to npm
+        id: changesets
+        uses: changesets/action@v1
+        with:
+          publish: npm run release
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+      # - name: Send a Discord notification if a publish happens
+      #   if: steps.changesets.outputs.published == 'true'
+      #   uses: Ilshidur/action-discord@0.3.2
+      #   with:
+      #     args: 'The project {{ EVENT_PAYLOAD.repository.full_name }} has been deployed.'

.gitignore

@@ -4,6 +4,7 @@
 dist
 tmp
 /out-tsc
+**/*.tgz
 
 # dependencies
 node_modules
@@ -16,6 +17,7 @@ node_modules
 *.launch
 .settings/
 *.sublime-workspace
+.obsidian
 
 # IDE - VSCode
 .vscode/*

.vscode/extensions.json

@@ -1,8 +0,0 @@
-{
-  "recommendations": [
-    "nrwl.angular-console",
-    "esbenp.prettier-vscode",
-    "firsttris.vscode-jest-runner",
-    "dbaeumer.vscode-eslint"
-  ]
-}

CONTRIBUTING.md

@@ -4,7 +4,7 @@ title: Developer docs for contributors
 
 ## Our repository
 
-https://github.com/datopian/portaljs
+https://github.com/datopian/datahub
 
 Structure:
 
@@ -17,7 +17,7 @@ Structure:
 
 ## How to contribute
 
-You can start by checking our [issues board](https://github.com/datopian/portaljs/issues).
+You can start by checking our [issues board](https://github.com/datopian/datahub/issues).
 
 If you'd like to work on one of the issues you can:
 
@@ -26,15 +26,16 @@ If you'd like to work on one of the issues you can:
 3. Clone the forked repository to your machine.
 4. Create a feature branch (e.g. `50-update-readme`, where `50` is the number of the related issue).
 5. Commit your changes to the feature branch.
-6. Push the feature branch to your forked repository.
+6. Add changeset file describing the changes. (See section below)
-7. Create a Pull Request against the original repository.
+7. Push the feature branch to your forked repository.
+8. Create a Pull Request against the original repository.
    - add a short description of the changes included in the PR
-8. Address review comments if requested by our demanding reviewers 😜.
+9. Address review comments if requested by our demanding reviewers 😜.
 
 If you have an idea for improvement, and it doesn't have a corresponding issue yet, simply submit a new one.
 
 > [!note]
-> Join our [Discord channel](https://discord.gg/rTxfCutu) do discuss existing issues and to ask for help.
+> Join our [Discord channel](https://discord.gg/KZSf3FG4EZ) do discuss existing issues and to ask for help.
 
 ## Nx
 
@@ -62,6 +63,7 @@ or you can use just:
 nx <target> <project>
 # e.g. npx nx serve ckan
 ```
+
 if you have the `nx` binary installed globally in your machine
 
 #### Running multiple tasks
@@ -174,3 +176,23 @@ To learn more see this [offical docs page](https://nx.dev/reference/nx-json).
 Each project also has it's own configuration file - `project.json`, where you can define and configure it's targets (and more).
 
 To learn more see this [offical docs page](https://nx.dev/reference/project-configuration).
+
+## Changesets and publishing packages
+
+> This monorepo is set up with changesets versioning tool. See their [github repository](https://github.com/changesets/changesets) to learn more.
+
+### What are Changesets?
+
+Changesets are files that describe the intention of a contributor to bump a version of the package according to their changes. Changeset file holds two key bits of information: a version type (following semver), and change information to be added to a changelog.
+
+### Adding changesets
+
+In the root directory of the repo, run:
+
+```
+npx changeset
+```
+
+Select the package that has been changed, the semver version that should be bumped with it and a description of your changes. Please make sure to add the most accurate but also concise information.
+
+To learn about semantic versioning standards see [this semver doc page](https://semver.org/).

README.md

@@ -1,31 +1,51 @@
-<h1 align="center">
+<p align="center">
-🌀 Portal.JS
+  Bugs, issues and suggestions re PortalJS framework
-<br />
+  <br />
-Rapidly build rich data portals using a modern frontend framework
+  <br /><a href="https://discord.gg/xfFDMPU9dC"><img src="https://dcbadge.vercel.app/api/server/xfFDMPU9dC" /></a>
-</h1>
+</p>
 
-* [What is Portal.JS ?](#What-is-Portal.JS)
+## PortalJS framework
-    * [Features](#Features)
-      * [For developers](#For-developers)
-* [Docs](#Docs)
-* [Community](#Community)
-* [Appendix](#Appendix)
-    * [What happened to Recline?](#What-happened-to-Recline?)
 
-# What is Portal.JS
+This repo and issue tracker are for
 
-🌀 Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. Portal.JS can be used to present a single dataset or build a full-scale data catalog/portal.
+- PortalJS 🌀 - https://www.portaljs.com/
+- DataHub Cloud ☁️ - https://datahub.io/
 
-Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com/) framework. Portal.JS assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
+### Issues
 
-## Features
+Found a bug: 👉 https://github.com/datopian/portaljs/issues/new
+
+### Discussions
+
+Got a suggestion, a question, want some support or just want to shoot the breeze 🙂
+
+Head to the discussion forum: 👉 https://github.com/datopian/portaljs/discussions
+
+### Chat on Discord
+
+If you would prefer to get help via live chat check out our discord 👉
+
+[Discord](https://discord.gg/xfFDMPU9dC)
+
+### Docs
+
+- For PortalJS go to https://www.portaljs.com/opensource
+- For DataHub Cloud – https://datahub.io/docs
+
+## PortalJS Cloud 🌀
+
+PortalJS Cloud 🌀 is a platform for rapidly creating rich data portal and publishing systems using a modern frontend approach. PortalJS Cloud can be used to publish a single dataset or build a full-scale data catalog/portal.
+
+PortalJS Cloud is built in JavaScript and React on top of the popular [Next.js](https://nextjs.org) framework. PortalJS Cloud assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/), GitHub, Frictionless Data Packages and more.
+
+### Features
 
 - 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. Wordpress) with a common internal API.
 - 👩‍💻 Developer friendly: built with familiar frontend tech (JavaScript, React, Next.js).
 - 🔋 Batteries included: full set of portal components out of the box e.g. catalog search, dataset showcase, blog, etc.
 - 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
 - 🧱 Extensible: quickly extend and develop/import your own React components
-- 📝 Well documented: full set of documentation plus the documentation of Next.js and Apollo.
+- 📝 Well documented: full set of documentation plus the documentation of Next.js.
 
 ### For developers
 
@@ -33,25 +53,3 @@ Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com
 - 🚀 Next.js framework: so everything in Next.js for free: Server Side Rendering, Static Site Generation, huge number of examples and integrations, etc.
   - Server Side Rendering (SSR) => Unlimited number of pages, SEO and more whilst still using React.
   - Static Site Generation (SSG) => Ultra-simple deployment, great performance, great lighthouse scores and more (good for small sites)
-
-#### **Check out the [Portal.JS website](https://portaljs.org/) for a gallery of live portals**
-
-___
-
-# Docs
-
-Access the Portal.JS documentation at:
-
-https://portaljs.org/docs
-
-- [Examples](https://portaljs.org/docs#examples)
-
-# Community
-
-If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
-
-# Appendix
-
-## What happened to Recline?
-
-Portal.JS used to be Recline(JS). If you are looking for the old Recline codebase it still exists:  see the [`recline` branch](https://github.com/datopian/portal.js/tree/recline). If you want context for the rename see [this issue](https://github.com/datopian/portal.js/issues/520).

examples/ckan-2021/README.md

@@ -2,7 +2,7 @@
 
 **🚩 UPDATE April 2023: This example is now deprecated - though still works!. Please use the [new CKAN examples](https://github.com/datopian/portaljs/tree/main/examples)**
 
-This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.org/blog/example-ckan-2021)
+This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.com/blog/example-ckan-2021)
 
 ## Developers
 

examples/ckan-example/.env

@@ -1 +0,0 @@
-DMS=https://demo.dev.datopian.com

examples/ckan-ssg/README.md

@@ -1,7 +1,7 @@
 This is a repo intended to serve as an example of a data catalog that get its data from a CKAN Instance.
 
 ```
-npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/ckan-example
+npx create-next-app <app-name> --example https://github.com/datopian/datahub/tree/main/examples/ckan-ssg
 cd <app-name>
 ```
 
@@ -19,7 +19,7 @@ npm run dev
 
 Congratulations, you now have something similar to this running on `http://localhost:4200`
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252297726865408/image.png?width=853&height=461)
-If yo go to any one of those pages by clicking on `More info` you will see something similar to this
+If you go to any one of those pages by clicking on `More info` you will see something similar to this
 ![](https://media.discordapp.net/attachments/1069718983604977754/1098252298074988595/image.png?width=853&height=461)
 
 ## Deployment

examples/ckan-ssg/package.json

@@ -10,6 +10,8 @@
   },
   "dependencies": {
     "@heroicons/react": "^2.0.17",
+    "@portaljs/ckan": "^0.0.2",
+    "@portaljs/remark-wiki-link": "^1.0.4",
     "next": "13.3.1",
     "next-seo": "^6.0.0",
     "octokit": "^2.0.14",
@@ -20,14 +22,14 @@
   },
   "devDependencies": {
     "@tailwindcss/typography": "^0.5.9",
-    "autoprefixer": "^10.4.14",
-    "postcss": "^8.4.23",
-    "tailwindcss": "^3.3.1",
-    "eslint": "8.39.0",
-    "eslint-config-next": "13.3.1",
-    "typescript": "5.0.4",
     "@types/node": "18.16.0",
     "@types/react": "18.0.38",
-    "@types/react-dom": "18.0.11"
+    "@types/react-dom": "18.0.11",
+    "autoprefixer": "^10.4.14",
+    "eslint": "8.39.0",
+    "eslint-config-next": "13.3.1",
+    "postcss": "^8.4.23",
+    "tailwindcss": "^3.3.1",
+    "typescript": "5.0.4"
   }
 }

examples/ckan-ssg/pages/@org/[org]/[dataset]/index.tsx

@@ -11,8 +11,9 @@ import {
   ServerIcon,
   UserIcon,
 } from '@heroicons/react/20/solid';
+import { CKAN } from '@portaljs/ckan';
 
-const dms = getConfig().publicRuntimeConfig.DMS;
+const backend_url = getConfig().publicRuntimeConfig.DMS;
 
 const formatter = new Intl.DateTimeFormat('en-US', {
   year: 'numeric',
@@ -25,14 +26,12 @@ const formatter = new Intl.DateTimeFormat('en-US', {
 });
 
 export const getServerSideProps: GetServerSideProps = async (context) => {
+  const ckan = new CKAN(backend_url)
   const { dataset } = context.query;
-  const response = await fetch(
+  const _dataset = await ckan.getDatasetDetails(dataset as string)
-    `${dms}/api/3/action/package_show?id=${dataset}`
-  );
-  const _dataset = await response.json();
   return {
     props: {
-      dataset: _dataset.result,
+      dataset: _dataset,
     },
   };
 };

examples/ckan-ssg/pages/index.tsx

@@ -1,7 +1,8 @@
 import getConfig from 'next/config';
 import styles from './index.module.css';
+import { CKAN } from '@portaljs/ckan';
 
-const dms = getConfig().publicRuntimeConfig.DMS
+const backend_url = getConfig().publicRuntimeConfig.DMS
 
 const formatter = new Intl.DateTimeFormat('en-US', {
   year: 'numeric',
@@ -15,12 +16,11 @@ const formatter = new Intl.DateTimeFormat('en-US', {
 
 
 export async function getServerSideProps() {
-  const response = await fetch(`${dms}/api/3/action/package_search`)
+  const ckan = new CKAN(backend_url)
-  const datasets = await response.json()
+  const { datasets } = await ckan.packageSearch({ limit: 1000, offset: 0, groups:[], orgs: [], tags: []})
-  const datasetsWithDetails = await Promise.all(datasets.result.results.map(async (dataset) => {
+  const datasetsWithDetails = await Promise.all(datasets.map(async (dataset) => {
-    const response = await fetch(`${dms}/api/3/action/package_show?id=` + dataset.name)
+    const _dataset = await ckan.getDatasetDetails(dataset.name)
-    const json = await response.json()
+    return _dataset
-    return json.result
   }))
 
   return {
@@ -79,7 +79,7 @@ export function Index({ datasets }) {
                 </thead>
                 <tbody className="divide-y divide-gray-200">
                   {datasets.map((dataset) => (
-                    <tr>
+                    <tr key={dataset.name}>
                       <td className="px-3 py-4 text-sm text-gray-500">
                           {dataset.title}
                       </td>

examples/ckan/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+  "extends": "next/core-web-vitals"
+}

examples/ckan/.gitignore

@@ -0,0 +1,35 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

examples/ckan/README.md

@@ -0,0 +1,38 @@
+This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
+
+[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
+
+The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
+
+This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

examples/ckan/components/DataRichDocument.tsx

@@ -1,6 +1,6 @@
 import { MDXRemote } from 'next-mdx-remote';
 import dynamic from 'next/dynamic';
-import { Mermaid } from '@flowershow/core';
+import { Mermaid } from '@portaljs/core';
 
 // Custom components/renderers to pass to MDX.
 // Since the MDX files aren't loaded by webpack, they have no knowledge of how
@@ -9,6 +9,7 @@ import { Mermaid } from '@flowershow/core';
 const components = {
   Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
   Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
+  FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
   mermaid: Mermaid,
   Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
   VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),

examples/ckan/content/test.md

@@ -0,0 +1,3 @@
+# Test
+
+Test Data Rich Stories

examples/ckan/lib/markdown.js

@@ -1,13 +1,13 @@
 import matter from "gray-matter";
 import mdxmermaid from "mdx-mermaid";
 import { h } from "hastscript";
-import remarkCallouts from "@flowershow/remark-callouts";
+import remarkCallouts from "@portaljs/remark-callouts";
-import remarkEmbed from "@flowershow/remark-embed";
+import remarkEmbed from "@portaljs/remark-embed";
 import remarkGfm from "remark-gfm";
 import remarkMath from "remark-math";
 import remarkSmartypants from "remark-smartypants";
 import remarkToc from "remark-toc";
-import remarkWikiLink from "@flowershow/remark-wiki-link";
+import remarkWikiLink from "@portaljs/remark-wiki-link";
 import rehypeAutolinkHeadings from "rehype-autolink-headings";
 import rehypeKatex from "rehype-katex";
 import rehypeSlug from "rehype-slug";

examples/ckan/lib/mddb.ts

@@ -1,4 +1,4 @@
-import { MarkdownDB } from "@flowershow/markdowndb";
+import { MarkdownDB } from "mddb";
 
 const dbPath = "markdown.db";
 

examples/ckan/next.config.js

@@ -0,0 +1,11 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  reactStrictMode: true,
+  publicRuntimeConfig: {
+    DMS: process.env.DMS
+      ? process.env.DMS.replace(/\/?$/, '')
+      : 'https://demo.dev.datopian.com/',
+  },
+};
+
+module.exports = nextConfig;

examples/ckan/package.json

@@ -0,0 +1,48 @@
+{
+  "name": "ckan",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "prebuild": "npm run mddb",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint",
+    "mddb": "mddb ./content"
+  },
+  "dependencies": {
+    "@githubocto/flat-ui": "^0.14.1",
+    "@heroicons/react": "^2.0.18",
+    "@portaljs/ckan": "^0.0.2",
+    "@portaljs/components": "0.1.6",
+    "@portaljs/core": "^1.0.5",
+    "@portaljs/remark-callouts": "^1.0.5",
+    "@portaljs/remark-embed": "^1.0.4",
+    "@portaljs/remark-wiki-link": "^1.0.4",
+    "@tailwindcss/typography": "^0.5.9",
+    "@types/node": "20.2.3",
+    "@types/react": "18.2.6",
+    "@types/react-dom": "18.2.4",
+    "autoprefixer": "10.4.14",
+    "eslint": "8.41.0",
+    "eslint-config-next": "13.4.3",
+    "isomorphic-unfetch": "^4.0.2",
+    "mddb": "^0.1.9",
+    "next": "13.4.3",
+    "next-mdx-remote": "^4.4.1",
+    "papaparse": "^5.4.1",
+    "postcss": "8.4.23",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-query": "^3.39.3",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
+    "tailwindcss": "3.3.2",
+    "typescript": "5.0.4"
+  }
+}

examples/ckan/pages/[org]/[dataset]/index.tsx

@@ -0,0 +1,179 @@
+import Head from "next/head";
+import { CKAN, Dataset } from "@portaljs/ckan";
+import {
+  ChevronRightIcon,
+  HomeIcon,
+  PaperClipIcon,
+} from "@heroicons/react/20/solid";
+import Link from "next/link";
+import getConfig from "next/config";
+
+const backend_url = getConfig().publicRuntimeConfig.DMS
+
+export const getServerSideProps = async (context: any) => {
+  try {
+    const datasetName = context.params?.dataset;
+    if (!datasetName) {
+      return {
+        notFound: true,
+      };
+    }
+    const ckan = new CKAN(backend_url);
+    const dataset = await ckan.getDatasetDetails(datasetName as string);
+    if (!dataset) {
+      return {
+        notFound: true,
+      };
+    }
+    return {
+      props: { dataset },
+    };
+  } catch {
+    return {
+      notFound: true,
+    };
+  }
+};
+
+export default function DatasetPage({
+  dataset,
+}: {
+  dataset: Dataset;
+}): JSX.Element {
+  return (
+    <>
+      <Head>
+        <title>{`${dataset.title || dataset.name} - Dataset`}</title>
+        <meta name="description" content="Generated by create next app" />
+        <link rel="icon" href="/favicon.ico" />
+      </Head>
+      <main className="flex min-h-screen flex-col items-center justify-between p-24 bg-zinc-900">
+        <div className="bg-white p-8 my-4 rounded-lg">
+          <nav className="flex px-4 py-8" aria-label="Breadcrumb">
+            <ol role="list" className="flex items-center space-x-4">
+              <li>
+                <div>
+                  <Link href="/" className="text-gray-400 hover:text-gray-500">
+                    <HomeIcon
+                      className="h-5 w-5 flex-shrink-0"
+                      aria-hidden="true"
+                    />
+                    <span className="sr-only">Home</span>
+                  </Link>
+                </div>
+              </li>
+              <li>
+                <div className="flex items-center">
+                  <ChevronRightIcon
+                    className="h-5 w-5 flex-shrink-0 text-gray-400"
+                    aria-hidden="true"
+                  />
+                  <span
+                    className="ml-4 text-sm font-medium text-gray-500 hover:text-gray-700"
+                    aria-current={"page"}
+                  >
+                    {dataset.name}
+                  </span>
+                </div>
+              </li>
+            </ol>
+          </nav>
+          {dataset && (
+            <div>
+              <div className="px-4 sm:px-0">
+                <h3 className="text-base font-semibold leading-7 text-gray-900">
+                  {dataset.title || dataset.name}
+                </h3>
+                <p className="mt-1 max-w-2xl text-sm leading-6 text-gray-500">
+                  Dataset details
+                </p>
+              </div>
+              <div className="mt-6 border-t border-gray-100">
+                <dl className="divide-y divide-gray-100">
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    <dt className="text-sm font-medium leading-6 text-gray-900">
+                      Title
+                    </dt>
+                    <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                      {dataset.title}
+                    </dd>
+                  </div>
+                  {dataset.tags && dataset.tags.length > 0 && (
+                    <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                      <dt className="text-sm font-medium leading-6 text-gray-900">
+                        Tags
+                      </dt>
+                      <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                        {dataset.tags.map((tag) => tag.display_name).join(", ")}
+                      </dd>
+                    </div>
+                  )}
+                  {dataset.tags && dataset.tags.length > 0 && (
+                    <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                      <dt className="text-sm font-medium leading-6 text-gray-900">
+                        URL
+                      </dt>
+                      <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                        {dataset.url}
+                      </dd>
+                    </div>
+                  )}
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    {dataset.notes && (
+                      <>
+                        <dt className="text-sm font-medium leading-6 text-gray-900">
+                          Description
+                        </dt>
+                        <dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
+                          {dataset.notes}
+                        </dd>
+                      </>
+                    )}
+                  </div>
+                  <div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
+                    <dt className="text-sm font-medium leading-6 text-gray-900">
+                      Files
+                    </dt>
+                    <dd className="mt-2 text-sm text-gray-900 sm:col-span-2 sm:mt-0">
+                      <ul
+                        role="list"
+                        className="divide-y divide-gray-100 rounded-md border border-gray-200"
+                      >
+                        {dataset.resources.map((resource) => (
+                          <li key={resource.id} className="flex items-center justify-between py-4 pl-4 pr-5 text-sm leading-6">
+                            <div className="flex w-0 flex-1 items-center">
+                              <PaperClipIcon
+                                className="h-5 w-5 flex-shrink-0 text-gray-400"
+                                aria-hidden="true"
+                              />
+                              <div className="ml-4 flex min-w-0 flex-1 gap-2">
+                                <span className="truncate font-medium">
+                                  {resource.name || resource.id}
+                                </span>
+                                <span className="flex-shrink-0 text-gray-400">
+                                  {resource.size}
+                                </span>
+                              </div>
+                            </div>
+                            <div className="ml-4 flex-shrink-0">
+                              <a
+                                href={resource.url}
+                                className="font-medium hover:text-indigo-500 mr-4"
+                              >
+                                Download
+                              </a>
+                            </div>
+                          </li>
+                        ))}
+                      </ul>
+                    </dd>
+                  </div>
+                </dl>
+              </div>
+            </div>
+          )}
+        </div>
+      </main>
+    </>
+  );
+}

examples/ckan/pages/_app.tsx

@@ -0,0 +1,7 @@
+import '@/styles/globals.css'
+import '@portaljs/ckan/styles.css'
+import type { AppProps } from 'next/app'
+
+export default function App({ Component, pageProps }: AppProps) {
+  return <Component {...pageProps} />
+}

examples/ckan/pages/_document.tsx

@@ -0,0 +1,13 @@
+import { Html, Head, Main, NextScript } from 'next/document'
+
+export default function Document() {
+  return (
+    <Html lang="en">
+      <Head />
+      <body>
+        <Main />
+        <NextScript />
+      </body>
+    </Html>
+  )
+}

examples/ckan/pages/api/cors.ts

@@ -0,0 +1,18 @@
+import fetch from 'isomorphic-unfetch';
+
+const Cors = async (req: any, res: any) => {
+  const { url } = req.query;
+  try {
+    const resProxy = await fetch(url, {
+      headers: {
+        Range: 'bytes=0-5132288',
+      },
+    });
+    const data = await resProxy.text();
+    return res.status(200).send(data);
+  } catch (error: any) {
+    res.status(400).send(error.toString());
+  }
+};
+
+export default Cors;

examples/ckan/pages/index.tsx

@@ -0,0 +1,64 @@
+import {
+  CKAN,
+  DatasetSearchForm,
+  ListOfDatasets,
+  PackageSearchOptions,
+  Organization,
+  Group,
+} from '@portaljs/ckan';
+import getConfig from 'next/config';
+import { useState } from 'react';
+
+const backend_url = getConfig().publicRuntimeConfig.DMS;
+
+export async function getServerSideProps() {
+  const ckan = new CKAN(backend_url);
+  const groups = await ckan.getGroupsWithDetails();
+  const orgs = await ckan.getOrgsWithDetails();
+  return {
+    props: {
+      groups,
+      orgs,
+    },
+  };
+}
+
+export default function Home({
+  orgs,
+  groups,
+}: {
+  orgs: Organization[];
+  groups: Group[];
+}) {
+  const ckan = new CKAN(backend_url);
+  const [options, setOptions] = useState<PackageSearchOptions>({
+    offset: 0,
+    limit: 5,
+    tags: [],
+    groups: [],
+    orgs: [],
+  });
+
+  return (
+    <div>
+      <main className="py-12 bg-zinc-900">
+        <DatasetSearchForm
+          options={options}
+          setOptions={setOptions}
+          groups={groups}
+          orgs={orgs}
+        />
+        <div
+          className="bg-white p-8 mx-auto my-4 rounded-lg"
+          style={{ width: 'min(1100px, 95vw)' }}
+        >
+          <ListOfDatasets
+            options={options}
+            setOptions={setOptions}
+            ckan={ckan}
+          />{' '}
+        </div>
+      </main>
+    </div>
+  );
+}

examples/ckan/pages/stories/[[...path]].tsx

@@ -0,0 +1,129 @@
+import { existsSync, promises as fs } from 'fs';
+import path from 'path';
+import parse from '../../lib/markdown';
+
+import DataRichDocument from '../../components/DataRichDocument';
+import clientPromise from '../../lib/mddb';
+import getConfig from 'next/config';
+import { CKAN } from '@portaljs/ckan';
+
+export const getStaticPaths = async () => {
+  const contentDir = path.join(process.cwd(), '/content/');
+  const contentFolders = await fs.readdir(contentDir, 'utf8');
+  const paths = contentFolders.map((folder: string) => ({
+    params: { path: [folder.split('.')[0]] },
+  }));
+  return {
+    paths,
+    fallback: false,
+  };
+};
+
+const backend_url = getConfig().publicRuntimeConfig.DMS;
+
+export const getStaticProps = async (context) => {
+  const mddb = await clientPromise;
+  const storyFile = await mddb.getFileByUrl(context.params.path);
+  const md = await fs.readFile(
+    `${process.cwd()}/${storyFile.file_path}`,
+    'utf8'
+  );
+
+  const ckan = new CKAN(backend_url);
+  const datasets = storyFile.metadata.datasets ? await Promise.all(
+    storyFile.metadata.datasets.map(
+      async (datasetName: string) => await ckan.getDatasetDetails(datasetName)
+    )
+  ) : [];
+  const orgs = storyFile.metadata.orgs ? await Promise.all(
+    storyFile.metadata.orgs.map(
+      async (orgName: string) => await ckan.getOrgDetails(orgName)
+    )
+  ) : [];
+
+  let { mdxSource, frontMatter } = await parse(md, '.mdx', { datasets, orgs });
+
+  return {
+    props: {
+      mdxSource,
+      frontMatter: JSON.stringify(frontMatter),
+    },
+  };
+};
+
+export default function DatasetPage({ mdxSource, frontMatter }) {
+  frontMatter = JSON.parse(frontMatter);
+  return (
+    <main className="flex min-h-screen flex-col justify-between p-16 bg-zinc-900">
+      <div className="bg-white p-8 my-4 rounded-lg">
+        <div className="prose mx-auto py-8">
+          <header>
+            <div className="mb-6">
+              <>
+                <h1 className="mb-2">{frontMatter.title}</h1>
+                {frontMatter.author && (
+                  <p className="my-0">
+                    <span className="font-semibold">Author: </span>
+                    <span className="my-0">{frontMatter.author}</span>
+                  </p>
+                )}
+                {frontMatter.description && (
+                  <p className="my-0">
+                    <span className="font-semibold">Description: </span>
+                    <span className="description my-0">
+                      {frontMatter.description}
+                    </span>
+                  </p>
+                )}
+                {frontMatter.modified && (
+                  <p className="my-0">
+                    <span className="font-semibold">Modified: </span>
+                    <span className="description my-0">
+                      {new Date(frontMatter.modified).toLocaleDateString()}
+                    </span>
+                  </p>
+                )}
+                {frontMatter.files && (
+                  <section className="py-6">
+                    <h2 className="mt-0">Data files</h2>
+                    <table className="table-auto">
+                      <thead>
+                        <tr>
+                          <th>File</th>
+                          <th>Format</th>
+                        </tr>
+                      </thead>
+                      <tbody>
+                        {frontMatter.files.map((f) => {
+                          const fileName = f.split('/').slice(-1);
+                          return (
+                            <tr key={`resources-list-${f}`}>
+                              <td>
+                                <a target="_blank" href={f}>
+                                  {fileName}
+                                </a>
+                              </td>
+                              <td>
+                                {fileName[0]
+                                  .split('.')
+                                  .slice(-1)[0]
+                                  .toUpperCase()}
+                              </td>
+                            </tr>
+                          );
+                        })}
+                      </tbody>
+                    </table>
+                  </section>
+                )}
+              </>
+            </div>
+          </header>
+          <main>
+            <DataRichDocument source={mdxSource} />
+          </main>
+        </div>
+      </div>
+    </main>
+  );
+}

examples/ckan/public/next.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>

examples/ckan/public/vercel.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 283 64"><path fill="black" d="M141 16c-11 0-19 7-19 18s9 18 20 18c7 0 13-3 16-7l-7-5c-2 3-6 4-9 4-5 0-9-3-10-7h28v-3c0-11-8-18-19-18zm-9 15c1-4 4-7 9-7s8 3 9 7h-18zm117-15c-11 0-19 7-19 18s9 18 20 18c6 0 12-3 16-7l-8-5c-2 3-5 4-8 4-5 0-9-3-11-7h28l1-3c0-11-8-18-19-18zm-10 15c2-4 5-7 10-7s8 3 9 7h-19zm-39 3c0 6 4 10 10 10 4 0 7-2 9-5l8 5c-3 5-9 8-17 8-11 0-19-7-19-18s8-18 19-18c8 0 14 3 17 8l-8 5c-2-3-5-5-9-5-6 0-10 4-10 10zm83-29v46h-9V5h9zM37 0l37 64H0L37 0zm92 5-27 48L74 5h10l18 30 17-30h10zm59 12v10l-3-1c-6 0-10 4-10 10v15h-9V17h9v9c0-5 6-9 13-9z"/></svg>

examples/ckan/styles/globals.css

@@ -0,0 +1,71 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+
+@import "@portaljs/remark-callouts/styles.css";
+
+/* mathjax */
+.math-inline > mjx-container > svg {
+  display: inline;
+  align-items: center;
+}
+
+/* smooth scrolling in modern browsers */
+html {
+  scroll-behavior: smooth !important;
+}
+
+/* tooltip fade-out clip */
+.tooltip-body::after {
+  content: "";
+  position: absolute;
+  right: 0;
+  top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
+  height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
+  width: 10rem;
+  background: linear-gradient(
+    to right,
+    rgba(255, 255, 255, 0),
+    rgba(255, 255, 255, 1) 100%
+  );
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
+  margin-left: -2rem !important;
+  padding-left: 2rem !important;
+  scroll-margin-top: 4.5rem;
+  position: relative;
+}
+
+.heading-link {
+  padding: 1px;
+  position: absolute;
+  left: 0;
+  top: 50%;
+  transform: translateY(-50%);
+  margin: auto 0;
+  border-radius: 5px;
+  background: #1e293b;
+  opacity: 0;
+  transition: opacity 0.2s;
+}
+
+.light .heading-link {
+  /* border: 1px solid #ab2b65; */
+  /* background: none; */
+  background: #e2e8f0;
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
+  opacity: 100;
+}
+
+.heading-link svg {
+  transform: scale(0.75);
+}
+
+@media screen and (max-width: 640px) {
+  .heading-link {
+    visibility: hidden;
+  }
+}

examples/ckan/tailwind.config.js

@@ -0,0 +1,18 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  content: [
+    './pages/**/*.{js,ts,jsx,tsx,mdx}',
+    './components/**/*.{js,ts,jsx,tsx,mdx}',
+    './app/**/*.{js,ts,jsx,tsx,mdx}',
+  ],
+  theme: {
+    extend: {
+      backgroundImage: {
+        'gradient-radial': 'radial-gradient(var(--tw-gradient-stops))',
+        'gradient-conic':
+          'conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))',
+      },
+    },
+  },
+  plugins: [require('@tailwindcss/typography')],
+}

examples/ckan/tsconfig.json

@@ -0,0 +1,23 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": ["dom", "dom.iterable", "esnext"],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": false,
+    "forceConsistentCasingInFileNames": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "preserve",
+    "incremental": true,
+    "paths": {
+      "@/*": ["./*"]
+    }
+  },
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
+  "exclude": ["node_modules"]
+}

examples/dataset-frictionless/README.md

@@ -1,6 +1,6 @@
 This example creates a portal/showcase for a single dataset. The dataset should be a [Frictionless dataset (data package)][fd] i.e. there should be a `datapackage.json`.
 
-[fd]: https://frictionlessdata.io/data-packages/
+[fd]: https://specs.frictionlessdata.io/data-package/
 
 ## How to use
 

examples/fivethirtyeight/README.md

@@ -1,3 +1,9 @@
+# PortalJS Demo replicating the FiveThirtyEight data portal
+
+## 👉 https://fivethirtyeight.portaljs.org 👈
+
+Here's a blog post we wrote about it: https://www.datopian.com/blog/fivethirtyeight-replica
+
 This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
 
 You might be asking why we did that, there are three main reasons:

examples/fivethirtyeight/components/Layout.tsx

@@ -0,0 +1,99 @@
+import { useState } from 'react';
+import { XMarkIcon } from '@heroicons/react/20/solid';
+import { Transition } from '@headlessui/react';
+
+export default function Layout({ children }: { children: React.ReactNode }) {
+  const [isShowing, setShow] = useState(true);
+  return (
+    <>
+      <Transition
+        show={isShowing}
+        enter="transition-opacity duration-75"
+        enterFrom="opacity-0"
+        enterTo="opacity-100"
+        leave="transition-opacity duration-150"
+        leaveFrom="opacity-100"
+        leaveTo="opacity-0"
+      >
+        <div className="flex items-center gap-x-6 bg-[#3c3c3c] px-6 py-2.5 sm:px-3.5 sm:before:flex-1">
+          <p className="text-sm leading-6 text-white">
+            This is a replica to the awesome{' '}
+            <a
+              className="hover:underline font-bold"
+              href="https://data.fivethirtyeight.com"
+            >
+              data.fivethirtyeight.com
+            </a>{' '}
+            website.{' '}
+            <a
+              className="hover:underline font-bold"
+              href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight#readme"
+            >
+              Read more here
+            </a>{' '}
+          </p>
+          <div className="flex flex-1 justify-end">
+            <button
+              type="button"
+              onClick={() => setShow(false)}
+              className="-m-3 p-3 focus-visible:outline-offset-[-4px]"
+            >
+              <span className="sr-only">Dismiss</span>
+              <XMarkIcon className="h-5 w-5 text-white" aria-hidden="true" />
+            </button>
+          </div>
+        </div>
+      </Transition>
+      <header className="max-w-5xl mx-auto mt-8 w-full">
+        <div className="border-b-2 pb-2.5 mx-2 border-zinc-800 flex justify-between">
+          <h1 className="flex gap-x-1 items-end">
+            <span className="sr-only">FiveThirtyEight</span>
+            <img
+              width="197"
+              height="25"
+              alt="FiveThirtyEight"
+              src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
+            />{' '}
+            <span className="-mb-0.5 text-[#3c3c3c]">replica</span>
+          </h1>
+          <div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
+            <a
+              className="hover:opacity-75 transition"
+              href="https://portaljs.com"
+            >
+              Built with 🌀PortalJS
+            </a>
+            <hr className="h-[80%] border border-[#3c3c3c] opacity-75 my-2"></hr>
+            <a
+              className="hover:opacity-75 transition"
+              href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
+            >
+              Github
+            </a>
+          </div>
+        </div>
+        <div className="mx-2 py-1.5 text-[14px] text-[#3c3c3c] md:hidden">
+          <ul className="flex gap-x-4">
+            <li>
+              <a
+                className="hover:opacity-75 transition"
+                href="https://portaljs.com"
+              >
+                PortalJS
+              </a>
+            </li>
+            <li>
+              <a
+                className="hover:opacity-75 transition"
+                href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
+              >
+                View on Github
+              </a>
+            </li>
+          </ul>
+        </div>
+      </header>
+      {children}
+    </>
+  );
+}

examples/fivethirtyeight/datasets.json

@@ -1,33 +1,4 @@
 [
-  {
-    "url": "https://github.com/fivethirtyeight/data/tree/master/polls",
-    "name": "polls",
-    "displayName": "<span class=\"lastword\">polls</span>",
-    "articles": [
-      {
-        "date": "2023-05-11T14:35:40.000Z",
-        "title": "Latest Polls",
-        "url": "https://projects.fivethirtyeight.com/polls/"
-      }
-    ],
-    "files": [
-      "https://projects.fivethirtyeight.com/polls-page/data/president_primary_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/president_primary_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/president_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/president_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/senate_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/senate_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/house_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/house_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/governor_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/governor_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/president_approval_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/generic_ballot_polls.csv",
-      "https://projects.fivethirtyeight.com/polls-page/data/generic_ballot_polls_historical.csv",
-      "https://projects.fivethirtyeight.com/2020-primary-data/pres_primary_avgs_2020.csv",
-      "https://projects.fivethirtyeight.com/2020-general-data/presidential_poll_averages_2020.csv"
-    ]
-  },
   {
     "url": "https://github.com/fivethirtyeight/data/tree/master/congress-generic-ballot",
     "name": "congress-generic-ballot",
@@ -195,6 +166,35 @@
       }
     ]
   },
+  {
+    "url": "https://github.com/fivethirtyeight/data/tree/master/polls",
+    "name": "polls",
+    "displayName": "<span class=\"lastword\">polls</span>",
+    "articles": [
+      {
+        "date": "2023-05-11T14:35:40.000Z",
+        "title": "Latest Polls",
+        "url": "https://projects.fivethirtyeight.com/polls/"
+      }
+    ],
+    "files": [
+      "https://projects.fivethirtyeight.com/polls-page/data/president_primary_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/president_primary_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/president_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/president_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/senate_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/senate_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/house_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/house_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/governor_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/governor_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/president_approval_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/generic_ballot_polls.csv",
+      "https://projects.fivethirtyeight.com/polls-page/data/generic_ballot_polls_historical.csv",
+      "https://projects.fivethirtyeight.com/2020-primary-data/pres_primary_avgs_2020.csv",
+      "https://projects.fivethirtyeight.com/2020-general-data/presidential_poll_averages_2020.csv"
+    ]
+  },
   {
     "url": "https://github.com/fivethirtyeight/data/tree/master/nfl-elo",
     "name": "nfl-elo",
@@ -1169,18 +1169,6 @@
       }
     ]
   },
-  {
-    "url": "https://github.com/fivethirtyeight/data/tree/master/undefeated-boxers",
-    "name": "undefeated-boxers",
-    "displayName": "undefeated-<span class=\"lastword\">boxers</span>",
-    "articles": [
-      {
-        "date": "2017-08-18T18:47:32.000Z",
-        "title": "Mayweather Is Defined By The Zero Next To His Name",
-        "url": "https://fivethirtyeight.com/features/mayweather-is-defined-by-the-zero-next-to-his-name/"
-      }
-    ]
-  },
   {
     "url": "https://github.com/fivethirtyeight/data/tree/master/chess-transfers",
     "name": "chess-transfers",
@@ -2139,6 +2127,18 @@
       }
     ]
   },
+  {
+    "url": "https://github.com/fivethirtyeight/data/tree/master/undefeated-boxers",
+    "name": "undefeated-boxers",
+    "displayName": "undefeated-<span class=\"lastword\">boxers</span>",
+    "articles": [
+      {
+        "date": "2017-08-18T18:47:32.000Z",
+        "title": "Mayweather Is Defined By The Zero Next To His Name",
+        "url": "https://fivethirtyeight.com/features/mayweather-is-defined-by-the-zero-next-to-his-name/"
+      }
+    ]
+  },
   {
     "url": "https://github.com/fivethirtyeight/data/tree/master/march-madness-predictions",
     "name": "march-madness-predictions",

examples/fivethirtyeight/package.json

@@ -9,7 +9,11 @@
     "lint": "next lint"
   },
   "dependencies": {
-    "@portaljs/components": "^0.1.0",
+    "@headlessui/react": "^1.7.14",
+    "@heroicons/react": "^2.0.18",
+    "@portaljs/components": "^0.1.8",
+    "@portaljs/core": "^1.0.5",
+    "@portaljs/remark-wiki-link": "^1.0.4",
     "@tailwindcss/typography": "^0.5.9",
     "@types/node": "20.1.1",
     "@types/react": "18.2.6",
@@ -26,12 +30,15 @@
     "react": "18.2.0",
     "react-dom": "18.2.0",
     "react-markdown": "^8.0.7",
+    "remark": "^14.0.3",
     "remark-code-frontmatter": "^1.0.0",
+    "remark-excerpt": "^1.0.0-beta.1",
     "remark-extract-frontmatter": "^3.2.0",
     "remark-frontmatter": "^4.0.1",
     "remark-gfm": "^3.0.1",
     "tailwindcss": "3.3.2",
     "timeago.js": "^4.0.2",
+    "to-vfile": "^7.2.4",
     "typescript": "5.0.4"
   }
 }

examples/fivethirtyeight/pages/_app.tsx

@@ -1,8 +1,49 @@
-import '@/styles/globals.css'
+import '@/styles/globals.css';
-import '@portaljs/components/styles.css'
+import '@portaljs/components/styles.css';
+import { useEffect } from 'react';
+import { pageview } from '@portaljs/core';
+import Script from 'next/script';
+import Head from 'next/head';
+import { useRouter } from 'next/router';
 
-import type { AppProps } from 'next/app'
+import type { AppProps } from 'next/app';
 
 export default function App({ Component, pageProps }: AppProps) {
-  return <Component {...pageProps} />
+  const router = useRouter();
+
+  useEffect(() => {
+    const handleRouteChange = (url: any) => {
+      pageview(url);
+    };
+    router.events.on('routeChangeComplete', handleRouteChange);
+    return () => {
+      router.events.off('routeChangeComplete', handleRouteChange);
+    };
+  }, [router.events]);
+  return (
+    <>
+      <Head>
+        <link rel="shortcut icon" href="/squared_logo.png" />
+      </Head>
+      <Script
+        strategy="afterInteractive"
+        src="https://www.googletagmanager.com/gtag/js?id=G-3N9SXTC7GS"
+      />
+      <Script
+        id="gtag-init"
+        strategy="afterInteractive"
+        dangerouslySetInnerHTML={{
+          __html: `
+          window.dataLayer = window.dataLayer || [];
+          function gtag(){dataLayer.push(arguments);}
+          gtag('js', new Date());
+          gtag('config', 'G-3N9SXTC7GS', {
+            page_path: window.location.pathname,
+          });
+        `,
+        }}
+      />
+      <Component {...pageProps} />
+    </>
+  );
 }

examples/fivethirtyeight/pages/_document.tsx

@@ -19,78 +19,9 @@ export default function Document() {
         />
       </Head>
       <body>
-        <div className="px-2 max-w-5xl mx-auto pb-2">
-          <div className="mt-2 px-2 bg-[#3c3c3c] text-white">
-            <div className="p-2 text-center">
-              This is a replica to the awesome{' '}
-              <a
-                className="hover:underline font-bold"
-                href="https://data.fivethirtyeight.com"
-              >
-                data.fivethirtyeight.com
-              </a>{' '}
-              website.{' '}
-              <a
-                className="hover:underline font-bold"
-                href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
-              >
-                Read more here
-              </a>{' '}
-            </div>
-          </div>
-        </div>
-        <header className="max-w-5xl mx-auto mt-8 w-full">
-          <div className="border-b-2 pb-2.5 mx-2 border-zinc-800 flex justify-between">
-            <h1 className="flex gap-x-1 items-end">
-              <span className="sr-only">FiveThirtyEight</span>
-              <img
-                width="197"
-                height="25"
-                alt="FiveThirtyEight"
-                src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
-              />{' '}
-              <span className="-mb-0.5 text-[#3c3c3c]">replica</span>
-            </h1>
-            <div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
-              <a
-                className="hover:opacity-75 transition"
-                href="https://portaljs.org"
-              >
-                Built with 🌀PortalJS
-              </a>
-              <hr className="h-[80%] border border-[#3c3c3c] opacity-75 my-2"></hr>
-              <a
-                className="hover:opacity-75 transition"
-                href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
-              >
-                Github
-              </a>
-            </div>
-          </div>
-          <div className="mx-2 py-1.5 text-[14px] text-[#3c3c3c] md:hidden">
-            <ul className="flex gap-x-4">
-              <li>
-                <a
-                  className="hover:opacity-75 transition"
-                  href="https://portaljs.org"
-                >
-                  PortalJS
-                </a>
-              </li>
-              <li>
-                <a
-                  className="hover:opacity-75 transition"
-                  href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
-                >
-                  View on Github
-                </a>
-              </li>
-            </ul>
-          </div>
-        </header>
         <Main />
-        <NextScript />
       </body>
+      <NextScript />
     </Html>
   );
 }

examples/fivethirtyeight/pages/datasets/[datasetName].tsx

@@ -7,10 +7,19 @@ import remarkGfm from 'remark-gfm';
 import extract from 'remark-extract-frontmatter';
 import { Dataset } from '..';
 import { GetStaticProps } from 'next';
-import { Table } from '@portaljs/components';
+import { FlatUiTable } from '@portaljs/components';
 import Breadcrumbs from '@/components/Breadcrumbs';
 import { ReactMarkdown } from 'react-markdown/lib/react-markdown';
 import remarkFrontmatter from 'remark-frontmatter';
+import Layout from '@/components/Layout';
+import { format } from 'timeago.js';
+
+// Request a weekday along with a long date
+const options = {
+  year: 'numeric',
+  month: 'long',
+  day: 'numeric',
+} as const;
 
 export default function DatasetPage({
   dataset,
@@ -22,54 +31,28 @@ export default function DatasetPage({
   return (
     <>
       <NextSeo title={`${dataset.name} page`} />
-      <main className="max-w-5xl px-2 prose mx-auto my-8 prose-thead:border-b-4 prose-table:max-w-5xl prose-table:overflow-scroll prose-thead:overflow-scroll prose-tbody:overflow-scroll prose-thead:pb-2 prose-thead:border-zinc-900 prose-th:uppercase prose-th:text-left prose-th:font-light prose-th:text-xs">
+      <Layout>
+        <main className="max-w-5xl px-2 prose mx-auto my-8 pb-8 prose-thead:border-b-4 prose-table:max-w-5xl prose-table:overflow-scroll prose-thead:overflow-scroll prose-tbody:overflow-scroll prose-thead:pb-2 prose-thead:border-zinc-900 prose-th:uppercase prose-th:text-left prose-th:font-light prose-th:text-xs prose-a:no-underline">
           <Breadcrumbs links={[{ title: dataset.name, href: '' }]} />
           <h1 className="uppercase mb-0 mt-16">{dataset.name}</h1>
-        <p className="mb-8">
+          <table className="w-full my-10 mb-8 hidden md:table">
-          <span className="font-semibold">Repository:</span>{' '}
-          <a target="_blank" href={dataset.url}>
-            {dataset.url}
-          </a>
-        </p>
-
-        <h2 className="mb-0 mt-10">FILES</h2>
-        <div className="inline-block min-w-full py-2 align-middle">
-          <table className="min-w-full divide-y divide-gray-300">
             <thead className="border-b-4 pb-2 border-zinc-900">
               <tr>
-                <th
+                <th className="uppercase text-left font-normal text-xs pb-3">
-                  className="uppercase text-left font-light text-xs pb-3"
+                  related content
-                  scope="col"
+                </th>
-                >
+                <th className="uppercase text-left font-normal text-xs pb-3">
-                  Name
+                  last updated
                 </th>
               </tr>
             </thead>
-            <tbody className="divide-y divide-gray-200">
+            <tbody>
-              {dataset.files?.map((file) => (
+              <DesktopItem key={dataset.name} dataset={dataset} />
-                <tr key={file}>
-                  <td className="whitespace-nowrap text-left py-4 text-sm text-gray-500">
-                    <a href={file}>{file.split('/').slice(-1)}</a>
-                  </td>
-                </tr>
-              ))}
             </tbody>
           </table>
-        </div>
+
-        {dataset.files && dataset.files.length > 0 && (
-          <>
-            <h2 className="mb-0 mt-10">DATA PREVIEWS</h2>
-            {dataset.files?.map((file) => (
-              <div key={file} className="preview-table my-8">
-                <h3>{file.split('/').slice(-1)}</h3>
-                <Table url={file} />
-              </div>
-            ))}
-          </>
-        )}
           {dataset.readme && (
             <>
-            <h2 className="uppercase font-black">Readme</h2>
               {dataset.readme && (
                 <ReactMarkdown
                   remarkPlugins={[
@@ -83,7 +66,111 @@ export default function DatasetPage({
               )}
             </>
           )}
+
+          <h2 className="mb-0 mt-10">Files</h2>
+          <div className="inline-block min-w-full py-2 align-middle">
+            <table className="min-w-full divide-y divide-gray-300">
+              <thead className="border-b-4 pb-2 border-zinc-900">
+                <tr>
+                  <th
+                    className="uppercase text-left font-light text-xs pb-3"
+                    scope="col"
+                  >
+                    Name
+                  </th>
+                  <th
+                    className="uppercase text-left font-light text-xs pb-3"
+                    scope="col"
+                  >
+                    Download
+                  </th>
+                </tr>
+              </thead>
+              <tbody className="divide-y divide-gray-200">
+                {dataset.files?.map((file) => (
+                  <tr key={file}>
+                    <td className="whitespace-nowrap text-left py-4 text-sm text-gray-500">
+                      <a href={`#${file.split('/').slice(-1)}`}>
+                        {file.split('/').slice(-1)}
+                      </a>
+                    </td>
+                    <td className="whitespace-nowrap py-4 text-sm text-gray-500">
+                      <a href={file}>
+                        <svg
+                          xmlns="http://www.w3.org/2000/svg"
+                          viewBox="0 0 24 24"
+                          fill="currentColor"
+                          className="w-8 h-8 text-blue-400 hover:text-blue-300 transition mt-1 ml-3"
+                        >
+                          <path
+                            fillRule="evenodd"
+                            d="M12 2.25c-5.385 0-9.75 4.365-9.75 9.75s4.365 9.75 9.75 9.75 9.75-4.365 9.75-9.75S17.385 2.25 12 2.25zm-.53 14.03a.75.75 0 001.06 0l3-3a.75.75 0 10-1.06-1.06l-1.72 1.72V8.25a.75.75 0 00-1.5 0v5.69l-1.72-1.72a.75.75 0 00-1.06 1.06l3 3z"
+                            clipRule="evenodd"
+                          />
+                        </svg>
+                      </a>
+                    </td>
+                  </tr>
+                ))}
+              </tbody>
+            </table>
+          </div>
+          {dataset.files && dataset.files.length > 0 && (
+            <>
+              <h2 className="mb-0 mt-8">Data Previews</h2>
+              {dataset.files?.map((file) => (
+                <div
+                  key={file}
+                  id={file.split('/').slice(-1).join('')}
+                  className="preview-table my-8"
+                >
+                  <h3>{file.split('/').slice(-1)}</h3>
+                  <FlatUiTable url={file} />
+                </div>
+              ))}
+            </>
+          )}
         </main>
+      </Layout>
+    </>
+  );
+}
+
+export function DesktopItem({ dataset }: { dataset: Dataset }) {
+  return (
+    <>
+      {dataset.articles.map((article, index) => (
+        <tr
+          key={article.url}
+          className={`${
+            index === dataset.articles.length - 1 ? 'border-b' : ''
+          } border-zinc-400`}
+        >
+          <td>
+            <a
+              className="py-8 font-bold hover:underline pr-2"
+              href={article.url}
+            >
+              {article.title}
+            </a>
+          </td>
+          <td className="py-8 font-light text-[14px] min-w-[138px] font-mono text-[#999]">
+            {format(article.date).includes('years')
+              ? new Date(article.date).toLocaleString('en-US', options)
+              : format(article.date)}
+          </td>
+          <td className="py-8 text-end">
+            {index === 0 && (
+              <a
+                className="ml-auto border border-zinc-900 font-light px-[25px] py-2.5 text-sm transition hover:bg-zinc-900 hover:text-white"
+                href={dataset.url}
+              >
+                info
+              </a>
+            )}
+          </td>
+        </tr>
+      ))}
     </>
   );
 }
@@ -101,6 +188,7 @@ export async function getStaticPaths() {
     fallback: false, // can also be true or 'blocking'
   };
 }
+// change href base check datahub-next
 
 export const getStaticProps: GetStaticProps = async ({ params }) => {
   const datasetsFile = path.join(process.cwd(), 'datasets.json');
@@ -110,15 +198,20 @@ export const getStaticProps: GetStaticProps = async ({ params }) => {
     (_dataset) => _dataset.name === params?.datasetName
   );
   const github_pat = getConfig().serverRuntimeConfig.github_pat;
-  const readmes = await Promise.all(['/README.md', '/readme.md', '/Readme.md'].map(async (readme) => await getProjectReadme(
+  const readmes = await Promise.all(
+    ['/README.md', '/readme.md', '/Readme.md'].map(
+      async (readme) =>
+        await getProjectReadme(
           'fivethirtyeight',
           'data',
           'master',
           dataset?.name + readme,
           github_pat
-  )));
+        )
-  const readme = readmes.find(item => item !== null)
+    )
-  if (!readme) console.log('Readme not found for ' + dataset?.name)
+  );
+  const readme = readmes.find((item) => item !== null);
+  if (!readme) console.log('Readme not found for ' + dataset?.name);
   return {
     props: {
       dataset: {

examples/fivethirtyeight/pages/index.tsx

@@ -4,6 +4,7 @@ import { format } from 'timeago.js';
 import { promises as fs } from 'fs';
 import path from 'path';
 import { NextSeo } from 'next-seo';
+import Layout from '@/components/Layout';
 
 const inter = Inter({ subsets: ['latin'] });
 
@@ -51,21 +52,12 @@ export function MobileItem({ dataset }: { dataset: Dataset }) {
         >
           info
         </a>
-        {/*
+        <a
-                <button>
+          className="ml-2 border border-[#3c3c3c] px-[25px] py-2.5 text-sm transition bg-[#3c3c3c] text-white hover:bg-zinc-900"
-                  <svg
+          href={`/datasets/${dataset.name}`}
-                    xmlns="http://www.w3.org/2000/svg"
-                    viewBox="0 0 24 24"
-                    fill="currentColor"
-                    className="w-12 h-12 text-blue-400 hover:text-blue-300 transition mt-1"
         >
-                    <path
+          explore
-                      fillRule="evenodd"
+        </a>
-                      d="M12 2.25c-5.385 0-9.75 4.365-9.75 9.75s4.365 9.75 9.75 9.75 9.75-4.365 9.75-9.75S17.385 2.25 12 2.25zm-.53 14.03a.75.75 0 001.06 0l3-3a.75.75 0 10-1.06-1.06l-1.72 1.72V8.25a.75.75 0 00-1.5 0v5.69l-1.72-1.72a.75.75 0 00-1.06 1.06l3 3z"
-                      clipRule="evenodd"
-                    />
-                  </svg>
-                </button> */}
       </div>
     </div>
   );
@@ -97,6 +89,16 @@ export function DesktopItem({ dataset }: { dataset: Dataset }) {
               ? new Date(article.date).toLocaleString('en-US', options)
               : format(article.date)}
           </td>
+          <td>
+            {index === 0 && (
+            <a
+              className="ml-2 border border-[#3c3c3c] px-[25px] py-2.5 text-sm transition bg-[#3c3c3c] text-white hover:bg-zinc-900"
+              href={`/datasets/${dataset.name}`}
+            >
+              explore
+            </a>
+            )}
+          </td>
           <td className="py-8">
             {index === 0 && (
               <a
@@ -107,23 +109,6 @@ export function DesktopItem({ dataset }: { dataset: Dataset }) {
               </a>
             )}
           </td>
-          {/*
-              <td>
-                <button>
-                  <svg
-                    xmlns="http://www.w3.org/2000/svg"
-                    viewBox="0 0 24 24"
-                    fill="currentColor"
-                    className="w-12 h-12 text-blue-400 hover:text-blue-300 transition mt-1"
-                  >
-                    <path
-                      fillRule="evenodd"
-                      d="M12 2.25c-5.385 0-9.75 4.365-9.75 9.75s4.365 9.75 9.75 9.75 9.75-4.365 9.75-9.75S17.385 2.25 12 2.25zm-.53 14.03a.75.75 0 001.06 0l3-3a.75.75 0 10-1.06-1.06l-1.72 1.72V8.25a.75.75 0 00-1.5 0v5.69l-1.72-1.72a.75.75 0 00-1.06 1.06l3 3z"
-                      clipRule="evenodd"
-                    />
-                  </svg>
-                </button>
-              </td>*/}
         </tr>
       ))}
     </>
@@ -143,6 +128,7 @@ export default function Home({ datasets }: { datasets: Dataset[] }) {
   return (
     <>
       <NextSeo title="FiveThirtyEight tribute by PortalJS" />
+      <Layout>
       <main
         className={`flex min-h-screen flex-col items-center max-w-5xl mx-auto pt-20 px-2.5 ${inter.className}`}
       >
@@ -206,6 +192,7 @@ export default function Home({ datasets }: { datasets: Dataset[] }) {
           .
         </p>
       </main>
+      </Layout>
     </>
   );
 }

examples/fivethirtyeight/styles/globals.css

@@ -3,6 +3,9 @@
 @tailwind utilities;
 
 .preview-table > div {
-  overflow-x: scroll;
   overflow-y: hidden;
 }
+
+.prose h1 {
+  font-size: 1.5em !important;
+}

examples/github-backed-catalog/README.md

@@ -6,7 +6,7 @@ A `datasets.json` file is used to specify which datasets are going to be part of
 
 The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
 
-You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
+You can read more about it on the [Data catalog with data on GitHub](https://portaljs.com/docs/examples/github-backed-catalog) blog post.
 
 ## Demo
 

examples/github-backed-catalog/components/DataRichDocument.tsx

@@ -0,0 +1,21 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@portaljs/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
+  Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
+  FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
+  mermaid: Mermaid,
+  Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
+  VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),
+  LineChart: dynamic(() => import('@portaljs/components').then(mod => mod.LineChart)),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/github-backed-catalog/datasets.json

@@ -15,6 +15,13 @@
     ],
     "readme": "README.md"
   },
+  {
+    "owner": "luccasmmg",
+    "branch": "main",
+    "repo": "test-data-repo-1",
+    "files": ["data_1.csv", "data_2.csv"],
+    "readme": "README.md"
+  },
   {
     "owner": "datasets",
     "branch": "main",

examples/github-backed-catalog/lib/markdown.js

@@ -0,0 +1,105 @@
+import matter from "gray-matter";
+import mdxmermaid from "mdx-mermaid";
+import { h } from "hastscript";
+import remarkCallouts from "@portaljs/remark-callouts";
+import remarkEmbed from "@portaljs/remark-embed";
+import remarkGfm from "remark-gfm";
+import remarkMath from "remark-math";
+import remarkSmartypants from "remark-smartypants";
+import remarkToc from "remark-toc";
+import remarkWikiLink from "@portaljs/remark-wiki-link";
+import rehypeAutolinkHeadings from "rehype-autolink-headings";
+import rehypeKatex from "rehype-katex";
+import rehypeSlug from "rehype-slug";
+import rehypePrismPlus from "rehype-prism-plus";
+
+import { serialize } from "next-mdx-remote/serialize";
+
+/**
+ * Parse a markdown or MDX file to an MDX source form + front matter data
+ *
+ * @source: the contents of a markdown or mdx file
+ * @format: used to indicate to next-mdx-remote which format to use (md or mdx)
+ * @returns: { mdxSource: mdxSource, frontMatter: ...}
+ */
+const parse = async function (source, format, scope) {
+  const { content, data, excerpt } = matter(source, {
+    excerpt: (file, options) => {
+      // Generate an excerpt for the file
+      file.excerpt = file.content.split("\n\n")[0];
+    },
+  });
+
+  const mdxSource = await serialize(
+    { value: content, path: format },
+    {
+      // Optionally pass remark/rehype plugins
+      mdxOptions: {
+        remarkPlugins: [
+          remarkEmbed,
+          remarkGfm,
+          [remarkSmartypants, { quotes: false, dashes: "oldschool" }],
+          remarkMath,
+          remarkCallouts,
+          remarkWikiLink,
+          [
+            remarkToc,
+            {
+              heading: "Table of contents",
+              tight: true,
+            },
+          ],
+          [mdxmermaid, {}],
+        ],
+        rehypePlugins: [
+          rehypeSlug,
+          [
+            rehypeAutolinkHeadings,
+            {
+              properties: { className: 'heading-link' },
+              test(element) {
+                return (
+                  ["h2", "h3", "h4", "h5", "h6"].includes(element.tagName) &&
+                  element.properties?.id !== "table-of-contents" &&
+                  element.properties?.className !== "blockquote-heading"
+                );
+              },
+              content() {
+                return [
+                  h(
+                    "svg",
+                    {
+                      xmlns: "http:www.w3.org/2000/svg",
+                      fill: "#ab2b65",
+                      viewBox: "0 0 20 20",
+                      className: "w-5 h-5",
+                    },
+                    [
+                      h("path", {
+                        fillRule: "evenodd",
+                        clipRule: "evenodd",
+                        d: "M9.493 2.853a.75.75 0 00-1.486-.205L7.545 6H4.198a.75.75 0 000 1.5h3.14l-.69 5H3.302a.75.75 0 000 1.5h3.14l-.435 3.148a.75.75 0 001.486.205L7.955 14h2.986l-.434 3.148a.75.75 0 001.486.205L12.456 14h3.346a.75.75 0 000-1.5h-3.14l.69-5h3.346a.75.75 0 000-1.5h-3.14l.435-3.147a.75.75 0 00-1.486-.205L12.045 6H9.059l.434-3.147zM8.852 7.5l-.69 5h2.986l.69-5H8.852z",
+                      }),
+                    ]
+                  ),
+                ];
+              },
+            },
+          ],
+          [rehypeKatex, { output: "mathml" }],
+          [rehypePrismPlus, { ignoreMissing: true }],
+        ],
+        format,
+      },
+      scope,
+    }
+  );
+
+  return {
+    mdxSource: mdxSource,
+    frontMatter: data,
+    excerpt,
+  };
+};
+
+export default parse;

examples/github-backed-catalog/lib/mddb.ts

@@ -0,0 +1,14 @@
+import { MarkdownDB } from "mddb";
+
+const dbPath = "markdown.db";
+
+const client = new MarkdownDB({
+  client: "sqlite3",
+  connection: {
+    filename: dbPath,
+  },
+});
+
+const clientPromise = client.init();
+
+export default clientPromise;

examples/github-backed-catalog/lib/octokit.ts

@@ -39,6 +39,32 @@ export async function getProjectReadme(
   }
 }
 
+export async function getProjectDatapackage(
+  owner: string,
+  repo: string,
+  branch: string,
+  github_pat?: string
+) {
+  const octokit = new Octokit({ auth: github_pat });
+  try {
+    const response = await octokit.rest.repos.getContent({
+      owner,
+      repo,
+      path: "datapackage.json",
+      ref: branch,
+    });
+    const data = response.data as { content?: string };
+    const fileContent = data.content ? data.content : "";
+    if (fileContent === "") {
+      return null;
+    }
+    const decodedContent = Buffer.from(fileContent, "base64").toString();
+    return JSON.parse(decodedContent);
+  } catch (error) {
+    return null
+  }
+}
+
 export async function getLastUpdated(
   owner: string,
   repo: string,
@@ -162,11 +188,20 @@ export async function getProject(project: GithubProject, github_pat?: string) {
     projectBase,
     github_pat
   );
+
+  const projectDatapackage = await getProjectDatapackage(
+    project.owner,
+    project.repo,
+    project.branch,
+    github_pat
+  );
+
   return {
     ...projectMetadata,
     files: projectData,
     readmeContent: projectReadme,
     last_updated,
     base_path: projectBase,
+    datapackage: projectDatapackage
   };
 }

examples/github-backed-catalog/package.json

@@ -10,19 +10,33 @@
     "prettier": "prettier --write ."
   },
   "dependencies": {
+    "@portaljs/components": "^0.1.6",
+    "@portaljs/core": "^1.0.5",
+    "@portaljs/remark-callouts": "^1.0.5",
+    "@portaljs/remark-embed": "^1.0.4",
+    "@portaljs/remark-wiki-link": "^1.0.4",
     "@types/node": "18.16.0",
     "@types/react": "18.0.38",
     "@types/react-dom": "18.0.11",
     "eslint": "8.39.0",
     "eslint-config-next": "13.3.1",
-    "next": "13.3.1",
+    "mddb": "^0.1.9",
+    "next": "13.4.3",
+    "next-mdx-remote": "^4.4.1",
     "next-seo": "^6.0.0",
     "octokit": "^2.0.14",
     "react": "18.2.0",
     "react-dom": "18.2.0",
     "react-markdown": "^8.0.7",
     "react-timeago": "^7.1.0",
+    "rehype-autolink-headings": "^6.1.1",
+    "rehype-katex": "^6.0.3",
+    "rehype-prism-plus": "^1.5.1",
+    "rehype-slug": "^5.1.0",
     "remark-gfm": "^3.0.1",
+    "remark-math": "^5.1.1",
+    "remark-smartypants": "^2.0.0",
+    "remark-toc": "^8.0.1",
     "typescript": "5.0.4"
   },
   "devDependencies": {

examples/github-backed-catalog/pages/@org/[org]/[...path].tsx

@@ -6,6 +6,8 @@ import { getProject, GithubProject } from "../../../lib/octokit";
 import ReactMarkdown from "react-markdown";
 import remarkGfm from "remark-gfm";
 import Breadcrumbs from "../../../components/_shared/Breadcrumbs";
+import parse from '../../../lib/markdown';
+import DataRichDocument from '../../../components/DataRichDocument'
 
 export default function ProjectPage({ project }) {
   const repoId = `@${project.repo_config.owner}/${project.repo_config.repo}`;
@@ -64,9 +66,7 @@ export default function ProjectPage({ project }) {
         <hr />
 
         <h2 className="uppercase font-black">Readme</h2>
-        <ReactMarkdown remarkPlugins={[remarkGfm]}>
+        <DataRichDocument source={project.mdxSource} />
-          {project.readmeContent}
-        </ReactMarkdown>
       </main>
     </>
   );
@@ -119,9 +119,10 @@ export async function getStaticProps({ params }) {
   });
   const github_pat = getConfig().serverRuntimeConfig.github_pat;
   const project = await getProject(repo, github_pat);
+  let { mdxSource, frontMatter } = await parse(project.readmeContent, '.mdx', { project });
   return {
     props: {
-      project: { ...project, repo_config: repo },
+      project: { ...project, repo_config: repo, mdxSource },
     },
   };
 }

examples/github-backed-catalog/pages/index.tsx

@@ -40,7 +40,7 @@ export function Datasets({ projects }) {
               <Link
                 target="_blank"
                 className="underline"
-                href="https://portaljs.org/"
+                href="https://portaljs.com/"
               >
                 🌀 PortalJS
               </Link>

examples/github-backed-catalog/pages/styles.css

@@ -78,3 +78,72 @@ pre {
   color: rgba(55, 65, 81, 1);
   width: 100%;
 }
+
+@import "@portaljs/remark-callouts/styles.css";
+
+/* mathjax */
+.math-inline > mjx-container > svg {
+  display: inline;
+  align-items: center;
+}
+
+/* smooth scrolling in modern browsers */
+html {
+  scroll-behavior: smooth !important;
+}
+
+/* tooltip fade-out clip */
+.tooltip-body::after {
+  content: "";
+  position: absolute;
+  right: 0;
+  top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
+  height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
+  width: 10rem;
+  background: linear-gradient(
+    to right,
+    rgba(255, 255, 255, 0),
+    rgba(255, 255, 255, 1) 100%
+  );
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
+  margin-left: -2rem !important;
+  padding-left: 2rem !important;
+  scroll-margin-top: 4.5rem;
+  position: relative;
+}
+
+.heading-link {
+  padding: 1px;
+  position: absolute;
+  left: 0;
+  top: 50%;
+  transform: translateY(-50%);
+  margin: auto 0;
+  border-radius: 5px;
+  background: #1e293b;
+  opacity: 0;
+  transition: opacity 0.2s;
+}
+
+.light .heading-link {
+  /* border: 1px solid #ab2b65; */
+  /* background: none; */
+  background: #e2e8f0;
+}
+
+:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
+  opacity: 100;
+}
+
+.heading-link svg {
+  transform: scale(0.75);
+}
+
+@media screen and (max-width: 640px) {
+  .heading-link {
+    visibility: hidden;
+  }
+}
+

examples/learn-example/README.md

@@ -1 +0,0 @@
-PortalJS Learn Example - https://portaljs.org/docs

examples/learn/README.md

@@ -0,0 +1 @@
+PortalJS Learn Example - https://portaljs.com/docs

examples/learn/components/DataRichDocument.tsx

@@ -0,0 +1,21 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@portaljs/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
+  Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
+  mermaid: Mermaid,
+  Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
+  VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),
+  LineChart: dynamic(() => import('@portaljs/components').then(mod => mod.LineChart)),
+  FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/learn/lib/markdown.js

@@ -0,0 +1,105 @@
+import matter from "gray-matter";
+import mdxmermaid from "mdx-mermaid";
+import { h } from "hastscript";
+import remarkCallouts from "@portaljs/remark-callouts";
+import remarkEmbed from "@portaljs/remark-embed";
+import remarkGfm from "remark-gfm";
+import remarkMath from "remark-math";
+import remarkSmartypants from "remark-smartypants";
+import remarkToc from "remark-toc";
+import remarkWikiLink from "@portaljs/remark-wiki-link";
+import rehypeAutolinkHeadings from "rehype-autolink-headings";
+import rehypeKatex from "rehype-katex";
+import rehypeSlug from "rehype-slug";
+import rehypePrismPlus from "rehype-prism-plus";
+
+import { serialize } from "next-mdx-remote/serialize";
+
+/**
+ * Parse a markdown or MDX file to an MDX source form + front matter data
+ *
+ * @source: the contents of a markdown or mdx file
+ * @format: used to indicate to next-mdx-remote which format to use (md or mdx)
+ * @returns: { mdxSource: mdxSource, frontMatter: ...}
+ */
+const parse = async function (source, format, scope) {
+  const { content, data, excerpt } = matter(source, {
+    excerpt: (file, options) => {
+      // Generate an excerpt for the file
+      file.excerpt = file.content.split("\n\n")[0];
+    },
+  });
+
+  const mdxSource = await serialize(
+    { value: content, path: format },
+    {
+      // Optionally pass remark/rehype plugins
+      mdxOptions: {
+        remarkPlugins: [
+          remarkEmbed,
+          remarkGfm,
+          [remarkSmartypants, { quotes: false, dashes: "oldschool" }],
+          remarkMath,
+          remarkCallouts,
+          remarkWikiLink,
+          [
+            remarkToc,
+            {
+              heading: "Table of contents",
+              tight: true,
+            },
+          ],
+          [mdxmermaid, {}],
+        ],
+        rehypePlugins: [
+          rehypeSlug,
+          [
+            rehypeAutolinkHeadings,
+            {
+              properties: { className: 'heading-link' },
+              test(element) {
+                return (
+                  ["h2", "h3", "h4", "h5", "h6"].includes(element.tagName) &&
+                  element.properties?.id !== "table-of-contents" &&
+                  element.properties?.className !== "blockquote-heading"
+                );
+              },
+              content() {
+                return [
+                  h(
+                    "svg",
+                    {
+                      xmlns: "http:www.w3.org/2000/svg",
+                      fill: "#ab2b65",
+                      viewBox: "0 0 20 20",
+                      className: "w-5 h-5",
+                    },
+                    [
+                      h("path", {
+                        fillRule: "evenodd",
+                        clipRule: "evenodd",
+                        d: "M9.493 2.853a.75.75 0 00-1.486-.205L7.545 6H4.198a.75.75 0 000 1.5h3.14l-.69 5H3.302a.75.75 0 000 1.5h3.14l-.435 3.148a.75.75 0 001.486.205L7.955 14h2.986l-.434 3.148a.75.75 0 001.486.205L12.456 14h3.346a.75.75 0 000-1.5h-3.14l.69-5h3.346a.75.75 0 000-1.5h-3.14l.435-3.147a.75.75 0 00-1.486-.205L12.045 6H9.059l.434-3.147zM8.852 7.5l-.69 5h2.986l.69-5H8.852z",
+                      }),
+                    ]
+                  ),
+                ];
+              },
+            },
+          ],
+          [rehypeKatex, { output: "mathml" }],
+          [rehypePrismPlus, { ignoreMissing: true }],
+        ],
+        format,
+      },
+      scope,
+    }
+  );
+
+  return {
+    mdxSource: mdxSource,
+    frontMatter: data,
+    excerpt,
+  };
+};
+
+export default parse;

examples/learn/lib/mddb.ts

@@ -0,0 +1,14 @@
+import { MarkdownDB } from "mddb";
+
+const dbPath = "markdown.db";
+
+const client = new MarkdownDB({
+  client: "sqlite3",
+  connection: {
+    filename: dbPath,
+  },
+});
+
+const clientPromise = client.init();
+
+export default clientPromise;

examples/learn/next.config.js

@@ -0,0 +1,5 @@
+const nextConfig = {
+  swcMinify: false
+}
+
+module.exports = nextConfig

examples/learn/package.json

@@ -12,25 +12,27 @@
     "mddb": "mddb ./content"
   },
   "dependencies": {
-    "@flowershow/core": "^0.4.10",
+    "@githubocto/flat-ui": "^0.14.1",
-    "@flowershow/markdowndb": "^0.1.1",
-    "@flowershow/remark-callouts": "^1.0.0",
-    "@flowershow/remark-embed": "^1.0.0",
-    "@flowershow/remark-wiki-link": "^1.1.2",
     "@heroicons/react": "^2.0.17",
     "@opentelemetry/api": "^1.4.0",
-    "@portaljs/components": "^0.1.0",
+    "@portaljs/components": "^0.1.8",
+    "@portaljs/core": "^1.0.5",
+    "@portaljs/remark-callouts": "^1.0.5",
+    "@portaljs/remark-embed": "^1.0.4",
+    "@portaljs/remark-wiki-link": "^1.0.4",
     "@tanstack/react-table": "^8.8.5",
     "flexsearch": "0.7.21",
     "gray-matter": "^4.0.3",
     "hastscript": "^7.2.0",
+    "mddb": "^0.1.9",
     "mdx-mermaid": "2.0.0-rc7",
     "next": "13.2.1",
     "next-mdx-remote": "^4.4.1",
     "papaparse": "^5.4.1",
-    "react": "18.2.0",
+    "react": "^18.2.0",
-    "react-dom": "18.2.0",
+    "react-dom": "^18.2.0",
     "react-hook-form": "^7.43.9",
+    "react-query": "^3.39.3",
     "react-vega": "^7.6.0",
     "rehype-autolink-headings": "^6.1.1",
     "rehype-katex": "^6.0.3",
@@ -40,7 +42,9 @@
     "remark-math": "^5.1.1",
     "remark-smartypants": "^2.0.0",
     "remark-toc": "^8.0.1",
-    "typescript": "5.0.4"
+    "typescript": "5.0.4",
+    "vega": "5.25.0",
+    "vega-lite": "5.1.0"
   },
   "devDependencies": {
     "@tailwindcss/typography": "^0.5.9",

examples/learn/pages/[[...path]].tsx

@@ -81,7 +81,7 @@ export default function DatasetPage({ mdxSource, frontMatter }) {
               <p className="my-0">
                 <span className="font-semibold">Modified: </span>
                 <span className="description my-0">
-                  {new Date(frontMatter.modified).toLocaleDateString()}
+                  {new Date(frontMatter.modified).toLocaleDateString("en-US")}
                 </span>
               </p>
             )}

examples/learn/postcss.config.js

@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}

examples/learn/styles/globals.css

@@ -1,7 +1,7 @@
 @tailwind base;
 @tailwind components;
 @tailwind utilities;
-@import "@flowershow/remark-callouts/styles.css";
+@import "@portaljs/remark-callouts/styles.css";
 
 .w-5 {
   width: 1.25rem

examples/openspending/README.md

@@ -6,7 +6,7 @@ A `datasets.json` file is used to specify which datasets are going to be part of
 
 The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
 
-You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
+You can read more about it on the [Data catalog with data on GitHub](https://portaljs.com/docs/examples/github-backed-catalog) blog post.
 
 ## Demo
 

examples/openspending/__tests__/os-data.test.ts

@@ -1,45 +1,116 @@
-import { Octokit } from 'octokit';
+import { expect, test } from 'vitest';
-import { assert, expect, test } from 'vitest'
+import { getAllProjectsFromOrg, getProjectDataPackage } from '../lib/project';
-import { getProjectDataPackage } from '../lib/octokit';
+import { loadDataPackage } from '../lib/loader';
+import { getProjectMetadata } from '../lib/project';
+import { validate } from 'datapackage';
 
-export async function getAllDataPackagesFromOrg(
+test(
-  org: string,
+  'Test OS-Data',
-  branch?: string,
+  async () => {
-  github_pat?: string
+    const repos = await getAllProjectsFromOrg(
-) {
+      'os-data',
-  const octokit = new Octokit({ auth: github_pat });
+      'main',
-  const repos = await octokit.rest.repos.listForOrg({ org, type: 'public', per_page: 100 });
+      process.env.VITE_GITHUB_PAT
-  let failedDataPackages = [];
-  const datapackages = await Promise.all(
-    repos.data.map(async (_repo) => {
-      const datapackage = await getProjectDataPackage(
-        org,
-        _repo.name,
-        branch ? branch : 'main',
-        github_pat
     );
-      if (!datapackage) {
+    if (repos.failed.length > 0)
-        failedDataPackages.push(_repo.name)
+      console.log('Failed to get datapackage on', repos.failed);
-        return null
+    let failedDatapackages = await Promise.all(
-      };
+      repos.results.map(async (item) => {
-      return {...datapackage, repo: _repo.name};
+        try {
+          const { valid, errors } = await validate(item.datapackage);
+          return errors.length > 0 ? item.repo.name : null;
+        } catch {
+          return item.repo.name;
+        }
       })
     );
-  return {
+    failedDatapackages = failedDatapackages.filter((item) => item !== null);
-    datapackages: datapackages.filter((item) => item !== null),
+    if (failedDatapackages.length > 0) {
-    failedDataPackages,
+      console.log('Failed to validate datapackage on ', failedDatapackages);
-  };
+    } else {
-}
+      console.log('No invalid packages');
+    }
+  },
+  { timeout: 100000 }
+);
 
-test('Test OS-Data', async () => {
+test(
-  const repos = await getAllDataPackagesFromOrg('os-data', 'main', process.env.VITE_GITHUB_PAT)
+  'Test Gift-Data',
-  if (repos.failedDataPackages.length > 0) console.log(repos.failedDataPackages)
+  async () => {
-  expect(repos.failedDataPackages.length).toBe(0)
+    const repos = await getAllProjectsFromOrg(
-}, {timeout: 100000})
+      'gift-data',
-
+      'main',
-test('Test Gift-Data', async () => {
+      process.env.VITE_GITHUB_PAT
-  const repos = await getAllDataPackagesFromOrg('gift-data', 'main', process.env.VITE_GITHUB_PAT)
+    );
-  if (repos.failedDataPackages.length > 0) console.log(repos.failedDataPackages)
+    if (repos.failed.length > 0) console.log(repos.failed);
-  expect(repos.failedDataPackages.length).toBe(0)
+    let failedDatapackages = await Promise.all(
-}, {timeout: 100000})
+      repos.results.map(async (item) => {
+        try {
+          const { valid, errors } = await validate(item.datapackage);
+          return errors.length > 0 ? item.repo.name : null;
+        } catch {
+          return item.repo.name;
+        }
+      })
+    );
+    failedDatapackages = failedDatapackages.filter((item) => item !== null);
+    if (failedDatapackages.length > 0) {
+      console.log('Failed to validate datapackage on ', failedDatapackages);
+    } else {
+      console.log('No invalid packages');
+    }
+  },
+  { timeout: 100000 }
+);
 
+test(
+  'Test getting one dataset from github',
+  async () => {
+    const datapackage = await getProjectDataPackage(
+      'os-data',
+      'berlin-berlin',
+      'main',
+      process.env.VITE_GITHUB_PAT
+    );
+    const repo = await getProjectMetadata(
+      'os-data',
+      'berlin-berlin',
+      process.env.VITE_GITHUB_PAT
+    );
+    const project = loadDataPackage(datapackage, repo);
+    delete project['datapackage'];
+    delete project.files[0]['dialect'];
+    delete project.files[0]['schema'];
+    expect(project).toStrictEqual({
+      name: 'berlin-berlin',
+      title: 'Berlin-Berlin',
+      description: null,
+      owner: {
+        name: 'os-data',
+        logo: 'https://avatars.githubusercontent.com/u/13695166?v=4',
+        title: 'os-data',
+      },
+      repo: {
+        name: 'berlin-berlin',
+        full_name: 'os-data/berlin-berlin',
+        url: 'https://github.com/os-data/berlin-berlin',
+      },
+      files: [
+        {
+          name: 'berlin-gesamt',
+          format: 'csv',
+          path: 'https://storage.openspending.org/berlin-berlin/berlin-gesamt.csv',
+          mediatype: 'text/csv',
+          bytes: 81128743,
+          encoding: 'utf-8',
+        },
+      ],
+      author: 'Michael Peters <michael.peters@okfn.de>',
+      cityCode: 'Berlin',
+      countryCode: 'DE',
+      fiscalPeriod: { start: '2014-01-01', end: '2019-12-31' },
+      readme: '',
+    });
+  },
+  { timeout: 100000 }
+);

examples/openspending/components/DataRichDocument.tsx

@@ -0,0 +1,29 @@
+import { MDXRemote } from 'next-mdx-remote';
+import dynamic from 'next/dynamic';
+import { Mermaid } from '@portaljs/core';
+
+// Custom components/renderers to pass to MDX.
+// Since the MDX files aren't loaded by webpack, they have no knowledge of how
+// to handle import statements. Instead, you must include components in scope
+// here.
+const components = {
+  Table: dynamic(() => import('@portaljs/components').then((mod) => mod.Table)),
+  Catalog: dynamic(() =>
+    import('@portaljs/components').then((mod) => mod.Catalog)
+  ),
+  mermaid: Mermaid,
+  Vega: dynamic(() => import('@portaljs/components').then((mod) => mod.Vega)),
+  VegaLite: dynamic(() =>
+    import('@portaljs/components').then((mod) => mod.VegaLite)
+  ),
+  LineChart: dynamic(() =>
+    import('@portaljs/components').then((mod) => mod.LineChart)
+  ),
+  FlatUiTable: dynamic(() =>
+    import('@portaljs/components').then((mod) => mod.FlatUiTable)
+  ),
+} as any;
+
+export default function DRD({ source }: { source: any }) {
+  return <MDXRemote {...source} components={components} />;
+}

examples/openspending/components/DatasetCard.tsx

@@ -9,7 +9,7 @@ export default function DatasetCard({ dataset }: { dataset: Project }) {
       className="overflow-hidden rounded-xl border border-gray-200"
     >
       <Link
-        href=""
+        href={`/@${dataset.owner.name}/${dataset.repo.name}`}
         className="flex items-center gap-x-4 border-b border-gray-900/5 bg-gray-50 p-6"
       >
         <img
@@ -45,12 +45,12 @@ export default function DatasetCard({ dataset }: { dataset: Project }) {
           <dt className="text-gray-500">Fiscal Period</dt>
           <dd className="text-gray-700">
             {dataset.fiscalPeriod?.start &&
-              new Date(dataset.fiscalPeriod.start).getFullYear()}
+              new Date(dataset.fiscalPeriod.start).getUTCFullYear()}
             {dataset.fiscalPeriod?.end &&
               dataset.fiscalPeriod?.start !== dataset.fiscalPeriod?.end && (
                 <>
                   {' - '}
-                  {new Date(dataset.fiscalPeriod.end).getFullYear()}
+                  {new Date(dataset.fiscalPeriod.end).getUTCFullYear()}
                 </>
               )}
           </dd>
@@ -60,8 +60,8 @@ export default function DatasetCard({ dataset }: { dataset: Project }) {
           <dd className="flex items-start gap-x-2">
             <div className="font-medium text-gray-900">
               <Link
-                // TODO: where do we get the info needed for this link?
+                // TODO: this link may be incorrect for some datasets
-                href=""
+                href={`https://github.com/${dataset.owner.name}/${dataset.repo.name}/blob/main/datapackage.json`}
                 target="_blank"
                 className="flex items-center hover:text-gray-700"
               >

examples/openspending/components/DatasetsSearch.tsx

@@ -2,9 +2,30 @@ import { useForm } from 'react-hook-form';
 import DatasetsGrid from './DatasetsGrid';
 import { Project } from '../lib/project.interface';
 import { Index } from 'flexsearch';
+import {
+  ChevronDoubleLeftIcon,
+  ChevronDoubleRightIcon,
+  ChevronLeftIcon,
+  ChevronRightIcon,
+} from '@heroicons/react/24/solid';
+import { useState } from 'react';
+
+export default function DatasetsSearch({
+  datasets,
+  availableCountries,
+  minPeriod,
+  maxPeriod,
+}: {
+  datasets: Project[];
+  availableCountries;
+  minPeriod: string;
+  maxPeriod: string;
+}) {
+  const itemsPerPage = 6;
+  const [page, setPage] = useState(1);
 
-export default function DatasetsSearch({ datasets }: { datasets: Project[] }) {
   const index = new Index({ tokenize: 'full' });
+
   datasets.forEach((dataset: Project) =>
     index.add(
       dataset.name,
@@ -21,12 +42,53 @@ export default function DatasetsSearch({ datasets }: { datasets: Project[] }) {
     },
   });
 
-  const allCountries = datasets
+  const filteredDatasets = datasets
-    .map((item) => item.countryCode)
+    .filter((dataset: Project) =>
-    .filter((v) => v) //  Filters false values
+      watch().searchTerm && watch().searchTerm !== ''
-    .filter((v, i, a) => a.indexOf(v) === i) //  Remove duplicates
+        ? index.search(watch().searchTerm).includes(dataset.name)
-    //  TODO: title should be the full name
+        : true
-    .map((code) => ({ code, title: code }));
+    )
+    .filter((dataset) =>
+      watch().country && watch().country !== ''
+        ? dataset.countryCode === watch().country
+        : true
+    )
+    .filter((dataset) => {
+      const filterMinDate = watch().minDate;
+      const filterMaxDate = watch().maxDate;
+
+      const datasetMinDate = dataset.fiscalPeriod?.start;
+      const datasetMaxDate = dataset.fiscalPeriod?.end;
+
+      let datasetStartOverlaps = false;
+      if (datasetMinDate) {
+        datasetStartOverlaps =
+          datasetMinDate >= filterMinDate && datasetMinDate <= filterMaxDate;
+      }
+
+      let datasetEndOverlaps = false;
+      if (datasetMaxDate) {
+        datasetEndOverlaps =
+          datasetMaxDate >= filterMinDate && datasetMaxDate <= filterMaxDate;
+      }
+
+      if (filterMinDate && filterMaxDate) {
+        return datasetStartOverlaps || datasetEndOverlaps;
+      } else if (filterMinDate) {
+        return datasetMinDate >= filterMinDate;
+      } else if (filterMaxDate) {
+        return datasetMinDate <= filterMaxDate;
+      }
+
+      return true;
+    });
+
+  const paginatedDatasets = filteredDatasets.slice(
+    (page - 1) * itemsPerPage,
+    (page - 1) * itemsPerPage + itemsPerPage
+  );
+
+  const pageCount = Math.ceil(filteredDatasets.length / itemsPerPage) || 1;
 
   return (
     <>
@@ -37,7 +99,7 @@ export default function DatasetsSearch({ datasets }: { datasets: Project[] }) {
             <input
               placeholder="Search datasets"
               aria-label="Search datasets"
-              {...register('searchTerm')}
+              {...register('searchTerm', { onChange: () => setPage(1) })}
               className="h-[3em] relative w-full rounded-lg bg-white py-2 pl-3 pr-10 text-left shadow-md focus:outline-none focus-visible:border-emerald-500 focus-visible:ring-2 focus-visible:ring-white focus-visible:ring-opacity-75 focus-visible:ring-offset-2 focus-visible:ring-offset-emerald-400 sm:text-sm"
             />
             {watch().searchTerm !== '' && (
@@ -55,10 +117,10 @@ export default function DatasetsSearch({ datasets }: { datasets: Project[] }) {
           <label className="text-sm text-gray-600 font-medium">Country</label>
           <select
             className="h-[3em] w-full rounded-lg bg-white py-2 pl-3 pr-10 text-left shadow-md focus:outline-none focus-visible:border-emerald-500 focus-visible:ring-2 focus-visible:ring-white focus-visible:ring-opacity-75 focus-visible:ring-offset-2 focus-visible:ring-offset-emerald-400 sm:text-sm"
-            {...register('country')}
+            {...register('country', { onChange: () => setPage(1) })}
           >
             <option value="">All</option>
-            {allCountries.map((country) => {
+            {availableCountries.map((country) => {
               return (
                 <option key={country.code} value={country.code}>
                   {country.title}
@@ -68,73 +130,83 @@ export default function DatasetsSearch({ datasets }: { datasets: Project[] }) {
           </select>
         </div>
         <div className="sm:basis-1/6">
-          <label className="text-sm text-gray-600 font-medium">Min. date</label>
+          <label className="text-sm text-gray-600 font-medium">
+            Fiscal Period Start
+          </label>
           <div className="relative">
             <input
               aria-label="Min. date"
-              type="date"
+              type="text"
-              {...register('minDate')}
+              placeholder={minPeriod}
+              onFocus={(e) => (e.target.type = 'date')}
+              onBlur={(e) => (e.target.type = 'text')}
+              {...register('minDate', { onChange: () => setPage(1) })}
               className="h-[3em] w-full rounded-lg bg-white py-2 pl-3 pr-10 text-left shadow-md focus:outline-none focus-visible:border-emerald-500 focus-visible:ring-2 focus-visible:ring-white focus-visible:ring-opacity-75 focus-visible:ring-offset-2 focus-visible:ring-offset-emerald-400 sm:text-sm"
             />
-            {watch().minDate !== '' && (
-              <button
-                onClick={() => resetField('minDate')}
-                className="absolute right-3 top-1/2 -translate-y-1/2 text-gray-500"
-              >
-                <CloseIcon />
-              </button>
-            )}
           </div>
         </div>
         <div className="sm:basis-1/6">
-          <label className="text-sm text-gray-600 font-medium">Max. date</label>
+          <label className="text-sm text-gray-600 font-medium">
+            Fiscal Period End
+          </label>
           <div className="relative">
             <input
               aria-label="Max. date"
-              type="date"
+              type="text"
-              {...register('maxDate')}
+              placeholder={maxPeriod}
+              onFocus={(e) => (e.target.type = 'date')}
+              onBlur={(e) => (e.target.type = 'text')}
+              {...register('maxDate', { onChange: () => setPage(1) })}
               className="h-[3em] w-full rounded-lg bg-white py-2 pl-3 pr-10 text-left shadow-md focus:outline-none focus-visible:border-emerald-500 focus-visible:ring-2 focus-visible:ring-white focus-visible:ring-opacity-75 focus-visible:ring-offset-2 focus-visible:ring-offset-emerald-400 sm:text-sm"
             />
-            {watch().maxDate !== '' && (
+          </div>
+        </div>
+      </div>
+      <div className="mt-5 mb-5">
+        <span className="text-lg font-medium">
+          {filteredDatasets.length} datasets found
+        </span>
+      </div>
+      <div className="min-w-full align-middle">
+        <DatasetsGrid datasets={paginatedDatasets} />
+        <div className="w-full flex justify-center mt-10">
           <button
-                onClick={() => resetField('maxDate')}
+            onClick={() => setPage(1)}
-                className="absolute right-3 top-1/2 -translate-y-1/2 text-gray-500"
+            disabled={page <= 1}
+            className="disabled:text-gray-400"
           >
-                <CloseIcon />
+            <ChevronDoubleLeftIcon className="w-6 h-6" />
+          </button>
+          <button
+            onClick={() => {
+              if (page > 1) setPage((prev) => --prev);
+            }}
+            disabled={page <= 1}
+            className="disabled:text-gray-400"
+          >
+            <ChevronLeftIcon className="w-6 h-6" />
+          </button>
+          <span className="mx-5">
+            Page {page} of {pageCount}
+          </span>
+          <button
+            onClick={() => {
+              if (page < pageCount) setPage((prev) => ++prev);
+            }}
+            disabled={page >= pageCount}
+            className="disabled:text-gray-400"
+          >
+            <ChevronRightIcon className="w-6 h-6" />
+          </button>
+          <button
+            onClick={() => setPage(pageCount)}
+            disabled={page >= pageCount}
+            className="disabled:text-gray-400"
+          >
+            <ChevronDoubleRightIcon className="w-6 h-6" />
           </button>
-            )}
         </div>
       </div>
-      </div>
-      <div className="min-w-full mt-10 align-middle">
-        <DatasetsGrid
-          datasets={datasets
-            .filter((dataset: Project) =>
-              watch().searchTerm && watch().searchTerm !== ''
-                ? index.search(watch().searchTerm).includes(dataset.name)
-                : true
-            )
-            .filter((dataset) =>
-              watch().country && watch().country !== ''
-                ? dataset.countryCode === watch().country
-                : true
-            )
-            // TODO: Does that really makes sense?
-            // What if the fiscalPeriod is 2015-2017 and inputs are
-            // set to 2015-2016. It's going to be filtered out but
-            // it shouldn't.
-            .filter((dataset) =>
-              watch().minDate && watch().minDate !== ''
-                ? dataset.fiscalPeriod?.start >= watch().minDate
-                : true
-            )
-            .filter((dataset) =>
-              watch().maxDate && watch().maxDate !== ''
-                ? dataset.fiscalPeriod?.end <= watch().maxDate
-                : true
-            )}
-        />
-      </div>
     </>
   );
 }
@@ -153,9 +225,9 @@ const CloseIcon = () => {
           id="Vector"
           d="M18 18L12 12M12 12L6 6M12 12L18 6M12 12L6 18"
           stroke="currentColor"
-          stroke-width="2"
+          strokeWidth="2"
-          stroke-linecap="round"
+          strokeLinecap="round"
-          stroke-linejoin="round"
+          strokeLinejoin="round"
         />
       </g>
     </svg>

examples/openspending/components/FlatUiTable.tsx

@@ -0,0 +1,115 @@
+import { QueryClient, QueryClientProvider, useQuery } from 'react-query';
+import Papa from 'papaparse';
+import { Grid } from '@githubocto/flat-ui';
+
+const queryClient = new QueryClient();
+
+export async function getCsv(url: string, corsProxy?: string, range?: string) {
+  if (corsProxy) {
+    url = corsProxy + url
+  }
+  const response = await fetch(url, {
+    headers: {
+      Range: range ? `bytes=0-${range}` : 'bytes=0-512000',
+    },
+  });
+  const data = await response.text();
+  return data;
+}
+
+export async function parseCsv(file: string): Promise<any> {
+  return new Promise((resolve, reject) => {
+    Papa.parse(file, {
+      header: true,
+      dynamicTyping: true,
+      skipEmptyLines: true,
+      transform: (value: string): string => {
+        return value.trim();
+      },
+      complete: (results: any) => {
+        return resolve(results);
+      },
+      error: (error: any) => {
+        return reject(error);
+      },
+    });
+  });
+}
+
+const Spinning = () => {
+  return (
+    <div role="status w-fit mx-auto">
+      <svg
+        aria-hidden="true"
+        className="w-8 h-8 mr-2 text-gray-200 animate-spin dark:text-gray-600 fill-emerald-600"
+        viewBox="0 0 100 101"
+        fill="none"
+        xmlns="http://www.w3.org/2000/svg"
+      >
+        <path
+          d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z"
+          fill="currentColor"
+        />
+        <path
+          d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z"
+          fill="currentFill"
+        />
+      </svg>
+      <span className="sr-only">Loading...</span>
+    </div>
+  );
+};
+
+export interface FlatUiTableProps {
+  url?: string;
+  data?: { [key: string]: number | string }[];
+  rawCsv?: string;
+  range?: string;
+  corsProxy?: string;
+}
+export const FlatUiTable: React.FC<FlatUiTableProps> = ({
+  url,
+  data,
+  rawCsv,
+  corsProxy,
+  range
+}) => {
+  return (
+    // Provide the client to your App
+    <QueryClientProvider client={queryClient}>
+      <TableInner range={range} corsProxy={corsProxy} url={url} data={data} rawCsv={rawCsv} />
+    </QueryClientProvider>
+  );
+};
+
+const TableInner: React.FC<FlatUiTableProps> = ({ url, data, rawCsv, corsProxy, range }) => {
+  if (data) {
+    return (
+      <div className="w-full" style={{height: '500px'}}>
+        <Grid data={data} />
+      </div>
+    );
+  }
+  const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
+    ['dataCsv', url],
+    () => getCsv(url as string, corsProxy, range),
+    { enabled: !!url }
+  );
+  const { data: parsedData, isLoading: isParsing } = useQuery(
+    ['dataPreview', csvString],
+    () => parseCsv(rawCsv ? rawCsv as string : csvString as string),
+    { enabled: rawCsv ? true : !!csvString }
+  );
+  if (isParsing || isDownloadingCSV)
+    <div className="w-full">
+      <Spinning />
+    </div>;
+  if (parsedData)
+    return (
+      <div className="w-full" style={{height: '500px'}}>
+        <Grid data={parsedData.data} />
+      </div>
+    );
+  return <Spinning />
+};
+

examples/openspending/components/Header.tsx

@@ -1,53 +1,202 @@
-import Image from 'next/image'
+import Image from 'next/image';
-import { Button } from './Button'
+import { Container } from './Container';
-import { Container } from './Container'
+import logo from '../public/logo.svg';
-import logo from "../public/logo.svg"
+import Link from 'next/link';
-import Link from 'next/link'
+import { useRouter } from 'next/router';
-import { useRouter } from 'next/router'
+import { Bars3Icon } from '@heroicons/react/24/outline';
+import { useState } from 'react';
+import { Fragment } from 'react';
+import { Menu, Transition } from '@headlessui/react';
+import { ChevronDownIcon } from '@heroicons/react/20/solid';
 
 export function Header() {
+  const [menuOpen, setMenuOpen] = useState<boolean>(false);
+
   const router = useRouter();
 
   const isActive = (navLink) => {
-    return router.asPath.split("?")[0] == navLink.href;
+    return router.asPath.split('?')[0] == navLink.href;
-  }
+  };
 
   const navLinks = [
     {
-      title: "Home",
+      title: 'Datasets',
-      href: "/#header"
+      href: '/#datasets',
     },
     {
-      title: "Datasets",
+      title: 'Data Stories',
-      href: "/#datasets"
+      href: '/stories',
     },
     {
-      title: "Community",
+      title: 'Blog',
-      href: "https://community.openspending.org/"
+      href: '/blog',
-    }
+    },
-  ]
+    {
+      title: 'About',
+      href: '/about',
+      children: [
+        {
+          title: 'Fiscal Data Package',
+          href: '/about/fiscaldatapackage/',
+        },
+        {
+          title: 'Tools',
+          href: '/about/tools/',
+        },
+        {
+          title: 'Funders',
+          href: '/about/funders/',
+        },
+        {
+          title: 'Presentations',
+          href: '/about/presentations/',
+        },
+      ],
+    },
+    {
+      title: 'Contributing',
+      href: '/contributing',
+    },
+    {
+      title: 'Help',
+      href: '/help',
+    },
+    {
+      title: 'Resources',
+      href: '/resources',
+      children: [
+        {
+          title: 'Follow the money',
+          href: '/resources/journo',
+        },
+        {
+          title: 'Map of Spending Projects',
+          href: '/resources/map-of-spending-projects/',
+        },
+        {
+          title: 'Working Group On Open Spending Data',
+          href: '/resources/wg/',
+        },
+        {
+          title: 'UK Departamental Spending',
+          href: '/resources/gb-spending',
+        },
+      ],
+    },
+  ];
 
   return (
-    <header className="z-50 pb-5 lg:pt-11 sticky top-0 backdrop-blur" id="header">
+    <header className="relative z-50 pb-11 lg:pt-11">
-      <Container className="flex flex-wrap items-center justify-center sm:justify-between lg:flex-nowrap">
+      <Container className="flex flex-wrap justify-between lg:flex-nowrap mt-10 lg:mt-0">
-        <div className="mt-10 lg:mt-0 lg:grow lg:basis-0 flex items-center">
+        <Link href="/" className="lg:mt-0 lg:grow lg:basis-0 flex items-center">
           <Image src={logo} alt="OpenSpending" className="h-12 w-auto" />
+        </Link>
+        <ul className="hidden list-none sm:flex gap-x-5 text-base font-medium">
+          {navLinks.map((link, i) => (
+            <li key={`nav-link-${i}`}>
+              <Dropdown navItem={link} />
+            </li>
+          ))}
+        </ul>
+        <div className="sm:hidden sm:mt-10 lg:mt-0 lg:grow lg:basis-0 lg:justify-end">
+          <button onClick={() => setMenuOpen(!menuOpen)}>
+            <Bars3Icon className="w-8 h-8" />
+          </button>
         </div>
-        <ul className='list-none flex gap-x-5 text-base font-medium'>
+        {menuOpen && (
+          <div className={`sm:hidden basis-full mt-5 text-center`}>
+            <ul className="gap-x-5 text-base font-medium">
               {navLinks.map((link, i) => (
                 <li key={`nav-link-${i}`}>
                   <Link
-                className={`text-emerald-900 hover:text-emerald-600 ${isActive(link) ? "text-emerald-600" : ""}`}
+                    className={`text-emerald-900 hover:text-emerald-600 ${
+                      isActive(link) ? 'text-emerald-600' : ''
+                    }`}
                     href={link.href}
                     scroll={false}
                   >
                     {link.title}
                   </Link>
-            </li>))}
+                </li>
+              ))}
             </ul>
-        <div className="hidden sm:mt-10 sm:flex lg:mt-0 lg:grow lg:basis-0 lg:justify-end">
           </div>
+        )}
       </Container>
-    </header >
+    </header>
-  )
+  );
+}
+
+function classNames(...classes) {
+  return classes.filter(Boolean).join(' ');
+}
+
+function Dropdown({ navItem }: { navItem: any }) {
+  const [showDropDown, setShowDropDown] = useState(false);
+  return (
+    <Menu as="div" className="relative inline-block text-left">
+      {({ open }) => (
+        <>
+          <div>
+            <Menu.Button
+              onMouseEnter={() => setShowDropDown(true)}
+              onMouseLeave={() => setShowDropDown(false)}
+              className="text-emerald-900 hover:text-emerald-600 inline-flex w-full justify-center gap-x-1.5 px-3 py-2 text-sm font-semibold"
+            >
+              <Link href={navItem.href}>{navItem.title}</Link>
+              {navItem.children && (
+                <ChevronDownIcon
+                  className="-mr-1 h-5 w-5 text-gray-400"
+                  aria-hidden="true"
+                />
+              )}
+            </Menu.Button>
+          </div>
+
+          {navItem.children && (
+            <Transition
+              as={Fragment}
+              show={showDropDown}
+              enter="transition ease-out duration-100"
+              enterFrom="transform opacity-0 scale-95"
+              enterTo="transform opacity-100 scale-100"
+              leave="transition ease-in duration-75"
+              leaveFrom="transform opacity-100 scale-100"
+              leaveTo="transform opacity-0 scale-95"
+            >
+              <div>
+                <Menu.Items
+                  static
+                  onMouseEnter={() => setShowDropDown(true)}
+                  onMouseLeave={() => setShowDropDown(false)}
+                  className="absolute right-0 z-10 w-56 origin-top-right rounded-md bg-white shadow-lg ring-1 ring-black ring-opacity-5 focus:outline-none"
+                >
+                  <div className="py-1">
+                    {navItem.children.map((item) => (
+                      <Menu.Item key={item.href}>
+                        {({ active }) => (
+                          <a
+                            key={item.href}
+                            href={item.href}
+                            className={classNames(
+                              active
+                                ? 'bg-gray-100 text-emerald-900 hover:text-emerald-600'
+                                : 'text-gray-700',
+                              'block px-4 py-2 text-sm'
+                            )}
+                          >
+                            {item.title}
+                          </a>
+                        )}
+                      </Menu.Item>
+                    ))}
+                  </div>
+                </Menu.Items>
+              </div>
+            </Transition>
+          )}
+        </>
+      )}
+    </Menu>
+  );
 }

examples/openspending/components/Hero.tsx

@@ -1,9 +1,9 @@
-import { Button } from './Button'
+import { Button } from './Button';
-import { Container } from './Container'
+import { Container } from './Container';
 
-export function Hero() {
+export function Hero({ countriesCount, datasetsCount, filesCount }) {
   return (
-    <div className="relative pb-20 pt-10 sm:py-40">
+    <div className="relative pb-20 pt-10 sm:py-40" id="hero">
       <div className="absolute inset-x-0 -bottom-14 -top-48 overflow-hidden bg-green-50 bg-opacity-50">
         <div className="absolute inset-x-0 top-0 h-40 bg-gradient-to-b from-white" />
         <div className="absolute inset-x-0 bottom-0 h-40 bg-gradient-to-t from-white" />
@@ -15,33 +15,44 @@ export function Hero() {
           </h1>
           <div className="mt-6 space-y-6 font-display text-2xl tracking-tight text-emerald-900">
             <p>
-              By understanding how governments spend money in our name can we have a say
+              By understanding how governments spend money in our name can we
-              in how that money will affect our own lives. The journey starts here.
+              have a say in how that money will affect our own lives. The
+              journey starts here.
             </p>
             <p>
-              OpenSpending is a free, open and global platform to search, visualise and analyse
+              OpenSpending is a free, open and global platform to search,
-              fiscal data in the public sphere.
+              visualise and analyse fiscal data in the public sphere.
             </p>
           </div>
           <Button href="#datasets" className="mt-10">
             Search datasets
           </Button>
-          <dl className="mt-10 grid grid-cols-2 gap-x-10 gap-y-6 sm:mt-16 sm:gap-x-16 sm:gap-y-10 sm:text-center lg:auto-cols-auto lg:grid-flow-col lg:grid-cols-none lg:justify-start lg:text-left">
+          <dl className="mt-10 grid grid-cols-1 sm:grid-cols-3 gap-x-10 gap-y-6 sm:mt-16 sm:gap-x-16 sm:gap-y-10 sm:text-center lg:auto-cols-auto lg:grid-flow-col lg:grid-cols-none lg:justify-start lg:text-left">
             {[
-              ['Countries', '75'],
+              //  Added the plus sign because some datasets do not
-              ['Datasets', '2091'],
+              //  contain defined countries
-              ['Files', '9230'],
+              ['Countries', '+' + countriesCount],
+              ['Datasets', datasetsCount],
+              ['Files', filesCount],
             ].map(([name, value]) => (
               <div key={name}>
+                <div className='flex gap-x-2 items-center sm:hidden' key={name}>
+                  <dd className="mt-0.5 text-2xl font-semibold tracking-tight text-emerald-900">
+                    {value}
+                  </dd>
+                  <dt className="font-mono text-sm text-emerald-600">{name}</dt>
+                </div>
+                <div className='hidden sm:block' key={name}>
                   <dt className="font-mono text-sm text-emerald-600">{name}</dt>
                   <dd className="mt-0.5 text-2xl font-semibold tracking-tight text-emerald-900">
                     {value}
                   </dd>
                 </div>
+              </div>
             ))}
           </dl>
         </div>
       </Container>
     </div>
-  )
+  );
 }