Compare commits

..

1 Commits

Author SHA1 Message Date
Luccas Mateus de Medeiros Gomes
5a04d8e4c0 [alan-turing][m] - fix markdown 2023-05-02 15:26:38 -03:00
1614 changed files with 29778 additions and 659088 deletions

View File

@ -1,8 +0,0 @@
# Changesets
Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
with multi-package repos, or single-package repos to help you version and publish your code. You can
find the full documentation for it [in our repository](https://github.com/changesets/changesets)
We have a quick list of common questions to get you started engaging with this project in
[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

View File

@ -1,14 +0,0 @@
{
"$schema": "https://unpkg.com/@changesets/config@2.3.0/schema.json",
"changelog": [
"@changesets/changelog-github",
{ "repo": "datopian/portaljs" }
],
"commit": false,
"fixed": [],
"linked": [],
"access": "restricted",
"baseBranch": "main",
"updateInternalDependencies": "patch",
"ignore": []
}

View File

@ -1,39 +0,0 @@
name: Release
on:
push:
branches:
- main
concurrency: release-${{ github.ref }}
jobs:
release:
name: Release
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout@v3
- name: Setup Node.js 16.x
uses: actions/setup-node@v3
with:
node-version: 16.x
- name: Install Dependencies
run: npm ci
- name: Create Release Pull Request or Publish to npm
id: changesets
uses: changesets/action@v1
with:
publish: npm run release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
# - name: Send a Discord notification if a publish happens
# if: steps.changesets.outputs.published == 'true'
# uses: Ilshidur/action-discord@0.3.2
# with:
# args: 'The project {{ EVENT_PAYLOAD.repository.full_name }} has been deployed.'

2
.gitignore vendored
View File

@ -4,7 +4,6 @@
dist
tmp
/out-tsc
**/*.tgz
# dependencies
node_modules
@ -17,7 +16,6 @@ node_modules
*.launch
.settings/
*.sublime-workspace
.obsidian
# IDE - VSCode
.vscode/*

8
.vscode/extensions.json vendored Normal file
View File

@ -0,0 +1,8 @@
{
"recommendations": [
"nrwl.angular-console",
"esbenp.prettier-vscode",
"firsttris.vscode-jest-runner",
"dbaeumer.vscode-eslint"
]
}

View File

@ -4,7 +4,7 @@ title: Developer docs for contributors
## Our repository
https://github.com/datopian/datahub
https://github.com/datopian/portaljs
Structure:
@ -13,11 +13,11 @@ Structure:
- **dataset-frictionless**: Example utilizing a frictionless dataset as an example
- **site**: the website for the project, with a landing page and the docs
- **packages**:
- **portaljs-components**: the library of components for creating a data portal
- **portaljs-components**: the library of components for creating a data portal
## How to contribute
You can start by checking our [issues board](https://github.com/datopian/datahub/issues).
You can start by checking our [issues board](https://github.com/datopian/portaljs/issues).
If you'd like to work on one of the issues you can:
@ -26,16 +26,15 @@ If you'd like to work on one of the issues you can:
3. Clone the forked repository to your machine.
4. Create a feature branch (e.g. `50-update-readme`, where `50` is the number of the related issue).
5. Commit your changes to the feature branch.
6. Add changeset file describing the changes. (See section below)
7. Push the feature branch to your forked repository.
8. Create a Pull Request against the original repository.
6. Push the feature branch to your forked repository.
7. Create a Pull Request against the original repository.
- add a short description of the changes included in the PR
9. Address review comments if requested by our demanding reviewers 😜.
8. Address review comments if requested by our demanding reviewers 😜.
If you have an idea for improvement, and it doesn't have a corresponding issue yet, simply submit a new one.
> [!note]
> Join our [Discord channel](https://discord.gg/KZSf3FG4EZ) do discuss existing issues and to ask for help.
> Join our [Discord channel](https://discord.gg/rTxfCutu) do discuss existing issues and to ask for help.
## Nx
@ -63,7 +62,6 @@ or you can use just:
nx <target> <project>
# e.g. npx nx serve ckan
```
if you have the `nx` binary installed globally in your machine
#### Running multiple tasks
@ -111,7 +109,7 @@ To check code formatting in selected projects:
```sh
npx nx format:check --projects=<array of projects>
# npx nx format:check --projects=ckan,dataset-frictionless
# npx nx format:check --projects=ckan,dataset-frictionless
```
To check code formatting in all projects:
@ -126,7 +124,7 @@ To fix code formatting in selected projects:
```sh
npx nx format:write --projects=<array projects>
# npx nx format:write --projects=ckan,dataset-frictionless
# npx nx format:write --projects=ckan,dataset-frictionless
```
To fix formatting in all projects:
@ -176,23 +174,3 @@ To learn more see this [offical docs page](https://nx.dev/reference/nx-json).
Each project also has it's own configuration file - `project.json`, where you can define and configure it's targets (and more).
To learn more see this [offical docs page](https://nx.dev/reference/project-configuration).
## Changesets and publishing packages
> This monorepo is set up with changesets versioning tool. See their [github repository](https://github.com/changesets/changesets) to learn more.
### What are Changesets?
Changesets are files that describe the intention of a contributor to bump a version of the package according to their changes. Changeset file holds two key bits of information: a version type (following semver), and change information to be added to a changelog.
### Adding changesets
In the root directory of the repo, run:
```
npx changeset
```
Select the package that has been changed, the semver version that should be bumped with it and a description of your changes. Please make sure to add the most accurate but also concise information.
To learn about semantic versioning standards see [this semver doc page](https://semver.org/).

View File

@ -1,51 +1,31 @@
<p align="center">
Bugs, issues and suggestions re PortalJS framework
<br />
<br /><a href="https://discord.gg/xfFDMPU9dC"><img src="https://dcbadge.vercel.app/api/server/xfFDMPU9dC" /></a>
</p>
<h1 align="center">
🌀 Portal.JS
<br />
Rapidly build rich data portals using a modern frontend framework
</h1>
## PortalJS framework
* [What is Portal.JS ?](#What-is-Portal.JS)
* [Features](#Features)
* [For developers](#For-developers)
* [Docs](#Docs)
* [Community](#Community)
* [Appendix](#Appendix)
* [What happened to Recline?](#What-happened-to-Recline?)
This repo and issue tracker are for
# What is Portal.JS
- PortalJS 🌀 - https://www.portaljs.com/
- DataHub Cloud ☁️ - https://datahub.io/
🌀 Portal.JS is a framework for rapidly building rich data portal frontends using a modern frontend approach. Portal.JS can be used to present a single dataset or build a full-scale data catalog/portal.
### Issues
Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com/) framework. Portal.JS assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
Found a bug: 👉 https://github.com/datopian/portaljs/issues/new
### Discussions
Got a suggestion, a question, want some support or just want to shoot the breeze 🙂
Head to the discussion forum: 👉 https://github.com/datopian/portaljs/discussions
### Chat on Discord
If you would prefer to get help via live chat check out our discord 👉
[Discord](https://discord.gg/xfFDMPU9dC)
### Docs
- For PortalJS go to https://www.portaljs.com/opensource
- For DataHub Cloud https://datahub.io/docs
## PortalJS Cloud 🌀
PortalJS Cloud 🌀 is a platform for rapidly creating rich data portal and publishing systems using a modern frontend approach. PortalJS Cloud can be used to publish a single dataset or build a full-scale data catalog/portal.
PortalJS Cloud is built in JavaScript and React on top of the popular [Next.js](https://nextjs.org) framework. PortalJS Cloud assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/), GitHub, Frictionless Data Packages and more.
### Features
## Features
- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. Wordpress) with a common internal API.
- 👩‍💻 Developer friendly: built with familiar frontend tech (JavaScript, React, Next.js).
- 🔋 Batteries included: full set of portal components out of the box e.g. catalog search, dataset showcase, blog, etc.
- 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
- 🧱 Extensible: quickly extend and develop/import your own React components
- 📝 Well documented: full set of documentation plus the documentation of Next.js.
- 📝 Well documented: full set of documentation plus the documentation of Next.js and Apollo.
### For developers
@ -53,3 +33,25 @@ PortalJS Cloud is built in JavaScript and React on top of the popular [Next.js](
- 🚀 Next.js framework: so everything in Next.js for free: Server Side Rendering, Static Site Generation, huge number of examples and integrations, etc.
- Server Side Rendering (SSR) => Unlimited number of pages, SEO and more whilst still using React.
- Static Site Generation (SSG) => Ultra-simple deployment, great performance, great lighthouse scores and more (good for small sites)
#### **Check out the [Portal.JS website](https://portaljs.org/) for a gallery of live portals**
___
# Docs
Access the Portal.JS documentation at:
https://portaljs.org/docs
- [Examples](https://portaljs.org/docs#examples)
# Community
If you have questions about anything related to Portal.JS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portal.js/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
# Appendix
## What happened to Recline?
Portal.JS used to be Recline(JS). If you are looking for the old Recline codebase it still exists: see the [`recline` branch](https://github.com/datopian/portal.js/tree/recline). If you want context for the rename see [this issue](https://github.com/datopian/portal.js/issues/520).

View File

@ -0,0 +1,27 @@
## Intro
This page catalogues datasets annotated for hate speech, online abuse, and offensive language. They may be useful for e.g. training a natural language processing system to detect this language.
Its built on top of [PortalJS](https://portaljs.org/), it allows you to publish datasets, lists of offensive keywords and static pages, all of those are stored as markdown files inside the `content` folder.
- .md files inside `content/datasets/` will appear on the dataset list section of the homepage and be searchable as well as having a individual page in `datasets/<file name>`
- .md files inside `content/keywords/` will appear on the list of offensive keywords section of the homepage as well as having a individual page in `keywords/<file name>`
- .md files inside `content/` will be converted to static pages in the url `/<file name>` eg: `content/about.md` becomes `/about`
This is also a Next.JS project so you can use the following steps to run the website locally.
## Getting started
To get started first install the npm dependencies:
```bash
npm install
```
Next, run the development server:
```bash
npm run dev
```
Finally, open [http://localhost:3000](http://localhost:3000) in your browser to view the website.

View File

@ -21,7 +21,7 @@ export function Footer() {
<Container.Inner>
<div className="flex flex-col items-center justify-between gap-6 sm:flex-row">
<p className="text-sm font-medium text-zinc-800 dark:text-zinc-200">
Built with <a href='https://portaljs.com'>PortalJS 🌀</a>
Built with <a href='https://portaljs.org'>PortalJS 🌀</a>
</p>
<p className="text-sm text-zinc-400 dark:text-zinc-500">
&copy; {new Date().getFullYear()} Leon Derczynski. All rights

View File

@ -0,0 +1,109 @@
import { useRef } from 'react'
import { useRouter } from 'next/router'
import { Container } from '../components/Container'
function SunIcon(props) {
return (
<svg
viewBox="0 0 24 24"
strokeWidth="1.5"
strokeLinecap="round"
strokeLinejoin="round"
aria-hidden="true"
{...props}
>
<path d="M8 12.25A4.25 4.25 0 0 1 12.25 8v0a4.25 4.25 0 0 1 4.25 4.25v0a4.25 4.25 0 0 1-4.25 4.25v0A4.25 4.25 0 0 1 8 12.25v0Z" />
<path
d="M12.25 3v1.5M21.5 12.25H20M18.791 18.791l-1.06-1.06M18.791 5.709l-1.06 1.06M12.25 20v1.5M4.5 12.25H3M6.77 6.77 5.709 5.709M6.77 17.73l-1.061 1.061"
fill="none"
/>
</svg>
)
}
function MoonIcon(props) {
return (
<svg viewBox="0 0 24 24" aria-hidden="true" {...props}>
<path
d="M17.25 16.22a6.937 6.937 0 0 1-9.47-9.47 7.451 7.451 0 1 0 9.47 9.47ZM12.75 7C17 7 17 2.75 17 2.75S17 7 21.25 7C17 7 17 11.25 17 11.25S17 7 12.75 7Z"
strokeWidth="1.5"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}
function ModeToggle() {
function disableTransitionsTemporarily() {
document.documentElement.classList.add('[&_*]:!transition-none')
window.setTimeout(() => {
document.documentElement.classList.remove('[&_*]:!transition-none')
}, 0)
}
function toggleMode() {
disableTransitionsTemporarily()
let darkModeMediaQuery = window.matchMedia('(prefers-color-scheme: dark)')
let isSystemDarkMode = darkModeMediaQuery.matches
let isDarkMode = document.documentElement.classList.toggle('dark')
if (isDarkMode === isSystemDarkMode) {
delete window.localStorage.isDarkMode
} else {
window.localStorage.isDarkMode = isDarkMode
}
}
return (
<button
type="button"
aria-label="Toggle dark mode"
className="group rounded-full bg-white/90 px-3 py-2 shadow-lg shadow-zinc-800/5 ring-1 ring-zinc-900/5 backdrop-blur transition dark:bg-zinc-800/90 dark:ring-white/10 dark:hover:ring-white/20"
onClick={toggleMode}
>
<SunIcon className="h-6 w-6 fill-zinc-100 stroke-zinc-500 transition group-hover:fill-zinc-200 group-hover:stroke-zinc-700 dark:hidden [@media(prefers-color-scheme:dark)]:fill-teal-50 [@media(prefers-color-scheme:dark)]:stroke-teal-500 [@media(prefers-color-scheme:dark)]:group-hover:fill-teal-50 [@media(prefers-color-scheme:dark)]:group-hover:stroke-teal-600" />
<MoonIcon className="hidden h-6 w-6 fill-zinc-700 stroke-zinc-500 transition dark:block [@media(prefers-color-scheme:dark)]:group-hover:stroke-zinc-400 [@media_not_(prefers-color-scheme:dark)]:fill-teal-400/10 [@media_not_(prefers-color-scheme:dark)]:stroke-teal-500" />
</button>
)
}
export function Header() {
let isHomePage = useRouter().pathname === '/'
let headerRef = useRef()
return (
<>
<header
className="pointer-events-none relative z-50 flex flex-col"
style={{
height: 'var(--header-height)',
marginBottom: 'var(--header-mb)',
}}
>
<div
ref={headerRef}
className="top-0 z-10 h-16 pt-6"
style={{ position: 'var(--header-position)' }}
>
<Container
className="top-[var(--header-top,theme(spacing.6))] w-full"
style={{ position: 'var(--header-inner-position)' }}
>
<div className="relative flex gap-4">
<div className="flex justify-end md:flex-1">
<div className="pointer-events-auto">
<ModeToggle />
</div>
</div>
</div>
</Container>
</div>
</header>
{isHomePage && <div style={{ height: 'var(--content-offset)' }} />}
</>
)
}

View File

@ -0,0 +1,34 @@
---
title: Hate Speech Dataset Catalogue
---
This page catalogues datasets annotated for hate speech, online abuse, and offensive language. They may be useful for e.g. training a natural language processing system to detect this language.
The list is maintained by Leon Derczynski, Bertie Vidgen, Hannah Rose Kirk, Pica Johansson, Yi-Ling Chung, Mads Guldborg Kjeldgaard Kongsbak, Laila Sprejer, and Philine Zeinert.
We provide a list of datasets and keywords. If you would like to contribute to our catalogue or add your dataset, please see the instructions for contributing.
If you use these resources, please cite (and read!) our paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out. And if you would like to find other resources for researching online hate, visit The Alan Turing Institutes Online Hate Research Hub or read The Alan Turing Institutes Reading List on Online Hate and Abuse Research.
If youre looking for a good paper on online hate training datasets (beyond our paper, of course!) then have a look at Resources and benchmark corpora for hate speech detection: a systematic review by Poletto et al. in Language Resources and Evaluation.
## How to contribute
We accept entries to our catalogue based on pull requests to the content folder. The dataset must be avaliable for download to be included in the list. If you want to add an entry, follow these steps!
Please send just one dataset addition/edit at a time - edit it in, then save. This will make everyones life easier (including yours!)
- Go to the repo url file and click the "Add file" dropdown and then click on "Create new file".
![](https://i.imgur.com/2PR0ZgL.png)
- In the following page type `content/datasets/<name-of-the-file>.md`. if you want to add an entry to the datasets catalog or `content/keywords/<name-of-the-file>.md` if you want to add an entry to the lists of abusive keywords, if you want to just add an static page you can leave in the root of `content` it will automatically get assigned an url eg: `/content/about.md` becomes the `/about` page
![](https://i.imgur.com/rr3uSYu.png)
- Copy the contents of `templates/dataset.md` or `templates/keywords.md` respectively to the camp below, filling out the fields with the correct data format
![](https://i.imgur.com/x6JIjhz.png)
- Click on "Commit changes", on the popup make sure you give some brief detail on the proposed change. and then click on Propose changes
<img src='https://i.imgur.com/BxuxKEJ.png' style={{ maxWidth: '50%', margin: '0 auto' }}/>
- Submit the pull request on the next page when prompted.

View File

@ -1,13 +1,13 @@
import matter from "gray-matter";
import mdxmermaid from "mdx-mermaid";
import { h } from "hastscript";
import remarkCallouts from "@portaljs/remark-callouts";
import remarkEmbed from "@portaljs/remark-embed";
import remarkCallouts from "@flowershow/remark-callouts";
import remarkEmbed from "@flowershow/remark-embed";
import remarkGfm from "remark-gfm";
import remarkMath from "remark-math";
import remarkSmartypants from "remark-smartypants";
import remarkToc from "remark-toc";
import remarkWikiLink from "@portaljs/remark-wiki-link";
import remarkWikiLink from "@flowershow/remark-wiki-link";
import rehypeAutolinkHeadings from "rehype-autolink-headings";
import rehypeKatex from "rehype-katex";
import rehypeSlug from "rehype-slug";

View File

@ -1,4 +1,4 @@
import { MarkdownDB } from "mddb";
import { MarkdownDB } from "@flowershow/markdowndb";
const dbPath = "markdown.db";

Binary file not shown.

View File

@ -12,6 +12,11 @@
},
"browserslist": "defaults, not ie <= 11",
"dependencies": {
"@flowershow/core": "^0.4.10",
"@flowershow/markdowndb": "^0.1.1",
"@flowershow/remark-callouts": "^1.0.0",
"@flowershow/remark-embed": "^1.0.0",
"@flowershow/remark-wiki-link": "^1.1.2",
"@headlessui/react": "^1.7.13",
"@heroicons/react": "^2.0.17",
"@mapbox/rehype-prism": "^0.8.0",
@ -19,22 +24,22 @@
"@mdx-js/react": "^2.1.5",
"@next/mdx": "^13.0.2",
"@opentelemetry/api": "^1.4.0",
"@portaljs/core": "^1.0.5",
"@portaljs/remark-callouts": "^1.0.5",
"@portaljs/remark-embed": "^1.0.4",
"@portaljs/remark-wiki-link": "^1.0.4",
"@tailwindcss/forms": "^0.5.3",
"@tailwindcss/typography": "^0.5.4",
"@tanstack/react-table": "^8.8.5",
"@types/node": "18.16.0",
"@types/react": "18.2.0",
"@types/react-dom": "18.2.0",
"autoprefixer": "^10.4.12",
"clsx": "^1.2.1",
"eslint": "8.39.0",
"eslint-config-next": "13.3.1",
"fast-glob": "^3.2.11",
"feed": "^4.2.2",
"flexsearch": "^0.7.31",
"focus-visible": "^5.2.0",
"gray-matter": "^4.0.3",
"hastscript": "^7.2.0",
"mddb": "^0.1.9",
"mdx-mermaid": "^2.0.0-rc7",
"mermaid": "^10.1.0",
"next": "13.2.1",
@ -60,9 +65,6 @@
"tailwindcss": "^3.3.0"
},
"devDependencies": {
"@types/node": "18.16.0",
"@types/react": "18.2.0",
"@types/react-dom": "18.2.0",
"eslint": "8.26.0",
"eslint-config-next": "13.0.2",
"prettier": "^2.8.7",

View File

@ -2,10 +2,11 @@ import { Container } from '../components/Container'
import clientPromise from '../lib/mddb'
import { promises as fs } from 'fs';
import { MDXRemote } from 'next-mdx-remote'
import { serialize } from 'next-mdx-remote/serialize'
import { Card } from '../components/Card'
import Head from 'next/head'
import parse from '../lib/markdown'
import { Mermaid } from '@portaljs/core';
import { Mermaid } from '@flowershow/core';
export const getStaticProps = async ({ params }) => {
const urlPath = params.slug ? params.slug.join('/') : ''
@ -84,10 +85,10 @@ export default function DRDPage({ mdxSource }) {
<Head>
<title>{meta.title}</title>
</Head>
<Container className="mt-9 relative">
<Container className="mt-16 lg:mt-32">
<article>
<header className="flex flex-col">
<h1 className="text-4xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
<h1 className="mt-6 text-4xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
{meta.title}
</h1>
<Card as="article">

View File

@ -1,17 +1,25 @@
import { Head, Html, Main, NextScript } from 'next/document'
const modeScript = `
let darkModeMediaQuery = window.matchMedia('(prefers-color-scheme: dark)')
updateMode()
darkModeMediaQuery.addEventListener('change', updateModeWithoutTransitions)
window.addEventListener('storage', updateModeWithoutTransitions)
function updateMode() {
let isDarkMode = window.localStorage.isDarkMode === 'true'
let isSystemDarkMode = darkModeMediaQuery.matches
let isDarkMode = window.localStorage.isDarkMode === 'true' || (!('isDarkMode' in window.localStorage) && isSystemDarkMode)
if (isDarkMode) {
document.documentElement.classList.add('dark')
} else {
document.documentElement.classList.remove('dark')
}
if (isDarkMode === isSystemDarkMode) {
delete window.localStorage.isDarkMode
}
}
function disableTransitionsTemporarily() {

View File

@ -13,9 +13,7 @@ import { MDXRemote } from 'next-mdx-remote'
function DatasetCard({ dataset }) {
return (
<Card as="article">
<Card.Title>
<Link href={dataset.url}>{dataset.title}</Link>
</Card.Title>
<Card.Title><Link href={dataset.url}>{dataset.title}</Link></Card.Title>
<Card.Description>
<span className="font-semibold">Link to publication: </span>{' '}
<a
@ -77,9 +75,7 @@ function DatasetCard({ dataset }) {
function ListOfAbusiveKeywordsCard({ list }) {
return (
<Card as="article">
<Card.Title>
<Link href={list.url}>{list.title}</Link>
</Card.Title>
<Card.Title><Link href={list.url}>{list.title}</Link></Card.Title>
{list.description && (
<Card.Description>
<span className="font-semibold">List Description: </span>{' '}
@ -143,23 +139,17 @@ export default function Home({
<h1 className="text-4xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
{indexText.frontmatter.title}
</h1>
<article className="index-text prose mt-6 flex flex-col gap-y-2 text-base text-zinc-600 dark:prose-invert prose-h3:mt-4 prose-a:font-normal prose-a:text-zinc-600 prose-a:decoration-inherit prose-img:rounded-none dark:text-zinc-400 prose-a:dark:text-zinc-400 hover:prose-a:text-teal-600 hover:prose-a:dark:text-teal-900">
<article className="mt-6 index-text flex flex-col gap-y-2 text-base text-zinc-600 dark:text-zinc-400 prose dark:prose-invert">
<MDXRemote {...indexText} />
</article>
</div>
</Container>
<Container className="mt-12 md:mt-14">
<Container className="mt-24 md:mt-28">
<div className="mx-auto grid max-w-7xl grid-cols-1 gap-y-8 lg:max-w-none">
<h2
id="Datasets-header"
className="text-xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl"
>
<h2 className="text-xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
Datasets
</h2>
<form
onSubmit={handleSubmit(() => reset())}
className="rounded-2xl border border-zinc-100 px-4 py-6 dark:border-zinc-700/40 sm:p-6"
>
<form onSubmit={handleSubmit(() => reset())} className="rounded-2xl border border-zinc-100 px-4 py-6 dark:border-zinc-700/40 sm:p-6">
<p className="mt-2 text-lg font-semibold text-zinc-600 dark:text-zinc-100">
Search for datasets
</p>
@ -208,12 +198,7 @@ export default function Home({
</option>
))}
</select>
<button
type="submit"
className="inline-flex flex-none items-center justify-center gap-2 rounded-md bg-zinc-800 px-3 py-2 text-sm font-semibold text-zinc-100 outline-offset-2 transition hover:bg-zinc-700 active:bg-zinc-800 active:text-zinc-100/70 active:transition-none dark:bg-zinc-700 dark:hover:bg-zinc-600 dark:active:bg-zinc-700 dark:active:text-zinc-100/70"
>
Clear filters
</button>
<button type='submit' className='inline-flex items-center gap-2 justify-center rounded-md py-2 px-3 text-sm outline-offset-2 transition active:transition-none bg-zinc-800 font-semibold text-zinc-100 hover:bg-zinc-700 active:bg-zinc-800 active:text-zinc-100/70 dark:bg-zinc-700 dark:hover:bg-zinc-600 dark:active:bg-zinc-700 dark:active:text-zinc-100/70 flex-none'>Clear filters</button>
</div>
</form>
<div className="flex flex-col gap-16">
@ -240,7 +225,7 @@ export default function Home({
</div>
</Container>
<Container className="mt-16">
<h2 id="Keywords-header" className="text-xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
<h2 className="text-xl font-bold tracking-tight text-zinc-800 dark:text-zinc-100 sm:text-5xl">
Lists of Abusive Keywords
</h2>
<div className="mt-3 flex flex-col gap-16">

View File

Before

Width:  |  Height:  |  Size: 566 B

After

Width:  |  Height:  |  Size: 566 B

View File

@ -2,7 +2,6 @@
module.exports = {
content: [
"./app/**/*.{js,ts,jsx,tsx,mdx}",
"./content/**/*.{js,ts,jsx,tsx,mdx}",
"./pages/**/*.{js,ts,jsx,tsx,mdx}",
"./components/**/*.{js,ts,jsx,tsx,mdx}",
],

View File

@ -2,7 +2,7 @@
**🚩 UPDATE April 2023: This example is now deprecated - though still works!. Please use the [new CKAN examples](https://github.com/datopian/portaljs/tree/main/examples)**
This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.com/blog/example-ckan-2021)
This example shows how you can build a full data portal using a CKAN Backend with a Next.JS Frontend powered by Apollo, a full fledged guide is available as a [blog post](https://portaljs.org/blog/example-ckan-2021)
## Developers

View File

@ -0,0 +1 @@
DMS=https://demo.dev.datopian.com

View File

@ -1,7 +1,7 @@
This is a repo intended to serve as an example of a data catalog that get its data from a CKAN Instance.
```
npx create-next-app <app-name> --example https://github.com/datopian/datahub/tree/main/examples/ckan-ssg
npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/ckan-example
cd <app-name>
```
@ -19,7 +19,7 @@ npm run dev
Congratulations, you now have something similar to this running on `http://localhost:4200`
![](https://media.discordapp.net/attachments/1069718983604977754/1098252297726865408/image.png?width=853&height=461)
If you go to any one of those pages by clicking on `More info` you will see something similar to this
If yo go to any one of those pages by clicking on `More info` you will see something similar to this
![](https://media.discordapp.net/attachments/1069718983604977754/1098252298074988595/image.png?width=853&height=461)
## Deployment

View File

@ -10,26 +10,24 @@
},
"dependencies": {
"@heroicons/react": "^2.0.17",
"@portaljs/ckan": "^0.0.2",
"@portaljs/remark-wiki-link": "^1.0.4",
"@types/node": "18.16.0",
"@types/react": "18.0.38",
"@types/react-dom": "18.0.11",
"eslint": "8.39.0",
"eslint-config-next": "13.3.1",
"next": "13.3.1",
"next-seo": "^6.0.0",
"octokit": "^2.0.14",
"react": "18.2.0",
"react-dom": "18.2.0",
"react-markdown": "^8.0.7",
"remark-gfm": "^3.0.1"
"remark-gfm": "^3.0.1",
"typescript": "5.0.4"
},
"devDependencies": {
"@tailwindcss/typography": "^0.5.9",
"@types/node": "18.16.0",
"@types/react": "18.0.38",
"@types/react-dom": "18.0.11",
"autoprefixer": "^10.4.14",
"eslint": "8.39.0",
"eslint-config-next": "13.3.1",
"postcss": "^8.4.23",
"tailwindcss": "^3.3.1",
"typescript": "5.0.4"
"tailwindcss": "^3.3.1"
}
}

View File

@ -11,9 +11,8 @@ import {
ServerIcon,
UserIcon,
} from '@heroicons/react/20/solid';
import { CKAN } from '@portaljs/ckan';
const backend_url = getConfig().publicRuntimeConfig.DMS;
const dms = getConfig().publicRuntimeConfig.DMS;
const formatter = new Intl.DateTimeFormat('en-US', {
year: 'numeric',
@ -26,12 +25,14 @@ const formatter = new Intl.DateTimeFormat('en-US', {
});
export const getServerSideProps: GetServerSideProps = async (context) => {
const ckan = new CKAN(backend_url)
const { dataset } = context.query;
const _dataset = await ckan.getDatasetDetails(dataset as string)
const response = await fetch(
`${dms}/api/3/action/package_show?id=${dataset}`
);
const _dataset = await response.json();
return {
props: {
dataset: _dataset,
dataset: _dataset.result,
},
};
};

View File

@ -1,8 +1,7 @@
import getConfig from 'next/config';
import styles from './index.module.css';
import { CKAN } from '@portaljs/ckan';
const backend_url = getConfig().publicRuntimeConfig.DMS
const dms = getConfig().publicRuntimeConfig.DMS
const formatter = new Intl.DateTimeFormat('en-US', {
year: 'numeric',
@ -16,11 +15,12 @@ const formatter = new Intl.DateTimeFormat('en-US', {
export async function getServerSideProps() {
const ckan = new CKAN(backend_url)
const { datasets } = await ckan.packageSearch({ limit: 1000, offset: 0, groups:[], orgs: [], tags: []})
const datasetsWithDetails = await Promise.all(datasets.map(async (dataset) => {
const _dataset = await ckan.getDatasetDetails(dataset.name)
return _dataset
const response = await fetch(`${dms}/api/3/action/package_search`)
const datasets = await response.json()
const datasetsWithDetails = await Promise.all(datasets.result.results.map(async (dataset) => {
const response = await fetch(`${dms}/api/3/action/package_show?id=` + dataset.name)
const json = await response.json()
return json.result
}))
return {
@ -79,7 +79,7 @@ export function Index({ datasets }) {
</thead>
<tbody className="divide-y divide-gray-200">
{datasets.map((dataset) => (
<tr key={dataset.name}>
<tr>
<td className="px-3 py-4 text-sm text-gray-500">
{dataset.title}
</td>

View File

@ -1,35 +0,0 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts

View File

@ -1,38 +0,0 @@
This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

View File

@ -1,21 +0,0 @@
import { MDXRemote } from 'next-mdx-remote';
import dynamic from 'next/dynamic';
import { Mermaid } from '@portaljs/core';
// Custom components/renderers to pass to MDX.
// Since the MDX files aren't loaded by webpack, they have no knowledge of how
// to handle import statements. Instead, you must include components in scope
// here.
const components = {
Table: dynamic(() => import('@portaljs/components').then(mod => mod.Table)),
Catalog: dynamic(() => import('@portaljs/components').then(mod => mod.Catalog)),
FlatUiTable: dynamic(() => import('@portaljs/components').then(mod => mod.FlatUiTable)),
mermaid: Mermaid,
Vega: dynamic(() => import('@portaljs/components').then(mod => mod.Vega)),
VegaLite: dynamic(() => import('@portaljs/components').then(mod => mod.VegaLite)),
LineChart: dynamic(() => import('@portaljs/components').then(mod => mod.LineChart)),
} as any;
export default function DRD({ source }: { source: any }) {
return <MDXRemote {...source} components={components} />;
}

View File

@ -1,3 +0,0 @@
# Test
Test Data Rich Stories

View File

@ -1,105 +0,0 @@
import matter from "gray-matter";
import mdxmermaid from "mdx-mermaid";
import { h } from "hastscript";
import remarkCallouts from "@portaljs/remark-callouts";
import remarkEmbed from "@portaljs/remark-embed";
import remarkGfm from "remark-gfm";
import remarkMath from "remark-math";
import remarkSmartypants from "remark-smartypants";
import remarkToc from "remark-toc";
import remarkWikiLink from "@portaljs/remark-wiki-link";
import rehypeAutolinkHeadings from "rehype-autolink-headings";
import rehypeKatex from "rehype-katex";
import rehypeSlug from "rehype-slug";
import rehypePrismPlus from "rehype-prism-plus";
import { serialize } from "next-mdx-remote/serialize";
/**
* Parse a markdown or MDX file to an MDX source form + front matter data
*
* @source: the contents of a markdown or mdx file
* @format: used to indicate to next-mdx-remote which format to use (md or mdx)
* @returns: { mdxSource: mdxSource, frontMatter: ...}
*/
const parse = async function (source, format, scope) {
const { content, data, excerpt } = matter(source, {
excerpt: (file, options) => {
// Generate an excerpt for the file
file.excerpt = file.content.split("\n\n")[0];
},
});
const mdxSource = await serialize(
{ value: content, path: format },
{
// Optionally pass remark/rehype plugins
mdxOptions: {
remarkPlugins: [
remarkEmbed,
remarkGfm,
[remarkSmartypants, { quotes: false, dashes: "oldschool" }],
remarkMath,
remarkCallouts,
remarkWikiLink,
[
remarkToc,
{
heading: "Table of contents",
tight: true,
},
],
[mdxmermaid, {}],
],
rehypePlugins: [
rehypeSlug,
[
rehypeAutolinkHeadings,
{
properties: { className: 'heading-link' },
test(element) {
return (
["h2", "h3", "h4", "h5", "h6"].includes(element.tagName) &&
element.properties?.id !== "table-of-contents" &&
element.properties?.className !== "blockquote-heading"
);
},
content() {
return [
h(
"svg",
{
xmlns: "http:www.w3.org/2000/svg",
fill: "#ab2b65",
viewBox: "0 0 20 20",
className: "w-5 h-5",
},
[
h("path", {
fillRule: "evenodd",
clipRule: "evenodd",
d: "M9.493 2.853a.75.75 0 00-1.486-.205L7.545 6H4.198a.75.75 0 000 1.5h3.14l-.69 5H3.302a.75.75 0 000 1.5h3.14l-.435 3.148a.75.75 0 001.486.205L7.955 14h2.986l-.434 3.148a.75.75 0 001.486.205L12.456 14h3.346a.75.75 0 000-1.5h-3.14l.69-5h3.346a.75.75 0 000-1.5h-3.14l.435-3.147a.75.75 0 00-1.486-.205L12.045 6H9.059l.434-3.147zM8.852 7.5l-.69 5h2.986l.69-5H8.852z",
}),
]
),
];
},
},
],
[rehypeKatex, { output: "mathml" }],
[rehypePrismPlus, { ignoreMissing: true }],
],
format,
},
scope,
}
);
return {
mdxSource: mdxSource,
frontMatter: data,
excerpt,
};
};
export default parse;

View File

@ -1,14 +0,0 @@
import { MarkdownDB } from "mddb";
const dbPath = "markdown.db";
const client = new MarkdownDB({
client: "sqlite3",
connection: {
filename: dbPath,
},
});
const clientPromise = client.init();
export default clientPromise;

View File

@ -1,11 +0,0 @@
/** @type {import('next').NextConfig} */
const nextConfig = {
reactStrictMode: true,
publicRuntimeConfig: {
DMS: process.env.DMS
? process.env.DMS.replace(/\/?$/, '')
: 'https://demo.dev.datopian.com/',
},
};
module.exports = nextConfig;

File diff suppressed because it is too large Load Diff

View File

@ -1,48 +0,0 @@
{
"name": "ckan",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"prebuild": "npm run mddb",
"build": "next build",
"start": "next start",
"lint": "next lint",
"mddb": "mddb ./content"
},
"dependencies": {
"@githubocto/flat-ui": "^0.14.1",
"@heroicons/react": "^2.0.18",
"@portaljs/ckan": "^0.0.2",
"@portaljs/components": "0.1.6",
"@portaljs/core": "^1.0.5",
"@portaljs/remark-callouts": "^1.0.5",
"@portaljs/remark-embed": "^1.0.4",
"@portaljs/remark-wiki-link": "^1.0.4",
"@tailwindcss/typography": "^0.5.9",
"@types/node": "20.2.3",
"@types/react": "18.2.6",
"@types/react-dom": "18.2.4",
"autoprefixer": "10.4.14",
"eslint": "8.41.0",
"eslint-config-next": "13.4.3",
"isomorphic-unfetch": "^4.0.2",
"mddb": "^0.1.9",
"next": "13.4.3",
"next-mdx-remote": "^4.4.1",
"papaparse": "^5.4.1",
"postcss": "8.4.23",
"react": "18.2.0",
"react-dom": "18.2.0",
"react-query": "^3.39.3",
"rehype-autolink-headings": "^6.1.1",
"rehype-katex": "^6.0.3",
"rehype-prism-plus": "^1.5.1",
"rehype-slug": "^5.1.0",
"remark-math": "^5.1.1",
"remark-smartypants": "^2.0.0",
"remark-toc": "^8.0.1",
"tailwindcss": "3.3.2",
"typescript": "5.0.4"
}
}

View File

@ -1,179 +0,0 @@
import Head from "next/head";
import { CKAN, Dataset } from "@portaljs/ckan";
import {
ChevronRightIcon,
HomeIcon,
PaperClipIcon,
} from "@heroicons/react/20/solid";
import Link from "next/link";
import getConfig from "next/config";
const backend_url = getConfig().publicRuntimeConfig.DMS
export const getServerSideProps = async (context: any) => {
try {
const datasetName = context.params?.dataset;
if (!datasetName) {
return {
notFound: true,
};
}
const ckan = new CKAN(backend_url);
const dataset = await ckan.getDatasetDetails(datasetName as string);
if (!dataset) {
return {
notFound: true,
};
}
return {
props: { dataset },
};
} catch {
return {
notFound: true,
};
}
};
export default function DatasetPage({
dataset,
}: {
dataset: Dataset;
}): JSX.Element {
return (
<>
<Head>
<title>{`${dataset.title || dataset.name} - Dataset`}</title>
<meta name="description" content="Generated by create next app" />
<link rel="icon" href="/favicon.ico" />
</Head>
<main className="flex min-h-screen flex-col items-center justify-between p-24 bg-zinc-900">
<div className="bg-white p-8 my-4 rounded-lg">
<nav className="flex px-4 py-8" aria-label="Breadcrumb">
<ol role="list" className="flex items-center space-x-4">
<li>
<div>
<Link href="/" className="text-gray-400 hover:text-gray-500">
<HomeIcon
className="h-5 w-5 flex-shrink-0"
aria-hidden="true"
/>
<span className="sr-only">Home</span>
</Link>
</div>
</li>
<li>
<div className="flex items-center">
<ChevronRightIcon
className="h-5 w-5 flex-shrink-0 text-gray-400"
aria-hidden="true"
/>
<span
className="ml-4 text-sm font-medium text-gray-500 hover:text-gray-700"
aria-current={"page"}
>
{dataset.name}
</span>
</div>
</li>
</ol>
</nav>
{dataset && (
<div>
<div className="px-4 sm:px-0">
<h3 className="text-base font-semibold leading-7 text-gray-900">
{dataset.title || dataset.name}
</h3>
<p className="mt-1 max-w-2xl text-sm leading-6 text-gray-500">
Dataset details
</p>
</div>
<div className="mt-6 border-t border-gray-100">
<dl className="divide-y divide-gray-100">
<div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
<dt className="text-sm font-medium leading-6 text-gray-900">
Title
</dt>
<dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
{dataset.title}
</dd>
</div>
{dataset.tags && dataset.tags.length > 0 && (
<div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
<dt className="text-sm font-medium leading-6 text-gray-900">
Tags
</dt>
<dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
{dataset.tags.map((tag) => tag.display_name).join(", ")}
</dd>
</div>
)}
{dataset.tags && dataset.tags.length > 0 && (
<div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
<dt className="text-sm font-medium leading-6 text-gray-900">
URL
</dt>
<dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
{dataset.url}
</dd>
</div>
)}
<div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
{dataset.notes && (
<>
<dt className="text-sm font-medium leading-6 text-gray-900">
Description
</dt>
<dd className="mt-1 text-sm leading-6 text-gray-700 sm:col-span-2 sm:mt-0">
{dataset.notes}
</dd>
</>
)}
</div>
<div className="px-4 py-6 sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0">
<dt className="text-sm font-medium leading-6 text-gray-900">
Files
</dt>
<dd className="mt-2 text-sm text-gray-900 sm:col-span-2 sm:mt-0">
<ul
role="list"
className="divide-y divide-gray-100 rounded-md border border-gray-200"
>
{dataset.resources.map((resource) => (
<li key={resource.id} className="flex items-center justify-between py-4 pl-4 pr-5 text-sm leading-6">
<div className="flex w-0 flex-1 items-center">
<PaperClipIcon
className="h-5 w-5 flex-shrink-0 text-gray-400"
aria-hidden="true"
/>
<div className="ml-4 flex min-w-0 flex-1 gap-2">
<span className="truncate font-medium">
{resource.name || resource.id}
</span>
<span className="flex-shrink-0 text-gray-400">
{resource.size}
</span>
</div>
</div>
<div className="ml-4 flex-shrink-0">
<a
href={resource.url}
className="font-medium hover:text-indigo-500 mr-4"
>
Download
</a>
</div>
</li>
))}
</ul>
</dd>
</div>
</dl>
</div>
</div>
)}
</div>
</main>
</>
);
}

View File

@ -1,7 +0,0 @@
import '@/styles/globals.css'
import '@portaljs/ckan/styles.css'
import type { AppProps } from 'next/app'
export default function App({ Component, pageProps }: AppProps) {
return <Component {...pageProps} />
}

View File

@ -1,13 +0,0 @@
import { Html, Head, Main, NextScript } from 'next/document'
export default function Document() {
return (
<Html lang="en">
<Head />
<body>
<Main />
<NextScript />
</body>
</Html>
)
}

View File

@ -1,18 +0,0 @@
import fetch from 'isomorphic-unfetch';
const Cors = async (req: any, res: any) => {
const { url } = req.query;
try {
const resProxy = await fetch(url, {
headers: {
Range: 'bytes=0-5132288',
},
});
const data = await resProxy.text();
return res.status(200).send(data);
} catch (error: any) {
res.status(400).send(error.toString());
}
};
export default Cors;

View File

@ -1,64 +0,0 @@
import {
CKAN,
DatasetSearchForm,
ListOfDatasets,
PackageSearchOptions,
Organization,
Group,
} from '@portaljs/ckan';
import getConfig from 'next/config';
import { useState } from 'react';
const backend_url = getConfig().publicRuntimeConfig.DMS;
export async function getServerSideProps() {
const ckan = new CKAN(backend_url);
const groups = await ckan.getGroupsWithDetails();
const orgs = await ckan.getOrgsWithDetails();
return {
props: {
groups,
orgs,
},
};
}
export default function Home({
orgs,
groups,
}: {
orgs: Organization[];
groups: Group[];
}) {
const ckan = new CKAN(backend_url);
const [options, setOptions] = useState<PackageSearchOptions>({
offset: 0,
limit: 5,
tags: [],
groups: [],
orgs: [],
});
return (
<div>
<main className="py-12 bg-zinc-900">
<DatasetSearchForm
options={options}
setOptions={setOptions}
groups={groups}
orgs={orgs}
/>
<div
className="bg-white p-8 mx-auto my-4 rounded-lg"
style={{ width: 'min(1100px, 95vw)' }}
>
<ListOfDatasets
options={options}
setOptions={setOptions}
ckan={ckan}
/>{' '}
</div>
</main>
</div>
);
}

View File

@ -1,129 +0,0 @@
import { existsSync, promises as fs } from 'fs';
import path from 'path';
import parse from '../../lib/markdown';
import DataRichDocument from '../../components/DataRichDocument';
import clientPromise from '../../lib/mddb';
import getConfig from 'next/config';
import { CKAN } from '@portaljs/ckan';
export const getStaticPaths = async () => {
const contentDir = path.join(process.cwd(), '/content/');
const contentFolders = await fs.readdir(contentDir, 'utf8');
const paths = contentFolders.map((folder: string) => ({
params: { path: [folder.split('.')[0]] },
}));
return {
paths,
fallback: false,
};
};
const backend_url = getConfig().publicRuntimeConfig.DMS;
export const getStaticProps = async (context) => {
const mddb = await clientPromise;
const storyFile = await mddb.getFileByUrl(context.params.path);
const md = await fs.readFile(
`${process.cwd()}/${storyFile.file_path}`,
'utf8'
);
const ckan = new CKAN(backend_url);
const datasets = storyFile.metadata.datasets ? await Promise.all(
storyFile.metadata.datasets.map(
async (datasetName: string) => await ckan.getDatasetDetails(datasetName)
)
) : [];
const orgs = storyFile.metadata.orgs ? await Promise.all(
storyFile.metadata.orgs.map(
async (orgName: string) => await ckan.getOrgDetails(orgName)
)
) : [];
let { mdxSource, frontMatter } = await parse(md, '.mdx', { datasets, orgs });
return {
props: {
mdxSource,
frontMatter: JSON.stringify(frontMatter),
},
};
};
export default function DatasetPage({ mdxSource, frontMatter }) {
frontMatter = JSON.parse(frontMatter);
return (
<main className="flex min-h-screen flex-col justify-between p-16 bg-zinc-900">
<div className="bg-white p-8 my-4 rounded-lg">
<div className="prose mx-auto py-8">
<header>
<div className="mb-6">
<>
<h1 className="mb-2">{frontMatter.title}</h1>
{frontMatter.author && (
<p className="my-0">
<span className="font-semibold">Author: </span>
<span className="my-0">{frontMatter.author}</span>
</p>
)}
{frontMatter.description && (
<p className="my-0">
<span className="font-semibold">Description: </span>
<span className="description my-0">
{frontMatter.description}
</span>
</p>
)}
{frontMatter.modified && (
<p className="my-0">
<span className="font-semibold">Modified: </span>
<span className="description my-0">
{new Date(frontMatter.modified).toLocaleDateString()}
</span>
</p>
)}
{frontMatter.files && (
<section className="py-6">
<h2 className="mt-0">Data files</h2>
<table className="table-auto">
<thead>
<tr>
<th>File</th>
<th>Format</th>
</tr>
</thead>
<tbody>
{frontMatter.files.map((f) => {
const fileName = f.split('/').slice(-1);
return (
<tr key={`resources-list-${f}`}>
<td>
<a target="_blank" href={f}>
{fileName}
</a>
</td>
<td>
{fileName[0]
.split('.')
.slice(-1)[0]
.toUpperCase()}
</td>
</tr>
);
})}
</tbody>
</table>
</section>
)}
</>
</div>
</header>
<main>
<DataRichDocument source={mdxSource} />
</main>
</div>
</div>
</main>
);
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 25 KiB

View File

@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>

Before

Width:  |  Height:  |  Size: 1.3 KiB

View File

@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 283 64"><path fill="black" d="M141 16c-11 0-19 7-19 18s9 18 20 18c7 0 13-3 16-7l-7-5c-2 3-6 4-9 4-5 0-9-3-10-7h28v-3c0-11-8-18-19-18zm-9 15c1-4 4-7 9-7s8 3 9 7h-18zm117-15c-11 0-19 7-19 18s9 18 20 18c6 0 12-3 16-7l-8-5c-2 3-5 4-8 4-5 0-9-3-11-7h28l1-3c0-11-8-18-19-18zm-10 15c2-4 5-7 10-7s8 3 9 7h-19zm-39 3c0 6 4 10 10 10 4 0 7-2 9-5l8 5c-3 5-9 8-17 8-11 0-19-7-19-18s8-18 19-18c8 0 14 3 17 8l-8 5c-2-3-5-5-9-5-6 0-10 4-10 10zm83-29v46h-9V5h9zM37 0l37 64H0L37 0zm92 5-27 48L74 5h10l18 30 17-30h10zm59 12v10l-3-1c-6 0-10 4-10 10v15h-9V17h9v9c0-5 6-9 13-9z"/></svg>

Before

Width:  |  Height:  |  Size: 629 B

View File

@ -1,71 +0,0 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
@import "@portaljs/remark-callouts/styles.css";
/* mathjax */
.math-inline > mjx-container > svg {
display: inline;
align-items: center;
}
/* smooth scrolling in modern browsers */
html {
scroll-behavior: smooth !important;
}
/* tooltip fade-out clip */
.tooltip-body::after {
content: "";
position: absolute;
right: 0;
top: 3.6rem; /* multiple of $line-height used on the tooltip body (defined in tooltipBodyStyle) */
height: 1.2rem; /* ($top + $height)/$line-height is the number of lines we want to clip tooltip text at*/
width: 10rem;
background: linear-gradient(
to right,
rgba(255, 255, 255, 0),
rgba(255, 255, 255, 1) 100%
);
}
:is(h2, h3, h4, h5, h6):not(.blogitem-title) {
margin-left: -2rem !important;
padding-left: 2rem !important;
scroll-margin-top: 4.5rem;
position: relative;
}
.heading-link {
padding: 1px;
position: absolute;
left: 0;
top: 50%;
transform: translateY(-50%);
margin: auto 0;
border-radius: 5px;
background: #1e293b;
opacity: 0;
transition: opacity 0.2s;
}
.light .heading-link {
/* border: 1px solid #ab2b65; */
/* background: none; */
background: #e2e8f0;
}
:is(h2, h3, h4, h5, h6):not(.blogitem-title):hover .heading-link {
opacity: 100;
}
.heading-link svg {
transform: scale(0.75);
}
@media screen and (max-width: 640px) {
.heading-link {
visibility: hidden;
}
}

View File

@ -1,18 +0,0 @@
/** @type {import('tailwindcss').Config} */
module.exports = {
content: [
'./pages/**/*.{js,ts,jsx,tsx,mdx}',
'./components/**/*.{js,ts,jsx,tsx,mdx}',
'./app/**/*.{js,ts,jsx,tsx,mdx}',
],
theme: {
extend: {
backgroundImage: {
'gradient-radial': 'radial-gradient(var(--tw-gradient-stops))',
'gradient-conic':
'conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))',
},
},
},
plugins: [require('@tailwindcss/typography')],
}

View File

@ -1,23 +0,0 @@
{
"compilerOptions": {
"target": "es5",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": false,
"forceConsistentCasingInFileNames": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "node",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"paths": {
"@/*": ["./*"]
}
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
"exclude": ["node_modules"]
}

View File

@ -1,6 +1,6 @@
This example creates a portal/showcase for a single dataset. The dataset should be a [Frictionless dataset (data package)][fd] i.e. there should be a `datapackage.json`.
[fd]: https://specs.frictionlessdata.io/data-package/
[fd]: https://frictionlessdata.io/data-packages/
## How to use

View File

@ -1,3 +0,0 @@
{
"extends": "next/core-web-vitals"
}

View File

@ -1,35 +0,0 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts

View File

@ -1,50 +0,0 @@
# PortalJS Demo replicating the FiveThirtyEight data portal
## 👉 https://fivethirtyeight.portaljs.org 👈
Here's a blog post we wrote about it: https://www.datopian.com/blog/fivethirtyeight-replica
This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
You might be asking why we did that, there are three main reasons:
- The website has a great UI, with multiple datasets being displayed elegantly and with simplicity.
- PortalJS allows us to add more functionality to it e.g dataset previews and search functionality.
- The project follows our same principles of open sourcing and free data, with every dataset being publicly available on Github.
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.
[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.
The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

View File

@ -1,23 +0,0 @@
import Link from "next/link";
function HomeIcon({ className = "" }) {
return <div className={`inline-block w-4 ${className}`}><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"> <path d="M 12 2 A 1 1 0 0 0 11.289062 2.296875 L 1.203125 11.097656 A 0.5 0.5 0 0 0 1 11.5 A 0.5 0.5 0 0 0 1.5 12 L 4 12 L 4 20 C 4 20.552 4.448 21 5 21 L 9 21 C 9.552 21 10 20.552 10 20 L 10 14 L 14 14 L 14 20 C 14 20.552 14.448 21 15 21 L 19 21 C 19.552 21 20 20.552 20 20 L 20 12 L 22.5 12 A 0.5 0.5 0 0 0 23 11.5 A 0.5 0.5 0 0 0 22.796875 11.097656 L 12.716797 2.3027344 A 1 1 0 0 0 12.710938 2.296875 A 1 1 0 0 0 12 2 z"/></svg></div>
}
export default function Breadcrumbs({ links }: { links: { title: string, href?: string, target?: string }[] }) {
const current = links.at(-1);
return <div className="flex items-center uppercase font-black text-xs">
<Link className="flex items-center" href='/'><HomeIcon /></Link>
{/* {links.length > 1 && links.slice(0, -1).map((link) => {
return <>
<span className="mx-4">/</span>
<Link href={link.href}>{link.title}</Link>
</>
})} */}
<span className="mx-4">/</span>
<span>{current?.title}</span>
</div >
}

View File

@ -1,99 +0,0 @@
import { useState } from 'react';
import { XMarkIcon } from '@heroicons/react/20/solid';
import { Transition } from '@headlessui/react';
export default function Layout({ children }: { children: React.ReactNode }) {
const [isShowing, setShow] = useState(true);
return (
<>
<Transition
show={isShowing}
enter="transition-opacity duration-75"
enterFrom="opacity-0"
enterTo="opacity-100"
leave="transition-opacity duration-150"
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div className="flex items-center gap-x-6 bg-[#3c3c3c] px-6 py-2.5 sm:px-3.5 sm:before:flex-1">
<p className="text-sm leading-6 text-white">
This is a replica to the awesome{' '}
<a
className="hover:underline font-bold"
href="https://data.fivethirtyeight.com"
>
data.fivethirtyeight.com
</a>{' '}
website.{' '}
<a
className="hover:underline font-bold"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight#readme"
>
Read more here
</a>{' '}
</p>
<div className="flex flex-1 justify-end">
<button
type="button"
onClick={() => setShow(false)}
className="-m-3 p-3 focus-visible:outline-offset-[-4px]"
>
<span className="sr-only">Dismiss</span>
<XMarkIcon className="h-5 w-5 text-white" aria-hidden="true" />
</button>
</div>
</div>
</Transition>
<header className="max-w-5xl mx-auto mt-8 w-full">
<div className="border-b-2 pb-2.5 mx-2 border-zinc-800 flex justify-between">
<h1 className="flex gap-x-1 items-end">
<span className="sr-only">FiveThirtyEight</span>
<img
width="197"
height="25"
alt="FiveThirtyEight"
src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
/>{' '}
<span className="-mb-0.5 text-[#3c3c3c]">replica</span>
</h1>
<div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
<a
className="hover:opacity-75 transition"
href="https://portaljs.com"
>
Built with 🌀PortalJS
</a>
<hr className="h-[80%] border border-[#3c3c3c] opacity-75 my-2"></hr>
<a
className="hover:opacity-75 transition"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
>
Github
</a>
</div>
</div>
<div className="mx-2 py-1.5 text-[14px] text-[#3c3c3c] md:hidden">
<ul className="flex gap-x-4">
<li>
<a
className="hover:opacity-75 transition"
href="https://portaljs.com"
>
PortalJS
</a>
</li>
<li>
<a
className="hover:opacity-75 transition"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
>
View on Github
</a>
</li>
</ul>
</div>
</header>
{children}
</>
);
}

File diff suppressed because it is too large Load Diff

View File

@ -1,38 +0,0 @@
import { Octokit } from 'octokit';
export interface GithubProject {
owner: string;
repo: string;
branch: string;
files: string[];
readme: string;
description?: string;
name?: string;
}
export async function getProjectReadme(
owner: string,
repo: string,
branch: string,
readme: string,
github_pat?: string
) {
const octokit = new Octokit({ auth: github_pat });
try {
const response = await octokit.rest.repos.getContent({
owner,
repo,
path: readme,
ref: branch,
});
const data = response.data as { content?: string };
const fileContent = data.content ? data.content : '';
if (fileContent === '') {
return null;
}
const decodedContent = Buffer.from(fileContent, 'base64').toString();
return decodedContent;
} catch (error) {
return null;
}
}

Some files were not shown because too many files have changed in this diff Show More