diff --git a/site/content/learn/ckan.md b/site/content/learn/ckan.md new file mode 100644 index 00000000..39df75ac --- /dev/null +++ b/site/content/learn/ckan.md @@ -0,0 +1,218 @@ +Live DEMOs: + +- https://catalog-portal-js.vercel.app +- https://ckan-enterprise-frontend.vercel.app/ + +## Create a Portal app for CKAN + +To create a Portal app, run the following command in your terminal: + +```console +npx create-next-app -e https://github.com/datopian/portal.js/tree/main/examples/ckan +``` + +> NB: Under the hood, this uses the tool called create-next-app, which bootstraps an app for you based on our CKAN example. + +## Guide + +### Styling 🎨 + +We use Tailwind as a CSS framework. Take a look at `/styles/globals.css` to see what we're importing from Tailwind bundle. You can also configure Tailwind using `tailwind.config.js` file. + +Have a look at Next.js support of CSS and ways of writing CSS: + +https://nextjs.org/docs/basic-features/built-in-css-support + +### Backend + +So far the app is running with mocked data behind. You can connect CMS and DMS backends easily via environment variables: + +```console +$ export DMS=http://ckan:5000 +$ export CMS=http://myblog.wordpress.com +``` + +> Note that we don't yet have implementations for the following CKAN features: +> +> - Activities +> - Auth +> - Groups +> - Facets + +### Routes + +These are the default routes set up in the "starter" app. + +- Home `/` +- Search `/search` +- Dataset `/@org/dataset` +- Resource `/@org/dataset/r/resource` +- Organization `/@org` +- Collection (aka group in CKAN) (?) - suggest to merge into org +- Static pages, eg, `/about` etc. from CMS or can do it without external CMS, e.g., in Next.js. + +### New Routes + +You can create new routes in `/pages` directory where each file is associated with a route based on its name. We suggest using [Next.JS docs][] for more detailed information. + +[next.js docs]: https://nextjs.org/docs/basic-features/pages + +### Data fetching + +We use Apollo client which allows us to query data with GraphQL. We have setup CKAN API for the demo (it uses demo.ckan.org as DMS): + +http://portal.datopian1.now.sh/ + +Note that we don't have Apollo Server but we connect CKAN API using [`apollo-link-rest`](https://www.apollographql.com/docs/link/links/rest/) module. You can see how it works in [lib/apolloClient.ts](https://github.com/datopian/portal/blob/master/lib/apolloClient.ts) and then have a look at [pages/\_app.tsx](https://github.com/datopian/portal/blob/master/pages/_app.tsx). + +For development/debugging purposes, we suggest installing the Chrome extension - https://chrome.google.com/webstore/detail/apollo-client-developer-t/jdkknkkbebbapilgoeccciglkfbmbnfm. + +### I18n configuration + +Portal.js is configured by default to support both `English` and `French` subpath for language translation. But for subsequent users, this following steps can be used to configure i18n for other languages; + +1. Update `next.config.js`, to add more languages to the i18n locales + +```js +i18n: { + locales: ['en', 'fr', 'nl-NL'], // add more language to the list + defaultLocale: 'en', // set the default language to use +}, +``` + +2. Create a folder for the language in `locales` --> `locales/en-Us` + +3. In the language folder, different namespace files (json) can be created for each translation. For the `index.js` use-case, I named it `common.json` + +```json +// locales/en/common.json +{ + "title" : "Portal js in English", +} + +// locales/fr/common.json +{ + "title" : "Portal js in French", +} +``` + +4. To use on pages using Server-side Props. + +```js +import { loadNamespaces } from './_app'; +import useTranslation from 'next-translate/useTranslation'; + +const Home: React.FC = ()=> { + const { t } = useTranslation(); + return ( +
{t(`common:title`)}
// we use common and title base on the common.json data + ); +}; + +export const getServerSideProps: GetServerSideProps = async ({ locale }) => { + ........ ........ + return { + props : { + _ns: await loadNamespaces(['common'], locale), + } + }; +}; + +``` + +5. Go to the browser and view the changes using language subpath like this `http://localhost:3000` and `http://localhost:3000/fr`. **Note** The subpath also activate chrome language Translator + +### Pre-fetch data in the server-side + +When visiting a dataset page, you may want to fetch the dataset metadata in the server-side. To do so, you can use `getServerSideProps` function from NextJS: + +```javascript +import { GetServerSideProps } from 'next'; +import { initializeApollo } from '../lib/apolloClient'; +import gql from 'graphql-tag'; + +const QUERY = gql` + query dataset($id: String) { + dataset(id: $id) @rest(type: "Response", path: "package_show?{args}") { + result + } + } +`; + +... + +export const getServerSideProps: GetServerSideProps = async (context) => { + const apolloClient = initializeApollo(); + + await apolloClient.query({ + query: QUERY, + variables: { + id: 'my-dataset' + }, + }); + + return { + props: { + initialApolloState: apolloClient.cache.extract(), + }, + }; +}; +``` + +This would fetch the data from DMS and save it in the Apollo cache so that we can query it again from the components. + +### Access data from a component + +Consider situation when rendering a component for org info on the dataset page. We already have pre-fetched dataset metadata that includes `organization` property with attributes such as `name`, `title` etc. We can now query only organization part for our `Org` component: + +```javascript +import { useQuery } from '@apollo/react-hooks'; +import gql from 'graphql-tag'; + +export const GET_ORG_QUERY = gql` + query dataset($id: String) { + dataset(id: $id) @rest(type: "Response", path: "package_show?{args}") { + result { + organization { + name + title + image_url + } + } + } + } +`; + +export default function Org({ variables }) { + const { loading, error, data } = useQuery( + GET_ORG_QUERY, + { + variables: { id: 'my-dataset' } + } + ); + + ... + + const { organization } = data.dataset.result; + + return ( + <> + {organization ? ( + <> + + + {organization.title || organization.name} + + + ) : ( + '' + )} + + ); +} +``` diff --git a/site/content/learn/deploy-to-gh-pages.md b/site/content/learn/deploy-to-gh-pages.md new file mode 100644 index 00000000..36a6319c --- /dev/null +++ b/site/content/learn/deploy-to-gh-pages.md @@ -0,0 +1,251 @@ +# Deploying data on Github using Portal.js and Github pages +--- +**Use Case:** +--- +You have some data in a Github repo and you'd like to deploy it online using "portal" so that it is easy for others to view, explore and use. +--- +Here we show how you can use portal.js plus github actions to deploy your dataset in minutes and keep it updated as you make changes. + +The example focuses on the case of a [Frictionless dataset][fd] but it works for any dataset type supported by portal.js. + +We provide three options on how to do this and recommend using the first one unless you really want to get hands on: + +* Deploying datasets automatically by setting up a github actions script. +* Deploying datasets from a local bash script with portal code commits +* Deploying datasets from a local bash script without portal code commits + +[fd]: https://frictionlessdata.io/data-packages/ + +## Deploy datasets automatically by setting up a github actions script + +The github actions below will automatically build and deploy a single page, Frictionless dataset to `gh-pages` branch. Follow the steps below to achieve this: + +1. Create a secret so we can automatically commit to gh-pages branch (see below) +2. Set up the github action to build portal to your dataset and deploy it (see below) +3. Wait for your page to build and then setup github pages (see below) +4. View the results: visit `https:///github.io//` + +### Step 1 + +In the dataset repository you want to deploy, create a github secret with the name `PORTAL_REPO_NAME` and the value should be the name of the repository. + +See steps on creating a secret [here](https://docs.github.com/en/actions/reference/encrypted-secrets) + + + +### Step 2 + +In the dataset repository you want deploy create a `.github/workflow` directory and add a `main.yml` file with the following content (you can also view/download this [action file here](scripts/actions/single-dataset-ssg.yml): + +```bash +name: github pages + +on: + push: + branches: + - master + - main + +jobs: + run: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Setup Node + uses: actions/setup-node@v2.1.2 + with: + node-version: '12.x' + + - name: Build datasets + env: + PORTAL_REPO_NAME: ${{ secrets.PORTAL_REPO_NAME }} + run: | + curl https://raw.githubusercontent.com/datopian/portal.js/main/site/public/scripts/single-dataset-no-commit.sh > portal.sh + git config --local user.email "$(git log --format='%ae' HEAD^!)" + git config --local user.name "$(git log --format='%an' HEAD^!)" + source ./portal.sh + +``` + +Then, commit and push your code. + +```bash +git add . +git commit -m "Build dataset page" +git push +``` + +### Step 3 + +Wait for a while as your page builds, and once you see the green check mark, navigate to your repository's github `pages` in settings, set the `source` to `gh-pages` and folder to `/root`: + + + + +## Deploy single dataset without commiting portal.js code + +Users who want to deploy datasets from a local bash script without saving/commiting the portal.js code, can use the script shown below. + +Using this script means you do not have access to the portal.js code used to generate the dataset page, and as such cannot modify/extend it. + +This script creates and commit only the build/output files to the gh-pages branch. Follow the steps below to achieve this. + +### Step 1 + +Clone/Pull the dataset repository you want deploy. For example: + +```bash +git clone https://github.com/datasets/finance-vix +cd finance-vix +``` + +### Step 2 + +In a terminal, export an env variable with the name of your dataset github repo. For example if deploying https://github.com/datasets/finance-vix, then export the name as: + +```bash +export PORTAL_REPO_NAME=finance-vix +``` + +### Step 3 + +In the dataset repository's root folder, create a file called `portal.sh` and paste the following [content](/scripts/single-dataset-no-commit.sh): + +```bash +#!/bin/bash +git checkout -b gh-pages +git rm -r --cached . +rm -rf portal +mkdir -p portal +npx create-next-app portal -e https://github.com/datopian/portal.js/tree/main/examples/dataset-frictionless +mkdir portal/public/dataset + +cp -a ./data portal/public/dataset +cp -a ./datapackage.json portal/public/dataset +cp -a ./README.md portal/public/dataset + +PORTAL_DATASET_PATH=$PWD"/portal/public/dataset" +export PORTAL_DATASET_PATH + +cd portal +assetPrefix='"/'$PORTAL_REPO_NAME'/"' +basePath='"/'$PORTAL_REPO_NAME'"' +echo 'module.exports = {assetPrefix:' ${assetPrefix}', basePath: '${basePath}' }' > next.config.js ## This ensures css and public folder works +yarn export + +cd .. +cp -R -a portal/out/* ./ +touch .nojekyll +git add $PWD'/_next' $PWD'/index.html' $PWD'/dataset' $PWD'/404.html' $PWD'/.nojekyll' $PWD'/favicon.ico' +git commit -m "Build new dataset page" +git push origin gh-pages +``` + +### Step 4 + +Run the bash script in a terminal with: + +```bash +source portal.sh +``` + +> Note: Use `source` instead of `bash` so that the script can work well with environment variables. + +### Step 5 + +Go to your repository's github `pages` in setting and set the Branch to gh-pages and folder to root: + + + +### Step 6 + +Open your deployed site at `https:///github.io/` + + +## Deploy single dataset with portal commit + +Users who want access to the portal.js code used for generating the dataset page can use the script shown in the following section. + +This script creates and commits the portal.js code to the root branch and also adds an automated script to deploy to gh-page. Follow the steps below to use this script. + +### Step 1 + +Create a Github Personal Access Token (PAT). See steps [here](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) + +### Step 2 + +In the dataset repository you want to deploy, create a github secret with the name `PORTAL_NEXT_TOKEN`. The value should be the PAT created in step 1. See steps on creating a secret [here](https://docs.github.com/en/actions/reference/encrypted-secrets) + +> Note: Without the PAT and the secret configured, the automatic build will fail. + +### Step 3 + +Clone/Pull the dataset repository you want deploy. For example: + +```bash +git clone https://github.com/datasets/finance-vix +cd finance-vix +``` + +### Step 4 + +In your computer's terminal/command prompt, export an environment variable with the name of your dataset's github repo. + +For example if you want to deploy the dataset at https://github.com/datasets/finance-vix, then export the name using the command: + +```bash +export PORTAL_REPO_NAME=finance-vix +``` + +### Step 5 + +Create a file called `portal.sh` and paste the following [content](/scripts/single-dataset-commit.sh): + +```bash +#!/bin/bash +rm -rf portal +mkdir -p portal +npx create-next-app portal -e https://github.com/datopian/portal.js/tree/main/examples/dataset-frictionless +mkdir portal/public/dataset + +cp -a ./data portal/public/dataset +cp -a ./datapackage.json portal/public/dataset +cp -a ./README.md portal/public/dataset + +PORTAL_DATASET_PATH=$PWD"/portal/public/dataset" +export PORTAL_DATASET_PATH + +mkdir -p .github && mkdir -p .github/workflows && touch .github/workflows/main.yml +curl https://raw.githubusercontent.com/datopian/portal.js/main/site/public/scripts/gh-page-builder-action.yml > .github/workflows/main.yml + +cd portal +assetPrefix='"/'$PORTAL_REPO_NAME'/"' +basePath='"/'$PORTAL_REPO_NAME'"' +echo 'module.exports = {assetPrefix:' ${assetPrefix}', basePath: '${basePath}' }' > next.config.js ## This ensures css and public folder works + +cd .. +git add . +git commit -m "Add dataset build feature" +git push +echo "Portal generated, please push your code to github" +``` + +### Step 6 + +Run the bash script with: + +```bash +source portal.sh +``` + +> Note: Use `source` instead of `bash` so that the script can work well with environment variables. + +### Step 7 + +Go to your repository's github `pages` in setting and set the Branch to gh-pages and folder to root: + + + +### Step 8 + +Open your deployed site at `https:///github.io/` diff --git a/site/content/learn/single-frictionless-dataset.md b/site/content/learn/single-frictionless-dataset.md new file mode 100644 index 00000000..dc623f95 --- /dev/null +++ b/site/content/learn/single-frictionless-dataset.md @@ -0,0 +1,32 @@ +Live Demo: +- https://portal-js.vercel.app/ + +## Create a single frictionless dataset portal + +The dataset should be a frictionless dataset i.e. it should have a [datapackage.json](https://specs.frictionlessdata.io/data-package/) + + +Create a frictionless dataset portal app from the default template by executing the following command in your terminal: +``` +$ npx create-next-app -e https://github.com/datopian/portal.js/tree/main/examples/dataset-frictionless +``` +> Choose a name for your portal when prompted e.g. your-portal + +Next, connect the frictionless dataset to `your-portal` by declaring the path to the directory level that contains the `datapackage.json` via an environment variable by executing the following command in your terminal: +``` +$ cd your-portal +$ export PORTAL_DATASET_PATH=path/to/your/dataset +``` +In `your-portal` directory, run the command below in your terminal to start the portal: +``` +$ yarn dev +``` +Open the page in your browser via the localhost url(usually http://localhost:3000) returned in the terminal to see your frictionless dataset portal. + +### Styling 🎨 + +We use Tailwind as a CSS framework. Take a look at `/styles/tailwind.css` to see what we're importing from Tailwind bundle. You can also configure Tailwind using `tailwind.config.js` file. + +Have a look at Next.js support of CSS and ways of writing CSS: + +https://nextjs.org/docs/basic-features/built-in-css-support