Compare commits

..

26 Commits

Author SHA1 Message Date
olayway
11144758ca [site/content/howto][xs]: list of guides in index page 2023-05-18 20:09:27 +02:00
olayway
f443dab92c [site/content][s]: add drd guide 2023-05-18 19:50:03 +02:00
olayway
d9be7d0f5b [site/content/assets][s]: ctreate howto subdir 2023-05-18 19:25:18 +02:00
olayway
7c21993b92 [site/content][s]: add markdown guide 2023-05-18 19:25:18 +02:00
olayway
3f06aa3f01 [site/content][s]: add comments guide 2023-05-18 15:03:26 +02:00
olayway
d191178ecf [site/content][s]: add blog guide 2023-05-18 15:03:26 +02:00
olayway
a0eb06ee19 [site/content][s]: add sitemap guide 2023-05-18 14:45:27 +02:00
olayway
539fd6c77e [site/content][xs]: add seo guide 2023-05-18 14:44:50 +02:00
olayway
8fa930b3f4 [content/howto][s]: add analytics guide 2023-05-18 14:43:51 +02:00
olayway
3be88f1cdb [content/howto][s]: create howto index page 2023-05-17 20:49:35 +02:00
olayway
d0ee06e8d0 [.gitignore][xs]: ignore .obsidian 2023-05-17 20:49:08 +02:00
Luccas Mateus
4e91e88f2b [examples/github-backed-catalog] - improvements after issue #871 (#887) 2023-05-17 11:10:18 -03:00
Luccas Mateus
ebcb93c996 [examples/turing][xs] - point out that markdown gets rendered 2023-05-16 07:22:45 -03:00
Luccas Mateus
1fc2499c71 [examples/538][xs] - change banner position + text + change README (#877)
* [examples/538][xs] - change banner position + text + change README

* [examples/538] - change banner background

* [examples/538][m] - changes after ola comments

* [example/538] - fix typo
2023-05-15 12:37:19 -03:00
Luccas Mateus
1af24ef57e 538 banner (#873)
* [example/538] - banner

* [example/538] - title on head
2023-05-12 14:50:31 -03:00
Luccas Mateus de Medeiros Gomes
698c06efda [site][xs] - fix logo on dark mode 2023-05-12 14:25:02 -03:00
Luccas Mateus de Medeiros Gomes
8792f295b0 [examples/turing][xs] - fix header 2023-05-12 08:26:02 -03:00
Luccas Mateus de Medeiros Gomes
3e6d01c4c7 [examples/538][xs] - go back to v01(only index page) 2023-05-11 19:17:41 -03:00
Luccas Mateus de Medeiros Gomes
7c943c1b31 [example/turing][sm] - forgot to add github on desktop 2023-05-11 17:16:56 -03:00
Luccas Mateus de Medeiros Gomes
7197a6686e [examples/turing][sm] - change view on github 2023-05-11 17:11:20 -03:00
Luccas Mateus de Medeiros Gomes
7822440f0d [examples/turing] - rename it to turing 2023-05-11 16:13:09 -03:00
Luccas Mateus de Medeiros Gomes
82773b5e8a [examples/538] - fix build 2023-05-11 13:28:33 -03:00
Luccas Mateus
1cfc4db528 [examples/538][m] - little fixes and renaming (#870) 2023-05-11 13:15:18 -03:00
João Demenech
336ff819dc OpenSpending Data Portal (#868)
* [#856,openspending][xl]: initial commit

* [examples/openspending][xs] - remove console.logs

---------

Co-authored-by: Luccas Mateus de Medeiros Gomes <luccasmmg@gmail.com>
2023-05-10 18:20:47 -03:00
Luccas Mateus
f610c953e7 [example/538] - individual pages (#865)
* [example/538] - individual pages

* [examples/538][sm] - force inclusion of classes

* [examples/538] - changes requested by demenech
2023-05-10 18:13:16 -03:00
João Demenech
3f350f8fcd [#810, github-backed example][xl]: improve looks, improve README, rename from simple-example to github-backed (#864) 2023-05-09 19:19:36 -03:00
132 changed files with 8476 additions and 779 deletions

1
.gitignore vendored
View File

@@ -16,6 +16,7 @@ node_modules
*.launch
.settings/
*.sublime-workspace
.obsidian
# IDE - VSCode
.vscode/*

View File

@@ -1,38 +0,0 @@
import { Html, Head, Main, NextScript } from 'next/document';
export default function Document() {
return (
<Html lang="en">
<Head>
<link
rel="icon"
type="image/x-icon"
href="https://projects.fivethirtyeight.com/shared/favicon.ico"
/>
</Head>
<body>
<header className="max-w-5xl mx-auto mt-8 w-full">
<div className="border-b-2 pb-2.5 mx-2 border-zinc-800">
<h1>
<span className="sr-only">FiveThirtyEight</span>
<a
className="flex gap-x-2 items-center"
href="http://fivethirtyeight.com"
>
<img
width="197"
height="25"
alt="FiveThirtyEight"
src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
/>{' '}
by PortalJS
</a>
</h1>
</div>
</header>
<Main />
<NextScript />
</body>
</Html>
);
}

View File

@@ -1,4 +1,10 @@
This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
You might be asking why we did that, there are three main reasons:
- The website has a great UI, with multiple datasets being displayed elegantly and with simplicity.
- PortalJS allows us to add more functionality to it e.g dataset previews and search functionality.
- The project follows our same principles of open sourcing and free data, with every dataset being publicly available on Github.
## Getting Started

View File

@@ -1,46 +1,11 @@
[
{
"url": "https://github.com/fivethirtyeight/data/tree/master/nba-forecasts",
"name": "nba-forecasts",
"displayName": "nba-<span class=\"lastword\">forecasts</span>",
"articles": [
{
"date": "2023-05-08T22:33:43.000Z",
"title": "2022-23 NBA Predictions",
"url": "https://projects.fivethirtyeight.com/2023-nba-predictions/"
}
],
"files": [
"https://projects.fivethirtyeight.com/nba-model/nba_elo.csv",
"https://projects.fivethirtyeight.com/nba-model/nba_elo_latest.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/soccer-spi",
"name": "soccer-spi",
"displayName": "soccer-<span class=\"lastword\">spi</span>",
"articles": [
{
"date": "2023-05-08T22:17:18.000Z",
"title": "Club Soccer Predictions",
"url": "https://projects.fivethirtyeight.com/soccer-predictions/"
}
],
"files": [
"https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv",
"https://projects.fivethirtyeight.com/soccer-api/club/spi_matches_latest.csv",
"https://projects.fivethirtyeight.com/soccer-api/club/spi_global_rankings.csv",
"https://projects.fivethirtyeight.com/soccer-api/international/spi_matches_intl.csv",
"https://projects.fivethirtyeight.com/soccer-api/international/spi_global_rankings_intl.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/polls",
"name": "polls",
"displayName": "<span class=\"lastword\">polls</span>",
"articles": [
{
"date": "2023-05-08T20:36:59.000Z",
"date": "2023-05-11T14:35:40.000Z",
"title": "Latest Polls",
"url": "https://projects.fivethirtyeight.com/polls/"
}
@@ -63,13 +28,45 @@
"https://projects.fivethirtyeight.com/2020-general-data/presidential_poll_averages_2020.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/congress-generic-ballot",
"name": "congress-generic-ballot",
"displayName": "congress-generic-<span class=\"lastword\">ballot</span>",
"articles": [
{
"date": "2023-05-11T14:35:40.000Z",
"title": "Do Voters Want Democrats Or Republicans In Congress?",
"url": "https://projects.fivethirtyeight.com/congress-generic-ballot-polls/"
}
],
"files": [
"https://projects.fivethirtyeight.com/generic-ballot-data/generic_polllist.csv",
"https://projects.fivethirtyeight.com/polls/data/generic_ballot_averages.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/nba-forecasts",
"name": "nba-forecasts",
"displayName": "nba-<span class=\"lastword\">forecasts</span>",
"articles": [
{
"date": "2023-05-11T11:15:46.000Z",
"title": "2022-23 NBA Predictions",
"url": "https://projects.fivethirtyeight.com/2023-nba-predictions/"
}
],
"files": [
"https://projects.fivethirtyeight.com/nba-model/nba_elo.csv",
"https://projects.fivethirtyeight.com/nba-model/nba_elo_latest.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/nba-raptor",
"name": "nba-raptor",
"displayName": "nba-<span class=\"lastword\">raptor</span>",
"articles": [
{
"date": "2023-05-08T11:15:48.000Z",
"date": "2023-05-11T11:13:20.000Z",
"title": "The Best NBA Players, According To RAPTOR",
"rowspan": 3,
"url": "https://projects.fivethirtyeight.com/nba-player-ratings/"
@@ -92,13 +89,32 @@
"https://projects.fivethirtyeight.com/nba-model/2023/latest_RAPTOR_by_player.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/soccer-spi",
"name": "soccer-spi",
"displayName": "soccer-<span class=\"lastword\">spi</span>",
"articles": [
{
"date": "2023-05-11T05:25:51.000Z",
"title": "Club Soccer Predictions",
"url": "https://projects.fivethirtyeight.com/soccer-predictions/"
}
],
"files": [
"https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv",
"https://projects.fivethirtyeight.com/soccer-api/club/spi_matches_latest.csv",
"https://projects.fivethirtyeight.com/soccer-api/club/spi_global_rankings.csv",
"https://projects.fivethirtyeight.com/soccer-api/international/spi_matches_intl.csv",
"https://projects.fivethirtyeight.com/soccer-api/international/spi_global_rankings_intl.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/nhl-forecasts",
"name": "nhl-forecasts",
"displayName": "nhl-<span class=\"lastword\">forecasts</span>",
"articles": [
{
"date": "2023-05-08T04:18:20.000Z",
"date": "2023-05-11T04:53:22.000Z",
"title": "2022-23 NHL Predictions",
"url": "https://projects.fivethirtyeight.com/2023-nhl-predictions/"
}
@@ -114,7 +130,7 @@
"displayName": "mlb-<span class=\"lastword\">elo</span>",
"articles": [
{
"date": "2023-05-08T02:25:55.000Z",
"date": "2023-05-11T02:35:49.000Z",
"title": "2023 MLB Predictions",
"url": "https://projects.fivethirtyeight.com/2023-mlb-predictions/"
}
@@ -124,22 +140,6 @@
"https://projects.fivethirtyeight.com/mlb-api/mlb_elo_latest.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/congress-generic-ballot",
"name": "congress-generic-ballot",
"displayName": "congress-generic-<span class=\"lastword\">ballot</span>",
"articles": [
{
"date": "2023-05-02T13:48:41.000Z",
"title": "Do Voters Want Democrats Or Republicans In Congress?",
"url": "https://projects.fivethirtyeight.com/congress-generic-ballot-polls/"
}
],
"files": [
"https://projects.fivethirtyeight.com/generic-ballot-data/generic_polllist.csv",
"https://projects.fivethirtyeight.com/polls/data/generic_ballot_averages.csv"
]
},
{
"url": "https://github.com/fivethirtyeight/data/tree/master/congress-demographics",
"name": "congress-demographics",

View File

@@ -16,6 +16,7 @@
"autoprefixer": "10.4.14",
"eslint": "8.40.0",
"eslint-config-next": "13.4.1",
"flexsearch": "^0.7.31",
"next": "13.4.1",
"next-mdx-remote": "^4.4.1",
"next-seo": "^6.0.0",
@@ -2610,6 +2611,11 @@
"resolved": "https://registry.npmjs.org/flatted/-/flatted-3.2.7.tgz",
"integrity": "sha512-5nqDSxl8nn5BSNxyR3n4I6eDmbolI6WT+QqR547RwxQapgjQBmtktdP+HTBb/a/zLsbzERTONyUB5pefh5TtjQ=="
},
"node_modules/flexsearch": {
"version": "0.7.31",
"resolved": "https://registry.npmjs.org/flexsearch/-/flexsearch-0.7.31.tgz",
"integrity": "sha512-XGozTsMPYkm+6b5QL3Z9wQcJjNYxp0CYn3U1gO7dwD6PAqU1SVWZxI9CCg3z+ml3YfqdPnrBehaBrnH2AGKbNA=="
},
"node_modules/for-each": {
"version": "0.3.3",
"resolved": "https://registry.npmjs.org/for-each/-/for-each-0.3.3.tgz",

View File

@@ -17,6 +17,7 @@
"autoprefixer": "10.4.14",
"eslint": "8.40.0",
"eslint-config-next": "13.4.1",
"flexsearch": "^0.7.31",
"next": "13.4.1",
"next-mdx-remote": "^4.4.1",
"next-seo": "^6.0.0",

View File

@@ -0,0 +1,96 @@
import { Html, Head, Main, NextScript } from 'next/document';
export default function Document() {
return (
<Html lang="en">
<Head>
<link
rel="icon"
type="image/x-icon"
href="https://projects.fivethirtyeight.com/shared/favicon.ico"
/>
<meta
property="og:image"
content="https://portaljs-fivethirtyeight.vercel.app/share_image.png"
/>
<meta
property="twitter:image"
content="https://portaljs-fivethirtyeight.vercel.app/share_image.png"
/>
</Head>
<body>
<div className="px-2 max-w-5xl mx-auto pb-2">
<div className="mt-2 px-2 bg-[#3c3c3c] text-white">
<div className="p-2 text-center">
This is a replica to the awesome{' '}
<a
className="hover:underline font-bold"
href="https://data.fivethirtyeight.com"
>
data.fivethirtyeight.com
</a>{' '}
website.{' '}
<a
className="hover:underline font-bold"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
>
Read more here
</a>{' '}
</div>
</div>
</div>
<header className="max-w-5xl mx-auto mt-8 w-full">
<div className="border-b-2 pb-2.5 mx-2 border-zinc-800 flex justify-between">
<h1 className="flex gap-x-1 items-end">
<span className="sr-only">FiveThirtyEight</span>
<img
width="197"
height="25"
alt="FiveThirtyEight"
src="data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0MjEgNTMuNzYiPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojMDEwMTAxO308L3N0eWxlPjwvZGVmcz48dGl0bGU+QXJ0Ym9hcmQgOTU8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTAgMGgyNXY4SDl2MTBoMTV2OEg5djE3SDBWMHpNMzEgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdIMzF6bTUtMzZoOHY4aC04ek0xNzkgMzZoNVYxOGgtNXYtOGgxM3YyNmg0djdoLTE3em01LTM2aDh2OGgtOHpNMzE2IDM2aDVWMThoLTV2LThoMTN2MjZoNHY3aC0xN3ptNS0zNmg4djhoLTh6TTU0IDI3VjEwaDh2MTVsNCA5Ljk4aDFMNzEgMjVWMTBoOHYxN2wtNyAxNkg2MWwtNy0xNnpNMTExIDQzSDk3LjQyQzg5LjIzIDQzIDg1IDM5LjE5IDg1IDMxLjE3VjIyYzAtNy41NyA0LjMtMTMgMTMtMTMgOS4zMyAwIDEzIDUuMDcgMTMgMTR2N0g5NHYxLjc0YzAgMi42MiAxIDQuMjYgMy40MiA0LjI2SDExMXpNOTQgMjNoOHYtMS41NWMwLTIuNjItMS4wNi01LjQ1LTQuMTMtNS40NS0yLjc5IDAtMy44NyAyLjItMy44NyA1LjQ1ek0xMjUgOGgtMTBWMGgyOXY4aC0xMHYzNWgtOVY4ek0yMDIgNDNWMTBoOHY0YzEuMTQtMi40NSAzLjc1LTQgNy4yMi00SDIyMHY4aC02Yy0yLjg0IDAtNCAuOTQtNCAzLjlWNDN6TTI0NSA0M2gtNC44NEMyMzMuMDUgNDMgMjMwIDM5LjMxIDIzMCAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0gyNDV6TTQyMSA0M2gtNC44NEM0MDkuMDUgNDMgNDA2IDM5LjMxIDQwNiAzMS44NVYxOGgtNnYtOGg2VjNoOHY3aDd2OGgtN2wtLjA3IDEzLjkzYzAgMi4yMi45MyA0LjA3IDMuNjYgNC4wN0g0MjF6TTI1NC4yNiA1My43Nmw0LjYxLTkuNUwyNTEgMjdWMTBoOHYxNWw0IDEwaDFsNC0xMFYxMGg4djE3bC0xMi4zIDI2Ljc2aC05LjQ0ek0yODQgMGgyNXY4aC0xNnY5aDE1djhoLTE1djEwaDE2djhoLTI1VjB6TTMzNyA0OHYtMmgxNi4xYzIgMCAyLjktLjE4IDIuOS0xLjI3di0uMzRjMC0xLjA4LS45MS0xLjM5LTIuOS0xLjM5SDM0MHYtNWw1LTVjLTUuMjktMS40OC04LTUuNDMtOC0xMXYtMWMwLTcuNTYgNC40NC0xMiAxNC0xMmEyMS45MyAyMS45MyAwIDAgMSA1Ljk1IDFMMzYxIDRsNSAzLTQgNmMxLjM3IDEuOTMgMyA0LjkzIDMgOHYxYzAgNy0zLjMgMTAuNjYtMTIgMTFsLTMgNGg2YzUuOTIgMCA5IDIuNjIgOSA3LjY4di4xMWMwIDUuMDYtMi43MSA4LjIxLTguNjIgOC4yMWgtMTNjLTQuMjkgMC02LjM4LTEuODQtNi4zOC01em0xOS0yNXYtM2MwLTMuMy0xLjMzLTQtNS00cy01IC43LTUgNHYzYzAgMy4zIDEuMzkgNCA1IDRzNS0uNyA1LTR6TTM4MCA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuNC00IDctNCA2LjI2IDAgOSAzLjA4IDkgMTAuNzZWNDNoLThWMjJjMC0zLjEzLTEuMDctNS00LTVzLTQgMS44Ny00IDV6TTE1NyA0M2gtOFYwaDh2MTRjMS4xNC0yLjY3IDMuOTEtNCA3LjQ5LTQgNi4yNiAwIDguNTEgMy4xMyA4LjUxIDEwLjgxVjQzaC04VjIxYzAtMy4xMy0xLjA3LTQuNDQtNC00LjQ0cy00IDIuMjYtNCA1LjM5eiIvPjwvc3ZnPg=="
/>{' '}
<span className="-mb-0.5 text-[#3c3c3c]">replica</span>
</h1>
<div className="md:flex items-center gap-x-3 text-[#3c3c3c] -mb-1 hidden">
<a
className="hover:opacity-75 transition"
href="https://portaljs.org"
>
Built with 🌀PortalJS
</a>
<hr className="h-[80%] border border-[#3c3c3c] opacity-75 my-2"></hr>
<a
className="hover:opacity-75 transition"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
>
Github
</a>
</div>
</div>
<div className="mx-2 py-1.5 text-[14px] text-[#3c3c3c] md:hidden">
<ul className="flex gap-x-4">
<li>
<a
className="hover:opacity-75 transition"
href="https://portaljs.org"
>
PortalJS
</a>
</li>
<li>
<a
className="hover:opacity-75 transition"
href="https://github.com/datopian/portaljs/tree/main/examples/fivethirtyeight"
>
View on Github
</a>
</li>
</ul>
</div>
</header>
<Main />
<NextScript />
</body>
</Html>
);
}

View File

@@ -3,6 +3,7 @@ import { Inter } from 'next/font/google';
import { format } from 'timeago.js';
import { promises as fs } from 'fs';
import path from 'path';
import { NextSeo } from 'next-seo';
const inter = Inter({ subsets: ['latin'] });
@@ -20,34 +21,36 @@ export interface Dataset {
files?: string[];
}
// Request a weekday along with a long date
const options = {
year: 'numeric',
month: 'long',
day: 'numeric',
} as const;
export function MobileItem({ dataset }: { dataset: Dataset }) {
return (
<div className="flex gap-x-2 pb-2 py-4 items-center justify-between border-b border-zinc-600">
<div className="flex flex-col">
<span className="font-light">{dataset.name}</span>
<span className="font-mono font-light">{dataset.name}</span>
{dataset.articles.map((article) => (
<div key={article.title} className="py-1 flex flex-col">
<span className="font-bold hover:underline">{article.title}</span>
<span className="font-light text-base">
{format(article.date)}
{format(article.date).includes('years')
? new Date(article.date).toLocaleString('en-US', options)
: format(article.date)}
</span>{' '}
</div>
))}
</div>
<div className="flex flex-col justify-start">
<a
className="border border-zinc-900 font-light px-4 py-1 text-sm transition hover:bg-zinc-900 hover:text-white"
className="ml-2 border border-zinc-900 font-light px-4 py-1 text-sm transition hover:bg-zinc-900 hover:text-white"
href={dataset.url}
target="_blank"
>
info
</a>
<a
className="ml-2 border border-zinc-900 font-light px-4 py-1 text-sm transition hover:bg-zinc-900 hover:text-white"
href={`/datasets/${dataset.name}`}
>
explore
</a>
{/*
<button>
<svg
@@ -78,33 +81,29 @@ export function DesktopItem({ dataset }: { dataset: Dataset }) {
index === dataset.articles.length - 1 ? 'border-b' : ''
} border-zinc-400`}
>
<td className="py-8 font-light">{index === 0 ? dataset.name : ''}</td>
<td className="py-8 font-light font-mono text-[13px] text-zinc-700">
{index === 0 ? dataset.name : ''}
</td>
<td>
<a className="py-8 font-bold hover:underline" href={article.url}>
<a
className="py-8 font-bold hover:underline pr-2"
href={article.url}
>
{article.title}
</a>
</td>
<td className="py-8 font-light text-base min-w-[120px]">
{format(article.date)}
</td>
<td className="py-8">
{index === 0 && (
<a
className="border border-zinc-900 font-light px-[25px] py-2.5 text-sm transition hover:bg-zinc-900 hover:text-white"
href={dataset.url}
target="_blank"
>
info
</a>
)}
<td className="py-8 font-light text-[14px] min-w-[138px] font-mono text-[#999]">
{format(article.date).includes('years')
? new Date(article.date).toLocaleString('en-US', options)
: format(article.date)}
</td>
<td className="py-8">
{index === 0 && (
<a
className="ml-2 border border-zinc-900 font-light px-[25px] py-2.5 text-sm transition hover:bg-zinc-900 hover:text-white"
href={`/datasets/${dataset.name}`}
href={dataset.url}
>
explore
info
</a>
)}
</td>
@@ -143,6 +142,7 @@ export async function getStaticProps() {
export default function Home({ datasets }: { datasets: Dataset[] }) {
return (
<>
<NextSeo title="FiveThirtyEight tribute by PortalJS" />
<main
className={`flex min-h-screen flex-col items-center max-w-5xl mx-auto pt-20 px-2.5 ${inter.className}`}
>
@@ -150,7 +150,7 @@ export default function Home({ datasets }: { datasets: Dataset[] }) {
<h1 className="text-[40px] font-bold text-zinc-800 text-center">
Our Data
</h1>
<p className="max-w-2xl text-lg text-center text-zinc-700">
<p className="max-w-[600px] text-[17px] text-center text-[#6d6f71]">
Were sharing the data and code behind some of our articles and
graphics. We hope youll use it to check our work and to create
stories and visualizations of&nbsp;your&nbsp;own.
@@ -164,13 +164,13 @@ export default function Home({ datasets }: { datasets: Dataset[] }) {
<table className="w-full mt-10 mb-4 hidden md:table">
<thead className="border-b-4 pb-2 border-zinc-900">
<tr>
<th className="uppercase text-left font-light text-xs pb-3">
<th className="uppercase text-left font-normal text-xs pb-3">
data set
</th>
<th className="uppercase text-left font-light text-xs pb-3">
<th className="uppercase text-left font-normal text-xs pb-3">
related content
</th>
<th className="uppercase text-left font-light text-xs pb-3">
<th className="uppercase text-left font-normal text-xs pb-3">
last updated
</th>
</tr>

View File

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 25 KiB

View File

Before

Width:  |  Height:  |  Size: 1.3 KiB

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 81 KiB

View File

Before

Width:  |  Height:  |  Size: 629 B

After

Width:  |  Height:  |  Size: 629 B

View File

@@ -0,0 +1,29 @@
{
"extends": ["next", "next/core-web-vitals"],
"ignorePatterns": ["!**/*", ".next/**/*"],
"overrides": [
{
"files": ["*.ts", "*.tsx", "*.js", "*.jsx"],
"rules": {
"@next/next/no-html-link-for-pages": [
"error",
"examples/simple-example/pages"
]
}
},
{
"files": ["*.ts", "*.tsx"],
"rules": {}
},
{
"files": ["*.js", "*.jsx"],
"rules": {}
}
],
"rules": {
"@next/next/no-html-link-for-pages": "off"
},
"env": {
"jest": true
}
}

View File

@@ -0,0 +1,7 @@
node_modules
**/.next/**
**/_next/**
**/dist/**
**/__tmp__/**
lerna.json
.github

View File

@@ -0,0 +1 @@
{}

View File

@@ -0,0 +1,101 @@
# A data catalog with data on GitHub
This example showcases a simple data catalog that get its data from a list of GitHub repos that serve as datasets.
A `datasets.json` file is used to specify which datasets are going to be part of the data catalog.
The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
## Demo
https://example.portaljs.org/
## Deploy your own
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fgithub-backed-catalog)
By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own GitHub/GitLab/Bitbucket account and automatically deploy everything.
## How to use
### Install
Execute `create-next-app` to bootstrap the example:
```
npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/github-backed-catalog
cd <app-name>
```
### Set environment variables
This project uses the GitHub API, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
```
GITHUB_PAT=<github token>
```
### Change datasets
You can change the datasets that will be displayed in the data catalog by editing the file `datasets.json`. Some examples can be found inside [this repo](https://github.com/datasets).
### Run in development mode
Run the app using:
```
npm run dev
```
Open http://localhost:3000 from your browser. You should see something similar to this:
![](https://i.imgur.com/jAljJ9C.png)
If click on the `info` button for a dataset you will see a page similar to this:
![](https://i.imgur.com/AoJd4O0.png)
## Notes
### Structure of `datasets.json`
The `datasets.json` file is simply a list of datasets, below you can see a minimal example of a dataset:
```json
{
"owner": "fivethirtyeight",
"repo": "data",
"branch": "master",
"files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
"readme": "nba-raptor/README.md"
}
```
It has:
- A `owner` which is going to be the github repo owner
- A `repo` which is going to be the github repo name
- A `branch` which is going to be the branch to which we need to get the files and the readme
- A list of `files` which is going to be a list of paths with files that you want to show to the world
- A `readme` which is going to be the path to your data description, it can also be a subpath eg: `example/README.md`
You can also add:
- A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
- A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
### Extra commands
You can also build the project for production with:
```
npm run build
```
And run the production build with:
```
npm run start
```

View File

@@ -0,0 +1,28 @@
import Link from "next/link";
import HomeIcon from "../icons/HomeIcon";
export default function Breadcrumbs({
links,
}: {
links: { title: string; href?: string; target?: string }[];
}) {
const current = links.at(-1);
return (
<div className="flex items-center uppercase font-black text-xs">
<Link className="flex items-center" href="/">
<HomeIcon />
</Link>
{/* {links.length > 1 && links.slice(0, -1).map((link) => {
return <>
<span className="mx-4">/</span>
<Link href={link.href}>{link.title}</Link>
</>
})} */}
<span className="mx-4">/</span>
<span>{current.title}</span>
</div>
);
}

View File

@@ -0,0 +1,13 @@
export default function ExternalLinkIcon({ className = "" }) {
return (
<div className={`inline-block w-4 ${className}`}>
<svg
xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 64 64"
fill="currentColor"
>
<path d="M 40 10 C 38.896 10 38 10.896 38 12 C 38 13.104 38.896 14 40 14 L 47.171875 14 L 30.585938 30.585938 C 29.804938 31.366938 29.804938 32.633063 30.585938 33.414062 C 30.976938 33.805063 31.488 34 32 34 C 32.512 34 33.023063 33.805062 33.414062 33.414062 L 50 16.828125 L 50 24 C 50 25.104 50.896 26 52 26 C 53.104 26 54 25.104 54 24 L 54 12 C 54 10.896 53.104 10 52 10 L 40 10 z M 18 12 C 14.691 12 12 14.691 12 18 L 12 46 C 12 49.309 14.691 52 18 52 L 46 52 C 49.309 52 52 49.309 52 46 L 52 34 C 52 32.896 51.104 32 50 32 C 48.896 32 48 32.896 48 34 L 48 46 C 48 47.103 47.103 48 46 48 L 18 48 C 16.897 48 16 47.103 16 46 L 16 18 C 16 16.897 16.897 16 18 16 L 30 16 C 31.104 16 32 15.104 32 14 C 32 12.896 31.104 12 30 12 L 18 12 z" />
</svg>
</div>
);
}

View File

@@ -0,0 +1,10 @@
export default function HomeIcon({ className = "" }) {
return (
<div className={`inline-block w-4 ${className}`}>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
{" "}
<path d="M 12 2 A 1 1 0 0 0 11.289062 2.296875 L 1.203125 11.097656 A 0.5 0.5 0 0 0 1 11.5 A 0.5 0.5 0 0 0 1.5 12 L 4 12 L 4 20 C 4 20.552 4.448 21 5 21 L 9 21 C 9.552 21 10 20.552 10 20 L 10 14 L 14 14 L 14 20 C 14 20.552 14.448 21 15 21 L 19 21 C 19.552 21 20 20.552 20 20 L 20 12 L 22.5 12 A 0.5 0.5 0 0 0 23 11.5 A 0.5 0.5 0 0 0 22.796875 11.097656 L 12.716797 2.3027344 A 1 1 0 0 0 12.710938 2.296875 A 1 1 0 0 0 12 2 z" />
</svg>
</div>
);
}

View File

@@ -19,10 +19,7 @@
"owner": "datasets",
"branch": "main",
"repo": "investor-flow-of-funds-us",
"files": [
"data/monthly.csv",
"data/weekly.csv"
],
"files": ["data/monthly.csv", "data/weekly.csv"],
"readme": "README.md"
},
{
@@ -38,7 +35,10 @@
"owner": "fivethirtyeight",
"repo": "data",
"branch": "master",
"files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
"files": [
"nba-raptor/historical_RAPTOR_by_player.csv",
"nba-raptor/historical_RAPTOR_by_team.csv"
],
"readme": "nba-raptor/README.md"
}
]

View File

@@ -0,0 +1,6 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
declare module "*.svg" {
const content: any;
export const ReactComponent: any;
export default content;
}

View File

@@ -0,0 +1,172 @@
import { Octokit } from "octokit";
export interface GithubProject {
owner: string;
repo: string;
branch: string;
files: string[];
readme: string;
description?: string;
name?: string;
}
export async function getProjectReadme(
owner: string,
repo: string,
branch: string,
readme: string,
github_pat?: string
) {
const octokit = new Octokit({ auth: github_pat });
try {
const response = await octokit.rest.repos.getContent({
owner,
repo,
path: readme,
ref: branch,
});
const data = response.data as { content?: string };
const fileContent = data.content ? data.content : "";
if (fileContent === "") {
return null;
}
const decodedContent = Buffer.from(fileContent, "base64").toString();
return decodedContent;
} catch (error) {
throw new Error(
"Couldn't get project readme please make sure that you are pointing to a valid repo and that the repo in question contains a README.md"
);
}
}
export async function getLastUpdated(
owner: string,
repo: string,
branch: string,
readme: string,
github_pat?: string
) {
const octokit = new Octokit({ auth: github_pat });
try {
const response = await octokit.rest.repos.listCommits({
owner,
repo,
ref: branch,
});
return response.data[0].commit.committer.date;
} catch (error) {
throw new Error(
"Couldn't get project list of commits please make sure that you are pointing to a valid repo"
);
}
}
export async function getProjectMetadata(
owner: string,
repo: string,
github_pat?: string
) {
const octokit = new Octokit({ auth: github_pat });
try {
const response = await octokit.rest.repos.get({
owner,
repo,
});
return response.data;
} catch (error) {
throw new Error(
"Couldn't get project metadata please make sure that you are pointing to a valid repo"
);
}
}
export async function getRepoContents(
owner: string,
repo: string,
branch: string,
files: string[],
github_pat?: string
) {
const octokit = new Octokit({ auth: github_pat });
try {
const contents = [];
for (const path of files) {
const response = await octokit.rest.repos.getContent({
owner,
repo,
ref: branch,
path: path,
});
const data = response.data as {
download_url?: string;
name: string;
size: number;
};
contents.push({
download_url: data.download_url,
name: data.name,
size: data.size,
});
}
return contents;
} catch (error) {
if (
error.message ===
'This endpoint can only return blobs smaller than 100 MB in size. The requested blob is too large to fetch via the API, but you can always clone the repository via Git to obtain it.: {"resource":"Blob","field":"data","code":"too_large"}'
) {
throw new Error(
`The requested files ${files.join(
", "
)} are too big making it impossible to fetch via Github API`
);
}
throw new Error(
"Couldn't get project contents please make sure that you are pointing to a valid repo"
);
}
}
export async function getProject(project: GithubProject, github_pat?: string) {
const projectMetadata = await getProjectMetadata(
project.owner,
project.repo,
github_pat
);
if (!projectMetadata) {
return null;
}
const projectReadme = await getProjectReadme(
project.owner,
project.repo,
project.branch,
project.readme,
github_pat
);
let projectData = [];
if (project.files) {
projectData = await getRepoContents(
project.owner,
project.repo,
project.branch,
project.files,
github_pat
);
}
const projectBase =
project.readme && project.readme.split("/").length > 1
? project.readme.split("/").slice(0, -1).join("/")
: "/";
const last_updated = await getLastUpdated(
project.owner,
project.repo,
project.branch,
projectBase,
github_pat
);
return {
...projectMetadata,
files: projectData,
readmeContent: projectReadme,
last_updated,
base_path: projectBase,
};
}

View File

@@ -0,0 +1,17 @@
const nextConfig = {
async rewrites() {
return {
beforeFiles: [
{
source: "/@:org/:project*",
destination: "/@org/:org/:project*",
},
],
};
},
serverRuntimeConfig: {
github_pat: process.env.GITHUB_PAT ? process.env.GITHUB_PAT : null,
},
};
module.exports = nextConfig;

View File

@@ -19,6 +19,7 @@
"react": "18.2.0",
"react-dom": "18.2.0",
"react-markdown": "^8.0.7",
"react-timeago": "^7.1.0",
"remark-gfm": "^3.0.1",
"typescript": "5.0.4"
},
@@ -26,6 +27,7 @@
"@tailwindcss/typography": "^0.5.9",
"autoprefixer": "^10.4.14",
"postcss": "^8.4.23",
"prettier": "2.8.8",
"tailwindcss": "^3.3.1"
}
},
@@ -4676,6 +4678,21 @@
"node": ">= 0.8.0"
}
},
"node_modules/prettier": {
"version": "2.8.8",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-2.8.8.tgz",
"integrity": "sha512-tdN8qQGvNjw4CHbY+XXk0JgCXn9QiF21a55rBe5LJAU+kDyC4WQn4+awm2Xfk2lQMk5fKup9XgzTZtGkjBdP9Q==",
"dev": true,
"bin": {
"prettier": "bin-prettier.js"
},
"engines": {
"node": ">=10.13.0"
},
"funding": {
"url": "https://github.com/prettier/prettier?sponsor=1"
}
},
"node_modules/prop-types": {
"version": "15.8.1",
"resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz",
@@ -4797,6 +4814,14 @@
"resolved": "https://registry.npmjs.org/react-is/-/react-is-18.2.0.tgz",
"integrity": "sha512-xWGDIW6x921xtzPkhiULtthJHoJvBbF3q26fzloPCK0hsvxtPVelvftw3zjbHWSkR2km9Z+4uxbDDK/6Zw9B8w=="
},
"node_modules/react-timeago": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/react-timeago/-/react-timeago-7.1.0.tgz",
"integrity": "sha512-rouF7MiEm55fH791Y8cg+VobIJgx8gtNJ+gjr86R4ZqO1WKPkXiXjdT/lRzrvEkUzsxT1exHqV2V+Zdi114H3A==",
"peerDependencies": {
"react": "^16.0.0 || ^17.0.0 || ^18.0.0"
}
},
"node_modules/read-cache": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/read-cache/-/read-cache-1.0.0.tgz",

View File

@@ -0,0 +1,35 @@
{
"name": "my-app",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "next lint",
"prettier": "prettier --write ."
},
"dependencies": {
"@types/node": "18.16.0",
"@types/react": "18.0.38",
"@types/react-dom": "18.0.11",
"eslint": "8.39.0",
"eslint-config-next": "13.3.1",
"next": "13.3.1",
"next-seo": "^6.0.0",
"octokit": "^2.0.14",
"react": "18.2.0",
"react-dom": "18.2.0",
"react-markdown": "^8.0.7",
"react-timeago": "^7.1.0",
"remark-gfm": "^3.0.1",
"typescript": "5.0.4"
},
"devDependencies": {
"@tailwindcss/typography": "^0.5.9",
"autoprefixer": "^10.4.14",
"postcss": "^8.4.23",
"prettier": "2.8.8",
"tailwindcss": "^3.3.1"
}
}

View File

@@ -0,0 +1,127 @@
import { NextSeo } from "next-seo";
import { promises as fs } from "fs";
import path from "path";
import getConfig from "next/config";
import { getProject, GithubProject } from "../../../lib/octokit";
import ReactMarkdown from "react-markdown";
import remarkGfm from "remark-gfm";
import Breadcrumbs from "../../../components/_shared/Breadcrumbs";
export default function ProjectPage({ project }) {
const repoId = `@${project.repo_config.owner}/${project.repo_config.repo}`;
return (
<>
<NextSeo
title={`${repoId}${
project.base_path !== "/" ? "/" + project.base_path : ""
} - GitHub Datasets`}
/>
<main className="prose mx-auto my-8">
<Breadcrumbs links={[{ title: repoId, href: "" }]} />
<h1 className="mb-0 mt-16">{project.repo_config.name || repoId}</h1>
<p className="mb-8">
<span className="font-semibold">Repository:</span>{" "}
<a target="_blank" href={project.html_url}>
{project.html_url}
</a>
</p>
<h2 className="mb-0 mt-10">Files</h2>
<div className="inline-block min-w-full py-2 align-middle">
<table className="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Name
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Size
</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200">
{project.files.map((file) => (
<tr key={file.download_url}>
<td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
<a href={file.download_url}>{file.name}</a>
</td>
<td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
{file.size} Bytes
</td>
</tr>
))}
</tbody>
</table>
</div>
<hr />
<h2 className="uppercase font-black">Readme</h2>
<ReactMarkdown remarkPlugins={[remarkGfm]}>
{project.readmeContent}
</ReactMarkdown>
</main>
</>
);
}
// Generates `/posts/1` and `/posts/2`
export async function getStaticPaths() {
const jsonDirectory = path.join(process.cwd(), "datasets.json");
const repos = await fs.readFile(jsonDirectory, "utf8");
return {
paths: JSON.parse(repos).map((repo) => {
const projectPath =
repo.readme && repo.readme.split("/").length > 1
? repo.readme.split("/").slice(0, -1)
: null;
let path = [repo.repo];
if (projectPath) {
projectPath.forEach((element) => {
path.push(element);
});
}
return {
params: { org: repo.owner, path },
};
}),
fallback: false, // can also be true or 'blocking'
};
}
export async function getStaticProps({ params }) {
const jsonDirectory = path.join(process.cwd(), "datasets.json");
const reposFile = await fs.readFile(jsonDirectory, "utf8");
const repos: GithubProject[] = JSON.parse(reposFile);
const repo = repos.find((_repo) => {
const projectPath =
_repo.readme && _repo.readme.split("/").length > 1
? _repo.readme.split("/").slice(0, -1)
: null;
let path = [_repo.repo];
if (projectPath) {
projectPath.forEach((element) => {
path.push(element);
});
}
return (
_repo.owner == params.org &&
JSON.stringify(path) === JSON.stringify(params.path)
);
});
const github_pat = getConfig().serverRuntimeConfig.github_pat;
const project = await getProject(repo, github_pat);
return {
props: {
project: { ...project, repo_config: repo },
},
};
}

View File

@@ -0,0 +1,18 @@
import { AppProps } from "next/app";
import Head from "next/head";
import "./styles.css";
function CustomApp({ Component, pageProps }: AppProps) {
return (
<>
<Head>
<title>GitHub Datasets</title>
</Head>
<main className="app">
<Component {...pageProps} />
</main>
</>
);
}
export default CustomApp;

View File

@@ -0,0 +1,139 @@
import { promises as fs } from "fs";
import path from "path";
import { getProject } from "../lib/octokit";
import getConfig from "next/config";
import ExternalLinkIcon from "../components/icons/ExternalLinkIcon";
import TimeAgo from "react-timeago";
import Link from "next/link";
import { NextSeo } from "next-seo";
export async function getStaticProps() {
const jsonDirectory = path.join(process.cwd(), "/datasets.json");
const repos = await fs.readFile(jsonDirectory, "utf8");
const github_pat = getConfig().serverRuntimeConfig.github_pat;
const projects = await Promise.all(
JSON.parse(repos).map(async (repo) => {
const project = await getProject(repo, github_pat);
return { ...project, repo_config: repo };
})
);
return {
props: {
projects,
},
};
}
export function Datasets({ projects }) {
return (
<>
<NextSeo title="GitHub Datasets" />
<div className="bg-white min-h-screen">
<div className="mx-auto max-w-7xl px-6 py-16 sm:py-24 lg:px-8">
<div className="text-center">
<h2 className="text-3xl font-bold leading-10 tracking-tight">
GitHub Datasets
</h2>
<p className="mt-3 mx-auto max-w-2xl text-base leading-7 text-gray-500">
Data catalog with datasets hosted on GitHub by{" "}
<Link
target="_blank"
className="underline"
href="https://portaljs.org/"
>
🌀 PortalJS
</Link>
</p>
</div>
<div className="mt-20">
<div className="-mx-4 -my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
<div className="inline-block min-w-full py-2 align-middle sm:px-6 lg:px-8">
<table className="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Name
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Repository
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Description
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Last updated
</th>
<th
scope="col"
className="relative py-3.5 pl-3 pr-4 sm:pr-0"
></th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200">
{projects.map((project) => (
<tr key={project.id}>
<td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
{project.repo_config.name
? project.repo_config.name
: project.full_name +
(project.base_path === "/"
? ""
: "/" + project.base_path)}
</td>
<td className="whitespace-nowrap px-3 py-6 text-sm group text-gray-500 hover:text-gray-900 transition-all duration-250">
<a
href={project.html_url}
target="_blank"
className="flex items-center"
>
@{project.full_name}{" "}
<ExternalLinkIcon className="ml-1" />
</a>
</td>
<td className="px-3 py-4 text-sm text-gray-500">
{project.repo_config.description
? project.repo_config.description
: project.description}
</td>
<td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
<TimeAgo date={new Date(project.last_updated)} />
</td>
<td className="relative whitespace-nowrap py-6 pl-3 pr-4 text-right text-sm font-medium sm:pr-0">
<a
href={`/@${project.repo_config.owner}/${
project.repo_config.repo
}/${
project.base_path === "/" ? "" : project.base_path
}`}
className="border border-gray-900 text-gray-900 px-4 py-2 transition-all hover:bg-gray-900 hover:text-white"
>
info
</a>
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</>
);
}
export default Datasets;

View File

@@ -0,0 +1,80 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
html {
-webkit-text-size-adjust: 100%;
font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont,
Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif,
Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;
line-height: 1.5;
tab-size: 4;
scroll-behavior: smooth;
}
body {
font-family: inherit;
line-height: inherit;
margin: 0;
}
h1,
h2,
p,
pre {
margin: 0;
}
*,
::before,
::after {
box-sizing: border-box;
border-width: 0;
border-style: solid;
border-color: currentColor;
}
h1,
h2 {
font-size: inherit;
font-weight: inherit;
}
a {
color: inherit;
text-decoration: inherit;
}
pre {
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
}
svg {
display: block;
vertical-align: middle;
shape-rendering: auto;
text-rendering: optimizeLegibility;
}
pre {
background-color: rgba(55, 65, 81, 1);
border-radius: 0.25rem;
color: rgba(229, 231, 235, 1);
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
overflow: scroll;
padding: 0.5rem 0.75rem;
}
.shadow {
box-shadow: 0 0 #0000, 0 0 #0000, 0 10px 15px -3px rgba(0, 0, 0, 0.1),
0 4px 6px -2px rgba(0, 0, 0, 0.05);
}
.rounded {
border-radius: 1.5rem;
}
.wrapper {
width: 100%;
}
.container {
margin-left: auto;
margin-right: auto;
max-width: 768px;
padding-bottom: 3rem;
padding-left: 1rem;
padding-right: 1rem;
color: rgba(55, 65, 81, 1);
width: 100%;
}

View File

@@ -0,0 +1,6 @@
module.exports = {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
};

View File

@@ -0,0 +1,12 @@
/** @type {import('tailwindcss').Config} */
module.exports = {
content: [
"./app/**/*.{js,ts,jsx,tsx,mdx}",
"./pages/**/*.{js,ts,jsx,tsx,mdx}",
"./components/**/*.{js,ts,jsx,tsx,mdx}",
],
theme: {
extend: {},
},
plugins: [require("@tailwindcss/typography")],
};

35
examples/openspending/.gitignore vendored Normal file
View File

@@ -0,0 +1,35 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts

View File

@@ -0,0 +1,102 @@
# A data catalog with data on GitHub
This example showcases a simple data catalog that get its data from a list of GitHub repos that serve as datasets.
A `datasets.json` file is used to specify which datasets are going to be part of the data catalog.
The application contains an index page, which lists all the datasets specified in the `datasets.json` file, and users can see more information about each dataset, such as the list of data files in it and the README, by clicking the "info" button on the list.
You can read more about it on the [Data catalog with data on GitHub](https://portaljs.org/docs/examples/github-backed-catalog) blog post.
## Demo
https://example.portaljs.org/
## Deploy your own
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fgithub-backed-catalog)
By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own GitHub/GitLab/Bitbucket account and automatically deploy everything.
## How to use
### Install
Execute `create-next-app` to bootstrap the example:
```
npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/github-backed-catalog
cd <app-name>
```
### Set environment variables
This project uses the GitHub API, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
```
GITHUB_PAT=<github token>
```
### Change datasets
You can change the datasets that will be displayed in the data catalog by editing the file `datasets.json`. Some examples can be found inside [this repo](https://github.com/datasets).
### Run in development mode
Run the app using:
```
npm run dev
```
Open http://localhost:3000 from your browser. You should see something similar to this:
![](https://i.imgur.com/jAljJ9C.png)
If click on the `info` button for a dataset you will see a page similar to this:
![](https://i.imgur.com/AoJd4O0.png)
## Notes
### Structure of `datasets.json`
The `datasets.json` file is simply a list of datasets, below you can see a minimal example of a dataset:
```json
{
"owner": "fivethirtyeight",
"repo": "data",
"branch": "master",
"files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
"readme": "nba-raptor/README.md"
}
```
It has:
- A `owner` which is going to be the github repo owner
- A `repo` which is going to be the github repo name
- A `branch` which is going to be the branch to which we need to get the files and the readme
- A list of `files` which is going to be a list of paths with files that you want to show to the world
- A `readme` which is going to be the path to your data description, it can also be a subpath eg: `example/README.md`
You can also add:
- A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
- A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
### Extra commands
You can also build the project for production with:
```
npm run build
```
And run the production build with:
```
npm run start
```

View File

@@ -0,0 +1,15 @@
import Link from 'next/link'
import clsx from 'clsx'
export function Button({ href, className = "", ...props }) {
className = clsx(
'inline-flex justify-center rounded-2xl bg-emerald-600 p-4 text-base font-semibold text-white hover:bg-emerald-500 focus:outline-none focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-emerald-500 active:text-white/70',
className
)
return href ? (
<Link href={href} className={className} {...props} />
) : (
<button className={className} {...props} />
)
}

View File

@@ -0,0 +1,10 @@
import clsx from 'clsx'
export function Container({ className = "", ...props }) {
return (
<div
className={clsx('mx-auto max-w-7xl px-4 sm:px-6 lg:px-8', className)}
{...props}
/>
)
}

View File

@@ -0,0 +1,54 @@
import Image from 'next/image'
import { Button } from './Button'
import { Container } from './Container'
import logo from "../public/logo.svg"
import Link from 'next/link'
import { useRouter } from 'next/router'
export function Header() {
const router = useRouter();
const isActive = (navLink) => {
return router.asPath.split("?")[0] == navLink.href;
}
const navLinks = [
{
title: "Home",
href: "/#header"
},
{
title: "Datasets",
href: "/#datasets"
},
{
title: "Community",
href: "https://community.openspending.org/"
}
]
return (
<header className="z-50 pb-5 lg:pt-11 sticky top-0 backdrop-blur" id="header">
<Container className="flex flex-wrap items-center justify-center sm:justify-between lg:flex-nowrap">
<div className="mt-10 lg:mt-0 lg:grow lg:basis-0 flex items-center">
<Image src={logo} alt="OpenSpending" className="h-12 w-auto" />
</div>
<ul className='list-none flex gap-x-5 text-base font-medium'>
{navLinks.map((link, i) => (
<li key={`nav-link-${i}`}>
<Link
className={`text-emerald-900 hover:text-emerald-600 ${isActive(link) ? "text-emerald-600" : ""}`}
href={link.href}
scroll={false}
>
{link.title}
</Link>
</li>))}
</ul>
<div className="hidden sm:mt-10 sm:flex lg:mt-0 lg:grow lg:basis-0 lg:justify-end">
<Button href="#">View on GitHub</Button>
</div>
</Container>
</header >
)
}

View File

@@ -0,0 +1,47 @@
import { Button } from './Button'
import { Container } from './Container'
export function Hero() {
return (
<div className="relative pb-20 pt-10 sm:py-40">
<div className="absolute inset-x-0 -bottom-14 -top-48 overflow-hidden bg-green-50 bg-opacity-50">
<div className="absolute inset-x-0 top-0 h-40 bg-gradient-to-b from-white" />
<div className="absolute inset-x-0 bottom-0 h-40 bg-gradient-to-t from-white" />
</div>
<Container className="relative">
<div className="mx-auto max-w-2xl lg:max-w-4xl lg:px-12">
<h1 className="font-display text-5xl font-bold tracking-tighter text-emerald-600 sm:text-7xl">
It's our money!
</h1>
<div className="mt-6 space-y-6 font-display text-2xl tracking-tight text-emerald-900">
<p>
By understanding how governments spend money in our name can we have a say
in how that money will affect our own lives. The journey starts here.
</p>
<p>
OpenSpending is a free, open and global platform to search, visualise and analyse
fiscal data in the public sphere.
</p>
</div>
<Button href="#" className="mt-10 w-full sm:hidden">
View on GitHub
</Button>
<dl className="mt-10 grid grid-cols-2 gap-x-10 gap-y-6 sm:mt-16 sm:gap-x-16 sm:gap-y-10 sm:text-center lg:auto-cols-auto lg:grid-flow-col lg:grid-cols-none lg:justify-start lg:text-left">
{[
['Countries', '75'],
['Datasets', '2091'],
['Files', '9230'],
].map(([name, value]) => (
<div key={name}>
<dt className="font-mono text-sm text-emerald-600">{name}</dt>
<dd className="mt-0.5 text-2xl font-semibold tracking-tight text-emerald-900">
{value}
</dd>
</div>
))}
</dl>
</div>
</Container>
</div>
)
}

View File

@@ -0,0 +1,20 @@
import Link from "next/link";
import HomeIcon from "../icons/HomeIcon";
export default function Breadcrumbs({ links }: { links: { title: string, href?: string, target?: string }[] }) {
const current = links.at(-1);
return <div className="flex items-center uppercase font-black text-xs">
<Link className="flex items-center" href='/'><HomeIcon /></Link>
{/* {links.length > 1 && links.slice(0, -1).map((link) => {
return <>
<span className="mx-4">/</span>
<Link href={link.href}>{link.title}</Link>
</>
})} */}
<span className="mx-4">/</span>
<span>{current.title}</span>
</div >
}

View File

@@ -0,0 +1,3 @@
export default function ExternalLinkIcon({ className = "" }) {
return <div className={`inline-block w-4 ${className}`}><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" fill="currentColor"><path d="M 40 10 C 38.896 10 38 10.896 38 12 C 38 13.104 38.896 14 40 14 L 47.171875 14 L 30.585938 30.585938 C 29.804938 31.366938 29.804938 32.633063 30.585938 33.414062 C 30.976938 33.805063 31.488 34 32 34 C 32.512 34 33.023063 33.805062 33.414062 33.414062 L 50 16.828125 L 50 24 C 50 25.104 50.896 26 52 26 C 53.104 26 54 25.104 54 24 L 54 12 C 54 10.896 53.104 10 52 10 L 40 10 z M 18 12 C 14.691 12 12 14.691 12 18 L 12 46 C 12 49.309 14.691 52 18 52 L 46 52 C 49.309 52 52 49.309 52 46 L 52 34 C 52 32.896 51.104 32 50 32 C 48.896 32 48 32.896 48 34 L 48 46 C 48 47.103 47.103 48 46 48 L 18 48 C 16.897 48 16 47.103 16 46 L 16 18 C 16 16.897 16.897 16 18 16 L 30 16 C 31.104 16 32 15.104 32 14 C 32 12.896 31.104 12 30 12 L 18 12 z"/></svg></div>
}

View File

@@ -0,0 +1,3 @@
export default function HomeIcon({ className = "" }) {
return <div className={`inline-block w-4 ${className}`}><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"> <path d="M 12 2 A 1 1 0 0 0 11.289062 2.296875 L 1.203125 11.097656 A 0.5 0.5 0 0 0 1 11.5 A 0.5 0.5 0 0 0 1.5 12 L 4 12 L 4 20 C 4 20.552 4.448 21 5 21 L 9 21 C 9.552 21 10 20.552 10 20 L 10 14 L 14 14 L 14 20 C 14 20.552 14.448 21 15 21 L 19 21 C 19.552 21 20 20.552 20 20 L 20 12 L 22.5 12 A 0.5 0.5 0 0 0 23 11.5 A 0.5 0.5 0 0 0 22.796875 11.097656 L 12.716797 2.3027344 A 1 1 0 0 0 12.710938 2.296875 A 1 1 0 0 0 12 2 z"/></svg></div>
}

View File

@@ -0,0 +1,25 @@
[
{
"owner": "os-data",
"branch": "main",
"repo": "mongolia-budget-2016-2017",
"files": [
"data/mongolia-2017.csv",
"data/mongolia-2017__2017.csv"
]
},
{
"owner": "os-data",
"branch": "main",
"repo": "gb-country-regional-analysis",
"files": [
"data/cofog.csv",
"data/cofog_dejargonise.csv",
"data/cra.csv",
"data/departments.csv",
"data/nuts_pop.csv",
"data/pogs.csv"
],
"readme": "README.md"
}
]

View File

@@ -94,8 +94,16 @@ export async function getRepoContents(
ref: branch,
path: path,
});
const data = response.data as { download_url?: string, name: string, size: number };
contents.push({ download_url: data.download_url, name: data.name, size: data.size});
const data = response.data as {
download_url?: string;
name: string;
size: number;
};
contents.push({
download_url: data.download_url,
name: data.name,
size: data.size,
});
}
return contents;
} catch (error) {
@@ -120,9 +128,7 @@ export async function getProject(project: GithubProject, github_pat?: string) {
project.readme,
github_pat
);
if (!projectReadme) {
return null;
}
const projectData = await getRepoContents(
project.owner,
project.repo,
@@ -133,15 +139,26 @@ export async function getProject(project: GithubProject, github_pat?: string) {
if (!projectData) {
return null;
}
const projectBase = project.readme.split('/').length > 1
? project.readme.split('/').slice(0, -1).join('/')
: '/'
const last_updated = await getLastUpdated(
project.owner,
project.repo,
project.branch,
projectBase,
github_pat
);
return { ...projectMetadata, files: projectData, readmeContent: projectReadme, last_updated, base_path: projectBase };
let projectBase = "", last_updated = "";
if (projectReadme) {
projectBase =
project.readme.split('/').length > 1
? project.readme.split('/').slice(0, -1).join('/')
: '/';
last_updated = await getLastUpdated(
project.owner,
project.repo,
project.branch,
projectBase,
github_pat
);
}
return {
...projectMetadata,
files: projectData,
readmeContent: projectReadme,
last_updated,
base_path: projectBase,
};
}

5866
examples/openspending/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -12,14 +12,17 @@
"@types/node": "18.16.0",
"@types/react": "18.0.38",
"@types/react-dom": "18.0.11",
"clsx": "^1.2.1",
"eslint": "8.39.0",
"eslint-config-next": "13.3.1",
"next": "13.3.1",
"next-seo": "^6.0.0",
"octokit": "^2.0.14",
"prettier": "^2.8.8",
"react": "18.2.0",
"react-dom": "18.2.0",
"react-markdown": "^8.0.7",
"react-timeago": "^7.1.0",
"remark-gfm": "^3.0.1",
"typescript": "5.0.4"
},

View File

@@ -1,6 +1,3 @@
import Head from 'next/head';
import { useRouter } from 'next/router';
import { NextSeo } from 'next-seo';
import { promises as fs } from 'fs';
import path from 'path';
@@ -8,15 +5,20 @@ import getConfig from 'next/config';
import { getProject, GithubProject } from '../../../lib/octokit';
import ReactMarkdown from 'react-markdown';
import remarkGfm from 'remark-gfm';
import Link from 'next/link';
import Breadcrumbs from '../../../components/_shared/Breadcrumbs';
export default function ProjectPage({ project }) {
const repoId = `@${project.repo_config.owner}/${project.repo_config.repo}`
return (
<>
<NextSeo title={`PortalJS - @${project.repo_config.owner}/${project.repo_config.repo}${project.base_path !== '/' ? '/' + project.base_path : ''}`} />
<NextSeo title={`${repoId}${project.base_path !== '/' ? '/' + project.base_path : ''} - GitHub Datasets`} />
<main className="prose mx-auto my-8">
<Link href='/'>Back to homepage</Link>
<h1 className="mb-0">Data</h1>
<Breadcrumbs links={[{ title: repoId, href: "" }]} />
<h1 className="mb-0 mt-16">{project.repo_config.name || repoId}</h1>
<p className='mb-8'><span className='font-semibold'>Repository:</span> <a target="_blank" href={project.html_url}>{project.html_url}</a></p>
<h2 className="mb-0 mt-10">Files</h2>
<div className="inline-block min-w-full py-2 align-middle">
<table className="min-w-full divide-y divide-gray-300">
<thead>
@@ -50,10 +52,14 @@ export default function ProjectPage({ project }) {
</table>
</div>
<h1>Readme</h1>
<ReactMarkdown remarkPlugins={[remarkGfm]}>
{project.readmeContent}
</ReactMarkdown>
{project.readmeContent && <>
<hr />
<h2 className='uppercase font-black'>Readme</h2>
<ReactMarkdown remarkPlugins={[remarkGfm]}>
{project.readmeContent}
</ReactMarkdown>
</>}
</main>
</>
);
@@ -70,7 +76,7 @@ export async function getStaticPaths() {
return {
paths: JSON.parse(repos).map((repo) => {
const projectPath =
repo.readme.split('/').length > 1
repo.readme && repo.readme.split('/').length > 1
? repo.readme.split('/').slice(0, -1)
: null;
let path = [repo.repo];
@@ -96,7 +102,7 @@ export async function getStaticProps({ params }) {
const repos: GithubProject[] = JSON.parse(reposFile);
const repo = repos.find((_repo) => {
const projectPath =
_repo.readme.split('/').length > 1
_repo.readme && _repo.readme.split('/').length > 1
? _repo.readme.split('/').slice(0, -1)
: null;
let path = [_repo.repo];

View File

@@ -6,7 +6,7 @@ function CustomApp({ Component, pageProps }: AppProps) {
return (
<>
<Head>
<title>Welcome to simple-example!</title>
<title>GitHub Datasets</title>
</Head>
<main className="app">
<Component {...pageProps} />

View File

@@ -0,0 +1,130 @@
import { promises as fs } from 'fs';
import path from 'path';
import { getProject } from '../lib/octokit';
import getConfig from 'next/config';
import ExternalLinkIcon from '../components/icons/ExternalLinkIcon';
import TimeAgo from 'react-timeago';
import Link from 'next/link';
import { Hero } from '../components/Hero';
import { Header } from '../components/Header';
import { Container } from '../components/Container';
export async function getStaticProps() {
const jsonDirectory = path.join(
process.cwd(),
'/datasets.json'
);
const repos = await fs.readFile(jsonDirectory, 'utf8');
const github_pat = getConfig().serverRuntimeConfig.github_pat;
const projects = await Promise.all(
(JSON.parse(repos)).map(async (repo) => {
const project = await getProject(repo, github_pat);
return { ...project, repo_config: repo };
})
);
return {
props: {
projects,
},
};
}
export function Datasets({ projects }) {
return (
<div className="bg-white min-h-screen">
<Header />
<Hero />
<section
className="py-20 sm:py-32"
>
<Container>
<div className="mx-auto max-w-2xl lg:mx-0">
<h2
id="datasets"
className="font-display text-4xl font-medium tracking-tighter text-emerald-600 sm:text-5xl"
>
Datasets
</h2>
<p className="mt-4 font-display text-2xl tracking-tight text-emerald-900">
Find spending data about countries all around the world.
</p>
</div>
<div className="mt-5">
<div className="-mx-4 -my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
<div className="inline-block min-w-full py-2 align-middle sm:px-6 lg:px-8">
<table className="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Name
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Repository
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Description
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Last updated
</th>
<th
scope="col"
className="relative py-3.5 pl-3 pr-4 sm:pr-0"
></th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200">
{projects.map((project) => (
<tr key={project.id}>
<td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
{project.repo_config.name
? project.repo_config.name
: project.full_name + (project.base_path === '/' ? '' : '/' + project.base_path)}
</td>
<td className="whitespace-nowrap px-3 py-6 text-sm group text-gray-500 hover:text-gray-900 transition-all duration-250">
<a href={project.html_url} target="_blank" className='flex items-center'>@{project.full_name} <ExternalLinkIcon className='ml-1' /></a>
</td>
<td className="px-3 py-4 text-sm text-gray-500">
{project.repo_config.description
? project.repo_config.description
: project.description}
</td>
<td className="whitespace-nowrap px-3 py-6 text-sm text-gray-500">
<TimeAgo date={new Date(project.last_updated)} />
</td>
<td className="relative whitespace-nowrap py-6 pl-3 pr-4 text-right text-sm font-medium sm:pr-0">
<a
href={`/@${project.repo_config.owner}/${project.repo_config.repo}/${project.base_path === '/' ? '' : project.base_path}`}
className='border border-gray-900 text-gray-900 px-4 py-2 transition-all hover:bg-gray-900 hover:text-white'
>
info
</a>
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
</div>
</Container>
</section>
</div>
);
}
export default Datasets;

View File

@@ -0,0 +1,80 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
html {
-webkit-text-size-adjust: 100%;
font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont,
Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif,
Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;
line-height: 1.5;
tab-size: 4;
scroll-behavior: smooth;
}
body {
font-family: inherit;
line-height: inherit;
margin: 0;
}
h1,
h2,
p,
pre {
margin: 0;
}
*,
::before,
::after {
box-sizing: border-box;
border-width: 0;
border-style: solid;
border-color: currentColor;
}
h1,
h2 {
font-size: inherit;
font-weight: inherit;
}
a {
color: inherit;
text-decoration: inherit;
}
pre {
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
}
svg {
display: block;
vertical-align: middle;
shape-rendering: auto;
text-rendering: optimizeLegibility;
}
pre {
background-color: rgba(55, 65, 81, 1);
border-radius: 0.25rem;
color: rgba(229, 231, 235, 1);
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
overflow: scroll;
padding: 0.5rem 0.75rem;
}
.shadow {
box-shadow: 0 0 #0000, 0 0 #0000, 0 10px 15px -3px rgba(0, 0, 0, 0.1),
0 4px 6px -2px rgba(0, 0, 0, 0.05);
}
.rounded {
border-radius: 1.5rem;
}
.wrapper {
width: 100%;
}
.container {
margin-left: auto;
margin-right: auto;
max-width: 768px;
padding-bottom: 3rem;
padding-left: 1rem;
padding-right: 1rem;
color: rgba(55, 65, 81, 1);
width: 100%;
}

View File

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.6 KiB

View File

@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "es5",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": false,
"forceConsistentCasingInFileNames": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "node",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
"exclude": ["node_modules"]
}

View File

@@ -1,75 +0,0 @@
This is a repo intended to serve as a simple example of a data catalog that get its data from a series of github repos, you can init an example just like this one by.
- Creating a new project with `create-next-app` like so:
```
npx create-next-app <app-name> --example https://github.com/datopian/portaljs/tree/main/examples/simple-example
cd <app-name>
```
- This project uses the github api, which for anonymous users will cap at 50 requests per hour, so you might want to get a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to a `.env` file inside the folder like so
```
GITHUB_PAT=<github token>
```
- Edit the file `datasets.json` to your liking, some examples can be found inside this [repo](https://github.com/datasets)
- Run the app using:
```
npm run dev
```
Congratulations, you now have something similar to this running on `http://localhost:3000`
![](https://i.imgur.com/jAljJ9C.png)
If yo go to any one of those pages by clicking on `More info` you will see something similar to this
![](https://i.imgur.com/AoJd4O0.png)
## Deployment
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fdatopian%2Fportaljs%2Ftree%2Fmain%2Fexamples%2Fsimple-example)
By clicking on this button, you will be redirected to a page which will allow you to clone the content into your own github/gitlab/bitbucket account and automatically deploy everything.
## Structure of `datasets.json`
The `datasets.json` file is simply a list of datasets, below you can see a minimal example of a dataset
```json
{
"owner": "fivethirtyeight",
"repo": "data",
"branch": "master",
"files": ["nba-raptor/historical_RAPTOR_by_player.csv", "nba-raptor/historical_RAPTOR_by_team.csv"],
"readme": "nba-raptor/README.md"
}
```
It has
- A `owner` which is going to be the github repo owner
- A `repo` which is going to be the github repo name
- A `branch` which is going to be the branch to which we need to get the files and the readme
- A list of `files` which is going to be a list of paths with files that you want to show to the world
- A `readme` which is going to be the path to your data description, it can also be a subpath eg: `example/README.md`
You can also add
- A `description` which is useful if you have more than one dataset for each repo, if not provided we are just going to use the repo description
- A `Name` which is useful if you want to give your dataset a nice name, if not provided we are going to use the junction of the `owner` the `repo` + the path of the README, in the exaple above it will be `fivethirtyeight/data/nba-raptor`
## Extra commands
You can also build the project for production with
```
npm run build
```
And run using the production build like so:
```
npm run start
```

View File

@@ -1,121 +0,0 @@
import { promises as fs } from 'fs';
import path from 'path';
import { getProject } from '../lib/octokit';
import getConfig from 'next/config';
export async function getStaticProps() {
const jsonDirectory = path.join(
process.cwd(),
'/datasets.json'
);
const repos = await fs.readFile(jsonDirectory, 'utf8');
const github_pat = getConfig().serverRuntimeConfig.github_pat;
const projects = await Promise.all(
(JSON.parse(repos)).map(async (repo) => {
const project = await getProject(repo, github_pat);
return { ...project, repo_config: repo };
})
);
return {
props: {
projects,
},
};
}
const formatter = new Intl.DateTimeFormat('en-US', {
year: 'numeric',
month: 'long',
day: 'numeric',
hour: 'numeric',
minute: 'numeric',
second: 'numeric',
timeZone: 'UTC',
});
export function Datasets({ projects }) {
return (
<div className="bg-white">
<div className="mx-auto max-w-7xl px-6 py-16 sm:py-24 lg:px-8">
<h2 className="text-2xl font-bold leading-10 tracking-tight">
My Datasets
</h2>
<p className="mt-6 max-w-2xl text-base leading-7 text-gray-600">
Here is a list of all my datasets for easy access and sharing
</p>
<div className="mt-20">
<div className="-mx-4 -my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
<div className="inline-block min-w-full py-2 align-middle sm:px-6 lg:px-8">
<table className="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Name
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Repo
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Description
</th>
<th
scope="col"
className="px-3 py-3.5 text-left text-sm font-semibold text-gray-900"
>
Last updated
</th>
<th
scope="col"
className="relative py-3.5 pl-3 pr-4 sm:pr-0"
></th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200">
{projects.map((project) => (
<tr key={project.id}>
<td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
{project.repo_config.name
? project.repo_config.name
: project.full_name + (project.base_path === '/' ? '' : '/' + project.base_path)}
</td>
<td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
<a href={project.html_url}>{project.full_name}</a>
</td>
<td className="px-3 py-4 text-sm text-gray-500">
{project.repo_config.description
? project.repo_config.description
: project.description}
</td>
<td className="whitespace-nowrap px-3 py-4 text-sm text-gray-500">
{formatter.format(new Date(project.last_updated))}
</td>
<td className="relative whitespace-nowrap py-4 pl-3 pr-4 text-right text-sm font-medium sm:pr-0">
<a
href={`/@${project.repo_config.owner}/${project.repo_config.repo}/${project.base_path === '/' ? '' : project.base_path}`}
>
More info
</a>
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
);
}
export default Datasets;

View File

@@ -1,403 +0,0 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
html {
-webkit-text-size-adjust: 100%;
font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont,
Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif,
Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;
line-height: 1.5;
tab-size: 4;
scroll-behavior: smooth;
}
body {
font-family: inherit;
line-height: inherit;
margin: 0;
}
h1,
h2,
p,
pre {
margin: 0;
}
*,
::before,
::after {
box-sizing: border-box;
border-width: 0;
border-style: solid;
border-color: currentColor;
}
h1,
h2 {
font-size: inherit;
font-weight: inherit;
}
a {
color: inherit;
text-decoration: inherit;
}
pre {
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
}
svg {
display: block;
vertical-align: middle;
shape-rendering: auto;
text-rendering: optimizeLegibility;
}
pre {
background-color: rgba(55, 65, 81, 1);
border-radius: 0.25rem;
color: rgba(229, 231, 235, 1);
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas,
Liberation Mono, Courier New, monospace;
overflow: scroll;
padding: 0.5rem 0.75rem;
}
.shadow {
box-shadow: 0 0 #0000, 0 0 #0000, 0 10px 15px -3px rgba(0, 0, 0, 0.1),
0 4px 6px -2px rgba(0, 0, 0, 0.05);
}
.rounded {
border-radius: 1.5rem;
}
.wrapper {
width: 100%;
}
.container {
margin-left: auto;
margin-right: auto;
max-width: 768px;
padding-bottom: 3rem;
padding-left: 1rem;
padding-right: 1rem;
color: rgba(55, 65, 81, 1);
width: 100%;
}
#welcome {
margin-top: 2.5rem;
}
#welcome h1 {
font-size: 3rem;
font-weight: 500;
letter-spacing: -0.025em;
line-height: 1;
}
#welcome span {
display: block;
font-size: 1.875rem;
font-weight: 300;
line-height: 2.25rem;
margin-bottom: 0.5rem;
}
#hero {
align-items: center;
background-color: hsla(214, 62%, 21%, 1);
border: none;
box-sizing: border-box;
color: rgba(55, 65, 81, 1);
display: grid;
grid-template-columns: 1fr;
margin-top: 3.5rem;
}
#hero .text-container {
color: rgba(255, 255, 255, 1);
padding: 3rem 2rem;
}
#hero .text-container h2 {
font-size: 1.5rem;
line-height: 2rem;
position: relative;
}
#hero .text-container h2 svg {
color: hsla(162, 47%, 50%, 1);
height: 2rem;
left: -0.25rem;
position: absolute;
top: 0;
width: 2rem;
}
#hero .text-container h2 span {
margin-left: 2.5rem;
}
#hero .text-container a {
background-color: rgba(255, 255, 255, 1);
border-radius: 0.75rem;
color: rgba(55, 65, 81, 1);
display: inline-block;
margin-top: 1.5rem;
padding: 1rem 2rem;
text-decoration: inherit;
}
#hero .logo-container {
display: none;
justify-content: center;
padding-left: 2rem;
padding-right: 2rem;
}
#hero .logo-container svg {
color: rgba(255, 255, 255, 1);
width: 66.666667%;
}
#middle-content {
align-items: flex-start;
display: grid;
gap: 4rem;
grid-template-columns: 1fr;
margin-top: 3.5rem;
}
#learning-materials {
padding: 2.5rem 2rem;
}
#learning-materials h2 {
font-weight: 500;
font-size: 1.25rem;
letter-spacing: -0.025em;
line-height: 1.75rem;
padding-left: 1rem;
padding-right: 1rem;
}
.list-item-link {
align-items: center;
border-radius: 0.75rem;
display: flex;
margin-top: 1rem;
padding: 1rem;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
width: 100%;
}
.list-item-link svg:first-child {
margin-right: 1rem;
height: 1.5rem;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
width: 1.5rem;
}
.list-item-link > span {
flex-grow: 1;
font-weight: 400;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
}
.list-item-link > span > span {
color: rgba(107, 114, 128, 1);
display: block;
flex-grow: 1;
font-size: 0.75rem;
font-weight: 300;
line-height: 1rem;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
}
.list-item-link svg:last-child {
height: 1rem;
transition-property: all;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
width: 1rem;
}
.list-item-link:hover {
color: rgba(255, 255, 255, 1);
background-color: hsla(162, 47%, 50%, 1);
}
.list-item-link:hover > span {
}
.list-item-link:hover > span > span {
color: rgba(243, 244, 246, 1);
}
.list-item-link:hover svg:last-child {
transform: translateX(0.25rem);
}
#other-links {
}
.button-pill {
padding: 1.5rem 2rem;
transition-duration: 300ms;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
align-items: center;
display: flex;
}
.button-pill svg {
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
flex-shrink: 0;
width: 3rem;
}
.button-pill > span {
letter-spacing: -0.025em;
font-weight: 400;
font-size: 1.125rem;
line-height: 1.75rem;
padding-left: 1rem;
padding-right: 1rem;
}
.button-pill span span {
display: block;
font-size: 0.875rem;
font-weight: 300;
line-height: 1.25rem;
}
.button-pill:hover svg,
.button-pill:hover {
color: rgba(255, 255, 255, 1) !important;
}
#nx-console:hover {
background-color: rgba(0, 122, 204, 1);
}
#nx-console svg {
color: rgba(0, 122, 204, 1);
}
#nx-repo:hover {
background-color: rgba(24, 23, 23, 1);
}
#nx-repo svg {
color: rgba(24, 23, 23, 1);
}
#nx-cloud {
margin-bottom: 2rem;
margin-top: 2rem;
padding: 2.5rem 2rem;
}
#nx-cloud > div {
align-items: center;
display: flex;
}
#nx-cloud > div svg {
border-radius: 0.375rem;
flex-shrink: 0;
width: 3rem;
}
#nx-cloud > div h2 {
font-size: 1.125rem;
font-weight: 400;
letter-spacing: -0.025em;
line-height: 1.75rem;
padding-left: 1rem;
padding-right: 1rem;
}
#nx-cloud > div h2 span {
display: block;
font-size: 0.875rem;
font-weight: 300;
line-height: 1.25rem;
}
#nx-cloud p {
font-size: 1rem;
line-height: 1.5rem;
margin-top: 1rem;
}
#nx-cloud pre {
margin-top: 1rem;
}
#nx-cloud a {
color: rgba(107, 114, 128, 1);
display: block;
font-size: 0.875rem;
line-height: 1.25rem;
margin-top: 1.5rem;
text-align: right;
}
#nx-cloud a:hover {
text-decoration: underline;
}
#commands {
padding: 2.5rem 2rem;
margin-top: 3.5rem;
}
#commands h2 {
font-size: 1.25rem;
font-weight: 400;
letter-spacing: -0.025em;
line-height: 1.75rem;
padding-left: 1rem;
padding-right: 1rem;
}
#commands p {
font-size: 1rem;
font-weight: 300;
line-height: 1.5rem;
margin-top: 1rem;
padding-left: 1rem;
padding-right: 1rem;
}
details {
align-items: center;
display: flex;
margin-top: 1rem;
padding-left: 1rem;
padding-right: 1rem;
width: 100%;
}
details pre > span {
color: rgba(181, 181, 181, 1);
display: block;
}
summary {
border-radius: 0.5rem;
display: flex;
font-weight: 400;
padding: 0.5rem;
cursor: pointer;
transition-property: background-color, border-color, color, fill, stroke,
opacity, box-shadow, transform, filter, backdrop-filter,
-webkit-backdrop-filter;
transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
transition-duration: 150ms;
}
summary:hover {
background-color: rgba(243, 244, 246, 1);
}
summary svg {
height: 1.5rem;
margin-right: 1rem;
width: 1.5rem;
}
#love {
color: rgba(107, 114, 128, 1);
font-size: 0.875rem;
line-height: 1.25rem;
margin-top: 3.5rem;
opacity: 0.6;
text-align: center;
}
#love svg {
color: rgba(252, 165, 165, 1);
width: 1.25rem;
height: 1.25rem;
display: inline;
margin-top: -0.25rem;
}
@media screen and (min-width: 768px) {
#hero {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
#hero .logo-container {
display: flex;
}
#middle-content {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
}

View File

@@ -140,7 +140,7 @@ function MobileNavigation(props) {
</div>
<nav className="mt-6">
<ul className="-my-2 divide-y divide-zinc-100 text-base text-zinc-800 dark:divide-zinc-100/5 dark:text-zinc-300">
<MobileNavItem href="https://github.com/leondz/hatespeechdata">
<MobileNavItem href="https://github.com/datopian/portaljs/tree/main/examples/turing">
View on Github <GithubIcon />
</MobileNavItem>
</ul>
@@ -179,7 +179,7 @@ function DesktopNavigation(props) {
return (
<nav {...props}>
<ul className="flex rounded-full bg-white/90 px-3 text-sm font-medium text-zinc-800 shadow-lg shadow-zinc-800/5 ring-1 ring-zinc-900/5 backdrop-blur dark:bg-zinc-800/90 dark:text-zinc-200 dark:ring-white/10">
<NavItem href="https://github.com/leondz/hatespeechdata">
<NavItem href="https://github.com/datopian/portaljs/tree/main/examples/turing">
View on Github <GithubIcon />
</NavItem>
</ul>

View File

@@ -36,7 +36,7 @@ In the following page type `content/datasets/<name-of-the-file>.md`. if you want
### Fill in content
Copy the contents of `templates/dataset.md` or `templates/keywords.md` respectively to the camp below, filling out the fields with the correct data format
Copy the contents of `templates/dataset.md` or `templates/keywords.md` respectively to the camp below, filling out the fields with the correct data format. Everything below the second `---` will automatically get rendered into the page, so you may add any standard markdown fields e.g tables, headings, lists...
![](https://i.imgur.com/x6JIjhz.png)

Some files were not shown because too many files have changed in this diff Show More