Datopian Tech
-
+ 
@@ -51,7 +51,6 @@ A DMS has a variety of features. This section provides an overview and links to
> [!tip] There are many ways to break down features and this is just one framing. We are thinking about others and if you have thoughts please get in touch.
-
- [Discovering and showcasing data (catalog and presenting)](/docs/dms/frontend)
- [Views on data](/docs/dms/views) including visualizing and previewing data as well [Data Explorers][explorer] and [Dashboards][]
- [Publishing data](/docs/dms/publish)
diff --git a/site/content/docs/index.md b/site/content/docs/index.md
index 84f79939..37d0a5df 100644
--- a/site/content/docs/index.md
+++ b/site/content/docs/index.md
@@ -1,53 +1,52 @@
-# 🌀 PortalJS: The JavaScript framework for data portals
+# Getting Started
-🌀 PortalJS is a framework for rapidly building rich data portal frontends using a modern frontend approach. PortalJS can be used to present a single dataset or build a full-scale data catalog/portal.
+Welcome to the PortalJS documentation!
-Built in JavaScript and React on top of the popular [Next.js](https://nextjs.com/) framework. PortalJS assumes a "decoupled" approach where the frontend is a separate service from the backend and interacts with backend(s) via an API. It can be used with any backend and has out of the box support for [CKAN](https://ckan.org/).
+If you have questions about anything related to PortalJS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portaljs/discussions) or on [our chat channel on Discord](https://discord.gg/EeyfGrGu4U).
-PortalJS offers a variety of examples (templates) to bootstrap your own data portal in just a few minutes. [Get started now!](#getting-started)
+## Setup
-## Features
+### Prerequisites
-- 🗺️ Unified sites: present data and content in one seamless site, pulling datasets from a DMS (e.g. CKAN) and content from a CMS (e.g. Wordpress) with a common internal API.
-- 👩‍💻 Developer friendly: built with familiar and modern frontend tech such as JavaScript, React and Next.js, so you can take advantage of everything these technologies have to offer: Server Side Rendering, Static Site Generation, huge number of examples and more!
-- 🔋 Batteries included: full set of portal components out of the box e.g. catalog search, dataset showcase, blog, etc.
-- 🎨 Easy to theme and customize: installable themes, use standard CSS and React+CSS tooling. Add new routes quickly.
-- 🧱 Extensible: quickly extend and develop/import your own React components
-- đź“ť Well documented: full set of documentation plus the documentation of Next.js and Apollo.
+- Node.js 14.18.0 or newer
+- MacOS, Windows (including WSL), and Linux are supported
-## What is a data portal?
+### Create a PortalJS app
-A Data Portal is a gateway to data. That gateway can be big or small, open or restricted. For example, data.gov is open to everyone, whilst an enterprise "intra" data portal is restricted to that enterprise (and perhaps even to certain people within it).
+To create a PortalJS app, open your terminal, cd into the directory you’d like to create the app in, and run the following command:
-A Data Portal's core purpose is to enable the rapid discovery and use of data. However, as a flexible, central point of truth on an organizations data assets, a Data Portal can become essential data infrastructure and be extended or integrated to provide many additional features:
+```bash
+npx create-next-app my-data-portal --example https://github.com/datopian/portaljs/tree/main/examples/basic-example
+```
-- Data storage and APIs
-- Data visualization and exploration
-- Data validation and schemas
-- Orchestration and integration of data
-- Data Lake coordination and organization
+> [!tip]
+> You may have noticed we used the command create-next-app. That’s because PortalJS is built on the awesome NextJS react javascript framework. That’s mean you can do everything you do with NextJS with PortalJS. Check out their docs to learn more.
-The rise of Data Portals reflect the rapid growth in the volume and variety of data that organizations hold and use. With so much data available internally (and externally) it is hard for users to discover and access the data they need. And with so many potential users and use-cases it is hard to anticipate what data will be needed, when.
+### Run the development server
-**[Read more about data portals](/docs/dms/data-portals)**
+You now have a new directory called `my-data-portal`. Let’s cd into it and then run the following command:
-## Getting started
+```bash
+npm run dev
+```
-To get started creating your own data portal with PortalJS, take a look at the examples below and analyze which one best suits your needs:
+This starts the NextJS (and hence PortalJS) "development server" on port 3000.
-### Examples
+Let's check it's working and what we have! Open http://localhost:3000 from your browser.
-* [Data catalog with data coming from CKAN](/docs/example-ckan)
-* [Simple data catalog](/docs/example-data-catalog)
+You should see a page like this when you access http://localhost:3000. This is the starter template page which shows the most simple data portal you could have: a simple README plus csv file.
-Then, simply follow the instructions on the given example page to use it as the template of your project.
+
-## Tutorials and guides
+### Editing the Page
-We are working on that! Tutorials coming out soon.
+Let’s try editing the starter page.
-## Getting Help
-
-If you have questions about anything related to PortalJS, you're always welcome to ask our community on [GitHub Discussions](https://github.com/datopian/portaljs/discussions) or on our [Discord server](https://discord.gg/EeyfGrGu4U).
+- Make sure the development server is still running.
+- Open content/index.md with your text editor.
+- Find the text that says “My Dataset” and change it to “My Awesome Dataset”.
+- Save the file.
+As soon as you save the file, the browser automatically updates the page with the new text:
+
diff --git a/site/content/docs/tutorial-create-data-portal-singe-dataset.md b/site/content/docs/tutorial-create-data-portal-singe-dataset.md
new file mode 100644
index 00000000..faca6e90
--- /dev/null
+++ b/site/content/docs/tutorial-create-data-portal-singe-dataset.md
@@ -0,0 +1,374 @@
+## Create a data portal with a single dataset
+
+Welcome to the PortalJS tutorials series. In this first tutorial, we are going to take a look at a simple data portal example built with PortalJS, understand its structure and learn how we can customize it, specially with data components.
+
+The resulting data portal is our _Hello World_ equivalent: a single dataset, and it looks like this:
+
+
+
+This tutorials series is sequential, so the next tutorials starts from where this one left, don't forget to save your progress, and, finally, let's get started!
+
+### Create a new PortalJS project
+
+First step is to create a new PortalJS project. To do that, please follow the instructions on the [Getting Started](#getting-started) section of the docs.
+
+Now, make sure you have the project running on your local environment (`npm run dev`) and access http://localhost:3000 on your browser. As you can see, the new project is not empty, it already contains some content which we will use as a base in this tutorial. Here's what the page looks like:
+
+
+
+
+### Basics
+
+As you can see, the page is very generic, and consists of a header, some text, a table and a line chart (built with Vega). Soon we are going to make it more interesting, but first, how did we end up with this?
+
+#### The content routing system
+
+Let's start by analyzing the main components of the folder strucutre of the project:
+
+```bash
+content/
+ my-dataset/
+ README.md
+public/
+ data_1.csv
+ data_2.csv
+```
+
+You see that `README.md` file inside the content folder? That's exactly what's being rendered on your browser. PortalJS uses a filesystem approach to content routing, this means that the folder structure inside the content folder determines the routes used to access the pages in the application, a page being a `.md` (Markdown) file, analogously to a HTML document. When the file is named "README.md", it means that it's an index file. Take a look at the following example:
+
+```bash
+content/
+ README.md # => Page rendered at /
+ folder-1/
+ README.md # => Page rendered at /folder-1
+ folder-2/
+ README.md # => Page rendered at /folder-2
+ folder-2-1/
+ README.md # => Page rendered at /folder-2/folder-2-1
+```
+
+INTERNAL NOTE: let's change that to index.md instead of README.md. Add examples of non-index pages? The MDX pipeline should be handling other .md files but it's not doing that rn. Maybe remove next paragraph
+
+Note that it's also possible to create non-index pages, but this is not going to be demonstrated on this tutorial for the sake of simplicity.
+
+#### The pages
+
+_Cool, a Markdown file becomes a page, but what is a Markdown file :thinking_face:?_
+
+Without getting into much detail, Markdown is a markup language which allows users to write structured and formatted text using a very simple syntax, with the beauty of not leaving the realm of plain text and keeping the document completely human-readable (opposite of, for instance, HTML, in which the document might get messy and very hard to read when it's not being rendered on a browser).
+
+It's not the intent of this tutorial to guide the user throught Markdown, but it's a requirement to understand it, so if you are not familiar with it we encourage you to take a look at [this guide](https://www.datopian.com/playbook/markdown) written by Datopian (Markdown is going to take over the world, seriously, you won't regret it!).
+
+Now that you are aware of Markdown documents and their application on PortalJS, let's hop to how this page you were seeing on your browser looks like behind the scenes. You probably noticed the cool chart and table on the page. Plain Markdown cannot do that, but the extended Markdown on PortalJS can.
+
+If you open `content/README.md` on your IDE or any text editor, you are going to come across the following content:
+
+```markdown
+# Data
+
+This is the README.md this project.
+
+## Table
+
+
+
+#### Simple line charts
+
+Let's use the `
+
+#### Complex charts
+
+Finally, PortalJS also supports the creation of visualizations with [Vega and VegaLite](https://vega.github.io/). This becomes specially interesting when it's desired to create more complex and custom visualizations. To demonstrate this, let's add a bar chart that compares Breaking Bad to Better Call Saul, a spin-off of the series, based on the data on Rotten Tomatoes. Here's the data in CSV format:
+
+```bash
+TV Show,Average Tomatometer,Average Audience Score
+Breaking Bad,0.96,0.97
+Better Call Saul,0.98,0.96
+```
+
+Add that to the file:
+
+```jsx=
+## Breaking Bad x Better Call Saul
+
+
+
+### Final results
+
+Here's the whole source code of the dataset page we built:
+
+```markdown
+# Breaking Bad Statistics
+
+**Data source:** https://openpsychometrics.org/tests/characters/stats/BB/
+
+Visualizations about the public perception of the Breaking Bad TV series and its characters.
+
+## Character Notability
+
+
+
+### Next steps
+
+Now that you already know how to create pages and render data components, we encourage you to play around with this project. You can try adding new visualizations, changing values, or creating a new page about something you find interesting.
+
+Finally, proceed to the next tutorial in the series.
\ No newline at end of file
diff --git a/site/package.json b/site/package.json
index 12b910ab..285d5c2b 100644
--- a/site/package.json
+++ b/site/package.json
@@ -4,7 +4,8 @@
"private": true,
"scripts": {
"dev": "next dev",
- "build": "npm run mddb && next build",
+ "build": "next build",
+ "prebuild": "npm run mddb && node ./scripts/fix-symlinks.mjs",
"start": "next start",
"mddb": "mddb content"
},
diff --git a/site/public/static/img/docs/dms/ckan-register.png b/site/public/static/img/docs/dms/ckan-register.png
new file mode 100644
index 00000000..8ba8f940
Binary files /dev/null and b/site/public/static/img/docs/dms/ckan-register.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/chart-builder.png b/site/public/static/img/docs/dms/data-explorer/chart-builder.png
new file mode 100644
index 00000000..fb0c045b
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/chart-builder.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/data-explorer.png b/site/public/static/img/docs/dms/data-explorer/data-explorer.png
new file mode 100644
index 00000000..5c0ae895
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/data-explorer.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/datapackage-views.png b/site/public/static/img/docs/dms/data-explorer/datapackage-views.png
new file mode 100644
index 00000000..7a7377c2
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/datapackage-views.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/date-picker.png b/site/public/static/img/docs/dms/data-explorer/date-picker.png
new file mode 100644
index 00000000..08d64042
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/date-picker.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/i18n-cookie.png b/site/public/static/img/docs/dms/data-explorer/i18n-cookie.png
new file mode 100644
index 00000000..c9fd302e
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/i18n-cookie.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/map-builder.png b/site/public/static/img/docs/dms/data-explorer/map-builder.png
new file mode 100644
index 00000000..56a06ef8
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/map-builder.png differ
diff --git a/site/public/static/img/docs/dms/data-explorer/query-builder.png b/site/public/static/img/docs/dms/data-explorer/query-builder.png
new file mode 100644
index 00000000..ebab0aa4
Binary files /dev/null and b/site/public/static/img/docs/dms/data-explorer/query-builder.png differ
diff --git a/site/public/static/img/docs/dms/dx/DX_architecture.jpg b/site/public/static/img/docs/dms/dx/DX_architecture.jpg
new file mode 100644
index 00000000..d3bbad0f
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/DX_architecture.jpg differ
diff --git a/site/public/static/img/docs/dms/dx/GitOps_Workflow.jpg b/site/public/static/img/docs/dms/dx/GitOps_Workflow.jpg
new file mode 100644
index 00000000..d54ba772
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/GitOps_Workflow.jpg differ
diff --git a/site/public/static/img/docs/dms/dx/logging-dashboard.png b/site/public/static/img/docs/dms/dx/logging-dashboard.png
new file mode 100644
index 00000000..a33195b5
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/logging-dashboard.png differ
diff --git a/site/public/static/img/docs/dms/dx/logging-query.png b/site/public/static/img/docs/dms/dx/logging-query.png
new file mode 100644
index 00000000..4afdd0d6
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/logging-query.png differ
diff --git a/site/public/static/img/docs/dms/dx/logging-service.png b/site/public/static/img/docs/dms/dx/logging-service.png
new file mode 100644
index 00000000..996038be
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/logging-service.png differ
diff --git a/site/public/static/img/docs/dms/dx/monitoring-console.png b/site/public/static/img/docs/dms/dx/monitoring-console.png
new file mode 100644
index 00000000..f1559960
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/monitoring-console.png differ
diff --git a/site/public/static/img/docs/dms/dx/monitoring-create-alert.png b/site/public/static/img/docs/dms/dx/monitoring-create-alert.png
new file mode 100644
index 00000000..9b4a28e4
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/monitoring-create-alert.png differ
diff --git a/site/public/static/img/docs/dms/dx/monitoring-create-uptime.png b/site/public/static/img/docs/dms/dx/monitoring-create-uptime.png
new file mode 100644
index 00000000..30b4a63c
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/monitoring-create-uptime.png differ
diff --git a/site/public/static/img/docs/dms/dx/monitoring-dashboard.png b/site/public/static/img/docs/dms/dx/monitoring-dashboard.png
new file mode 100644
index 00000000..b59ad9e6
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/monitoring-dashboard.png differ
diff --git a/site/public/static/img/docs/dms/dx/monitoring-log-based-metrics.png b/site/public/static/img/docs/dms/dx/monitoring-log-based-metrics.png
new file mode 100644
index 00000000..cabe8bf7
Binary files /dev/null and b/site/public/static/img/docs/dms/dx/monitoring-log-based-metrics.png differ
diff --git a/site/public/static/img/docs/dms/frontend-sequence-1.png b/site/public/static/img/docs/dms/frontend-sequence-1.png
new file mode 100644
index 00000000..124701d6
Binary files /dev/null and b/site/public/static/img/docs/dms/frontend-sequence-1.png differ
diff --git a/site/scripts/fix-symlinks.mjs b/site/scripts/fix-symlinks.mjs
new file mode 100644
index 00000000..8429b6d5
--- /dev/null
+++ b/site/scripts/fix-symlinks.mjs
@@ -0,0 +1,15 @@
+// Script executed before builds
+
+import { exec } from "child_process";
+import fs from "fs";
+
+// If Vercel environment is detected
+if (process.env.VERCEL_ENV) {
+ console.log(
+ "[scripts/fix-symlinks.mjs] Vercel environment detected. Fixing symlinks..."
+ );
+
+ // fs.unlinkSync('public/assets')
+ exec('cp -r ./content/assets ./public/')
+
+}