Version Packages (#1087)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

examples/fivethirtyeight/README.md

@@ -1,3 +1,9 @@
+# PortalJS Demo replicating the FiveThirtyEight data portal
+
+## 👉 https://fivethirtyeight.portaljs.org 👈
+
+Here's a blog post we wrote about it: https://www.datopian.com/blog/fivethirtyeight-replica
+
 This is a replica of the awesome data.fivethirtyeight.com using PortalJS.
 
 You might be asking why we did that, there are three main reasons:

packages/components/CHANGELOG.md

@@ -1,5 +1,79 @@
 # @portaljs/components
 
+## 0.6.0
+
+### Minor Changes
+
+- [`a044f56e`](https://github.com/datopian/portaljs/commit/a044f56e3cbe0519ddf9d24d78b0bb7eac917e1c) Thanks [@luccasmmg](https://github.com/luccasmmg)! - Added plotly components
+
+## 0.5.10
+
+### Patch Changes
+
+- [#1083](https://github.com/datopian/portaljs/pull/1083) [`86a2945e`](https://github.com/datopian/portaljs/commit/86a2945ee68dfcea0299984ca9cc9070d68fe1c2) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created integration with datastore api for table component
+
+## 0.5.9
+
+### Patch Changes
+
+- [#1081](https://github.com/datopian/portaljs/pull/1081) [`2bbf3134`](https://github.com/datopian/portaljs/commit/2bbf3134896df3ecc66560bdf95bece143614c7b) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed error to remove anchor from document
+
+## 0.5.8
+
+### Patch Changes
+
+- [#1079](https://github.com/datopian/portaljs/pull/1079) [`058d2367`](https://github.com/datopian/portaljs/commit/058d23678a024890f8a6d909ded9fc8fc11cf145) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Changed the download behaviour of the bucket viewer component and removed loading component while downloading
+
+## 0.5.7
+
+### Patch Changes
+
+- [#1077](https://github.com/datopian/portaljs/pull/1077) [`6d7acd27`](https://github.com/datopian/portaljs/commit/6d7acd27ed9299cbcc14eab906f2f0eb414656b8) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created property to present a component while is loading the download of the file and fixed download bug on pagination
+
+## 0.5.6
+
+### Patch Changes
+
+- [#1075](https://github.com/datopian/portaljs/pull/1075) [`26dcffc2`](https://github.com/datopian/portaljs/commit/26dcffc279057f80a579134e862085ba042c06c3) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed problem presenting the download component in the first load of the bucket viewer
+
+## 0.5.5
+
+### Patch Changes
+
+- [#1073](https://github.com/datopian/portaljs/pull/1073) [`cf24042a`](https://github.com/datopian/portaljs/commit/cf24042a910567e98eeb75ade42ce0149bdb62d1) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Fixed filter by startDate error
+
+## 0.5.4
+
+### Patch Changes
+
+- [#1071](https://github.com/datopian/portaljs/pull/1071) [`27c99add`](https://github.com/datopian/portaljs/commit/27c99adde8fa36ad2c2e03f227f93aa62454eefa) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Added pagination and filter properties for the BucketViewer component
+
+## 0.5.3
+
+### Patch Changes
+
+- [#1066](https://github.com/datopian/portaljs/pull/1066) [`dd03a493`](https://github.com/datopian/portaljs/commit/dd03a493beca5459d1ef447b2df505609fc64e95) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created Iframe component
+
+## 0.5.2
+
+### Patch Changes
+
+- [#1063](https://github.com/datopian/portaljs/pull/1063) [`b13e3ade`](https://github.com/datopian/portaljs/commit/b13e3ade3ccefe7dffe84f824bdedd3e512ce499) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created auto zoom configuration for the map component
+
+## 0.5.1
+
+### Patch Changes
+
+- [#1061](https://github.com/datopian/portaljs/pull/1061) [`4ddfc112`](https://github.com/datopian/portaljs/commit/4ddfc1126a3f0b8137ea47a08a36c56b7373b8f6) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Created the style property in the Map component
+
+## 0.5.0
+
+### Minor Changes
+
+- [#1055](https://github.com/datopian/portaljs/pull/1055) [`712f4a3b`](https://github.com/datopian/portaljs/commit/712f4a3b0f074e654879bb75059f51e06b422b32) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Creation of BucketViewer component to show the data of public buckets
+
+- [#1057](https://github.com/datopian/portaljs/pull/1057) [`61c750b7`](https://github.com/datopian/portaljs/commit/61c750b7e11fe52bf04d25f192440ee1bb307404) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Exporting BucketViewer to be accessed out of the folder
+
 ## 0.4.0
 
 ### Minor Changes

packages/components/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@portaljs/components",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "type": "module",
   "description": "https://portaljs.org",
   "keywords": [
@@ -29,6 +29,8 @@
     "@githubocto/flat-ui": "^0.14.1",
     "@heroicons/react": "^2.0.17",
     "@planet/maps": "^8.1.0",
+    "@react-pdf-viewer/core": "3.6.0",
+    "@react-pdf-viewer/default-layout": "3.6.0",
     "@tanstack/react-table": "^8.8.5",
     "ag-grid-react": "^30.0.4",
     "chroma-js": "^2.4.2",
@@ -37,19 +39,19 @@
     "next-mdx-remote": "^4.4.1",
     "ol": "^7.4.0",
     "papaparse": "^5.4.1",
+    "pdfjs-dist": "2.15.349",
+    "plotly.js": "^2.30.1",
     "postcss-url": "^10.1.3",
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-hook-form": "^7.43.9",
     "react-leaflet": "^4.2.1",
+    "react-plotly.js": "^2.6.0",
     "react-query": "^3.39.3",
     "react-vega": "^7.6.0",
     "vega": "5.25.0",
     "vega-lite": "5.1.0",
     "vitest": "^0.31.4",
-    "@react-pdf-viewer/core": "3.6.0",
-    "@react-pdf-viewer/default-layout": "3.6.0",
-    "pdfjs-dist": "2.15.349",
     "xlsx": "^0.18.5"
   },
   "devDependencies": {

packages/components/src/components/BucketViewer.tsx

@@ -0,0 +1,222 @@
+import { CSSProperties, ReactNode, useEffect, useState } from 'react';
+import LoadingSpinner from './LoadingSpinner';
+
+export interface BucketViewerFilterSearchedDataEvent {
+  startDate?: Date;
+  endDate?: Date;
+}
+
+export interface BucketViewerProps {
+  onLoadTotalNumberOfItems?: (total: number) => void;
+  domain: string;
+  downloadConfig?: {
+    hoverOfTheFileComponent?: ReactNode;
+  };
+  suffix?: string;
+  className?: string;
+  paginationConfig?: BucketViewerPaginationConfig;
+  filterState?: BucketViewerFilterSearchedDataEvent;
+  dataMapperFn: (rawData: Response) => Promise<BucketViewerData[]>;
+}
+
+export interface BucketViewerPaginationConfig {
+  containerClassName?: string;
+  containerStyles?: CSSProperties;
+  itemsPerPage: number;
+}
+
+export interface BucketViewerData {
+  fileName: string;
+  downloadFileUri: string;
+  dateProps?: {
+    date: Date;
+    dateFormatter?: (date: Date) => string;
+  };
+}
+
+export function BucketViewer({
+  domain,
+  suffix,
+  dataMapperFn,
+  className,
+  filterState,
+  paginationConfig,
+  downloadConfig,
+  onLoadTotalNumberOfItems,
+}: BucketViewerProps) {
+  suffix = suffix ?? '/';
+
+  const { hoverOfTheFileComponent } = downloadConfig ?? {};
+  const [isLoading, setIsLoading] = useState<boolean>(false);
+  const [showDownloadComponentOnLine, setShowDownloadComponentOnLine] =
+    useState(-1);
+  const [currentPage, setCurrentPage] = useState<number>(0);
+  const [lastPage, setLastPage] = useState<number>(0);
+  const [bucketFiles, setBucketFiles] = useState<BucketViewerData[]>([]);
+  const [paginatedData, setPaginatedData] = useState<BucketViewerData[]>([]);
+  const [filteredData, setFilteredData] = useState<BucketViewerData[]>([]);
+
+  useEffect(() => {
+    setIsLoading(true);
+    fetch(`${domain}${suffix}`)
+      .then((res) => dataMapperFn(res))
+      .then((data) => {
+        setBucketFiles(data);
+        setFilteredData(data);
+      })
+      .finally(() => setIsLoading(false));
+  }, [domain, suffix]);
+
+  useEffect(() => {
+    if (paginationConfig) {
+      const startIndex = paginationConfig
+        ? currentPage * paginationConfig.itemsPerPage
+        : 0;
+
+      const endIndex = paginationConfig
+        ? startIndex + paginationConfig.itemsPerPage
+        : 0;
+
+      setLastPage(
+        Math.ceil(filteredData.length / paginationConfig.itemsPerPage) - 1
+      );
+      setPaginatedData(filteredData.slice(startIndex, endIndex));
+    }
+  }, [currentPage, filteredData]);
+
+  useEffect(() => {
+    if (onLoadTotalNumberOfItems) onLoadTotalNumberOfItems(filteredData.length);
+  }, [filteredData]);
+
+  useEffect(() => {
+    if (!filterState) return;
+
+    if (filterState.startDate && filterState.endDate) {
+      setFilteredData(
+        bucketFiles.filter(({ dateProps }) =>
+          dateProps
+            ? dateProps.date.getTime() >= filterState.startDate.getTime() &&
+              dateProps.date.getTime() <= filterState.endDate.getTime()
+            : true
+        )
+      );
+    } else if (filterState.startDate) {
+      setFilteredData(
+        bucketFiles.filter(({ dateProps }) =>
+          dateProps
+            ? dateProps.date.getTime() >= filterState.startDate.getTime()
+            : true
+        )
+      );
+    } else if (filterState.endDate) {
+      setFilteredData(
+        bucketFiles.filter(({ dateProps }) =>
+          dateProps
+            ? dateProps.date.getTime() <= filterState.endDate.getTime()
+            : true
+        )
+      );
+    } else {
+      setFilteredData(bucketFiles);
+    }
+  }, [filterState]);
+
+  return isLoading ? (
+    <div className="w-full flex items-center justify-center h-[300px]">
+      <LoadingSpinner />
+    </div>
+  ) : bucketFiles ? (
+    <>
+      {...(paginationConfig && bucketFiles ? paginatedData : filteredData)?.map(
+        (data, i) => (
+          <ul
+            onClick={() => {
+              const a: HTMLAnchorElement = document.createElement('a');
+              a.href = data.downloadFileUri;
+              a.target = `_blank`;
+              a.download = data.fileName;
+              document.body.appendChild(a);
+              a.click();
+              document.body.removeChild(a);
+            }}
+            key={i}
+            onMouseEnter={() => setShowDownloadComponentOnLine(i)}
+            onMouseLeave={() => setShowDownloadComponentOnLine(undefined)}
+            className={`${
+              className ??
+              'mb-2 border-b-[2px] border-b-[red] hover:cursor-pointer'
+            }`}
+          >
+            {hoverOfTheFileComponent && showDownloadComponentOnLine === i ? (
+              hoverOfTheFileComponent
+            ) : (
+              <></>
+            )}
+            <div className="flex justify-between w-full items-center">
+              <div>
+                <li>{data.fileName}</li>
+                {data.dateProps && data.dateProps.dateFormatter ? (
+                  <li>{data.dateProps.dateFormatter(data.dateProps.date)}</li>
+                ) : (
+                  <></>
+                )}
+              </div>
+            </div>
+          </ul>
+        )
+      )}
+      {paginationConfig ? (
+        <ul
+          className={
+            paginationConfig.containerClassName
+              ? paginationConfig.containerClassName
+              : 'flex justify-end gap-x-[0.5rem] w-full'
+          }
+          style={paginationConfig.containerStyles ?? {}}
+        >
+          <li>
+            <button
+              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
+              disabled={currentPage === 0}
+              onClick={() => setCurrentPage(0)}
+            >
+              First
+            </button>
+          </li>
+          <li>
+            <button
+              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
+              onClick={() => setCurrentPage(currentPage - 1)}
+              disabled={currentPage === 0}
+            >
+              Previous
+            </button>
+          </li>
+          <label>{currentPage + 1}</label>
+
+          <li>
+            <button
+              onClick={() => setCurrentPage(currentPage + 1)}
+              disabled={currentPage >= lastPage}
+              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
+            >
+              Next
+            </button>
+          </li>
+
+          <li>
+            <button
+              onClick={() => setCurrentPage(lastPage)}
+              disabled={currentPage >= lastPage}
+              className="hover:cursor-pointer hover:disabled:cursor-not-allowed"
+            >
+              Last
+            </button>
+          </li>
+        </ul>
+      ) : (
+        <></>
+      )}
+    </>
+  ) : null;
+}

packages/components/src/components/Iframe.tsx

@@ -0,0 +1,14 @@
+import { CSSProperties } from "react";
+
+export interface IframeProps {
+  url: string;
+  style?: CSSProperties;
+}
+
+export function Iframe({
+  url, style
+}: IframeProps) {
+  return (
+    <iframe src={url} style={style ?? { width: `100%`, height: `100%` }}></iframe>
+  );
+}

packages/components/src/components/Map.tsx

@@ -1,4 +1,4 @@
-import { useEffect, useState } from 'react';
+import { CSSProperties, useEffect, useState } from 'react';
 import LoadingSpinner from './LoadingSpinner';
 import loadData from '../lib/loadData';
 import chroma from 'chroma-js';
@@ -30,6 +30,10 @@ export type MapProps = {
   title?: string;
   center?: { latitude: number | undefined; longitude: number | undefined };
   zoom?: number;
+  style?: CSSProperties;
+  autoZoomConfiguration?: {
+    layerName: string
+  }
 };
 
 export function Map({
@@ -44,6 +48,8 @@ export function Map({
   center = { latitude: 45, longitude: 45 },
   zoom = 2,
   title = '',
+  style = {},
+  autoZoomConfiguration = undefined,
 }: MapProps) {
   const [isLoading, setIsLoading] = useState<boolean>(false);
   const [layersData, setLayersData] = useState<any>([]);
@@ -96,6 +102,7 @@ export function Map({
       zoom={zoom}
       scrollWheelZoom={false}
       className="h-80 w-full"
+      style={style ?? {}}
       // @ts-ignore
       whenReady={(map: any) => {
         //  Enable zoom using scroll wheel
@@ -115,6 +122,24 @@ export function Map({
         };
 
         if (title) info.addTo(map.target);
+        if(!autoZoomConfiguration) return;
+
+        let layerToZoomBounds = L.latLngBounds(L.latLng(0, 0), L.latLng(0, 0));
+
+        layers.forEach((layer) => {
+          if(layer.name === autoZoomConfiguration.layerName) {
+            const data = layersData.find(
+              (layerData) => layerData.name === layer.name
+            )?.data;
+
+            if (data) {
+              layerToZoomBounds = L.geoJSON(data).getBounds();
+              return;
+            }
+          }
+        });
+
+        map.target.fitBounds(layerToZoomBounds);
       }}
     >
       <TileLayer

packages/components/src/components/Plotly.tsx

@@ -0,0 +1,9 @@
+import Plot, { PlotParams } from "react-plotly.js";
+
+export const Plotly: React.FC<PlotParams> = (props) => {
+  return (
+    <div>
+      <Plot {...props} />
+    </div>
+  );
+};

packages/components/src/components/PlotlyBarChart.tsx

@@ -0,0 +1,157 @@
+import { QueryClient, QueryClientProvider, useQuery } from "react-query";
+import { Plotly } from "./Plotly";
+import Papa, { ParseConfig } from "papaparse";
+import LoadingSpinner from "./LoadingSpinner";
+
+const queryClient = new QueryClient();
+
+async function getCsv(url: string, bytes: number) {
+  const response = await fetch(url, {
+    headers: {
+      Range: `bytes=0-${bytes}`,
+    },
+  });
+  const data = await response.text();
+  return data;
+}
+
+async function parseCsv(
+  file: string,
+  parsingConfig: ParseConfig,
+): Promise<any> {
+  return new Promise((resolve, reject) => {
+    Papa.parse(file, {
+      ...parsingConfig,
+      header: true,
+      dynamicTyping: true,
+      skipEmptyLines: true,
+      transform: (value: string): string => {
+        return value.trim();
+      },
+      complete: (results: any) => {
+        return resolve(results);
+      },
+      error: (error: any) => {
+        return reject(error);
+      },
+    });
+  });
+}
+
+export interface PlotlyBarChartProps {
+  url?: string;
+  data?: { [key: string]: number | string }[];
+  rawCsv?: string;
+  randomId?: number;
+  bytes?: number;
+  parsingConfig?: ParseConfig;
+  xAxis: string;
+  yAxis: string;
+  lineLabel?: string;
+  title?: string;
+}
+
+export const PlotlyBarChart: React.FC<PlotlyBarChartProps> = ({
+  url,
+  data,
+  rawCsv,
+  bytes = 5132288,
+  parsingConfig = {},
+  xAxis,
+  yAxis,
+  lineLabel,
+  title = "",
+}) => {
+  const randomId = Math.random();
+  return (
+    // Provide the client to your App
+    <QueryClientProvider client={queryClient}>
+      <PlotlyBarChartInner
+        url={url}
+        data={data}
+        rawCsv={rawCsv}
+        randomId={randomId}
+        bytes={bytes}
+        parsingConfig={parsingConfig}
+        xAxis={xAxis}
+        yAxis={yAxis}
+        lineLabel={lineLabel ?? yAxis}
+        title={title}
+      />
+    </QueryClientProvider>
+  );
+};
+
+const PlotlyBarChartInner: React.FC<PlotlyBarChartProps> = ({
+  url,
+  data,
+  rawCsv,
+  randomId,
+  bytes,
+  parsingConfig,
+  xAxis,
+  yAxis,
+  lineLabel,
+  title,
+}) => {
+  if (data) {
+    return (
+      <div className="w-full" style={{ height: "500px" }}>
+        <Plotly
+          layout={{
+            title,
+          }}
+          data={[
+            {
+              x: data.map((d) => d[xAxis]),
+              y: data.map((d) => d[yAxis]),
+              type: "bar",
+              name: lineLabel,
+            },
+          ]}
+        />
+      </div>
+    );
+  }
+  const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
+    ["dataCsv", url, randomId],
+    () => getCsv(url as string, bytes ?? 5132288),
+    { enabled: !!url },
+  );
+  const { data: parsedData, isLoading: isParsing } = useQuery(
+    ["dataPreview", csvString, randomId],
+    () =>
+      parseCsv(
+        rawCsv ? (rawCsv as string) : (csvString as string),
+        parsingConfig ?? {},
+      ),
+    { enabled: rawCsv ? true : !!csvString },
+  );
+  if (isParsing || isDownloadingCSV)
+    <div className="w-full flex justify-center items-center h-[500px]">
+      <LoadingSpinner />
+    </div>;
+  if (parsedData)
+    return (
+      <div className="w-full" style={{ height: "500px" }}>
+        <Plotly
+          layout={{
+            title,
+          }}
+          data={[
+            {
+              x: parsedData.data.map((d: any) => d[xAxis]),
+              y: parsedData.data.map((d: any) => d[yAxis]),
+              type: "bar",
+              name: lineLabel,
+            },
+          ]}
+        />
+      </div>
+    );
+  return (
+    <div className="w-full flex justify-center items-center h-[500px]">
+      <LoadingSpinner />
+    </div>
+  );
+};

packages/components/src/components/PlotlyLineChart.tsx

@@ -0,0 +1,157 @@
+import { QueryClient, QueryClientProvider, useQuery } from "react-query";
+import { Plotly } from "./Plotly";
+import Papa, { ParseConfig } from "papaparse";
+import LoadingSpinner from "./LoadingSpinner";
+
+const queryClient = new QueryClient();
+
+async function getCsv(url: string, bytes: number) {
+  const response = await fetch(url, {
+    headers: {
+      Range: `bytes=0-${bytes}`,
+    },
+  });
+  const data = await response.text();
+  return data;
+}
+
+async function parseCsv(
+  file: string,
+  parsingConfig: ParseConfig,
+): Promise<any> {
+  return new Promise((resolve, reject) => {
+    Papa.parse(file, {
+      ...parsingConfig,
+      header: true,
+      dynamicTyping: true,
+      skipEmptyLines: true,
+      transform: (value: string): string => {
+        return value.trim();
+      },
+      complete: (results: any) => {
+        return resolve(results);
+      },
+      error: (error: any) => {
+        return reject(error);
+      },
+    });
+  });
+}
+
+export interface PlotlyLineChartProps {
+  url?: string;
+  data?: { [key: string]: number | string }[];
+  rawCsv?: string;
+  randomId?: number;
+  bytes?: number;
+  parsingConfig?: ParseConfig;
+  xAxis: string;
+  yAxis: string;
+  lineLabel?: string;
+  title?: string;
+}
+
+export const PlotlyLineChart: React.FC<PlotlyLineChartProps> = ({
+  url,
+  data,
+  rawCsv,
+  bytes = 5132288,
+  parsingConfig = {},
+  xAxis,
+  yAxis,
+  lineLabel,
+  title = "",
+}) => {
+  const randomId = Math.random();
+  return (
+    // Provide the client to your App
+    <QueryClientProvider client={queryClient}>
+      <LineChartInner
+        url={url}
+        data={data}
+        rawCsv={rawCsv}
+        randomId={randomId}
+        bytes={bytes}
+        parsingConfig={parsingConfig}
+        xAxis={xAxis}
+        yAxis={yAxis}
+        lineLabel={lineLabel ?? yAxis}
+        title={title}
+      />
+    </QueryClientProvider>
+  );
+};
+
+const LineChartInner: React.FC<PlotlyLineChartProps> = ({
+  url,
+  data,
+  rawCsv,
+  randomId,
+  bytes,
+  parsingConfig,
+  xAxis,
+  yAxis,
+  lineLabel,
+  title,
+}) => {
+  if (data) {
+    return (
+      <div className="w-full" style={{ height: "500px" }}>
+        <Plotly
+          layout={{
+            title,
+          }}
+          data={[
+            {
+              x: data.map((d) => d[xAxis]),
+              y: data.map((d) => d[yAxis]),
+              mode: "lines",
+              name: lineLabel,
+            },
+          ]}
+        />
+      </div>
+    );
+  }
+  const { data: csvString, isLoading: isDownloadingCSV } = useQuery(
+    ["dataCsv", url, randomId],
+    () => getCsv(url as string, bytes ?? 5132288),
+    { enabled: !!url },
+  );
+  const { data: parsedData, isLoading: isParsing } = useQuery(
+    ["dataPreview", csvString, randomId],
+    () =>
+      parseCsv(
+        rawCsv ? (rawCsv as string) : (csvString as string),
+        parsingConfig ?? {},
+      ),
+    { enabled: rawCsv ? true : !!csvString },
+  );
+  if (isParsing || isDownloadingCSV)
+    <div className="w-full flex justify-center items-center h-[500px]">
+      <LoadingSpinner />
+    </div>;
+  if (parsedData)
+    return (
+      <div className="w-full" style={{ height: "500px" }}>
+        <Plotly
+          layout={{
+            title,
+          }}
+          data={[
+            {
+              x: parsedData.data.map((d: any) => d[xAxis]),
+              y: parsedData.data.map((d: any) => d[yAxis]),
+              mode: "lines",
+              name: lineLabel,
+            },
+          ]}
+        />
+      </div>
+    );
+  return (
+    <div className="w-full flex justify-center items-center h-[500px]">
+      <LoadingSpinner />
+    </div>
+  );
+};

packages/components/src/components/Table.tsx

@@ -6,6 +6,8 @@ import {
   getFilteredRowModel,
   getPaginationRowModel,
   getSortedRowModel,
+  PaginationState,
+  Table as ReactTable,
   useReactTable,
 } from '@tanstack/react-table';
 
@@ -25,12 +27,19 @@ import DebouncedInput from './DebouncedInput';
 import loadData from '../lib/loadData';
 import LoadingSpinner from './LoadingSpinner';
 
+export type TableData = { cols: {key: string, name: string}[]; data: any[]; total: number };
+
 export type TableProps = {
   data?: Array<{ [key: string]: number | string }>;
   cols?: Array<{ [key: string]: string }>;
   csv?: string;
   url?: string;
   fullWidth?: boolean;
+  datastoreConfig?: {
+    dataStoreURI: string;
+    rowsPerPage?: number;
+    dataMapperFn: (data) => Promise<TableData> | TableData;
+  };
 };
 
 export const Table = ({
@@ -39,8 +48,28 @@ export const Table = ({
   csv = '',
   url = '',
   fullWidth = false,
+  datastoreConfig,
 }: TableProps) => {
   const [isLoading, setIsLoading] = useState<boolean>(false);
+  const [pageMap, setPageMap] = useState(new Map<number, boolean>());
+  const {
+    dataMapperFn,
+    dataStoreURI,
+    rowsPerPage = 10,
+  } = datastoreConfig ?? {};
+
+  const [globalFilter, setGlobalFilter] = useState('');
+  const [isLoadingPage, setIsLoadingPage] = useState<boolean>(false);
+  const [totalOfRows, setTotalOfRows] = useState<number>(0);
+
+  const [{ pageIndex, pageSize }, setPagination] = useState<PaginationState>({
+    pageIndex: 0,
+    pageSize: rowsPerPage,
+  });
+
+  const [lastIndex, setLastIndex] = useState(pageSize);
+  const [startIndex, setStartIndex] = useState(0);
+  const [hasSorted, setHasSorted] = useState(false);
 
   if (csv) {
     const out = parseCsv(csv);
@@ -62,9 +91,43 @@ export const Table = ({
     );
   }, [data, cols]);
 
-  const [globalFilter, setGlobalFilter] = useState('');
+  let table: ReactTable<unknown>;
 
-  const table = useReactTable({
+  if (datastoreConfig) {
+    useEffect(() => {
+      setIsLoading(true);
+      fetch(`${dataStoreURI}&limit=${rowsPerPage}&offset=0`)
+        .then((res) => res.json())
+        .then(async (res) => {
+          const { data, cols, total } = await dataMapperFn(res);
+          setData(data);
+          setCols(cols);
+          setTotalOfRows(Math.ceil(total / rowsPerPage));
+          pageMap.set(0, true);
+        })
+        .finally(() => setIsLoading(false));
+    }, [dataStoreURI]);
+
+    table = useReactTable({
+      data,
+      pageCount: totalOfRows,
+      columns: tableCols,
+      getCoreRowModel: getCoreRowModel(),
+      state: {
+        pagination: { pageIndex, pageSize },
+      },
+      getFilteredRowModel: getFilteredRowModel(),
+      manualPagination: true,
+      onPaginationChange: setPagination,
+      getSortedRowModel: getSortedRowModel(),
+    });
+
+    useEffect(() => {
+      if (!hasSorted) return;
+      queryDataByText(globalFilter);
+    }, [table.getState().sorting]);
+  } else {
+    table = useReactTable({
       data,
       columns: tableCols,
       getCoreRowModel: getCoreRowModel(),
@@ -77,6 +140,7 @@ export const Table = ({
       getPaginationRowModel: getPaginationRowModel(),
       getSortedRowModel: getSortedRowModel(),
     });
+  }
 
   useEffect(() => {
     if (url) {
@@ -91,6 +155,70 @@ export const Table = ({
     }
   }, [url]);
 
+  const queryDataByText = (filter) => {
+    setIsLoadingPage(true);
+    const sortedParam = getSortParam();
+    fetch(
+      `${dataStoreURI}&limit=${rowsPerPage}&offset=0&q=${filter}${sortedParam}`
+    )
+      .then((res) => res.json())
+      .then(async (res) => {
+        const { data, total = 0 } = await dataMapperFn(res);
+        setTotalOfRows(Math.ceil(total / rowsPerPage));
+        setData(data);
+        const newMap = new Map();
+        newMap.set(0, true);
+        setPageMap(newMap);
+        table.setPageIndex(0);
+        setStartIndex(0);
+        setLastIndex(pageSize);
+      })
+      .finally(() => setIsLoadingPage(false));
+  };
+
+  const getSortParam = () => {
+    const sort = table.getState().sorting;
+    return sort.length == 0
+      ? ``
+      : '&sort=' +
+          sort
+            .map(
+              (x, i) =>
+                `${x.id}${
+                  i === sort.length - 1 ? (x.desc ? ` desc` : ` asc`) : `,`
+                }`
+            )
+            .reduce((x1, x2) => x1 + x2);
+  };
+
+  const queryPaginatedData = (newPageIndex) => {
+    let newStartIndex = newPageIndex * pageSize;
+    setStartIndex(newStartIndex);
+    setLastIndex(newStartIndex + pageSize);
+
+    if (!pageMap.get(newPageIndex)) pageMap.set(newPageIndex, true);
+    else return;
+
+    const sortedParam = getSortParam();
+
+    setIsLoadingPage(true);
+    fetch(
+      `${dataStoreURI}&limit=${rowsPerPage}&offset=${
+        newStartIndex + pageSize
+      }&q=${globalFilter}${sortedParam}`
+    )
+      .then((res) => res.json())
+      .then(async (res) => {
+        const { data: responseData } = await dataMapperFn(res);
+        responseData.forEach((e) => {
+          data[newStartIndex] = e;
+          newStartIndex++;
+        });
+        setData([...data]);
+      })
+      .finally(() => setIsLoadingPage(false));
+  };
+
   return isLoading ? (
     <div className="w-full h-full min-h-[500px] flex items-center justify-center">
       <LoadingSpinner />
@@ -99,7 +227,10 @@ export const Table = ({
     <div className={`${fullWidth ? 'w-[90vw] ml-[calc(50%-45vw)]' : 'w-full'}`}>
       <DebouncedInput
         value={globalFilter ?? ''}
-        onChange={(value: any) => setGlobalFilter(String(value))}
+        onChange={(value: any) => {
+          if (datastoreConfig) queryDataByText(String(value));
+          setGlobalFilter(String(value));
+        }}
         className="p-2 text-sm shadow border border-block"
         placeholder="Search all columns..."
       />
@@ -114,7 +245,10 @@ export const Table = ({
                       className: h.column.getCanSort()
                         ? 'cursor-pointer select-none'
                         : '',
-                      onClick: h.column.getToggleSortingHandler(),
+                      onClick: (v) => {
+                        setHasSorted(true);
+                        h.column.getToggleSortingHandler()(v);
+                      },
                     }}
                   >
                     {flexRender(h.column.columnDef.header, h.getContext())}
@@ -135,7 +269,19 @@ export const Table = ({
           ))}
         </thead>
         <tbody>
-          {table.getRowModel().rows.map((r) => (
+          {datastoreConfig && isLoadingPage ? (
+            <tr>
+              <td colSpan={cols.length} rowSpan={cols.length}>
+                <div className="w-full h-full flex items-center justify-center pt-6">
+                  <LoadingSpinner />
+                </div>
+              </td>
+            </tr>
+          ) : (
+            (datastoreConfig
+              ? table.getRowModel().rows.slice(startIndex, lastIndex)
+              : table.getRowModel().rows
+            ).map((r) => (
               <tr key={r.id} className="border-b border-b-slate-200">
                 {r.getVisibleCells().map((c) => (
                   <td key={c.id} className="py-2">
@@ -143,7 +289,8 @@ export const Table = ({
                   </td>
                 ))}
               </tr>
-          ))}
+            ))
+          )}
         </tbody>
       </table>
       <div className="flex gap-2 items-center justify-center mt-10">
@@ -151,7 +298,10 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanPreviousPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => table.setPageIndex(0)}
+          onClick={() => {
+            if (datastoreConfig) queryPaginatedData(0);
+            table.setPageIndex(0);
+          }}
           disabled={!table.getCanPreviousPage()}
         >
           <ChevronDoubleLeftIcon />
@@ -160,7 +310,12 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanPreviousPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => table.previousPage()}
+          onClick={() => {
+            if (datastoreConfig) {
+              queryPaginatedData(table.getState().pagination.pageIndex - 1);
+            }
+            table.previousPage();
+          }}
           disabled={!table.getCanPreviousPage()}
         >
           <ChevronLeftIcon />
@@ -176,7 +331,11 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanNextPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => table.nextPage()}
+          onClick={() => {
+            if (datastoreConfig)
+              queryPaginatedData(table.getState().pagination.pageIndex + 1);
+            table.nextPage();
+          }}
           disabled={!table.getCanNextPage()}
         >
           <ChevronRightIcon />
@@ -185,7 +344,11 @@ export const Table = ({
           className={`w-6 h-6 ${
             !table.getCanNextPage() ? 'opacity-25' : 'opacity-100'
           }`}
-          onClick={() => table.setPageIndex(table.getPageCount() - 1)}
+          onClick={() => {
+            const pageIndexToNavigate = table.getPageCount() - 1;
+            if (datastoreConfig) queryPaginatedData(pageIndexToNavigate);
+            table.setPageIndex(pageIndexToNavigate);
+          }}
           disabled={!table.getCanNextPage()}
         >
           <ChevronDoubleRightIcon />

packages/components/src/index.ts

@@ -8,3 +8,8 @@ export * from './components/OpenLayers/OpenLayers';
 export * from './components/Map';
 export * from './components/PdfViewer';
 export * from "./components/Excel";
+export * from "./components/BucketViewer";
+export * from "./components/Iframe";
+export * from "./components/Plotly";
+export * from "./components/PlotlyLineChart";
+export * from "./components/PlotlyBarChart";

packages/components/stories/BarChartPlotly.stories.ts

@@ -0,0 +1,74 @@
+import type { Meta, StoryObj } from '@storybook/react';
+
+import { PlotlyBarChart, PlotlyBarChartProps } from '../src/components/PlotlyBarChart';
+
+// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
+const meta: Meta = {
+  title: 'Components/PlotlyBarChart',
+  component: PlotlyBarChart,
+  tags: ['autodocs'],
+  argTypes: {
+    url: {
+      description:
+        'CSV Url to be parsed and used as data source',
+    },
+    data: {
+      description:
+        'Data to be displayed. as an array of key value pairs \n\n E.g.: [{ year: 1850, temperature: -0.41765878 }, { year: 1851, temperature: -0.2333498 }, ...]',
+    },
+    rawCsv: {
+      description:
+        'Raw csv data to be parsed and used as data source',
+    },
+    bytes: {
+      description:
+        'How many bytes to read from the url',
+    },
+    parsingConfig: {
+      description: 'If using url or rawCsv, this parsing config will be used to parse the data. Optional, check https://www.papaparse.com/ for more info',
+    },
+    title: {
+      description: 'Title to display on the chart. Optional.',
+    },
+    lineLabel: {
+      description: 'Label to display on the line, Optional, will use yAxis if not provided',
+    },
+    xAxis: {
+      description:
+        'Name of the X axis on the data. Required when the "data" parameter is an URL.',
+    },
+    yAxis: {
+      description:
+        'Name of the Y axis on the data. Required when the "data" parameter is an URL.',
+    },
+  },
+};
+
+export default meta;
+
+type Story = StoryObj<PlotlyBarChartProps>;
+
+export const FromDataPoints: Story = {
+  name: 'Line chart from array of data points',
+  args: {
+    data: [
+      {year: '1850', temperature: -0.41765878},
+      {year: '1851', temperature: -0.2333498},
+      {year: '1852', temperature: -0.22939907},
+      {year: '1853', temperature: -0.27035445},
+      {year: '1854', temperature: -0.29163003},
+    ],
+    xAxis: 'year',
+    yAxis: 'temperature',
+  },
+};
+
+export const FromURL: Story = {
+  name: 'Line chart from URL',
+  args: {
+    title: 'Apple Stock Prices',
+    url: 'https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv',
+    xAxis: 'Date',
+    yAxis: 'AAPL.Open',
+  },
+};

packages/components/stories/BucketViewer.stories.ts

@@ -0,0 +1,97 @@
+import { type Meta, type StoryObj } from '@storybook/react';
+
+import {
+  BucketViewer,
+  BucketViewerProps,
+} from '../src/components/BucketViewer';
+import LoadingSpinner from '../src/components/LoadingSpinner';
+
+// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
+const meta: Meta = {
+  title: 'Components/BucketViewer',
+  component: BucketViewer,
+  tags: ['autodocs'],
+  argTypes: {
+    domain: {
+      description: 'Bucket domain URI',
+    },
+    suffix: {
+      description: 'Suffix of bucket domain',
+    },
+    downloadConfig: {
+      description: `Bucket file download configuration`,
+    },
+    filterState: {
+      description: `State with values used to filter the bucket files`,
+    },
+    paginationConfig: {
+      description: `Configuration to show and stylise the pagination on the component`,
+    },
+  },
+};
+
+export default meta;
+
+type Story = StoryObj<BucketViewerProps>;
+
+// More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
+export const Normal: Story = {
+  name: 'Bucket viewer',
+  args: {
+    domain: 'https://ssen-smart-meter.datopian.workers.dev',
+    suffix: '/',
+    dataMapperFn: async (rawData: Response) => {
+      const result = await rawData.json();
+      return result.objects.map((e) => ({
+        downloadFileUri: e.downloadLink,
+        fileName: e.key.replace(/^(\w+\/)/g, ''),
+        dateProps: {
+          date: new Date(e.uploaded),
+          dateFormatter: (date) => date.toLocaleDateString(),
+        },
+      }));
+    },
+  },
+};
+
+export const WithPagination: Story = {
+  name: 'With pagination',
+  args: {
+    domain: 'https://ssen-smart-meter.datopian.workers.dev',
+    suffix: '/',
+    paginationConfig: {
+      itemsPerPage: 3,
+    },
+    dataMapperFn: async (rawData: Response) => {
+      const result = await rawData.json();
+      return result.objects.map((e) => ({
+        downloadFileUri: e.downloadLink,
+        fileName: e.key.replace(/^(\w+\/)/g, ''),
+        dateProps: {
+          date: new Date(e.uploaded),
+          dateFormatter: (date) => date.toLocaleDateString(),
+        },
+      }));
+    },
+  },
+};
+
+export const WithComponentOnHoverOfEachBucketFile: Story = {
+  name: 'With component on hover of each bucket file',
+  args: {
+    domain: 'https://ssen-smart-meter.datopian.workers.dev',
+    suffix: '/',
+    downloadConfig: { hoverOfTheFileComponent: `HOVER COMPONENT` },
+    dataMapperFn: async (rawData: Response) => {
+      const result = await rawData.json();
+      return result.objects.map((e) => ({
+        downloadFileUri: e.downloadLink,
+        fileName: e.key.replace(/^(\w+\/)/g, ''),
+        dateProps: {
+          date: new Date(e.uploaded),
+          dateFormatter: (date) => date.toLocaleDateString(),
+        },
+      }));
+    },
+  },
+};

packages/components/stories/FlatUiTable.stories.ts

@@ -64,6 +64,6 @@ export const FromRawCSV: Story = {
 export const FromURL: Story = {
   name: 'Table from URL',
   args: {
-    url: 'https://ckan-dev.sse.datopian.com/datastore/dump/601c9cf0-595e-46d8-88fc-d1ab2904e2db',
+    url: 'https://storage.openspending.org/alberta-budget/__os_imported__alberta_total.csv',
   },
 };

packages/components/stories/Iframe.stories.ts

@@ -0,0 +1,31 @@
+import { type Meta, type StoryObj } from '@storybook/react';
+
+import { Iframe, IframeProps } from '../src/components/Iframe';
+
+const meta: Meta = {
+  title: 'Components/Iframe',
+  component: Iframe,
+  tags: ['autodocs'],
+  argTypes: {
+    url: {
+      description:
+        'Page to display inside of the component',
+    },
+    style: {
+      description:
+        'Style of the component',
+    },
+  },
+};
+
+export default meta;
+
+type Story = StoryObj<IframeProps>;
+
+export const Normal: Story = {
+  name: 'Iframe',
+  args: {
+    url: 'https://app.powerbi.com/view?r=eyJrIjoiYzBmN2Q2MzYtYzE3MS00ODkxLWE5OWMtZTQ2MjBlMDljMDk4IiwidCI6Ijk1M2IwZjgzLTFjZTYtNDVjMy04MmM5LTFkODQ3ZTM3MjMzOSIsImMiOjh9',
+    style: {width: `100%`, height: `100%`}
+  },
+};

packages/components/stories/LineChartPlotly.stories.ts

@@ -0,0 +1,74 @@
+import type { Meta, StoryObj } from '@storybook/react';
+
+import { PlotlyLineChart, PlotlyLineChartProps } from '../src/components/PlotlyLineChart';
+
+// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
+const meta: Meta = {
+  title: 'Components/PlotlyLineChart',
+  component: PlotlyLineChart,
+  tags: ['autodocs'],
+  argTypes: {
+    url: {
+      description:
+        'CSV Url to be parsed and used as data source',
+    },
+    data: {
+      description:
+        'Data to be displayed. as an array of key value pairs \n\n E.g.: [{ year: 1850, temperature: -0.41765878 }, { year: 1851, temperature: -0.2333498 }, ...]',
+    },
+    rawCsv: {
+      description:
+        'Raw csv data to be parsed and used as data source',
+    },
+    bytes: {
+      description:
+        'How many bytes to read from the url',
+    },
+    parsingConfig: {
+      description: 'If using url or rawCsv, this parsing config will be used to parse the data. Optional, check https://www.papaparse.com/ for more info',
+    },
+    title: {
+      description: 'Title to display on the chart. Optional.',
+    },
+    lineLabel: {
+      description: 'Label to display on the line, Optional, will use yAxis if not provided',
+    },
+    xAxis: {
+      description:
+        'Name of the X axis on the data. Required when the "data" parameter is an URL.',
+    },
+    yAxis: {
+      description:
+        'Name of the Y axis on the data. Required when the "data" parameter is an URL.',
+    },
+  },
+};
+
+export default meta;
+
+type Story = StoryObj<PlotlyLineChartProps>;
+
+export const FromDataPoints: Story = {
+  name: 'Line chart from array of data points',
+  args: {
+    data: [
+      {year: '1850', temperature: -0.41765878},
+      {year: '1851', temperature: -0.2333498},
+      {year: '1852', temperature: -0.22939907},
+      {year: '1853', temperature: -0.27035445},
+      {year: '1854', temperature: -0.29163003},
+    ],
+    xAxis: 'year',
+    yAxis: 'temperature',
+  },
+};
+
+export const FromURL: Story = {
+  name: 'Line chart from URL',
+  args: {
+    title: 'Oil Price x Year',
+    url: 'https://raw.githubusercontent.com/datasets/oil-prices/main/data/wti-year.csv',
+    xAxis: 'Date',
+    yAxis: 'Price',
+  },
+};

packages/components/stories/Map.stories.ts

@@ -21,6 +21,12 @@ const meta: Meta = {
     zoom: {
       description: 'Zoom level',
     },
+    style: {
+      description: "Styles for the container"
+    },
+    autoZoomConfiguration: {
+      description: "Configuration to auto zoom in the specified layer data"
+    }
   },
 };
 
@@ -88,4 +94,32 @@ export const GeoJSONMultipleLayers: Story = {
     center: { latitude: 45, longitude: 0 },
     zoom: 2,
   },
+}
+
+export const GeoJSONMultipleLayersWithAutoZoomInSpecifiedLayer: Story = {
+  name: 'GeoJSON polygons and points map with auto zoom in the points layer',
+  args: {
+    layers: [
+      {
+        data: 'https://opendata.arcgis.com/datasets/9c58741995174fbcb017cf46c8a42f4b_25.geojson',
+        name: 'Points',
+        tooltip: true,
+      },
+      {
+        data: 'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_geography_marine_polys.geojson',
+        name: 'Polygons',
+        tooltip: true,
+        colorScale: {
+          starting: '#ff0000',
+          ending: '#00ff00',
+        },
+      },
+    ],
+    title: 'Polygons and points',
+    center: { latitude: 45, longitude: 0 },
+    zoom: 2,
+    autoZoomConfiguration: {
+      layerName: 'Points'
+    }
+  },
 };

packages/components/stories/Plotly.stories.ts

@@ -0,0 +1,39 @@
+import type { Meta, StoryObj } from '@storybook/react';
+
+import { Plotly } from '../src/components/Plotly';
+
+// More on how to set up stories at: https://storybook.js.org/docs/react/writing-stories/introduction
+const meta: Meta = {
+  title: 'Components/Plotly',
+  component: Plotly,
+  tags: ['autodocs'],
+};
+
+export default meta;
+
+type Story = StoryObj<any>;
+
+// More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
+export const Primary: Story = {
+  name: 'Chart built with Plotly',
+  args: {
+    data: [
+      {
+        x: [1, 2, 3],
+        y: [2, 6, 3],
+        type: 'scatter',
+        mode: 'lines+markers',
+        marker: { color: 'red' },
+      },
+    ],
+    layout: {
+      title: 'Chart built with Plotly',
+      xaxis: {
+        title: 'x Axis',
+      },
+      yaxis: {
+        title: 'y Axis',
+      },
+    },
+  },
+};

packages/components/stories/Table.stories.ts

@@ -9,17 +9,22 @@ const meta: Meta = {
   tags: ['autodocs'],
   argTypes: {
     data: {
-      description: "Data to be displayed in the table, must also set \"cols\" to work."
+      description:
+        'Data to be displayed in the table, must also set "cols" to work.',
     },
     cols: {
-      description: "Columns to be displayed in the table, must also set \"data\" to work."
+      description:
+        'Columns to be displayed in the table, must also set "data" to work.',
     },
     csv: {
-      description: "CSV data as string.",
+      description: 'CSV data as string.',
     },
     url: {
-      description: "Fetch the data from a CSV file remotely."
-    }
+      description: 'Fetch the data from a CSV file remotely.',
+    },
+    datastoreConfig: {
+      description: `Configuration to use CKAN's datastore API extension integrated with the component`,
+    },
   },
 };
 
@@ -29,7 +34,7 @@ type Story = StoryObj<TableProps>;
 
 // More on writing stories with args: https://storybook.js.org/docs/react/writing-stories/args
 export const FromColumnsAndData: Story = {
-  name: "Table from columns and data",
+  name: 'Table from columns and data',
   args: {
     data: [
       { id: 1, lastName: 'Snow', firstName: 'Jon', age: 35 },
@@ -49,21 +54,40 @@ export const FromColumnsAndData: Story = {
   },
 };
 
+export const WithDataStoreIntegration: Story = {
+  name: 'Table with datastore integration',
+  args: {
+    datastoreConfig: {
+      dataStoreURI: `https://www.civicdata.com/api/action/datastore_search?resource_id=46ec0807-31ff-497f-bfa0-f31c796cdee8`,
+      dataMapperFn: ({
+        result,
+      }: {
+        result: { fields: { id }[]; records: []; total: number };
+      }) => {
+        return {
+          data: result.records,
+          cols: result.fields.map((x) => ({ key: x.id, name: x.id })),
+          total: result.total,
+        };
+      },
+    },
+  },
+};
+
 export const FromRawCSV: Story = {
-  name: "Table from raw CSV",
+  name: 'Table from raw CSV',
   args: {
     csv: `
     Year,Temp Anomaly
     1850,-0.418
     2020,0.923
-    `
-  }
+    `,
+  },
 };
 
 export const FromURL: Story = {
-  name: "Table from URL",
+  name: 'Table from URL',
   args: {
-    url: "https://raw.githubusercontent.com/datasets/finance-vix/main/data/vix-daily.csv"
-  }
+    url: 'https://raw.githubusercontent.com/datasets/finance-vix/main/data/vix-daily.csv',
+  },
 };
-

packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@portaljs/core",
-  "version": "1.0.8",
+  "version": "1.0.9",
   "description": "Core Portal.JS components, configs and utils.",
   "repository": {
     "type": "git",

packages/core/src/ui/analytics/GoogleAnalytics.tsx

@@ -0,0 +1,36 @@
+import Script from 'next/script.js'
+
+export interface GoogleAnalyticsProps {
+  googleAnalyticsId: string
+}
+
+export const GA = ({ googleAnalyticsId }: GoogleAnalyticsProps) => {
+  return (
+    <>
+      <Script
+        strategy="afterInteractive"
+        src={`https://www.googletagmanager.com/gtag/js?id=${googleAnalyticsId}`}
+      />
+
+      <Script strategy="afterInteractive" id="ga-script">
+        {`
+            window.dataLayer = window.dataLayer || [];
+            function gtag(){dataLayer.push(arguments);}
+            gtag('js', new Date());
+            gtag('config', '${googleAnalyticsId}');
+        `}
+      </Script>
+    </>
+  )
+}
+
+// https://developers.google.com/analytics/devguides/collection/gtagjs/events
+export const logEvent = (action, category, label, value) => {
+  // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+  // @ts-ignore
+  window.gtag?.('event', action, {
+    event_category: category,
+    event_label: label,
+    value: value,
+  })
+}

packages/core/src/ui/analytics/Plausible.tsx

@@ -0,0 +1,41 @@
+import Script from 'next/script.js'
+
+export interface PlausibleProps {
+  plausibleDataDomain: string
+  dataApi?: string
+  src?: string
+}
+
+/**
+ * Plausible analytics component.
+ * To proxy the requests through your own domain, you can use the dataApi and src attribute.
+ * See [Plausible docs](https://plausible.io/docs/proxy/guides/nextjs#step-2-adjust-your-deployed-script)
+ * for more information.
+ *
+ */
+export const Plausible = ({
+  plausibleDataDomain,
+  dataApi = undefined,
+  src = 'https://plausible.io/js/plausible.js',
+}: PlausibleProps) => {
+  return (
+    <>
+      <Script
+        strategy="lazyOnload"
+        data-domain={plausibleDataDomain}
+        data-api={dataApi}
+        src={src}
+      />
+      <Script strategy="lazyOnload" id="plausible-script">
+        {`
+            window.plausible = window.plausible || function() { (window.plausible.q = window.plausible.q || []).push(arguments) }
+        `}
+      </Script>
+    </>
+  )
+}
+
+// https://plausible.io/docs/custom-event-goals
+export const logEvent = (eventName, ...rest) => {
+  return window.plausible?.(eventName, ...rest)
+}

packages/core/src/ui/analytics/Posthog.tsx

@@ -0,0 +1,25 @@
+import Script from 'next/script.js'
+
+export interface PosthogProps {
+  posthogProjectApiKey: string
+  apiHost?: string
+}
+
+/**
+ * Posthog analytics component.
+ * See [Posthog docs](https://posthog.com/docs/libraries/js#option-1-add-javascript-snippet-to-your-html-badgerecommendedbadge) for more information.
+ *
+ */
+export const Posthog = ({
+  posthogProjectApiKey,
+  apiHost = 'https://app.posthog.com',
+}: PosthogProps) => {
+  return (
+    <Script strategy="lazyOnload" id="posthog-script">
+      {`
+          !function(t,e){var o,n,p,r;e.__SV||(window.posthog=e,e._i=[],e.init=function(i,s,a){function g(t,e){var o=e.split(".");2==o.length&&(t=t[o[0]],e=o[1]),t[e]=function(){t.push([e].concat(Array.prototype.slice.call(arguments,0)))}}(p=t.createElement("script")).type="text/javascript",p.async=!0,p.src=s.api_host+"/static/array.js",(r=t.getElementsByTagName("script")[0]).parentNode.insertBefore(p,r);var u=e;for(void 0!==a?u=e[a]=[]:a="posthog",u.people=u.people||[],u.toString=function(t){var e="posthog";return"posthog"!==a&&(e+="."+a),t||(e+=" (stub)"),e},u.people.toString=function(){return u.toString(1)+".people (stub)"},o="capture identify alias people.set people.set_once set_config register register_once unregister opt_out_capturing has_opted_out_capturing opt_in_capturing reset isFeatureEnabled onFeatureFlags".split(" "),n=0;n<o.length;n++)g(u,o[n]);e._i.push([i,s,a])},e.__SV=1)}(document,window.posthog||[]);
+          posthog.init('${posthogProjectApiKey}',{api_host:'${apiHost}'})
+      `}
+    </Script>
+  )
+}

packages/core/src/ui/analytics/SimpleAnalytics.tsx

@@ -0,0 +1,29 @@
+import Script from 'next/script.js'
+
+export interface SimpleAnalyticsProps {
+  src?: string
+}
+
+export const SimpleAnalytics = ({
+  src = 'https://scripts.simpleanalyticscdn.com/latest.js',
+}: SimpleAnalyticsProps) => {
+  return (
+    <>
+      <Script strategy="lazyOnload" id="sa-script">
+        {`
+            window.sa_event=window.sa_event||function(){var a=[].slice.call(arguments);window.sa_event.q?window.sa_event.q.push(a):window.sa_event.q=[a]};
+        `}
+      </Script>
+      <Script strategy="lazyOnload" src={src} />
+    </>
+  )
+}
+
+// https://docs.simpleanalytics.com/events
+export const logEvent = (eventName, callback) => {
+  if (callback) {
+    return window.sa_event?.(eventName, callback)
+  } else {
+    return window.sa_event?.(eventName)
+  }
+}

packages/core/src/ui/analytics/Umami.tsx

@@ -0,0 +1,20 @@
+import Script from 'next/script.js'
+
+export interface UmamiProps {
+  umamiWebsiteId: string
+  src?: string
+}
+
+export const Umami = ({
+  umamiWebsiteId,
+  src = 'https://analytics.umami.is/script.js',
+}: UmamiProps) => {
+  return (
+    <Script
+      async
+      defer
+      data-website-id={umamiWebsiteId}
+      src={src} // Replace with your umami instance
+    />
+  )
+}

packages/core/src/ui/analytics/index.tsx

@@ -0,0 +1,82 @@
+/* eslint-disable @typescript-eslint/no-explicit-any */
+import { GA, GoogleAnalyticsProps } from "./GoogleAnalytics";
+import { Plausible, PlausibleProps } from "./Plausible";
+import { SimpleAnalytics, SimpleAnalyticsProps } from "./SimpleAnalytics";
+import { Umami, UmamiProps } from "./Umami";
+import { Posthog, PosthogProps } from "./Posthog";
+
+declare global {
+    interface Window {
+        // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+        // @ts-ignore
+        gtag?: (...args: any[]) => void;
+        plausible?: (...args: any[]) => void;
+        sa_event?: (...args: any[]) => void;
+    }
+}
+
+export interface AnalyticsConfig {
+    googleAnalytics?: GoogleAnalyticsProps;
+    plausibleAnalytics?: PlausibleProps;
+    umamiAnalytics?: UmamiProps;
+    posthogAnalytics?: PosthogProps;
+    simpleAnalytics?: SimpleAnalyticsProps;
+}
+
+/**
+ * @example
+ * const analytics: AnalyticsConfig = {
+ *  plausibleDataDomain: '', // e.g. tailwind-nextjs-starter-blog.vercel.app
+ *  simpleAnalytics: false, // true or false
+ *  umamiWebsiteId: '', // e.g. 123e4567-e89b-12d3-a456-426614174000
+ *  posthogProjectApiKey: '', // e.g. AhnJK8392ndPOav87as450xd
+ *  googleAnalyticsId: '', // e.g. UA-000000-2 or G-XXXXXXX
+ * }
+ */
+export interface AnalyticsProps {
+    analyticsConfig: AnalyticsConfig;
+}
+
+const isProduction = true || process.env["NODE_ENV"] === "production";
+
+/**
+ * Supports Plausible, Simple Analytics, Umami, Posthog or Google Analytics.
+ * All components default to the hosted service, but can be configured to use a self-hosted
+ * or proxied version of the script by providing the `src` / `apiHost` props.
+ *
+ * Note: If you want to use an analytics provider you have to add it to the
+ * content security policy in the `next.config.js` file.
+ * @param {AnalyticsProps} { analytics }
+ * @return {*}
+ */
+export const Analytics = ({ analyticsConfig }: AnalyticsProps) => {
+    return (
+        <>
+            {isProduction && analyticsConfig.plausibleAnalytics && (
+                <Plausible {...analyticsConfig.plausibleAnalytics} />
+            )}
+            {isProduction && analyticsConfig.simpleAnalytics && (
+                <SimpleAnalytics {...analyticsConfig.simpleAnalytics} />
+            )}
+            {isProduction && analyticsConfig.posthogAnalytics && (
+                <Posthog {...analyticsConfig.posthogAnalytics} />
+            )}
+            {isProduction && analyticsConfig.umamiAnalytics && (
+                <Umami {...analyticsConfig.umamiAnalytics} />
+            )}
+            {isProduction && analyticsConfig.googleAnalytics && (
+                <GA {...analyticsConfig.googleAnalytics} />
+            )}
+        </>
+    );
+};
+
+export { GA, Plausible, SimpleAnalytics, Umami, Posthog };
+
+export type {
+    GoogleAnalyticsProps,
+    PlausibleProps,
+    UmamiProps,
+    PosthogProps,
+    SimpleAnalyticsProps,
+};

packages/core/src/ui/index.ts

@@ -21,3 +21,4 @@ export { SiteToc, NavItem, NavGroup } from "./SiteToc";
 export { Comments, CommentsConfig } from "./Comments";
 export { AuthorConfig } from "./types";
 export { Hero } from "./Hero";
+export { Analytics, AnalyticsConfig } from "./analytics";

packages/core/src/utils/gtag.ts

@@ -7,6 +7,8 @@ export const pageview = ({
   analyticsID: string;
 }) => {
   if (typeof window.gtag !== undefined) {
+    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+    // @ts-ignore
     window.gtag("config", analyticsID, {
       page_path: url,
     });
@@ -16,6 +18,8 @@ export const pageview = ({
 // https://developers.google.com/analytics/devguides/collection/gtagjs/events
 export const event = ({ action, category, label, value }) => {
   if (typeof window.gtag !== undefined) {
+    // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+    // @ts-ignore
     window.gtag("event", action, {
       event_category: category,
       event_label: label,

packages/remark-wiki-link/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @portaljs/remark-wiki-link
 
+## 1.1.2
+
+### Patch Changes
+
+- [#1040](https://github.com/datopian/portaljs/pull/1040) [`85bb6cb9`](https://github.com/datopian/portaljs/commit/85bb6cb98c53bedc2add3d014927570b5dd1bbdf) Thanks [@Gutts-n](https://github.com/Gutts-n)! - Changed regex to permit any symbols other than #
+
 ## 1.1.1
 
 ### Patch Changes

packages/remark-wiki-link/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@portaljs/remark-wiki-link",
-  "version": "1.1.1",
+  "version": "1.1.2",
   "description": "Parse and render wiki-style links in markdown especially Obsidian style links.",
   "repository": {
     "type": "git",

packages/remark-wiki-link/src/lib/fromMarkdown.ts

@@ -79,7 +79,7 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
       data: { isEmbed, target, alias },
     } = wikiLink;
     // eslint-disable-next-line no-useless-escape
-    const wikiLinkWithHeadingPattern = /([\p{Letter}\d\s\/\.-_]*)(#.*)?/u;
+    const wikiLinkWithHeadingPattern = /^(.*?)(#.*)?$/u;
     const [, path, heading = ""] = target.match(wikiLinkWithHeadingPattern);
 
     const possibleWikiLinkPermalinks = wikiLinkResolver(path);
@@ -116,7 +116,7 @@ function fromMarkdown(opts: FromMarkdownOptions = {}) {
 
     // remove leading # if the target is a heading on the same page
     const displayName = alias || target.replace(/^#/, "");
-    const headingId = heading.replace(/\s+/, "-").toLowerCase();
+    const headingId = heading.replace(/\s+/g, "-").toLowerCase();
     let classNames = wikiLinkClassName;
     if (!matchingPermalink) {
       classNames += " " + newClassName;

packages/remark-wiki-link/src/lib/html.ts

@@ -64,7 +64,7 @@ function html(opts: HtmlOptions = {}) {
     const { target, alias } = wikiLink;
     const isEmbed = token.isType === "embed";
     // eslint-disable-next-line no-useless-escape
-    const wikiLinkWithHeadingPattern = /([\w\s\/\.-]*)(#.*)?/;
+    const wikiLinkWithHeadingPattern = /^(.*?)(#.*)?$/u;
     const [, path, heading = ""] = target.match(wikiLinkWithHeadingPattern);
 
     const possibleWikiLinkPermalinks = wikiLinkResolver(path);
@@ -99,7 +99,7 @@ function html(opts: HtmlOptions = {}) {
     // remove leading # if the target is a heading on the same page
     const displayName = alias || target.replace(/^#/, "");
     // replace spaces with dashes and lowercase headings
-    const headingId = heading.replace(/\s+/, "-").toLowerCase();
+    const headingId = heading.replace(/\s+/g, "-").toLowerCase();
     let classNames = wikiLinkClassName;
     if (!matchingPermalink) {
       classNames += " " + newClassName;

packages/remark-wiki-link/test/getPermalinks.spec.ts

@@ -6,7 +6,7 @@ import { getPermalinks } from "../src/utils";
 // const markdownFolder = path.join(__dirname, "/fixtures/content");
 const markdownFolder = path.join(
   ".",
-  "/packages/remark-wiki-link/test/fixtures/content"
+  "test/fixtures/content"
 );
 
 describe("getPermalinks", () => {

packages/remark-wiki-link/test/micromarkExtensionWikiLink.spec.ts

@@ -321,4 +321,14 @@ describe("micromark-extension-wiki-link", () => {
       );
     });
   });
+
+  describe("Links with special characters", () => {
+    test("parses a link with special characters and symbols", () => {
+      const serialized = micromark("[[li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#LI NK-W(i)th-àcèô íã_a(n)d_uNdErlinE!:ª%@'*º$ °~./\\]]", "ascii", {
+        extensions: [syntax()],
+        htmlExtensions: [html() as any],
+      });
+      expect(serialized).toBe(`<p><a href="li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô-íã_a(n)d_underline!:ª%@'*º$-°~./\\" class="internal new">li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#LI NK-W(i)th-àcèô íã_a(n)d_uNdErlinE!:ª%@'*º$ °~./\\</a></p>`);
+    });
+  })
 });

packages/remark-wiki-link/test/remarkWikiLink.spec.ts

@@ -361,6 +361,32 @@ describe("remark-wiki-link", () => {
     });
   });
 
+  describe("Links with special characters", () => {
+    test("parses a link with special characters and symbols", () => {
+      const processor = unified().use(markdown).use(wikiLinkPlugin);
+
+      let ast = processor.parse("[[li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô íã_a(n)D_UNDERLINE!:ª%@'*º$ °~./\\]]");
+      ast = processor.runSync(ast);
+      expect(select("wikiLink", ast)).not.toEqual(null);
+
+      visit(ast, "wikiLink", (node: Node) => {
+        expect(node.data?.exists).toEqual(false);
+        expect(node.data?.permalink).toEqual("li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\");
+        expect(node.data?.alias).toEqual(null);
+        expect(node.data?.hName).toEqual("a");
+        expect((node.data?.hProperties as any).className).toEqual(
+          "internal new"
+        );
+        expect((node.data?.hProperties as any).href).toEqual(
+          "li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô-íã_a(n)d_underline!:ª%@'*º$-°~./\\"
+        );
+        expect((node.data?.hChildren as any)[0].value).toEqual(
+          "li nk-w(i)th-àcèô íã_a(n)d_underline!:ª%@'*º$ °~./\\#li-nk-w(i)th-àcèô íã_a(n)D_UNDERLINE!:ª%@'*º$ °~./\\"
+        );
+      })
+    });
+  })
+
   describe("invalid wiki links", () => {
     test("doesn't parse a wiki link with two missing closing brackets", () => {
       const processor = unified().use(markdown).use(wikiLinkPlugin);
@@ -560,3 +586,4 @@ describe("remark-wiki-link", () => {
     });
   });
 });
+

site/content/blog/enhancing-geospatial-data-visualization-with-portaljs.md

@@ -1,15 +1,17 @@
 ---
-title: 'Enhancing Geospatial Data Visualization with PortalJS'
+title: 'Adding Maps to PortalJS: Enhancing Geospatial Data Visualization with PortalJS'
 date: 2023-07-18
 authors: ['João Demenech', 'Luccas Mateus', 'Yoana Popova']
 filetype: 'blog'
 ---
 
-Are you keen on building rich and interactive data portals? Do you find value in the power and flexibility of JavaScript, Nextjs, and React? In that case, allow us to introduce you to [PortalJS](https://portaljs.org/), a state-of-the-art framework leveraging these technologies to help you build amazing data portals.
+This post walks you though adding maps and geospatial visualizations to PortalJS.
 
-Perhaps you already understand that the effective data visualization lies in the adept utilization of various data components. Within [PortalJS](https://portaljs.org/), we take data visualization a step further. It's not just about displaying data - it's about telling a captivating story through the strategic orchestration of a diverse array of data components.
+Are you interested in building rich and interactive data portals? Do you find value in the power and flexibility of JavaScript, Nextjs, and React? If so, [PortalJS](https://portaljs.org/) is for you. It's a state-of-the-art framework leveraging these technologies to help you build rich data portals.
 
-We are now eager to share our latest enhancement to [PortalJS](https://portaljs.org/): maps, a powerful tool for visualizing geospatial data. In this post, we will to take you on a tour of our experiments and progress in enhancing map functionalities on [PortalJS](https://portaljs.org/). Our journey into this innovative feature is still in its early stages, with new facets being unveiled and refined as we perfect our API. Still, this exciting development opens a new avenue for visualizing data, enhancing your ability to convey complex geospatial information with clarity and precision.
+Effective data visualization lies in the use  of various data components. Within [PortalJS](https://portaljs.org/), we take data visualization a step further. It's not just about displaying data - it's about telling a story through combining a variety of data components.
+
+In this post we will share our latest enhancement to PortalJS: maps, a powerful tool for visualizing geospatial data. In this post, we will to take you on a tour of our experiments and progress in enhancing map functionalities on PortalJS. The journey is still in its early stages, with new facets being unveiled and refined as we perfect our API.
 
 ## Exploring Map Formats
 

site/content/blog/markdowndb-launch.md

@@ -0,0 +1,149 @@
+---
+title: 'Announcing MarkdownDB: an open source tool to create an SQL API to your markdown files! 🚀'
+description: MarkdownDB - an open source library to transform markdown content into sql-queryable data. Build rich markdown-powered sites easily and reliably. New dedicated website at markdowndb.com
+date: 2023-10-11
+authors: ['Ola Rubaj']
+filetype: blog
+---
+
+Hello, dear readers!
+
+We're excited to announce the official launch of MarkdownDB, an open source library to transform markdown content into sql-queryable data. Get a rich SQL API to your markdown files in seconds and build rich markdown-powered sites easily and reliably.
+
+We've also created a new dedicated website:
+
+https://markdowndb.com/
+
+## 🔍 What is MarkdownDB?
+
+MarkdownDB transforms your Markdown content into a queryable, lightweight SQLite database. Imagine being able to treat your collection of markdown files like entries in a database! Ever thought about fetching documents with specific tags or created in the past week? Or, say, pulling up all tasks across documents marked with the "⏭️" emoji? With MarkdownDB, all of this (and more) is not just possible, but a breeze.
+
+## 🌟 Features
+
+- Rich metadata extracted including frontmatter, links and more.
+- Lightweight and fast indexing 1000s of files in seconds.
+- Open source and extensible via plugin system.
+
+## 🚀 How it works
+
+### You have a folder of markdown content
+
+For example, your blog posts. Each file can have a YAML frontmatter header with metadata like title, date, tags, etc.
+
+```md
+---
+title: My first blog post
+date: 2021-01-01
+tags: [a, b, c]
+author: John Doe
+---
+
+# My first blog post
+
+This is my first blog post.
+I'm using MarkdownDB to manage my blog posts.
+```
+
+### Index the files with MarkdownDB
+
+Use the npm `mddb` package to index Markdown files into an SQLite database. This will create a `markdown.db` file in the current directory.
+
+```bash
+# npx mddb <path-to-folder-with-your-md-files>
+npx mddb ./blog
+```
+
+### Query your files with SQL...
+
+E.g. get all the files with with tag `a`.
+
+```sql
+SELECT files.*
+FROM files
+INNER JOIN file_tags ON files._id = file_tags.file
+WHERE file_tags.tag = 'a'
+```
+
+### ...or using MarkdownDB Node.js API in a framework of your choice!
+
+Use our Node API to query your data for your blog, wiki, docs, digital garden, or anything you want!
+
+E.g. here is an example of a Next.js page:
+
+```js
+// @/pages/blog/index.js
+import React from 'react';
+import clientPromise from '@/lib/mddb.mjs';
+
+export default function Blog({ blogs }) {
+  return (
+    <div>
+      <h1>Blog</h1>
+      <ul>
+        {blogs.map((blog) => (
+          <li key={blog.id}>
+            <a href={blog.url_path}>{blog.title}</a>
+          </li>
+        ))}
+      </ul>
+    </div>
+  );
+}
+
+export const getStaticProps = async () => {
+  const mddb = await clientPromise;
+  // get all files that are not marked as draft in the frontmatter
+  const blogFiles = await mddb.getFiles({
+    frontmatter: {
+      draft: false,
+    },
+  });
+
+  const blogsList = blogFiles.map(({ metadata, url_path }) => ({
+    ...metadata,
+    url_path,
+  }));
+
+  return {
+    props: {
+      blogs: blogsList,
+    },
+  };
+};
+```
+
+And the imported library with MarkdownDB database connection:
+
+```js
+// @/lib/mddb.mjs
+import { MarkdownDB } from 'mddb';
+
+const dbPath = 'markdown.db';
+
+const client = new MarkdownDB({
+  client: 'sqlite3',
+  connection: {
+    filename: dbPath,
+  },
+});
+
+const clientPromise = client.init();
+
+export default clientPromise;
+```
+
+### Build your blog, wiki, docs, digital garden, or anything you want
+
+And share it with the world!
+
+![[markdowbdb-launch-site-example.png]]
+
+👉 [Read the full tutorial](https://markdowndb.com/blog/basic-tutorial)
+
+---
+
+Find out more on our new official website: https://markdowndb.com/
+
+Check out the source on GitHub: https://github.com/datopian/markdowndb
+
+— Ola Rubaj, Developer at Datopian

site/content/blog/summer-updates-2023.md

@@ -2,6 +2,7 @@
 title: What We Shipped in Jul-Aug 2023
 authors: ['ola-rubaj']
 date: 2023-09-2
+filetype: blog
 ---
 
 Hey everyone! 👋 Summer has been in full swing, and while I've managed to catch some vacation vibes, I've also been deep into code. I'm super excited to share some of the latest updates and features we've rolled out over the past two months. Let's dive in:

site/content/blog/the-open-spending-revamp-behind-the-scenes.md

@@ -0,0 +1,34 @@
+---
+title: 'The OpenSpending Revamp: Behind the Scenes'
+date: 2023-10-13
+authors: ['Luccas Mateus', 'João Demenech']
+filetype: 'blog'
+---
+
+_This post was originally published on [the Datopian blog](http://datopian.com/blog/the-open-spending-revamp-behind-the-scenes)._
+
+In our last article, we explored [the Open Spending revamp](https://www.datopian.com/blog/the-open-spending-revamp). Now, let's dive into the tech stack that makes it tick. We'll unpack how PortalJS, Cloudflare R2, Frictionless Data Packages, and Octokit come together to power this next-level data portal. From our Javascript framework PortalJS, that shapes the user experience, to Cloudflare R2, the robust storage solution that secures the data, we'll examine how each piece of technology contributes to the bigger picture. We'll also delve into the roles of Frictionless Data Packages for metadata management and Octokit for automating dataset metadata retrieval. Read on for the inside scoop!
+
+## The Core: PortalJS
+
+At the core of the revamped OpenSpending website is [PortalJS](https://portaljs.org), a JavaScript library that's a game-changer in building powerful data portals with data visualizations. What makes it so special? Well, it's packed with reusable React components that make our lives - and yours - a whole lot easier. Take, for example, our sleek CSV previews; they're brought to life by PortalJS' [FlatUI Component](https://storybook.portaljs.org/?path=/story/components-flatuitable--from-url). It helps transform raw numbers into visuals that you can easily understand and use. Curious to know more? Check out the [official PortalJS website](https://portaljs.org).
+
+![Data visualization](/assets/blog/2023-10-13-the-open-spending-revamp-behind-the-scenes/data-visualization.png)
+
+## Metadata: Frictionless Data Packages
+
+Storing metadata might seem like a backstage operation, but it is pivotal. We chose Frictionless Data Packages, housed in the `os-data` GitHub organization as repositories, to serve this purpose. Frictionless Data Packages offer a simple but powerful format for cataloging and packaging a collection of data - in our scenario, that's primarily tabular data. These aren't merely storage bins - they align with FAIR principles, ensuring that the data is easily Findable, Accessible, Interoperable, and Reusable. This alignment positions them as an ideal solution for publishing datasets designed to be both openly accessible and highly usable. Learn more from their [official documentation](https://framework.frictionlessdata.io/).
+
+## The Link: Octokit
+
+Can you imagine having to manually gather metadata for each dataset from multiple GitHub repositories? Sounds tedious, right? That’s why we used Octokit, a GitHub API client for Node.js. This tool takes care of the heavy lifting, automating the metadata retrieval process for us. If you're intrigued by Octokit's capabilities, you can discover more in its [GitHub repository](https://github.com/octokit/octokit.js). To explore the datasets we've been working on, take a look at [OpenSpending Datasets](https://github.com/os-data).
+
+## Storage: Cloudflare R2
+
+When it comes to data storage, Cloudflare R2 emerges as our choice, defined by its blend of speed and reliability. This service empowers developers to securely store large amounts of blob data without the costly egress bandwidth fees associated with typical cloud storage services. For a comprehensive understanding of Cloudflare R2, their [blog post](https://cloudflare.net/news/news-details/2021/Cloudflare-Announces-R2-Storage-Rapid-and-Reliable-S3-Compatible-Object-Storage-Designed-for-the-Edge/default.aspx) serves as an excellent resource.
+
+## In Closing
+
+In closing, we invite you to explore the architecture and code that power this project. It's all openly accessible in our [GitHub repository](https://github.com/datopian/portaljs/tree/main/examples/openspending). Should you want to experience the end result firsthand, feel free to visit [openspending.org](https://www.openspending.org/). If you encounter any issues or have suggestions to improve the project, we welcome your contributions via our [GitHub issues page](https://github.com/datopian/portaljs/issues). For real-time assistance and to engage with our community, don't hesitate to join our [Discord Channel](https://discord.com/invite/EeyfGrGu4U). Thank you for taking the time to read about our work! We look forward to fostering a collaborative environment where knowledge is freely shared and continually enriched. ♥️
+
+![FlatUiTable Code Snippet](/assets/blog/2023-10-13-the-open-spending-revamp-behind-the-scenes/code-example.png)

site/content/config.js

@@ -23,8 +23,7 @@ const config = {
     { name: 'Guide', href: '/guide' },
     {
       name: 'Examples',
-      href: 'https://github.com/datopian/portaljs/tree/main/examples',
-      target: '_blank',
+      href: '/examples/'
     },
     {
       name: 'Components',
@@ -70,7 +69,7 @@ const config = {
     },
   },
   github: 'https://github.com/datopian/portaljs',
-  discord: 'https://discord.gg/EeyfGrGu4U',
+  discord: 'https://discord.gg/xfFDMPU9dC',
   tableOfContents: true,
   analytics: 'G-96GWZHMH57',
   // editLinkShow: true,

site/content/examples/index.md

@@ -0,0 +1,5 @@
+# Examples
+
+For now, see the examples folder in github:
+
+https://github.com/datopian/portaljs/tree/main/examples

tools/tsconfig.tools.json

@@ -1,12 +0,0 @@
-{
-  "extends": "../tsconfig.base.json",
-  "compilerOptions": {
-    "outDir": "../dist/out-tsc/tools",
-    "rootDir": ".",
-    "module": "commonjs",
-    "target": "es5",
-    "types": ["node"],
-    "importHelpers": false
-  },
-  "include": ["**/*.ts"]
-}