diff --git a/site/content/assets/blog/output.png b/site/content/assets/blog/output.png new file mode 100644 index 00000000..e06d184e Binary files /dev/null and b/site/content/assets/blog/output.png differ diff --git a/site/content/assets/blog/sqlite-viewer.png b/site/content/assets/blog/sqlite-viewer.png new file mode 100644 index 00000000..f712749d Binary files /dev/null and b/site/content/assets/blog/sqlite-viewer.png differ diff --git a/site/content/blog/markdowndb-basics-tutorial-2023.md b/site/content/blog/markdowndb-basics-tutorial-2023.md index 1d3c97aa..a646880b 100644 --- a/site/content/blog/markdowndb-basics-tutorial-2023.md +++ b/site/content/blog/markdowndb-basics-tutorial-2023.md @@ -9,32 +9,261 @@ We've just released our first tutorial that covers the fundamentals of [Markdown ![Gif](/assets/blog/markdowndb.gif) -[👉 Check it out!](https://github.com/datopian/markdowndb/blob/main/examples/basic-example/README.md) - ## What is MarkdownDB? -MarkdownDB can parse your markdown files, extract structured data, such as frontmatter, tags, back- and forward links, and create an index in a local SQLite database. It also provides a lightweight JavaScript API for querying the database and importing files into your application. With MarkdownDB, you have a powerful tool that allows you to efficiently query your markdown data. +MarkdownDB can parse your markdown files, extract structured data (such as frontmatter, tags, back- and forward links and more), and create an index in a local SQLite database. It also provides a lightweight JavaScript API for querying the database and importing files into your application. With MarkdownDB, you have a powerful tool that allows you to efficiently query your markdown data. [🔍 Click here to learn more!](https://github.com/datopian/markdowndb) ## What you're going to learn -This tutorial guides you through the steps of creating SQLite database of our GitHub projects, using markdown files and MarkdownDB package. In addition, it demonstrates how to write a simple script to query a database for the projects and display them in an attractive format on the terminal. +This tutorial guides you through the steps of creating a small project catalog. In our case, we used our open-source projects on GitHub - but you could use anything! We use simple markdown files to describe each project and then display them with the MarkdownDB package. -Key highlights from the tutorial include: +## Step 1: Create a markdown file for each project -1. Creating markdown files with some basic information for each GH project. -2. Indexing markdown files into an SQLite database. -3. Exploring the SQLite database using SQLite viewer or in the command line. -4. Querying the database in a simple Node.js app and displaing them in the terminal. -5. Adding metadata, like name, numbers of stars or number of forks, to project files. -6. Pretty printing the output in the terminal. +First, let's create a markdown file for each project. You can use real project details or make up some examples. For the sake of this tutorial, we'll create files for some of the projects we've built at Datopian. -By the end of the tutorial, you'll have a simple catalog of a few of our GitHub projects using markdown files and the MarkdownDB package. +```bash +mkdir projects +cd projects +touch markdowndb.md portaljs.md flowershow.md datapipes.md giftless.md data-cli.md +``` -[👉 Check it out!](https://github.com/datopian/markdowndb/blob/main/examples/basic-example/README.md) +In each file we'll write some short info about a given project, like so: +```md +# What is MarkdownDB + +MarkdownDB is a javascript library for treating markdown files as a database -- as a "MarkdownDB". Specifically, it: + +- Parses your markdown files to extract structured data (frontmatter, tags, etc) and creates an index in a local SQLite database +- Provides a lightweight javascript API for querying the database and importing files into your application +``` + +## Step 2: Index markdown files into SQLite database + +Once we have prepared our markdown files, we can store them (or more precisely - their metadata) in a database, so that we can then query it later for specific project files. + +```bash +# npx @flowershow/markdowndb +npx @flowershow/markdowndb ./projects +``` + +The above command will output a `markdown.db` file in the directory where it was executed. So, in our case, the folder structure will look like this: + +``` +. +├── markdown.db +└── projects + ├── data-cli.md + ├── ... + └── portaljs.md +``` + +## Step 3: Explore the SQLite database + +Now, let's explore the database. We can do it with any SQLite viewer, e.g. https://sqlitebrowser.org/. When we open the `markdown.db` file in the viewer, we'll see a list of tables: + +- `files`: containing metadata of our markdown, +- `file_tags`: containing tags set in the frontmatter of our markdown files, +- `links`: containing wiki links, i.e. links between our markdown files, +- `tags`: containing all tags used in our markdown files. + +In our case, the `files` table will look like this: + +![[/assets/blog/sqlite-viewer.png]] + +You can also explore the database from the command line, e.g.: + +```bash +sqlite3 markdown.db +``` + +And then run SQL queries, e.g.: + +```sql +SELECT * FROM files; +``` + +Which will output: + +```bash +27ce406aac24e59af7a9f3c0c0a437c1d024152b|projects/data-cli.md|md|data-cli||{} +26b1b0b06a4f450646f9e22fc18ec069bf577d8c|projects/datapipes.md|md|datapipes||{} +dafdd0daf71a1b06db1988c57848dc36947375f4|projects/flowershow.md|md|flowershow||{} +32c8db33fb8758516bfefb6ab1f22d03b1e53a08|projects/giftless.md|md|giftless||{} +7e01cae193f12f5a4a9be2b89f22b429761bd313|projects/markdowndb.md|md|markdowndb||{} +5445349c6822704d6f531a83c2aca2e4f90de864|projects/portaljs.md|md|portaljs||{} +``` + +## Step 4: Query the database in the Node.js app + +Now, let's write a simple script that will query the database for our projects and display them on the terminal. + +First, let's create a new Node.js project: + +```bash +mkdir projects-list +cd projects-list +npm init -y +``` + +Then, let's install the `@flowershow/markdowndb` package: + +```bash +npm install @flowershow/markdowndb +``` + +Now, let's create a new file `index.js` and add the following code: + +```js +import { MarkdownDB } from '@flowershow/markdowndb'; + +// change this to the path to your markdown.db file +const dbPath = 'markdown.db'; + +const client = new MarkdownDB({ + client: 'sqlite3', + connection: { + filename: dbPath, + }, +}); + +const mddb = await client.init(); + +// get all projects +const projects = await mddb.getFiles(); + +console.log(JSON.stringify(projects, null, 2)); + +process.exit(0); +``` + +Since we're using ES6 modules, we also need to add `"type": "module"` to our `package.json` file. + +Before we run the above script, we need to make sure that the `dbPath` variable is pointing to our `markdown.db` file. If you want to store the database outside of your project folder, you can update the `dbPath` variable to point to the correct location. If you want to have it inside your project folder, you can copy it there, or simply re-run the `npx @flowershow/markdowndb ` command from within your project folder. + +Now, let's run the script: + +```bash +node index.js +``` + +It should output the JSON with all our projects. + +```json +[ + { + "_id": "7e01cae193f12f5a4a9be2b89f22b429761bd313", + "file_path": "projects/markdowndb.md", + "extension": "md", + "url_path": "markdowndb", + "filetype": null, + "metadata": {} + }, + ... +] +``` + +## Step 5: Add metadata to project files + +Now, let's add some metadata to our project files. We'll use frontmatter for that. Since we're creating a catalog of our GitHub projects, we'll add the following frontmatter fields to each file: + +```md --- +name: markdowndb +description: Javascript library for treating markdown files as a database. +stars: 6 +forks: 0 +--- +``` + +After adding the metadata, we need to re-index our markdown files into the database: + +```bash +npx @flowershow/markdowndb ../projects +``` + +Now, if we run our script again, we'll see that the `metadata` field in the output contains the metadata we've added to our project files: + +```json +[ + { + "_id": "7e01cae193f12f5a4a9be2b89f22b429761bd313", + "file_path": "projects/markdowndb.md", + "extension": "md", + "url_path": "markdowndb", + "metadata": { + "name": "markdowndb", + "description": "Javascript library for treating markdown files as a database", + "stars": 6, + "forks": 0 + } + }, + ... +] +``` + +## Step 6: Pretty print the output + +Now, let's make the output a bit more readable. We'll use the `columnify` package for that: + +```bash +npm install columnify +``` + +And then we'll update our `index.js` file: + +```js {2,16-38} +import { MarkdownDB } from '@flowershow/markdowndb'; +import columnify from 'columnify'; + +const dbPath = 'markdown.db'; + +const client = new MarkdownDB({ + client: 'sqlite3', + connection: { + filename: dbPath, + }, +}); + +const mddb = await client.init(); +const projects = await mddb.getFiles(); + +// console.log(JSON.stringify(projects, null, 2)); + +const projects2 = projects.map((project) => { + const { file_path, metadata } = project; + return { + file_path, + ...metadata, + }; +}); + +const columns = columnify(projects2, { + truncate: true, + columnSplitter: ' | ', + config: { + description: { + maxWidth: 80, + }, + }, +}); + +console.log('\n'); +console.log(columns); +console.log('\n'); + +process.exit(0); +``` + +The above script will output the following to the terminal: + +![[/assets/blog/output.png]] + +## Done! + +That's it! We've just created a simple catalog of our GitHub projects using markdown files and the MarkdownDB package. You can find the full code for this tutorial [here](https://github.com/datopian/markdowndb/tree/main/examples/basic-example). We look forward to seeing the amazing applications you'll build with this tool!