Plaintext references management with Markdown and Zotero

My workflow for capturing and managing research references without reliance on closed apps or services.

Josh Carpenter
Josh Carpenter
Capturing references with Zotero, citing them in Markdown notes, and then publishing to the web. We’ll cover publishing in a later post.

The need for better tools

In 2019 I decided to learn more climate change. As I read and took notes, I tried to track my sources by pasting URLs next to this fact or that figure. But I quickly realized this wouldn’t scale: ad hoc citations were quickly cluttering up my documents; I was inconsistent about what information I captured; and there was no centralized system for managing shared references across documents.

I needed a workflow that let me 1) easily capture references, 2) with full metadata (e.g. author, date, title, url), 3) into a distinct, canonical repository, and 4) easily insert citations into my notes.

I also wanted this workflow to be 100% plain text. Using formats such as JSON and Markdown would make it possible to create scripted publishing workflows that I could tune to exactly my liking. And keeping my data out of proprietary apps and services helped ensured I would always have access to it.

After fussing with various tools, I arrived at the following workflow for capturing and managing references. It’s not perfect, but it checks the above boxes, and I’m using it with hundreds of references. In a future post I’ll share my workflow for publishing the web.

At a high level, my workflow looks like the following:

  1. Capture and manage references with Zotero. Each reference receives a unique “citation key” ID.
  2. Write notes in Markdown. 
  3. Create citations in Markdown notes by pasting in citation keys, wrapped in square brackets.
  4. Publish notes to the web, with citations rendered inline, using Eleventy and Citeproc.

The steps to recreate the workflow are as follows.

Step 1: Install Zotero

My climate change references in Zotero

Zotero is a free open-source desktop app for managing research references. Each reference is stored as an item with detailed metadata such as title, author, year published, etc. Zotero includes tools for capturing references for books, reports, websites, and more. It’s available for Mac, Linux and Windows.

Step 2: Install Better BibTeX

Better BibTeX (BBT) is an extension for Zotero that adds powerful features for text-based workflows.

  1. Download the latest Better BibTex release
  2. In Zotero, under Tools > Add-ons, select Install Add-on From File from the gear icon in top-right, and the select the downloaded Better BibTeX .xpi file.
  3. Restart Zotero. When it re-opens the Better BibTeX setup wizard will appear. Confirm the default options.

Step 3: Set Zotero preferences

Zotero and Better BibTeX have a long list of options that we can configure. The screenshots above show my exact preferences for the important preferences sections. But the essential options are as follows:

Preferences tab Option Set to
Better BibTeX
Citation keys Force citation key to plain text Check
Citation keys Keep keys unique Within each library
Export > BibTeX Export unicode as plain-text latext commands Check
Export > Quick-Copy Quick-Copy format Cite Keys
Export > Miscellaneous Apply title-casing to titles Check
Automatic export Automatic export On Change
Export
Default Format Better BibTeX Quick Copy

Step 4: Export our references to JSON 

{"id":"blog-Block-WhatIt-17","accessed":{"date-parts":[[2019,11,9]]},"author":[{"family":"Block","given":"Erin"}],"citation-key":"blog-Block-WhatIt-17","container-title":"Waterstudio","issued":{"date-parts":[[2017,9,22]]},"language":"en","title":"What It Takes to Make a Brand New Island","type":"post-weblog","URL":"https://www.waterstudio.nl/what-it-takes-to-make-a-brand-new-island/"}

We need to export our Zotero library to JSON so that later on, in the publishing step, we can use this JSON—along with our Markdown notes—to render our notes as HTML, along with references rendered as citations and/or bibliographies. Once exported, Zotero will automatically update the JSON whenver we make a change.

  1. Select File > Export Library
  2. Set Format to Better CSL JSON, and check Keep updated in export options.
  3. Name and save the file.

Step 5: Capture references!

Using Zotero Connector to capture an article from Chrome. The Connector has automatically filled out all the fields for us.

In addition to manually creating references using File > New Item, we can automatically generate detailed references from the web or publications with the following tools:

Add websites from the browser

If you create references from websites, the Zotero Connector browser extension is a must-have time saver. It lives as a button in the browser toolbar, and when clicked saves a detailed reference for the current site, complete with a “snapshot” offline capture of the page. Per the docs:

With the click of a button, Zotero can automatically create an item of the appropriate type and populate the metadata fields, download a full-text PDF if available, and attach useful links…

Add books, papers, articles (etc) by publication identifier

Enter an ISBN, DOI, PMID, or arXiv ID into the Add Item(s) by Identifier tool and Zotero will create a detaiked reference for the publication. This is a huge time saver. Look for a toolbar button that looks like a wand (for some reason there’s no menu command).

Step 6: Cite references in Markdown

Citation keys (in pink) in a Markdown document

As we create references (“items”, in Zotero parlance), each item is assigned a unique Citation Key. To create a citation in Markdown, we paste the reference’s citation key, preceded by an ampersand, and wrapped in square brackets. For example, if our reference’s citation key is brandWholeEarth, we enter [@brandWholeEarth] in our Markdown note.

To specify where in the reference our citation comes from, we can optionally add a locator: [@brandWholeEarth, p. 67]. There are locators for chapter, figure, page, section, etc.  For more information on the citation key syntax, see the Pandoc Markdown citation extension documentation.

Copy citation keys with Cmd-Shift-C

To copy citation keys from Zotero, select one or more references in the library and hit Cmd-Shift-C (Mac). This will copy the citation key(s) to the clipboard, and we can then paste them into our notes. Alternatively, we can drag and drop references from Zotero into our notes. For these features to work, we need to set the following preferences (which we already did in Step 3):

  • Export > Default Format > Better BibTeX Quick Copy: Cite Keys
  • Better BibTeX > Export > Quick Copy > Quick-Copy format: Cite Keys

Editing items in Zotero after citing them in Markdown may break the citations. By default, when we edit an item in Zotero, BBT automatically updates the citation key. If the key changes and we’ve already used the old version, the citation will break at publish time unless we first manually update it. To help avoid this, BBT has a feature wherein we can fix (“pin”) individual keys, so they don’t change when we edit their items. See the BBT docs for details.

Conclusion: It works! But the UX could be better.

A discrete citation key UX that I hacked together in Typora. Keys collapse until hovered over or clicked.

This workflow is pretty good! Our references are properly managed, and because everything is in plain text, we control our own data. But the writing experience could be better. Citation keys quickly clutter our Markdown as we add citations. The whole point of Markdown is to minimize the visual noise of HTML elements so that words have focus. It would be better if citation keys collapsed by default, previewed on hover, and became editable when clicked on. I hacked together an implementation of this using Typora , but it’s not ideal. It requires I wrap citation keys in otherwise-unnecessary == mark tags, and I generally prefer VSCode as my text editor . It would also be better if hovering showed the full reference; not just the citation key.

I was obsessed enough about nailing this workflow that I spent over a year building my own text editor with citations integration, but with a new baby in the family my spare time is suddenly short, and I’m not sure I’ll ship the app. So for now I tolerate cluttered Markdown in VSCode.

In a follow up post, I’ll show how we can publish our notes to the web, with citations rendered inline, using using Node, Eleventy and Citeproc.

  1. I use the GitHub Flavored Markdown spec syntax, plus a few Pandoc Markdown extensions for footnotes, implicit figures, and citations.↩︎
  2. Your available locators are determined by which Locale you choose to use, when we come to the publish step.↩︎
  3. Credit to Dave Smith for his zotpick-applescript repo. My fork just adds a brackets parameter.↩︎
  4. Automator actions are saved to user/Library/Services↩︎
  5. The citation picker is enabled by Better BibTeX’s Cite as You Write feature.↩︎
  6. Modern Apple still enables automation workflows on the Mac platform, but it wraps them in lots of permissions.↩︎
  7. Due to a known VS Code Info.plist issue on Mac, the script will try to paste the selected key into Electron, instead of VS Code↩︎
  8. Typora is built on Electron and enables us to write custom themes using CSS↩︎
  9. Collapsing citation keys in VSCode would be great, but it’s not possible until it lands the ability to collapse ranges, per this issue.↩︎
Josh Carpenter

Hello! I'm a Staff UX Designer for Google Maps, based in Vancouver, Canada. Previously I led design teams at Google and Mozilla working to bring virtual and augmented reality to the open web. Since 2019 I've been traveling and working on a series of personal projects, including an Electron-based Markdown editor and a deep dive into climate change.

Twitter | GitHub | LinkedIn | Resume

Portfolio

Posts

Projects