External note-taking with MultiMarkdown and Metadata


(Peter Punk) #1

Hello,

Some days ago I tried MultiMarkdown and found out that one can put Metadata in a YAML Header at the top of a note in form of key: value pairs. (https://hiltmon.com/blog/2012/06/18/markdown-metadata/)

The look reminded me of Key Attributes and the Text Pane in Tinderbox.

As I like to take notes outside of Tinderbox and was looking for a way to capture “meta” stuff right along with the note creation I made a Keyboard Maestro Macro which creates a fresh Markdown file from a BBEdit Template, ready with my new YAML Header (I divide Header and “real” content by “—” on a new line. Not sure if this is necessary.) All files created by this macro are saved to a Finder folder which is a WatchFolder in Tinderbox.

My Plan now is to extract the YAML values, put them in User Attributes and then remove the complete Header from $Text. Then I can do stuff in Tinderbox (look through the notes, reorganize them, fill User Attributes and maybe add new ones). But I don’t want the notes to only live in Tinderbox so the next step would be Export. I want Markdown files with the User Attributes in form of a YAML Header and beneath this $Text (structure just like the input in the WatchFolder).

Now I would have notes which can be accessed from any app (which can handle Markdown or plain text). The neat thing about a YAML Header in MultiMarkdown is: it’s not there. It’s hidden in rendered view. If I index the exported notes in DEVONthink then I would have “normal” Markdown notes - plus the hidden User Attributes exported from Tinderbox. Hidden but searchable.

But I don’t want to use the exported notes in DEVONthink in the first line. My starting point was to split up revision of notes and actual usage (e.g. Attribute Browser) into two separate Tinderbox documents. This way the state of a note should always be clear. So I plan to make the folder with the exported notes a WatchFolder in another Tinderbox document and repat the extracting of YAML values and setting of User Attributes there.

In order to keep everything clean and clear I thought of:

  • moving the File to another Finder folder (to stop reloading $Text)
  • changing $File to the new location in Finder (for easy access if needed)
  • moving the processed notes to another $Container (just tidy up)

I have this code so far but it doesn’t work the way I thought it would. If I use it as $Edict of the “Imported from Finder” Prototype I have to click every note to process it (or do I have to wait much longer?). And used as $OnAdd of the WatchFolder it only moves the notes to the other container.

$MyString=’"’+$File.replace("~/","/Users/ian/")+’"’ ; runCommand(“mv “+$MyString+” /Users/ian/moved/”) ; if $Text.contains("—") {$MyList=$Text.replace(";",":").split("\n"); $MyList.each(n){if n.beginsWith(“title: “){$YAMLtitle=n.replace(“title: “,””); $YAMLtitle=$YAMLtitle.replace(”:”,";")}}; $MyList.each(n){if n.beginsWith(“created: “){$YAMLcreated=n.replace(“created: “,””); $YAMLcreated=$YAMLcreated.replace(”:”,";")}}; $MyList.each(n){if n.beginsWith(“category: “){$YAMLcategory=n.replace(“category: “,””); $YAMLcategory=$YAMLcategory.replace(”:”,";")}}; $MyList.each(n){if n.beginsWith(“state: “){$YAMLstate=n.replace(“state: “,””); $YAMLstate=$YAMLstate.replace(”:”,";")}}}; if $Text.contains("—") {$Text=$Text.split("—").at(1)} ; $Container=processed ; $File=$File.replace("/Test/","/moved/") ; $EdictDisabled=true ;

What am I missing? Is this a good way to go at all? What do you think of splitting different steps to different Tinderbox documents? Does it make sense?

Ian


(Mark Anderson) #2

Could you link to some specimen input on which you’d run this code (or paste some text here)? That would make it significantly easier to check/test.


(Peter Punk) #3

Here’s a Tinderbox. The other day I managed to extract values and to export but now am failing again. import, review and export notes.tbx (60.4 KB)


(Mark Anderson) #4

In the TBX you posted, the code contains a number of corrected a number of curly quotes (possibly changed via the forum software) and added some whitespace formatting so the process is clearer. This, I think, your original code equates to this:

$MyString='"'+$File.replace("~/","/Users/ian/")+'"';
runCommand("mv "+$MyString+" /Users/ian/moved/");
if $Text.contains("---"){
   $MyList=$Text.replace(";",":").split("\n");
   $MyList.each(n){
      if n.beginsWith("title: "){
         $YAMLtitle=n.replace("title: ","");
         $YAMLtitle=$YAMLtitle.replace(":",";");
      };
   };
   $MyList.each(n){
      if n.beginsWith("created: "){
         $YAMLcreated=n.replace("created: ","");
         $YAMLcreated=$YAMLcreated.replace(":",";");
      };
   };
   $MyList.each(n){
      if n.beginsWith("category: "){
         $YAMLcategory=n.replace("category: ","");
         $YAMLcategory=$YAMLcategory.replace(":",";");
      };
   };
   $MyList.each(n){
      if n.beginsWith("state: "){
         $YAMLstate=n.replace("state: ","");
         $YAMLstate=$YAMLstate.replace(":",";");
      };
   };
};
if $Text.contains("---"){
   $Text=$Text.split("---").at(1);
};
$Container="/processed";
$File=$File.replace("/Test/","/moved/");
$EdictDisabled=true;

If so, this can be expressed a bit more concisely:

$MyString='"'+$File.replace("~/","/Users/ian/")+'"';
runCommand("mv "+$MyString+" /Users/ian/moved/");
if $Text.contains("---"){
   $MyList=$Text.replace(";",":").split("\n");
   $MyList.each(n){
      if n.beginsWith("title: "){
         $YAMLtitle=n.replace("title: ","");
         $YAMLtitle=$YAMLtitle.replace(":",";");
      };
      if n.beginsWith("created: "){
         $YAMLcreated=n.replace("created: ","");
         $YAMLcreated=$YAMLcreated.replace(":",";");
      };
      if n.beginsWith("category: "){
         $YAMLcategory=n.replace("category: ","");
         $YAMLcategory=$YAMLcategory.replace(":",";");
      };
      if n.beginsWith("state: "){
         $YAMLstate=n.replace("state: ","");
         $YAMLstate=$YAMLstate.replace(":",";");
      };
   };
   $Text=$Text.split("---").at(1);
};
$Container="/processed";
$File=$File.replace("/Test/","/moved/");
$EdictDisabled=true;

This seems to work for me and uses fewer loops to achieve the same end. I think the main problem is that you may have used a normal note’s $Text to work on your code. That is not good as straight quotes are turned to curly quotes (which action code treats differently) and triple hyphens become an em-dash.

You may need to change the paths, container values, etc. but the essential part of extracting the data into the YAML attributes works.

Do say if I’ve misunderstood the task, or you still have problems.


(Peter Punk) #5

It’s working. Thanks a lot :smiley:

What I don’t get is exporting. How do I get rid of the HTML Markup?


(Mark Anderson) #6

HTML mark-up where? I’d mainly tested the $Text-processing part of your code


(Peter Punk) #7

You’re right, forgot to ask for it. I’ve

  • build a template (it’s in the former linked file),
  • set it in the HTML Inspector of the Prototype and
  • changed the extension to “.md”

But in HTML Preview and in exports I still have the HTML Markup. I had already managed to export the desired plain text with Markdown extension but broke it somehow again. Could you tell me what’s missing? Does export depend on other templates?


(Paul Walters) #8

Your test file has one template (my MultiMarkDown Template), but does your real file have others?

What command are you using to export? File > Export > as HTML | as Outline | as Text?


(Mark Anderson) #9

I don’t have the copy of Markdown set in your YBX so am using Tinderbox v7.5.4’s internal version - though I don’t think that affects the HTML code aspect of this - just noting it in case.

Something odd is happening, normally ^text(plain)^ emits $Text with no mark-up for paragraphs and such. If I remove the $HTMLPreviewCommand (to use MD) in your prototype then the processed note’s HTML view shows no mark-up. IOW, it appears it is being added by the MD-based pre-parsing of the exported data.

On a hunch, I re-enabled the $HTMLPreviewCommand but altered the template’s code for inserting text from text(plain)^ to value($text)^. My understanding has always been those two export commands are functionally the same. Yet in this case the latter doesn’t add paragraph mark-up, which is want I think you want.

MD is an area in which I’m inexpert plus the $HTMLPreviewCommand stuff happens under the hood where I as a user can’t peer. Perhaps @PaulWalters an clarify why we get these differing results from ostensibly (functionally) the same export command.

Edit: for clarity, test described in this post explicitly using v7.5.4b328.


(Paul Walters) #10

^text(plain)^ and ^value($Text)^ definitely produce different results. See this simple example.

Text MD Test.tbx (60.0 KB) <— use this version not the original post

With the former, Tinderbox 7 adds HTML markup to the “HTML” preview regardless of the template or $HTMLPreviewCommand. With the latter, one gets the plain text without markup. I don’t understand why there is a difference.

Note: using $HTMLPreviewCommand and a template seems to cause conflicts. You can use one or the other but not both – my example uses both.


(Mark Anderson) #11

Thanks for that @PaulWalters, as I’m aware on my lack of MD smarts. I suspect this disparity of outcome falls into the (current) context of unexpected consequences, and your observation about either/or with respect to MD & templates is pertinent.

My suspicion [sic] at this point is current functionality assumed people simple wanted to write MD code in $Text as see MD-rendered in preview mode. However, here the OP’s task - of making a YAML header, shows a need for a bit of extra polish the the MD support … which I’m sure will come, even if not today.

Serendipitously value($Text) gives a work-around as at v7.5.4.

[pinging @eastgate re a possible feature issue]


(Paul Walters) #12

So we need to be clear about a few things:

  1. Markdown and MultiMarkdown were originally intended to be engines to produce canonical HTML code. You compose using Markdown or MultiMarkdown syntax and the engines convert that to HTML. So:
    markdown code --> HTML --> render in a browser
  2. The landscape has become confused in a sense because the middle step either gets hidden or forgotten in most markdown “editors” and the last step no longer requires a browser. Viz., DayOne, Bear, etc. aren’t browsers.
  3. The usage:
    attribute: value
    in a document is not something I would consider canonical YAML (YAML is a rather more sophisticated data serialization language). The syntax used by the OP is actually part of Fletcher’s MultiMarkdown syntax spec, and just happens to look a bit like a YAML snippet. The OP has to use the multimarkdown engine in their $HTMLPreviewCommand because Gruber markdown does not recognize the attribute:value syntax. Fletcher engines ignore the lines with that format and do not output them. The intent is to use these attributes for internal tracking of document versions or the like.

I think Tinderbox is doing already exactly what it needs to do. The feature is Preview not Export. If we want to do preview-only with Markdown- or MultiMarkdown-styled text, then we use $HTMLPreviewCommand; if we want to export we use very simple templates – e.g., ^value($Text)^ and that’s that.

(@pat also recently introduced a hybrid mode between the two modes, using the Marked app.)

I also think aTbRef already explains what’s going on with ^text(plain)^.


(Peter Punk) #13

Got it working with ^value($Text)^. Thanks a lot to both of you! That could have caused me some serious headache :sweat_smile:

I’ve noticed that e.g. a note with $Name “Möbel und Kartons auspacken” is exported to “M_bel_und_Kartons_auspac.md”

Is there a way to export to filenames with umlauts or special characters? If not is there a best practice to workaround?


(Paul Walters) #14

I can’t explain the weird truncation**, but if you have an agent, or a stamp, or merely hand hack setting

$HTMLExportFileName = $Name

then all will be well.

** perhaps a left over from more restrictive days when output couldn’t have special characters, spaces, and length was limited? If you are going to use the exports in a browser then the limitations make sense.


(Mark Anderson) #15

Check $HTMLFileNameMaxLength, which as at v7.5.4 has a default of 24 characters (i.e. for filename without extension).


(eastgate) #16

When Tinderbox was designed, many Web servers couldn’t cope with files with diacritical marks in their names. Others could only handle file names of a certain length — as I recall, DOS servers, still common, were limited to 8 character file names!

We tried to respect those limitations, which mattered to people.