Export individual HTML files from agent results

satikusala · April 27, 2021, 7:55pm

To the best of my knowledge the plan is to have an updated markdown rendering engine in TBX 9. You’ll be able to create markdown files natively in TBX and export them as markdown as well, if all goes as expected. So keep an eye out for the 9.0 release.

To be clear, between now an then you can write markdown in TBX 8.9 and export it as markdown, but in TBX 8.9 and below the preview won’t be as nice as what is expected in TBX 9.0. All you need to do is create a markdown template. Let me know if you need help with this.

ShiJianhui · April 27, 2021, 8:24pm

Thank you for your quick response!
Good to know that new version will get better markdown support! Most of my notes are written with some HTML code inside them, so it would require some additional work to rewrite them into markdown. From what I know after watching last few Tinderbox Meetups, TBX 9 is just around the corner and should land in a few weeks time (of course If everything goes according to plan or those plans won’t change), so I will just patiently wait for it as my case is not so urgent.
Thank you for offering to help, I really appreciate it!

satikusala · April 27, 2021, 8:37pm

Me too! I’d love to have my cake and eat it too, i.e. have both HTML and markdown in my notes. I’m still testing, but I think it may work.

sumnerg · April 28, 2021, 3:15am

Yes, you can do something like this to have underscores instead of blanks:

-- Have an export folder ready. Select Tinderbox notes and run. 
-- NB. Will overwrite existing files of same name.

set text item delimiters to "_"

set theFolder to (choose folder with prompt "Choose a folder to receive the exported MD files")

tell front document of application "Tinderbox 8"
	repeat with aNote in selections
		tell aNote
			set theFilePath to POSIX path of theFolder & (words of (value of attribute "Name" as text) as text) & ".html" -- name file after note
			set theHTML to evaluate with "exportedString(this,$HTMLExportTemplate)"
			do shell script "touch " & quoted form of theFilePath -- create file if doesn't already exist
			do shell script "echo " & quoted form of theHTML & "> " & quoted form of theFilePath -- write to file
		end tell
	end repeat
end tell

If you don’t want underscores and instead want to remove the spaces between words then just remove the first line in the script. Or keep that line and remove the underscore between the quotes.

sumnerg · April 28, 2021, 3:36am

Thanks for heads up on difficulty with Pandoc on M1 Macs. Presume you have already seen this and you couldn’t get it to work.

@satikusala You’ll be able to create markdown files natively in TBX and export them as markdown as well

Write in Markdown and export to Markdown? Or also write in rich text and export from that to both html and Markdown “natively” (i.e. without messing about trying to install Pandoc and get it to work)?

ShiJianhui · April 28, 2021, 5:53am

I’ve given it another chance and successfully installed pandoc! Thanks for help! I’m still new to macOS (M1 is my first Mac ever), so there are still many things that I don’t quite understand. Converting HTML files one by one through terminal works great, but with few hundreds of notes it will take some time, so I tried to apply your Markdown export script You wrote in this thread (I’ve only changed pandoc directory), but with almost no results i.e. after running script, I get empty markdown files.
I think I will just wait for the TBX 9
Thanks again!

sumnerg · April 28, 2021, 11:00am

From everything I’ve read an M1 Mac is a great place to start!

If you’ve got Pandoc working via the command line then it shouldn’t be that hard to get it working via script. What is the complete command you are using from the command line that works?

ShiJianhui · April 28, 2021, 12:06pm

It looks like this:

Mac-mini-Arek:pandoc shijianhui$ pwd
/Users/arkadiuszszlaga/pandoc
Mac-mini-Arek:pandoc shijianhui$ pandoc --version
pandoc 2.13
Compiled with pandoc-types 1.22, texmath 0.12.2, skylighting 0.10.5,
citeproc 0.3.0.9, ipynb 0.1.0.1
User data directory: /Users/shijianhui/.local/share/pandoc
Copyright (C) 2006-2021 John MacFarlane. Web: https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
Mac-mini-Arek:pandoc shijianhui$ ls
test.html
Mac-mini-Arek:pandoc shijianhui$ pandoc test.html -f html -t markdown -s -o test.md
Mac-mini-Arek:pandoc shijianhui$ ls
test.html test.md

Thanks again for help!

sumnerg · April 28, 2021, 12:19pm

And what Pandoc command did you use in the script that didn’t work (including path)?

ShiJianhui · April 28, 2021, 12:37pm

I’ve tried twice with different paths:

-- Have an export folder ready. Select Tinderbox notes and run. 
-- Assumes Pandoc is installed at /usr/local/bin/. See https://pandoc.org/installing.html
-- NB: overwrites any like-named .md files in that folder

set prefix to "@" -- character(s) used to distinguish internal links from external ones

set pandocCmd to "/Users/shijianhui/.local/share/pandoc -f html -t markdown_mmd" -- html to MultiMarkdown
set sedCmd to "sed -E 's/" & prefix & "(\\[.+\\]).+\\)/[\\1]/g;t'" -- grab anchor, surround by [[ ]]
set cmdStr to pandocCmd & " | " & sedCmd -- assemble the "pipe"

set theFolder to (choose folder with prompt "Choose a folder to receive the exported MD files")

tell front document of application "Tinderbox 8"
	repeat with aNote in selections
		tell aNote
			set theFilePath to POSIX path of theFolder & (value of attribute "Name") & ".md" -- name file after note
			set theHTML to evaluate with "exportedString(this,$HTMLExportTemplate)"
			set theMMD to do shell script "echo " & quoted form of theHTML & " | " & cmdStr
			do shell script "touch " & quoted form of theFilePath -- create file if doesn't exist
			do shell script "echo " & quoted form of theMMD & "> " & quoted form of theFilePath -- write to file
		end tell
	end repeat
end tell

and

-- Have an export folder ready. Select Tinderbox notes and run. 
-- Assumes Pandoc is installed at /usr/local/bin/. See https://pandoc.org/installing.html
-- NB: overwrites any like-named .md files in that folder

set prefix to "@" -- character(s) used to distinguish internal links from external ones

set pandocCmd to "/Users/shijianhui/pandoc -f html -t markdown_mmd" -- html to MultiMarkdown
set sedCmd to "sed -E 's/" & prefix & "(\\[.+\\]).+\\)/[\\1]/g;t'" -- grab anchor, surround by [[ ]]
set cmdStr to pandocCmd & " | " & sedCmd -- assemble the "pipe"

set theFolder to (choose folder with prompt "Choose a folder to receive the exported MD files")

tell front document of application "Tinderbox 8"
	repeat with aNote in selections
		tell aNote
			set theFilePath to POSIX path of theFolder & (value of attribute "Name") & ".md" -- name file after note
			set theHTML to evaluate with "exportedString(this,$HTMLExportTemplate)"
			set theMMD to do shell script "echo " & quoted form of theHTML & " | " & cmdStr
			do shell script "touch " & quoted form of theFilePath -- create file if doesn't exist
			do shell script "echo " & quoted form of theMMD & "> " & quoted form of theFilePath -- write to file
		end tell
	end repeat
end tell

I’m still learning AppleScript so there is an extremely high probability that I have made a silly mistake and don’t see it.

sumnerg · April 28, 2021, 1:01pm

The trick is to have the path point to where Pandoc is installed on your machine, which may vary depending on the method of installation.

Pre-M1, if Pandoc is installed via Homebrew, then the path used in the original AppleScript should work.

set pandocCmd to "/usr/local/bin/pandoc -f html -t markdown_mmd" -- html to MultiMarkdown

Sometimes the $PATH variable needs editing. When you enter
echo "$PATH"
(with the quotes around $PATH) in the command line what do you see? Does it include /user/local/bin ?

If you’re still stuck then maybe @Bernard-0 or someone else who knows more about the command line can help. It shouldn’t be too hard, and is probably worth it, because Pandoc does some useful things.

Bernard-0 · April 28, 2021, 2:25pm

Pandoc doesn’t mind filenames with spaces at all When you have a path with a blank space in it in the CL you need to escape it or add quotes:

cd ~/Dropbox/Michael\ Becker/

cd "~/Dropbox/Michael Becker"

Otherwise, it would read the empty space as the end of the path.

It’s the same in the M1 Mac

ShiJianhui · April 28, 2021, 3:17pm

I have reinstalled homebrew and pandoc and everything is working now as it should, thanks for your help!!!

mwra · April 28, 2021, 3:23pm

I’d add an ‘amen’ here. For those of us who use the Command Line less often this might seem like ‘dumb design’. But, spaces matter on the Command Line.

To expand for later readers who aren’t familiar with the Mac’s Unix Command Line…

A very simplistic analogy is that Tinderbox stores a list as one long string so needs the user (or you code) to insert semi-colons so the apps can process the string into a set of list items. In that context, the space in your command line code act as delimiters. In the CL this is normally how the computer separates the program called (which usually (always?) comes first) from input parameters. Consider this CL - exactly what it does isn’t important, just the layout of the code:

sort -k1,1 -k2,2 test.txt

Look for spaces and we see the sort operator getting three discrete inputs: k1,1, -k2,2, and `test.txt’. So we run our test and need to try a new one for which we provide an a file ‘text 1.txt’. so, unthinking, we type in:

sort -k1,1 -k2,2 test 1.txt

Now the sort gets four discrete inputs: k1,1, -k2,2, test, and `1.txt’. Not our intention! We need to tell the CL that ‘test 1.txt’ is all one input, so:

sort -k1,1 -k2,2 'test 1.txt'

So if doing lots of CL work I tend to avoid that bu not using spaces in the names, e.g. :

sort -k1,1 -k2,2 test_1.txt

You could still use quotes there, if its it’s a quasi-rule you’ve learned:

sort -k1,1 -k2,2 `test_1.txt`

The quotes aren’t needed, but, whatever. For the same reasons, if making OS folders where I’ll be using full or partial OS paths (e.g. partial to sub-folders) I tend to use all lowercase a-z0-9_ characters only. It saves lots of head-scratching.

Sorry, bit of a long answer, but coming to CL work via Tinderbox ‘just’ to do something in Tinderbox, I never got to sit in Unix 101 classes so the above sort of issues tripped me up. Plus no one tells you (until you screw up) as they fall in the “too obvious to warrant mention category” .

Oh, and CL experts feel free to correct me if the above is wrong. I do realise '' and "" quotes get treated slightly differently but I don’t think that nuance matters in this context.

satikusala · April 28, 2021, 8:17pm

Yes.

The RFF part is a little more problematic. If you use ^value($value)^ in your markdown export template TBX will convert RTF bullets to markdown bullets. As for the rest it will pass it through as straight text.

satikusala · April 28, 2021, 8:24pm

Just tested! Works great. Thanks did not know that.

satikusala · April 29, 2021, 2:33pm

@sumnerg, per your comment on this thread about being able to add commandline: Updating one note's attribure value with the attribute value's of another note that has the same name - #15 by sumnerg.

How would one go about merging your export individual HTML files script with your add “_” to spaces script (both above) with triggering the addition of a pandoc command line to take the HTML file and cover it to a .docx file, e.g. “pandoc -s test_1.html -o test_1.docx”? The test_1 would be the name of each individual exported note.

sumnerg · April 29, 2021, 2:35pm

@satikusala Do you want the result to be one .docx file including the contents of the selected notes or an individual .docx file for each note?

satikusala · April 29, 2021, 2:48pm

I can get the one integrated file natively withing Tinderbox, select the container, then the export selected note. Tinderbox will merge all the files into one doc and the pandoc command line will convert it to a docx.

What I’m missing is the ability to export a batch of individual files and convert the to the docx. So, I think all of your existing code will work, just need to figure out how to add the pandoc command line to it.

sumnerg · April 29, 2021, 2:58pm

Feeding individual files to pandoc would require scripting Finder.

If your goal is one .docx file with the contents of a Tinderbox container then wouldn’t it be easier just to add the pandoc command to a script that exports that container? (That is easily done, I think, if that is your goal).

Or are you trying to combine files exported from different TBX or something like that?

EDIT. I think I see. To convert to .docx it seems you have to write to a file first.(?)