Sending data to Quarto via runCommand()

[mwra (Mark Anderson)]

Thank you for the update. The whole point of runCommand() is you don’t need the AppleScript and Terminal. runCommand() ‘speaks’ directly to the Mac’s Unix shell. As you’d need to use runCommand to call your AppleScript, it is extra work for no reward.

I assume, because you didn’t report it as working, that the runCommand() example I gave you didn’t work. Looking at your AppleScript script example there is a path element (that wasn’t mentioned in your original example). So have you tried a command like:

var:string vCmd = "xargs quarto";
var:string vArgs= "preview ja.qmd";
var:string vPath = "~/Documents/Tinderbox9/2023quarto";
runCommand(vCmd, vArgs, vPath);

Depending on how big complex your Quarto file is, you can actually skip exporting that and just pass the exported data (currently saved to ja.md) directly into runCommand().

So a pseudo example (as I don’t have all the data & apps to try it):

var:string vCmd = "xargs quarto";
var:string vArgs= 'preview "'+exportedString("ja")+'"';
var:string vPath = "~/Documents/Tinderbox9/2023quarto";
runCommand(vCmd, vArgs, vPath);

I’d recommend getting the first example method (of the two in the this post) to work before trying the second. The whole point of the approach is to change only a little part of the process at a time.

It is difficult to be precise on what works as whilst we are all using Tinderbox, not all Mac unix shells are configured the same way. This is why starting with a Terminal command that works is a got starting play when trying to figure out a runCommand() value to use. The possible parts are:

• the shell command (e.g. like quarto) - does it run from anywhere
• the command’s flags (e.g. preview) that trigger different behaviours in the command
• other input such as data file(s) which might, in Terminal, be supplied via stdIn or as an argument (alongside the flag(s).
• the relative location of data file(s) to where runCommand() executes in the shell. By default this is you home folder (~), so if calling a data file (above) which isn’t at ~ you need to supply the path to it (either full path or path relative to ~.

Anyway, whilst I’m glad your AppleScript test worked, it is not part of the solution here. Think of runCommand() as similar to being able to type into Terminal directly from inside Tinderbox.

[mwra (Mark Anderson)]

Note: I’ve moved this to a thread of its own with a more useful title.

Your test note includes lots of Quarto customisation (i.e that other users might not have.). I’d suggest exporting the simplest note possible to Quarto as that increases the chance of other people being able to test your code. If a simple export works, then you can add in the code you really want confident that the export will work.

I hope that helps

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson,
Thank you very much for your detailed explanation.
I am sorry that all the questions are too elementary.
The big question for me at the moment is,
where in the Tinderbox should I put the command
which you have given me when I use runCommand()?
I do nt know this exact where adding in, that is a reason why
I experimented with a different approach using AppleScript.

How can I put this in my Tinderbox so that I can take my first steps,
where can I put my hand, where can I catch it, so that
I can start walking like a baby?

Please forgive me for bothering you and taking up valuable time
from you on an elementary matter.
I hope you can provide me with more information.
Sincerely, WAKAMATSU

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Mr. Dominique Renauld,
I have a little question.
It is about how you use quarto.
Are you processing documents written in Tinderbox with quarto?
Q01 :
If so, how do you change the extension in the export to specifying it?

Q02 : Do you use HTMLexportTemplate?
If you are using, when creating the “.qmd” extension
How to write extra description when exporting from html avoiding it?
Q03 :
If possible, a sample of your Tinderbox file
I would appreciate it if you could present it.
Yours, WAKAMATSU
P.S.
Export the file as Text, not HTML
quarto preview when renaming the file with a “.qmd” extension
Commands from Terminal were enabled cleanly.

[mwra (Mark Anderson)]

@WAKAMATSU

where in the Tinderbox should I put the command
which you have given me when I use runCommand()?

Ah. My apologies. First to understand is that runConmmand() is an [action code] operator (Action Code). So you can use runCommand() in any place where you use actions : rules, edicts, agent actions, on add actions, stamps.

My suggestion for initial testing is to use a stamp. To give you confidence, make a new note, add attribute MyNumber as a Displayed Attribute and apply this stamp (as from the documentation):

$MyNumber = runCommand("wc -w","hello world");

After running the stamp, the note’s $MyNumber is 2. Why, because the number of words in the string ‘hello world’ is 2.

Congratulations. You now know the workings of runCommand(). This is the first of the small steps you asked for. You have now passed a command from Tinderbox it the OS command line and got a calculated output without using the Terminal.

You should be see that what you pass as arguments to the runCommand() operator changes what gets done. Depending on your task you can use runCommand() with:

runCommand( ...);

or without:

$SomeAttribute = runCommand( ... );

a left side argument.

[dominiquerenauld]

Dear Monsieur WAKAMATSU,

To answer you:

Q01:

I don’t export my document. I don’t use any extension from Tinderbox. But, you can do it of course with Tinderbox. Instead of doing that, I just copy and paste my document — including my YAML and my separators — into Quarto. Then, I « knit » the whole and get a nice pdf.

But, I know that you directly « compile » your document from Tinderbox. There are one video Michael Becker @satikusala made about it and some reflections here in the Tinderbox forum.

Q02:

I don’t use HTMLexportTemplate.

Q03:

You’ll find below a file I made to illustrate my words and some pictures to see the whole.

Plan article Quarto.tbx (192.0 KB)

As you can see, I use a Latex template in my YAML because I do need to output a PDF that looks like an already accepted and published article. This trick helps me a lot. But, you can customize your export template within Tinderbox.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr. Mark Anderson,
Thank you for your detailed explanation.
I apologize for bothering you with my lack of understanding, everytime.
Last time, I set the general idea as “Action Code” and
I created a Stamp using the method which you presented,
but maybe because my writing style was bad, it did not work for me.
This time, after thoroughly understanding it, I will try runCommand again.
Yours, WAKAMATSU

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Mr.Dominique Renauld,
Thank you for your detailed answer.
I am happy to know that you are using the LaTeX Template.
I am a LaTeX enthusiast.
The reason is LaTeX description method and music
This is because it creates the same world as the specification method in the score.
I will read and learn from your tbx file.
Hurry up and thank you.
Sincerely yours, WAKAMATSU
P.S
My YAML LaTeX Template using lualatex engine.

[mwra (Mark Anderson)]

OK, here is the example from above:

And the TBX file shown above: runCommand-example.tbx (212.6 KB)

Test files for test #2 (zip): test.zip (29.9 KB)

Select note “Test” and run stamp “Example 1” on it. Observe that it works.

Next make a new TBX, and without just copy/pasting from my demo, re-create the same test for yourself and prove it works. You will now know the basic principle of runCommand().

Now, in the original file, read the note for test 2 and install the test folders/files from the zip. Select ‘Test 2’ and one at a time try all 4 variants of stamp “Example 2” (a through d). This exercise shows how you deconstruct a single command string into both the 2 and 3 argument forms of runCommand() use.

As before, try recreating the second set of tests in a new TBX so you get used to the syntax.

Note: runCommand() simply communicates with the macOS command line. If you pass the command line the wrong command it will not work: “garbage in, garbage out”. It is important to understand that for non-OS-default command line tools, it is not possible to know that they are installed the same way on every Mac. This is why using the Terminal app to test commands is a good start. If your command doesn’t work in Terminal, it will not work via runCommand(). Note also that a command in Terminal may need re-formatting (i.e. same information, different layout) to meet runCommand’s documented argument syntax.

Tip: to avoid having to code in long literal paths, the 3-argument form of runCommand helps a lot. Note, via the second example, that if using several directories, the base directory you pass as the third argument must be a common parent of the other folders. So, in example #2 it is ‘test’ from which only the ‘x’ and ‘y’ sub-paths are needed. In reality, ‘test’ might me a folder 14 levels deep in your documents, which is why the 3-argument version of runCommand() syntax is so useful.

Summary, until you can re-create this simple demo yourself, and make it work, don’t waste time on anything more complex as you will have no understanding of why it fails.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr. Mark Anderson,
Thank you for your detailed explanation.
I apologize again, for bothering you with my lack of understanding, everytime.
I understand the first step you taught me.
I tried to move on, but it was all failure.
I was trying to find out what I didn’t know.
「Code examples using attributes with a ‘My’ prefix」
「format(dataStr, formatStr[, additionalArguments])」
「List/Set.at(itemNum)」
「aTbRef’s naming conventions」
「String.replace(regexMatchStr, replacementStr)etc.etc…
There was a response to ja.qmd only once, “Coincidentally?”
The contents were only the information of “html export” was added.
For this reason, in the evening, I was going to give up the approach it.

After seeing your “how to approach” in your exercises,
I am determined to make even more efforts in the future.
I am very grateful to you.
Yours, WAKAMATSU

[sazanamix (Takeshi Otsuka)]

Dear Mr. Wakamatsu

One thing to keep in mind when using runcommand is that, unlike the normal terminal, the PATH of the command is limited.

var:string vCmd="echo $PATH;";
$MyString=runCommand(vCmd);
$MyString  -> /usr/bin:/bin:/usr/sbin:/sbin (in my case)

I think quarto is installed in /usr/local/bin, so when you call it, you need to set for PATH before calling quarto.

export PATH=/usr/local/bin:$PATH;quarto preview

But, calling quarto preview directly from Tinderbox did not work.

If you are calling a command that does not stop, the next runcommand waits for it. Also, exporting notes was not executed.

When quarto preview is run, the process remains(for the browser), so if you want to edit and preview again, you have to kill the process from the terminal.

After my trial and error, I was able to get the PDF and browser display to update automatically when I overwrite the qmd file from Tinderbox by running quarto preview in the terminal in advance and displaying it in the browser.

[mwra (Mark Anderson)]

@WAKAMATSU

I would like to execute the “quarto preview” command from Tinderbox via Terminal
I am searching for a way to do that.
From which attribute can the RunCommand be quoted?

I think it is now clear that if using runCommand() you don’t need to involve the Terminal as both call directly to the macOS Unix shell.

I looked at your original TBX and there were a lot of mis-configuration issues:

• the example note had no export template (I added one)
• the example note had been set to not export! (I re-enabled that)
• there was an export command set that could not run (I removed that.

Now the note can be exported and make a ‘note.qmd’ which I’m exporting to my home folder (~). As I have now got the homebrew Quarto installed, I used this command in Terminal (note how I first test in Terminal):

quarto preview note.qmd

and get feedback like:

Starting python3 kernel...Done

Executing 'note.ipynb'

pandoc 
  to: latex
  output-file: note.tex
  standalone: true
  pdf-engine: lualatex
  variables:
    graphics: true
    tables: true
  default-image-extension: pdf
  toc: true
  toc-depth: 3
  number-sections: true
  
metadata
  block-headings: true
  title: My title
  author: 若松久仁光
  lang: ja
  jupyter: python3
  documentclass: ltjsarticle
  classoption: 'lualatex,ja=standard'
  citations-hover: true
  footnotes-hover: true
  bibliography:
    - ./references1.bib
  editor: visual
  
File ./references1.bib not found in resource path

So, while the command failed as I don’t have any of your Python customisations installed, but it indicates that quarto is basically working. Next I added a stamp:

$Text(/log) = runCommand("quarto preview note.qmd");

Running this in Tinderbox returns this message:

/bin/sh: line 1: quarto: command not found

Clearly the environemnt variables don’'t have quarto in the PAT, but this command teels us where Quarto lives:

which quarto

which returns:

/usr/local/bin/quarto

So the stamp becomes:

$Text(/log) = runCommand("/usr/local/bin/quarto preview note.qmd");

which now runs quarto on the command line as desired but fails due to missing components (python, etc.) that are nothing to do with Tinderbox or runCommand().

I note that whilst drafting this, @sazanamix has added some useful information re the shell PATH variable.

Here is your file with the fixes to the export: quarto-test -1.tbx (225.5 KB)

If you export ‘note’ to your home folder and run the stamp, I believe quarto should compile for you.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson & Mr.Takeshi Otsuka,
Thank you for all your support.
I tried using the revised quarto-test-1.tbx.
From the terminal first by the method pointed out by Mr.Takeshi Otsuka
I started it, and once it was finished, I ran a stamp on Tinderbox.
Append to quarto-test-1.tbx and export note.qmd
Run Stamp again.
For the most part, I was able to confirm that it works up to Preview.
From now on, it will be time for daily flute practice in the morning.

I would like to express my sincere gratitude to you for reporting
that the function you taught has been fulfilled.
With warm regards,WAKAMATSU（August 23, 2023 11:03:39 JST）

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson,
Please excuse my rudimentary and silly question, again.
Q01 : From quarto-test-1.tbx, create a new child file in the current note,
To export to a similar series of note.qmd
How should I add Template > just-text?
I still don’t quite understand the expression of “^value($Text)^”.

Q02: Parallel instead of appending child files
If I create a new file at the same level,
could I continue using “^value($Text)^”?
How should I write it to work?
Yours faithfully, WAKAMATSU

[mwra (Mark Anderson)]

Q1. In you original test TBX, the ‘note’ you were exporting didn’t export any data because:

@WAKAMATSU

I still don’t quite understand the expression of “^value($Text)^”.

• $HTMLDon’tExport was true (non-default). This tell Tinderbox to ignore processing the note when it is part of an export call (whole document or only selected item(s)).

• your document had no export template. so, even having fixed the last point (i.e. $HTMLDon’tExport is now true), the note exports nothing its template—or lack of a template—defines not exportable data.

So, what sort of template does your document need? Not a normal HTML export template as you just want only the $Text in plain text from. In other words, the text as you see it in text with no further processing. There are two ways to specify that in export code:
^text(plain)^
^value($Text)^
In my template I use the last of the two, but either is OK. You will also see from the edited TBX I posted that the template I added, called ‘just-text’, that the only content of the template (i.e. the template note’s $Text) is:
^value($Text)^
When applied to a note and the note is exported, Tinderbox simply takes the contents of $Text, without additional processing and saves it as a file using the desired file name and extension—in this case ‘note.qmd’.

So that is what ^value($Text)^ does … the ‘how?’.

@WAKAMATSU

How should I add Template > just-text?

Now we know (see above) the point in the template, we simply make a new template in your TBX’s /Templates container. Here is the such a template shown in my earlier demo:

Now you have the template, it is available for use as a template. How to apply it. You can set it via action code:

// using only template name
$HTMLExportTemplate = "just-text";
// or
// using only template path
$HTMLExportTemplate = "/Templates/just-text";

As long as the template’s $Name is unique in the TBX, using just the template $Name is OK, otherwise use its full $Path.

Easier, at least until you are more practised with export is to set the export template via the Export Inspector using the pop-up illustrated below:

Q2. Using the template

@WAKAMATSU

instead of appending child files
If I create a new file at the same level,
could I continue using “^value($Text)^”?

I’m not sure if I fully understand, but I think your question is “how do I apply this new template to other notes being the test note above?”.

OK, a template is a note used to tell Tinderbox how to format the data exported from a note. Rather in the same way a prototype can configure one note or many notes, a template can be used for export by one note or by many notes. In the latter case all notes using the same template are precessed the same way, but using different data (from the note being processed for export). Thus in aTbRef >95% of the 2,000+ HTML pages are created via a single template being used by different notes.

Put simply export is a case of: source note data (the ingredients) + template (the recipe) → exported note

Where the note and the template live in the document are not an issue. Any note you wish to export must be allowed to export (see Q1 above) and have a template assigned. So, as you add new notes that will export to QMD files, give each one the correct template.

Tip: if you will have many such QMD notes, consider using a prototype for all such files. Then set the desired QMD export template in the prototype as this will set it automatically for any note using the prototype. Once set up, having the correct export template is one less thing to forget when making a QMD note.

Summary. Here, we’ve moved away from Quarto issue to basic export issues. Given the questions asked here, I would commend reading up on formatted (i.e. ‘HTML’ ) export. The problems raised come not from the Quarto problem but simply from not understanding how export works. I don’t mean that in a disrespectful manner, but simply that taking a little time to understand the general functioning of export will mean you will already know the answers to these questions when they arise.

I hope this helps.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson,
You truly deserve to be called a gentleman.
I am grateful to you for all the details you have given me.

As you pointed out, my knowledge is broken here and there.
I spent the afternoon trying to make up for my lack of knowledge.
The goal of what I want to do is a little bit more clear,
and there are so many things that I don’t know about this and that.
Really ashamed of myself.
I promise to study again with a serious mind.
In closing, I would like to thank you in advance,again.
Yours faithfully, WAKAMATSU
P.S.
Could you give me a little more time to learn?

[mwra (Mark Anderson)]

Thanks for those kind words. Please, do not be harsh on yourself: as I said, I wasn’t being judgemental, as much of knowledge is only of relevance when we find we need it and don’t have it!! At the same time, we can’t learn everything. I was—hopefully—politely trying to explain that your work had got to the point that understanding how export works would be a help (i.e. time not wasted). but that nuance perhaps did not travel, and if so, my apologies.

Something I’ve taken away from this thread is that the community would benefit from some better explanations for where we are exporting non-HTML data (and have no interest in the HTML-export derivation of the same).

Historically, Tinderbox exported to text (for non techie people) and HTML for those embracing the new medium. In the 20+ years since, “HTML Export” is now termed “Export” or “Formatted export” but is, unintentionally, sightly a case of HTML export minus the HTML. Fine for us who know the back-story. Less useful if you just need to generate some YAML or JSON … or a QMD file.

I’m coming to see we need some description/demos of exporting to neither ‘text’ nor ‘HTML’ but rather explicit exporting plain text with automation (e.g. insertion of attribute values in the output). It’s not that it can’t be done, but currently the explanation tends to take a detour though turning off HTML-adding features, and that is confusing/irrelevant in context. I don’t see that as a failure of the app (that has to support many different types of use) but more in terms of missing usage descriptions better matching contemporary use.

Good luck with the Quarto data export … which is where this all started.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson,
I got it.
Write to the newly added template name “multi-text”
^value($Text)^
^children(^value($Text)^)^
In this way, additional child notes test1 & test2 & yaml children were placed in note.
The export is created to pdf without problems, and from Safari it is “viewer.html” where I was able to check the preview in.
Thank you very much.
I apologize for the delay in reporting.
Yours, WAKAMATSU
P.S.
For export when 3 children are placed in the same column as note.
Not yet successful.

[mwra (Mark Anderson)]

@WAKAMATSU

Write to the newly added template name “multi-text”
^value($Text)^
^children(^value($Text)^)^

Yes, in the layout below, your template above will export the text of B, C and D as part of A, but the text of Y and Z are not exported. Why? Because A’s children only export their own text. If you want X and Y included, i.e. all descendants of A use this template code:

^value($Text)^
^children(/Templates/multi-text)^

An alternative is:

^value($Text)^
^descendants(^value($Text)^)^

In both cases, (exportable) notes are exported and added to the main output as read in $OutlineOrder order.

Also, in the above case, if you will export the whole document and not A as a single note export, you likely want to set A’s $HTMLExportChildren to true or set it in A’s Export Inspector:

@WAKAMATSU

For export when 3 children are placed in the same column as note.
Not yet successful.

I assume we are talking about notes P, Q and R in the example below which assume :

If you want to export P, Q and R as separate notes, use the your existing ‘just-text’ template. If you want P, Q and R in the same exported note, which is the parent/first note. The answer to the last depending on which source note will export. Do you want A to also contain (the $Text of) P, Q, R? It’s isn’t clear. An annotated screen grab, like above might help us through issues of translations.

Tip: with export, it a very good idea, to structure your outline as used in export. Otherwise you end up in unnecessarily complex export code. Put another way, if your outline doesn’t reflect the export structure, _does it need to be that way? This tip is informed by lessons learned (mistakes made@) planning complex export in the past

To if, for example, you want the contents of P exported by A, make P a child of A in the outline. Trust me: it will make your work easier.

[WAKAMATSU (WAKAMATSU kunimitsu)]

Dear Dr.Mark Anderson,
How are you doing?
Unfortunately, hot days continue in Japan as well.
I finally started experimenting to make the next step runCommand（） .
Call quarto $Text(/log) = runCommand(“/usr/local/bin/quarto preview note.qmd”);
When I run this, “[91mERROR: note.qmd not found” is displayed in the log.
Call quarto 2 $Text(/log 2) = runCommand(“/usr/local/bin/quarto preview note.qmd”,“~/Tinderbox2023/note.qmd”);
And I added the location specification of note.qmd.
For the time being, permission “777” is applied to the “Tinderbox2023” folder itself and its inside.
Occasionally, I don’t see any errors in log 2,
but when I update by appending to the file
I get a permission error like this:
/bin/sh: line 2: /Users/kuni33/Tinderbox2023/note.qmd: Permission denied

It seems that the command in quarto changes the “number” on the
“Preview allocation” site every time.
For this, Users > Library > .chache > quarto > jt >
{“port”: 51389, “secret”: “3a1a41cb-23b9-40b9-9afe-3fb3c858d7cf”}
It seems that it is one of the procedures in compiling the file
based on the allocation number described in
It seems that there is also a reason why Terminal emits an error.
By removing this cache once in Terminal, at last normal
It will run “quarto preview note.qmd”.

This is my question.
Q01 : Will it work if I write permissions to Stamps?
I remember that there was an article about “permission permission” somewhere
It’s yesterday and today, but I haven’t been able to find “that place”.

Q02 : Is there an order of priority for writing in runCommand()?
The location of how to give permissions is “first” or “last”
Where should I write it to make it work properly?

Yours faithfully, WAKAMATSU
P.S
Should the permissions be chmod 755?
I found an explanation in the Command Line in your A Tinderbox Reference File, again.