As a keen beginner, I’m trying to organise my notes in Obsidian using Tinderbox. With the help of AI (of course), I now have an export from Obsidian to Tinderbox at my disposal. The success I’ve had with this combination in tidying up my Obsidian notes has sparked new ideas. The faster the cycle of ‘identifying patterns, editing in Obsidian, next analysis’ runs, the more tedious it is to rebuild the analyses. But perhaps there is a better solution?
Here the more technical description of my constellation:
I’m building Tinderbox documents (~20k notes) from an Obsidian vault (Markdown files with YAML frontmatter) using an external Python script. The script parses the source files and builds a fresh .tbx from scratch each run (via xml.etree.ElementTree), so notes, attributes and prototype links always reflect the current state of the source — a one-way, source-of-truth-stays-external workflow.
That part works well. The problem: because the script builds the document from scratch and never reads the previous .tbx, anything that only exists inside Tinderbox — saved Agents, view configurations (outline sort order, chart axes, column selection), manually adjusted map layouts — gets wiped on every regeneration. That’s fine for a one-shot export, but it breaks down once I start actually using Tinderbox interactively (building agents, tuning report views) between regenerations of the source data.
Has anyone solved this pattern before? Specifically I’m wondering about:
Merge-on-rebuild: has anyone written a script (or is there a built-in mechanism) that reads an existing.tbx, extracts Agents/saved views/prototype definitions, and re-injects them into a freshly generated document — rather than a true incremental diff/update of notes?
Separating “structure” from “data”: is there an established pattern for keeping Agents, stamps, and view definitions in a stable container that survives even when sibling note containers are torn down and rebuilt? (I already keep generated prototypes in their own top-level container, but the whole document is still one file being overwritten.)
Incremental update instead of full rebuild: is there prior art for updating an existing .tbx in place from an external script (matching notes by a stable ID/path and only touching what changed), instead of regenerating the whole tree? I’d want to preserve everything Tinderbox-side that the script doesn’t know about.
Any general stories about round-tripping a large Tinderbox document against an external plain-text source of truth would be very welcome — even “don’t do it this way, do X instead.”
Happy to share more detail on the generator (attribute mapping, prototype resolution from tags, etc.) if useful.
Can you have your AI update the Python script to “upsert” an existing TBX from your Obsidian vault instead of regenerating the .tbx file every time? Presumably, by having the script update an existing TBX, it’ll preserve much of your Tinderbox-native work.
During my current tidying-up phase, many linked notes are being edited directly, which in turn triggers changes in related notes. Obsidian is very fast at this. Attempts to update individual files within the TBX file, however, result in the beach ball appearing. The Python script generates the complete TBX (20,000 files) and the prototypes in around 40 seconds. And TBX handles the finished file without any issues.
Later on, I’d naturally like to use the functionality of Tinderbox with my material. But I don’t like learning using ‘play materials’. So I’ll have to adapt my material to the right format first
Agents are just a special form of note, you can have a container holding all your. But, if you want different agents used on, for instance, different maps, then no you can’t have a single container for agents. The key to understanding this is the Outline view. Tinderbox stored a document in a TBX under a single root-level <item> XML element that replicated the Outline view seen in the comment.
But, why? You don’t need to write the whole document, unless there are other criteria/constraints you haven’t shared. You could just build out the outline or even just part of the outline as opposed to the whole TBX.
What I read here is a common mistake to which we are all prone. It is to over-engineer the inter-app communication because we assume it is a simple mapping. Better is to step back and first ask the bigger questions:
Why am I using more than one app here? This isn’t an “Is app X better than app Y?” zero-sum choice, but to properly understand what each app does that others can’t. IOW, don’t replicate unneeded information, indeed is more than one app needed. We often assume that where we started is the logic anchoring context but, after considering the process as a whole is it really getting you to your target. I’m stuck at “I like feature A in app X only but also feature B in app Y only” ask your self if like is the same as need. Don’t overrate familiarity if it stops to moving to your desired outcome.
Why replicate *all the data? Can I not replicate only the date I need?
Have you read/used the document offered here that was written to help the AI understand the structure of a TBX. Likely you’re writing everything as the AI doesn’t understand the internals of the TBX structure?
TL;DR, I’d pause on doing more automation until you have a better feel for what data belongs where and why. Gaining an understanding of that should enable you to design a better workflow. a weakness of AI is it will have a go at anything.It is less likely to tell you the overarching plan may be improved by more consideration of the needs/constraints. With deep enough pockets (to but more tokens) AI can try most things but it isn’t necessary the most useful or effective first approach.
Yes, I’m not really interested in making comparisons, but rather in combining tools.
I like to keep my data storage separate from the tools I use to work on it.
Obsidian works very well as an inbox across all my devices, and Devonthink manages the data storage brilliantly. Two separate worlds. But I obviously still need to delve a bit deeper into the TBX data structure to integrate this properly. Thanks for the tip about the data description for the AI. I’ll have a go at interviewing it.
I think what you want to do is to have the genie examine both your current Tinderbox document and the updates to your vault, and using both these sources create a fresh Tinderbox document that would better meet your needs.
As @mwra points out, just what the criteria for this merged document might be is a nice question. You’re teaching the genie to edit, which might require a greater understanding of exactly how and why you edit yourself.
Note to newcomers: @rolf911 is doing something unusual here. Instead of having the AI use MCP or scripting to read and revised a Tinderbox document, he has the genie write a fresh Tinderbox document from scratch. This is tricky in itself because it’s novel, and it necessarily depends on ill-documented aspects of the file format.
This is what the MCP interface does; it has tools to make new notes or modify existing notes.
The clever AI is given the one-off task of writing a Python script that writes a complete Obsidian Vault to a Tinderbox file. To assist with this, there is the Tinderbox file description and a link to the Excalibrain ontology.
Once the script works, no further AI support is needed.
The next Job for KI is to Write a Python Script write the modified items and attributes to an Obsidian Vault.
At the end of the process, there is a cycle between Obsidian and Tinderbox.
Since I tipped my TBX ‘for-AI-explainer’ doc into play, it is worth noting that the document is more up to date than the TBX syntax part of aTbRef on which it is based. Why? Well, making the explainer involved a lot of rinse-and-repeat black-box testing built off the aTbRef notes. This showed up lots of differences I’ve yet to reverse into aTbRef. Bear in mind, many features are written to TBX only when first used, so a blank new TBX is an incomplete XML reference listing. Plus, whilst a TBX is ‘just’ XML, changes aren’t documented in release notes as hitherto there hasn’t been a need.
IIRC, the order in which XML element attributes are written is consistent, at least since 5.x, as standardisation of (XML) attribute order was needed for the erstwhile Emberlight project.
Whilst one can build a new TBX from scratch, the clarification from @rolf911 suggests an edit/merge might be a better route as more data stored outside notes will get preserved. If nothing else it avoids a do-over of previous analytic work on each new file.
Tinderbox has no merge, but if the Obsidian notes are given a GUID (where? Note YAML front matter?) then automation (programming/scripting/AI:MCP) can use that GUID to see if a TBX note of that GUID exists and update/append to it rather that adding a new note.
Another reason to merge/add data rather than completely rebuild the TBX is if links are used. Obsidian uses post-Web style where links must have an in-text anchor and this is done using wiki-style square-bracket notion. Tinderbox offers a richer linking systems allowing both in-text links and note-to-note links (‘basic’ links). An example of the latter is when you drag a link between two note in Map view. Either way both forms are stored, hypertext-style in a linkable in the XML discrete from the per-note info. As long as merge inbound note data is appended to existing $Text, then existing basic and text type Tinderbox links should survive instant.
As this makes the planning seem more complex than it is. If the aim is to collect in Obsidian but to the thinking work in Tinderbox, then the process is broadly one way and likely additive to existing content. Thus a small amount of pre-planning will, I believe, have disproportionate gain.
thanks for the helpful hints.
arrived at the first Milestone
The PY-script generate bases on obsidian-vault and excalidraw-ontolgy the attributes and items in the tbx file. Works fine with text and now I trie to integrate the excalidraw Views of the “Backside” of my Notes.
Actual Workflow
Export Obsidian-Vault in Tinderbox Document (PY-Script)
First export Excalibrain-Ontology to Attributes
Second export Items
View in Tinderbox
use Text-Window to show Note
use Preview-Window to show Markdown-Layout, but no Grafik
use link (ID-Link) to edit Notes in Obsidian
Close Tinderbox
Update Tinderbox-Document (PY-Script)
Works with 19000 Notes and 2000 embedded Graphics (60 Seconds for update)
New Problem: Text-Window show Markdown and the Grafik. In the Preview-Window interpretet Markdown but no Grafik !?
I don’t understand the sync logic between the Text- and the Preview Window.
Thanks — that three-step breakdown helps. A follow-up on step 1 (“uses the Preview template to generate HTML or Markdown from the styled text”):
Our setup: $HTMLMarkdown is true, and $HTMLPreviewCommand points to our own Markdown→HTML converter (it handles Obsidian callouts, task lists, Mermaid, etc. that the built-in CommonMark doesn’t).
What we see: an inline image in the styled text — pasted/embedded, so it lives in the RTFD as an inline attachment — never reaches our $HTMLPreviewCommand. Not as a path, not as data, not even as the U+FFFC object-replacement character. Surrounding text and styling arrive fine. Our reading is that step 1’s styled-text→Markdown serialization has no Markdown representation for an inline image attachment and drops it before stdin.
Questions:
When $HTMLMarkdown is true, is the “Preview template” in step 1 a fixed built-in, or is it selectable/overridable — per note or per prototype, via $HTMLExportTemplate or some other attribute?
Is there any way to make an inline RTFD image survive step 1 — e.g. serialized as a data: URI inside an / , or written to a temp file and referenced?
Does any note/document context reach $HTMLPreviewCommand (env var, argument, working directory)? On 11.8 we measured argv = script path only, cwd = home, and no TBX_* environment variables.
Has any part of this preview pipeline changed since 11.8.0?
Like any attribute, $HTMLMarkdown can be set locally. To clarity, the articles here might be helpful: Inheritance of attribute values.
I think not, insofar as the app can’t guess your intent: view inline vs referenced content. But what is possible is to set up two forms of expert. One showing info, one using reference images. My suggestion is to make the first of those the default or in app use, then have a note, using cascade export to export the same content via different templates using external images.
Not in Markdown. There are down ways to do this; I had previously thought it intractable. I’ll look into it; in the interim, consider explicitly image references to a disk URL, perhaps using a Tinderbox macro for syntactic sugar.
You set the command line for $HTMLPreviewCommand, so you should be able to pass whatever arguments you like.
You can also inject metadata into the HTMLPreview template using ^value(). For example, you could set a javascript global to the string returned by $Path(this) . There is some discussion of this technique in the context of poster notes.