Toward emergent structure

Q & A - getting started with Tinderbox
21

Thank you for the generous offer. Not relevant to my situation, though, as I use DT almost entirely as a repository for static data, primarily imported PDFs.

22

This remeinds me. @andreas has a wonderful workflow that has DevonTHINK be the source of truth for some notes and keeps TBX updated. We demoed it a couple weeks back at a meetup. Perhaps he can review it again this weekend.

23

Fifth step. I’ve now got notes clustered on topical Adornments. Time to start writing, which means time to extract the structure I’ve created to Scrivener.

(I have not yet had a chance to look at @mwra’s exercise files for basic exports, here: Exercise files for learning basic export . Clearly they need to go to the top of my reading list.)

First attempt: Select notes, use File→Export→As Text→Scrivener. OPML file appears. But it’s empty. Huh?

(At this point my deadline is creating too much urgency to investigate the issue. Moving on…)

Second attempt: Select notes. Become confused by the order in which they appear in the “Text” sidebar in Map View. It isn’t the order in which they were selected or the order in which they appear in Outline view. Make a note to investigate later. File → Export → As Text → RTF. That works. Repeat for each topical cluster. Done.

24

Food for thought, it is at this point that I dove headlong into Tinderbox. I tried using Scrivener for years. I found that drift that occurred between my notes and my writing become unmanageable, so I built out templates that helped me automate structure (inc. linear docs, bullets lists, nested tables, media, indexes, glossaries) and output (HTML pages, docx, xlsx, pptx, markdown, json, csv…pretty much anything). To accomplish this I mentally consider 4 types of types: raw input, atomic, writing, and functional (have discussed these several times in meetups). With the templating, action code automation, and this mental contract all the thinking can be contained in Tinderbox. I only go out to “output software” when I want to publish and or need to collaborate with someone that is unfamiliar with the ecosystem. The whole process is extremely powerful, and dare I say liberating. However, have said this, it is not without it cognitive overload at time. Again, there are tons of meetups reviewing all of the above. Happy to do it again or jump on a private DM call with anyone interested in seeing the workflow.

25

I’m extremely comfortable in Scrivener. I am not at all comfortable in Tinderbox. While your observation is noted, I’m unlikely to take that path anytime soon.

1 Like
26

Completely get it. Writing time is not the moment to experiment with tools. IMO, flow beats architecture every time. I’ve figured out something that works for me, but it’s definitely not for everyone. :blush:

Scrivener in skilled hands is a seriously powerful platform. I know many community members who use it well — I just never quite found my groove there.

If you’d ever be open to sharing your workflow in a meetup sometime — no rush at all — I’d genuinely love to see how the pieces come together for you.

27

Resources, lessons learned, and future work

  • Thanks again to @mwra for the excellent aTBRef. In particular
    Full Operator List
    DEVONthink
    Action Code

  • Backups are good! (I thought I knew this already. :roll_eyes: )

  • Allow plenty of time! (This too. :roll_eyes: )

  • Experiment with watch folders and ways to adjust the information that is shared. Test DT → TBX updates, report bug if one exists.

  • Experiment with ways to import notes captured in other applications, in particular on the iPad.

  • Figure out Export. Start here: Exercise files for learning basic export

  • Experiment with alternative views, ways to scale to larger numbers of notes.

1 Like
28

Don’t worry unduly about Export until/unless you need it. Also, my Export exercises are for what used to be called HTML Export. Now it is called just Export as the technique allows pretty much any form of plain text-based markup. However, there are ways to export to DOCX or RTF. Those methods are not as powerful in scope as (HTML) Export but might suffice for Tinderbox to Scrivener. See more here.

29

Looking at this a little more closely, exporting “Entire Document” using this command successfully creates a Scrivener-importable .OPML file. Exporting “Selected Notes” creates an OPML file that Scrivener can’t read. Examining it with TextEdit, it contains the title of the first selected note, but that’s it.

What am I missing?

30

Forgetting who’s on what version, it might be pertinent that ‘Export Selected Note’ could only export the first selected note until v9.5.0. From that version each selected note is exported: see more on this feature. In the latter ‘legacy’ is pre-v9.5.0.

Aha, I see the issue. You are not missing anything as such. But you have unintentionally stumbled onto a dark corner where things have moved on a bit. In short, the OPML and the Scrivener built-in templates were designed for whole doc export—over a decade back and when the focus on such use was different†. For single note export:

  • add a new template to your document (in the /Templates container)
  • call the new template ‘Scrivener-note’.
  • delete any $Text context added by the app/prototype.
  • paste(-and-match-style) this code to $Text
^action($HTMLExportExtension=".opml")^
<opml version="1.0">
<head>
<title>^value(attributeEncode($Name))^</title>
<expansionState>0</expansionState>
</head>
<body>
<outline^if(Text)^ text="^value(attributeEncode($Name))^" _note="^value(attributeEncode($Text))^"^else^ text="^value(attributeEncode($Name))^"^endIf^^if($Checked)^ _status="checked"^endIf^>
</body>
</opml>
  • Set this template for notes where you want to export selections of note(s) to Scrivener.

This is a workaround for current templates. But, don’t rush to assume error. this isn’t a zero sum app is right vs. user is right. The app can (be configured to) do what the user wants. The user being able to signal the right contextual use to the app is less simple than in the mind of the user and that’s something more experienced users of the app might need to ponder on before there is a more permanent ‘fix’.

†.Checking back it is some years since OPML was discussed by users. A further complication that many miss is that whilst OPML is a standard of sorts, many apps that claim OPML import/export actually use undocumented customisation, requiring the user to tinker a bit to get a standard OPML template to work for [app name]'s implementation of it. Again not an issue of blame, it’s just how apps roll. Markdown is a similar pot-mess of conflicting 'flavour’s of the original (not is there even a defined Markdown standard).

1 Like
31

For me, the best two applications I found for this are DEVONThink and Obsdiain, as they work great on mobile and there are clean workflows for getting those notes into Tinderbox. As you point out, “sync” in our traditional expections can be out of sync.

32

If the app explicitly offers a “Selected Notes” menu option, it’s not unreasonable for the user to expect it to work.

33

I was being kind, in fact the scenario was user error [sic]. Selection Export does work, but the as the user did not bother to check the template, and then used in the context for which it was not intended, the outcome shouldn’t be a surprise. If the user had bothered to take a look at how OPML works, their poor choice would be become evident.

But, as I tried to point out, a blame game isn’t productive. I guess some folk always have to win.

34

The templates are effectively the same as those for general OPML export but labelled so as to assist the more tech-averse Tinderbox/Scrivener user looking to export data from one to the other. (Emphasis mine.)

  • Scrivener. Exports the selected scope in Scrivener (OPML) format. If necessary, the relevant built-in templates and prototype are added to the document to facilitate the export before it occurs. (Emphasis mine.)

Yes, I concede that I configured the export erroneously. However I’m not sure, based on the resources linked above, exactly how I was supposed to know that.

I’m not interested in “winning.” I am interested in using Tinderbox effectively, which includes pointing out areas where I find doing so especially difficult.

35

Yes, but combative questions and disparagement of the documentation doesn’t encourage people to give their free time and expertise to help. This is a user-to-user forum not company tech support so the help you get is other people’s goodwill.

A toolbox app like Tinderbox has a vast number of combinations, more than can be pre-tested and even within a given feature it’s hard to know, and thus pre-test, exactly how users will combine features. Meanwhile, the underlying OS, and people’s wider experience and thus perspective changes all the time. Therefore unexpected things may occur, but they aren’t necessarily bugs or negligence. A ‘bug’ is some code that does not work as it was designed to do. Occasions where something doesn’t work as the user expects is a concern but it is not a true ‘bug’: a common misconception. This why the phrasing of your questions is unhelpful. You don’t know enough about the app to make the assertions you do.

To put the OPML issue in perspective, I recall OPML last being a topic of interest about 10 years ago, and Scrivener is rarely mentioned except as an alternative writing tool. But, a way back some people wanted to move data between Tinderbox and Scrivener . Although OPML import has been supported since v3 at least (Nov 2005), features to OPML export—which requires some special text re-encoding appears in v4.6.0 (March 2009). Built-in OPML export templates arrived in November 2010, but ‘export selected note’ was not added until v6 in May 2014. That only exported a single note (first one, if a multiple selection), exporting a multiple selection only arrive with v9.5.0 (December 2022). Pre-made templates to assist configuring export to scrivener came in v5.10.0 (December 2011). Scrivener import/export was also reworked in the

So given little evidence of sustained use of OPML export since the early 2010s and the completely separate evolution of the ‘Export selected note’ †, it doesn’t surprise me at all that some disconnect had occurred between un-related tools in the tool box. That said you had a fix within hours of asking (I posted template code) and the Backstage crew are already looking at replacement versions for the built-in Scrivener/OPML‡ and likely that will roll with the next public release of the app when it comes.

I’ve only been using software for some 40 years but in many cases you’d be lucky to get any communication with an app’s developer or community. So, I think the claims of things being rubbishy are a bit strong. Yes, it’s frustrating if our first guess at a solution didn’t work, but even experienced user know that in an open-ended setting things may not always work as we assume they do. Happily, barring bona fide bugs—thankfully rare—most ‘unexpected’ have a solution or temporary workaround, q.v. your own OPML export issue.

†. FWIW, I recall this as a feature request by me as it made it easier to export a single aTbRef page if I spotted a typo or such. A hidden and powerful touch is that if the TBX has an export location set, the exported page automatically replaces or fills the location it would use on a full-document export. It was only after having the feature that its use for other export purposes—such as you exporting a single OPML note—came into use.

‡. Scrivener uses OPML, but as I previously noted few OPML-using apps follow the published OPML standard so can require specific non-standard coding in order for OPML to ‘just work’ with a given app. Why those apps choose [sic] to not follow the standard, you’d have to ask their developers

36

@kderbyshire, if I understand your first step correctly, you have:

  • PDFs of research articles imported into DEVONthink
  • notes on these articles as note records in DEVONthink

These PDFs and note records in DEVONthink represent your source of truth and you want to bring the notes over to Tinderbox in order to further work with your notes (analyse, filter/group by topic, arrange on a map, etc). Important metadata from your DEVONthink notes (like their tags) should also get transferred over to Tinderbox.

In addition, when changing a note’s tags or other metadata in DEVONthink, the corresponding note in Tinderbox should get updated and reflect these changes.

If that is what you would like to achieve, I’ve written two scripts that will do exactly that(*):

If your DEVONthink notes originally stem from annotations stored inside your PDF(s), you could use my DEVONthink Notes from PDF Annotations script to extract your annotations as DEVONthink notes. The script will transfer the annotation’s highlight color and can also transfer metadata such as a flag, rating, tags or custom metadata, and it can automatically fetch the article’s bibliographic metadata and create a deep link that points back to the PDF annotation.

In order to then export your DEVONthink notes to Tinderbox, select all of them (or simply select their hosting group(s)) and run my Exporting notes from DEVONthink to Tinderbox script. This will create corresponding Tinderbox notes for each of your DEVONthink notes. For each DEVONthink note, the script will also transfer its tags, flag, rating, color & label as well as many custom metadata, it can create Tinderbox links from Wiki links, and create deep links that directly point back to the DEVONthink record & PDF annotation.

If you change your note(s) in DEVONthink, simply select them and run the script again. This will update the corresponding Tinderbox notes so that they reflect the changes you’ve made to your notes in DEVONthink.

(*) That said, note that changing the note’s title will currently create a new note in Tinderbox (and not update the existing one) as the note’s path (which includes its title) is used to identify the note in Tinderbox. A future version of the script could use the DEVONthink note’s ID to lookup the note in Tinderbox which would allow to also change the note’s title.

2 Likes
37

The PDFs are in DevonThink, but the notes generally are not. I don’t find DT a congenial notetaking environment. But thank you for the suggestion.

38

Ah, I see, sorry for the confusion.

Just to be clear, the outlined approach would also work for any kind of workflow where your notes start from PDF annotations, created outside of DEVONthink. E.g., you’d annotate your PDFs with, say, GoodNotes on iPad and then export the PDFs and import them into DEVONthink.

The extracted annotation notes in DEVONthink could also be viewed as transient/intermediary and you could have your “source of truth” in Tinderbox.