Customising Bookends → Tinderbox import

[macgarvin (Malcolm MacGarvin)]

Thanks for your help on this Michael, Mark et al. Really appreciated.

A quick heads up. There was some unanticipated behaviour going on with the Bookends option-command drag and drop into Tinderbox which people need to know about (c.f. Max in the comments column during the meeting)

What I thought I had discovered was a workflow where, if I created a Bookends bibliographic format that just included the RIS notes and abstract field, this was somehow allowing the Tinderbox Edict to then access the full RIS format to grab the Author, Year and Reference Title for the Tinderbox Note Title, while just putting the RIS Note and Abstract fields into the Tinderbox Note Text, which was very convenient for subsequent work. I’ll call that ‘Note 1’.

I should have been suspicious. What was actually happening, for reasons I still don’t fully understand, is that this requires the same Bookends reference to have perhaps previously—and maybe forgotten about—or certainly later, dragged into Tinderbox with Bookend’s RIS bibliographic format selected. It is that which was allowing the edict, when it next runs, to incorporate the Author, Year and Title into ‘Note 1’, but also, in a subsequent edict run, to then replace the Note 1 text (which had just the Note and Abstract fields) with whatever RIS fields were copied over into the note with the RIS bibliographic format, at which point the two Notes are identical (and the RIS Note may have been lost, if that has not been incorporated into the RIS bibliographic format in Bookends)

In order to get what I actually wanted, I suppose that I could have eg an action that on adding a note checks to see if (in my case) it has "N - " which would identify the Note as a Bookends import with a RIS note field; then extract the text associated with "N - " and puts it in a new ‘Note’ attribute (because I don’t think the default Reference prototype has a RIS Note Attribute by default) and then replace the Note body text with the contents of this newly created Note Attribute and the Abstract Attribute?

Malcolm

[eastgate]

@macgarvin

What I thought I had discovered was a workflow where, if I created a Bookends bibliographic format that just included the RIS notes and abstract field, this was somehow allowing the Tinderbox Edict to then access the full RIS format to grab the Author, Year and Reference Title for the Tinderbox Note Title, while just putting the RIS Note and Abstract fields into the Tinderbox Note Text, which was very convenient for subsequent work. I’ll call that ‘Note 1’.

My initial reaction is that I myself would take a different approach.

1. Make sure the fields of interest — Note and Abstract — are included in the Bookends’s RIS format. If not, add them: Bookends makes editing export formats fairly easy.

2. Drag the reference into Tinderbox as usual.

3. Let the edict extract the relevant fields from $ReferenceRIS using the stream methods, storing them in user attributes $Abstract or $ReferenceNotes.

4. Alternatively, use an agent to find Reference notes that have a $ReferenceRIS but not $Abstract or $ReferenceNotes, and fill the user attributes as needed.

[mwra (Mark Anderson)]

The takeaway is that defaults are just that. Knowledge tools have a lot of dark/complex corners because someone needs those features. Any at time, that might be us. So if something doesn’t just work, we might ned to pop the hood and customise the default. Doing so casts no shade on us, or the person setting the defaults: they are just that—defaults. Plus, we like to think we all work the same way: evidence suggests otherwise.

I write this recalling my own aversion, in the past, to altering app defaults as it felt annoying and/or transgressive: why do others not do as I do? I’ve stopped worrying. Read the documentation and put settings to what your work needs. Plus, use the forums that most deeper tools have, to help hone your customisation.

[macgarvin (Malcolm MacGarvin)]

@eastgate

My initial reaction is that I myself would take a different approach.

1. Make sure the fields of interest — Note and Abstract — are included in the Bookends’s RIS format. If not, add them: Bookends makes editing export formats fairly easy.
2. Drag the reference into Tinderbox as usual.
3. Let the edict extract the relevant fields from $ReferenceRIS using the stream methods, storing them in user attributes $Abstract or $ReferenceNotes.
4. Alternatively, use an agent to find Reference notes that have a $ReferenceRIS but not $Abstract or $ReferenceNotes, and fill the user attributes as needed.

Sounds v. promising. So:

1. Yes, Note and Abstract are in the Bookend’s RIS format;

2. Reference option-command drag works, and confirmed that Notes and Abstract are present and correct in $ReferenceRIS;

3. $Abstract is already present and populated with the Abstract field, so no need for action there. I’ve added a ‘Note’ Attribute to my modified Reference prototype,

4. The edict $Note=$ReferenceRIS.following("N1 - "); then populates the new $Note field … but only the first paragraph, and of course there can be multiple notes coming from Bookends, in separate paragraphs, in the N1 field.

So what would people recommend as an elegant way of dealing with this? The next field is the Abstract, and that always starts with "AB. - ". Can that be used in aid to mark the limit? Alternatively, in Bookends, it would be possible to put a special character marker at the end of the N1 field, but that is then messing with the RIS style, as the RIS format may be used other than for Tinderbox, and in a couple of years I may have forgotten a lot!

[macgarvin (Malcolm MacGarvin)]

@mwra

Plus, use the forums that most deeper tools have, to help hone your customisation.

I wondered whether your solution here Extract a block of text from text could be a way forward but I found myself getting into deep water pretty quickly.

[satikusala (Michael Becker)]

@macgarvin here is a sample file:
TBX L - Reference Parsing.tbx (266.0 KB)
My Reference parsing is in the function ‘fReferenceParse.’ My code is below (some of it I’ve commented out). Yes, you can pull the abstract in $Abstract. You can also stream-parse text to put the values wherever you want them or explode them, as we reviewed last week.

As noted, I think it best for you to first configure Bookends the way you want it and then have TBX parse it into the right spots.

function fReferenceParse(){
$CitationKey="@"+$ReferenceRIS.following("LB - ");
$Name=$CitationKey;
$DisplayExpressionCache=$CitationKey.format("l")+": "+$ArticleTitle;
$Abstract=$ReferenceRIS.following("AB - ");
$URL=$ReferenceRIS.following("UR - ");
$ItemType=$ReferenceRIS.following("TY - ");
$Term+=$ReferenceRIS.following("KW - ");
var:set vSet=$Term;
var:set vSet2=;
vSet.each(x){
vSet2+=x.capitalize;
};
$Term=vSet2;
$ZoteroURI=$ReferenceRIS.following("LC - ");
$DT3URL=$ReferenceRIS.following("LD - DT3 - ");
//if($Text==;){
//$Text="Abstract: "+$Abstract;
//$Text+="\n\n"+"Q: How does this answers my research question?\n";
//$Text+="Q: What’s the Author’s question?\n";
//$Text+="Q: What’s the Author’s answer to their question?\n";
//$Text+="Q: Are there additional helpful details?";
//};
$PublicationYear=$ReferenceRIS.following("PY - ");
$DatePublished=$ReferenceRIS.following("DA - ");
//Parse Date Accessed...
var vInput=$ReferenceRIS.following("Y2 - ");
var:list vDate=vInput.split("/");
var vYear=vDate.at(o).year;
var vMonth=vDate.at(1);
var vDay=vDate.at(2);
var:list vTime=vDate.at(3).split(":");
var vHour=vTime.at(0)-8;
var vMin=vTime.at(1);
var vSec=vTime.at(2);
$DateAccessed=date(vYear, vMonth, vDay, vHour, vMin, vSec);
//I user $URL not $ReferneceURL for everything.
//$ReferenceURL="";
//Now that text is parsed can clear RIS.
//$ReferenceRIS="";
//Disable rule to reduce processing load in the future.
/$Rule="";
$IsDormant=true;
}

TBX L - Reference Parsing.tbx (266.8 KB)

[mwra (Mark Anderson)]

OK, a number of parts to this:

1. You want the note created by the reference drag Bookends → Tinderbox to have its $Text set replaced so it only holds the notes field (RIS field N1 - 'from the source reference.

2. Not stated but I assume you want the other source reference data, just not in the note’s $Text. So that rules out making a custom RIS format that only contains the note data.

3. The RIS format is no longer actively maintained and has no formal description for field values containing multiple paragraphs (i.e. line breaks). The age of the format suggests such an eventuality was not envisaged at the time the format was invented.

4. Bookends is placing the contents of it’s ‘notes’ field into the N1 RIS tag verbatim. If the source notes contain multiple paragraphs you get this sort of thing:

Here, above, the source notes contain 3 paragraphs. It comes after the AB (abstract) field and before the UR field

Note too that in Bookends default RIS export filter, the N1 filed is not the last in the record. It is followed by the UR (URL) field. Export data fields are only populated if the source has data for that field, but at the Tinderbox end, one should assume the UR field is/may be present.
Note: Bookends default

To see info about the Bookends configuration, in Bookends use menu Biblio ▸ Formats Manager… and in the resulting dialog scroll the list of formats to find ‘RIS.fmt’. See the Bookends manual for info on that and how to edit it. N.B. Bookends has a support format and its dev is responsive to support emails.

Tempting as it might be to make a custom format … Aha! that is the solution. Whilst Bookends uses its default RIS format to put data on the modified drag (which Tinderbox puts in $ReferenceRIS). But the $Text of the notes is the content of the Formatted Reference pane in Bookends (see the app’s lower status bar)

So make a new format in Bookends (ask there for how) that only exports the notes field. I called mine “Just Notes”:

Now I set the formatted reference pane to use the new format:

As seen here:

Importantly, this has no effect on Tinderbox collecting the $ReferenceRIS data and you have the correct $Text with zero work at the Tinderbox end. Really easy, once set up.

So, having made the custom export filter in Bookends, you just remember to select it in Bookends before doing any drag/drop. Creating the new format doesn’t affect other ones in Bookends and just adds to the 300+ formats that app has already pre-configured.

Great to find the solution was so simple, though admittedly I’ve explored under the hood in Bookends a fair bit. That app is very powerful at configuring export like this. But you do have to figure out what you want and do the actual work—there is no “do it for me” button.

Before anyone asks about Zotero, Mendeley, etc., I’ve no idea as I don’t have/use those apps. It can probably be done but it needs a user of those apps to explain how.

HTH

[macgarvin (Malcolm MacGarvin)]

Thank you @mwra for spending this amount of time on my question - these detailed notes will be really helpful for someone coming from Bookends trying to work out a work-flow into Tinderbox.

Sorry to come back to this but I had earlier gone down exactly your route, except that I was including the abstract as well as my notes (only) in a bespoke Bookends format, having found that the RIS fields were getting pulled into the Tinderbox note and populating the relevant fields - ideal for me. I then created an Edict to change the Tinderbox Note Title to abbreviated Author+Year+Title. But, for reasons I don’t understand, the capture of the RIS data became unreliable, which is when I switched strategy to modifying the RIS.fmt in Bookends to include the N1 Notes field, as as you saw in the meeting.

Keeping this as simple as possible to troubleshoot, if I replicate what you outline here, and do it by dragging the reference into a virgin Tinderbox doc, I don’t get the same behaviour as you. The note puts all the fields into the title, the abstract goes into the text field as required, but the RIS attributes are not populated. That means I can’t create the title. I’m assuming that there is no way I can have modified Tinderbox’s Reference Prototype?

Here’s a screenshot in Tinderbox of my outcome following your route:

And here’s a screenshot of the Bookends input:

I’ve attached my tinderbox file, although I’m not confident that helps.

New TND doc-Notes Only Bookend BiogFormat-MacGarvin.tbx (229.9 KB)

[macgarvin (Malcolm MacGarvin)]

Thank you @satikusala - I did use your work to find out how to get the N1 RIS note field into my User ‘Note’ Attribute. And I intend to come back and explore your Reference Parsing function in the context of Bookends. But I can only cope with one ball in the air at a time and I’d first like to try and resolve why @mwra ’ solution isn’t working for me.

[mwra (Mark Anderson)]

The first part is easy. Using the model before we simply make a new custom Bookends export format (or amend the old one. This, below, exports the Abstract followed by any Notes:

As to the other issue of setting the title, I’m stumped. I can’t replicate the issue shown in your file.

But, with my new format above, I see a possibly similar thing in that whilst the $Name and $Text of the new note are as expected, none of the extra attributes are filled in and $ReferenceRIS is empty. But if I change the display format for the formatted Reference in Bookends and repeat not only does the new reference populate the $ReferenceRIS, etc., but the existing one is updated too.

The challenge for the user is this backstage co-operation between Bookends is not documented for either app. It is not clear if the drag payload includes the RIS data or whether Tinderbox is using the Bookends pseudo-protocol to poll Bookends for the data and that is the process happening above. I’m afraid as this is ‘internal’ it isn’t possible for a user to investigate in greater depth.

Quite why a different styled citation at source (Bookends’s currently selected export format) should make the drag-drop fail is unclear. The new formats work fine in Bookends. Indeed, my “Just Notes” format from yesterday works fine dragging Bookends references into your test document. But today’s “Just abstract and Notes” format doesn’t populate $ReferenceRIS. Go figure…

To replicate, you’ll use Bookends (I’m on v14.2.8) to make two new formats, in each case based on the RIS format.

• In Bookends:
  • Open menu Biblio ▸ Formats Manager…
  • In the formats list select “RIS.fmt”, and at the bottom of the list click the + button to make a new format (based on the RIS one).
    • In the New Format dialog leave settings as presented except for the name. Change “Format 1.fmt” to “Just notes.fmt”. Click the OK button.
    • The new format is added to and selected in the Formats Manager dialog.
    • In the Field Order box delete all the codes and simple add a lowercase n (‘n’ is the code for the Notes field in Bookends).
  • Now repeat the last series of steps making a new RIS-based format using the name “Just Abstract and Notes”
    • In the Field Order box, add a b (for Abstract), a Return character (via the Special Characters popup under the bottom right of the code input box—see grab above), then on a new line an n (for Notes).
• Your Bookends now has the two new format. select record(s) with both abstract and Notes data.
• Select format “Just Notes” in the status bar of Bookends and select the ‘F’ icon if not already selected. The preview should show the record’s Notes data:

• Do a Cmd+Opt+drag of the record(s) Bookends → Tinderbox
• Observe the note is made, named with the source record’s title, the note’s $Text is the contents of the source’s Notes field and various Tinderbox reference attributes are populated including $ReferenceRIS
• Now select the Just Abstract and notes" format in Bookends and repeat the procedure.
• Observe that whilst $name and $Text are as expected, the only reference attribute populated in Tinderbox is $URL (the bookends:// pseudo-protocol inter-app link).

Whilst the issue might be caused at issues at either/both apps, given what user testing has revealed, it seems the first place to check under the hood is why Tinderbox does/doesn’t complete its attribute population task given some Bookends export types. Alternatively, the formats might be obfuscating internal breakage possible due to an unescaped character in the drag data payload—or data pulled via the pseudo-protocol link.

I hope this gives @eastgate enough to go on.

I’m going to split out this into a discrete thread so the main one can still discuss the meet-up.

[mwra (Mark Anderson)]

OK, some extra wrinkles emerge if making/using Bookends custom formats for this data transfer task. Importantly, this isn’t an issue of bugs/errors, but properly capturing our intent and ensuring such changes are correctly signalled to the apps.

1. Article types in formats. Reference Managers (RM) track reference ‘types’ as a book might need a different citation style from a conference paper, from a journal article. Bookends, like other RMs, takes a ‘journal’ type as the general default and the default to use if nothing is defined for the current references type. TL;DR—if the format doesn’t vary by type, just configure one type. So, having made a new format, delete all types except ‘journal’ and customise the fields for that.

2. Recognising changes. I had some odd/inconsistent effects testing for this thread. My takeaway is that after making a new style, close the Formats Manager and re-start Bookends. It may not be strictly necessary, but this extra task should ensure the new format is saved and then read back in for subsequent use.

I’d accept that I’m possibly not using Bookends export styles quite for their primary purpose. The building blocks are that the ‘text’ of that dragged data is the contents of the Bookends ‘styled reference’ data for the selected record. That data is styled, in terms of which field(s), etc.) by the currently selected export format set in Bookends. Thus if we make a custom format (see above) that includes all the fields we’d otherwise need to parse from RIS data—which I’ll wager was never intended for multi-line/paragraph values—then it saves messing around with action in Tinderbox. Once set up [sic], the user only has to select the right export format to ‘load’ the desired fields into the data passed to Tinderbox to use in a new note’s $Text.

Plain vs. Styled text. some may have noticed that Bookends supports styled text in the Abstract and notes fields (sand likely most others). However, the data passed via reference drag-drop is, I believe, plain-text only. If wishing to copy rich/styled text, another method such as AppleScript is likely the best route.

[mwra (Mark Anderson)]

Meanwhile, off-forum diagnosis of why some import is failing is continuing.

[mwra (Mark Anderson)]

I attach a zip of the two Bookends custom formats made for testing in this thread: Archive.zip (3.6 KB)

The zip contains files “Just Abstract and Notes.fmt” and “Just Notes.fmt”. To use them opening your Bookends support folder ^†, and place the files in the Custom Formats sub-folder, making that folder if not already present.

After adding the files and re-start Bookends. You can check if the formats have been found by opening the Format Manager dialog and checking the list of averrable formats.

The fields offered here arose for the original porter’s requirements. But, if you require more/different fields or a different order, the same idea holds true: make a new custom format in Bookends.

†. This is normally at ~/Library/Application Support/Bookends, but may be elsewhere if you have customised this. See the Bookends app’s menu Bookends ▸ Settings… and look at the General tab’s Folder Locations section.

[macgarvin (Malcolm MacGarvin)]

@mwra

I attach a zip of the two Bookends custom formats made for testing in this thread: Archive.zip (3.6 KB)

The zip contains files “Just Abstract and Notes.fmt” and “Just Notes.fmt”. To use them opening your Bookends support folder †, and place the files in the Custom Formats sub-folder, making that folder if not already present.

After adding the files and re-start Bookends. You can check if the formats have been found by opening the Format Manager dialog and checking the list of averrable formats.

@mwra - Using your formats imported into Bookends I can confirm that the behaviour is as you describe, and I now understand why I was baffled. I had been working with a biographic format with just the Note field, which served my purpose. I then experimented by adding the abstract, but when I dragged to Tinderbox, the same reference with just the note field was still present. When I later dragged new references with notes and abstract and encountered problems, I did not make the connection.

I can also report that using your formats the Edict I adapted for making the Note Title -

works as intended with some RIS formats (e.g. Journal, Book) but not with others that I have added using the field codes (e.g. Newspaper) as set out in the Wikipedia RIS formats entry. That’s because – in that case, Newspapers – it uses three fields, T1, T2 and T3 to deal with article title, ‘short’ article title/strap line and Newspaper section. Tinderbox puts the T2 newspaper section field into the Note Title rather than T1. I have attached a Bookends Library with one entry for each of the File Types I have adapted the RIS format, and a copy of the modified RIS.fmt file. I’ve checked that when this custom RIS.fmt file is deleted, Bookends reverts to the default RIS!

Archive.zip (81.5 KB)

[rogerbackhouse]

Thank you all for this very helpful thread, which explains how I can solve a problem I had. In BE I used User20 to hold a DevonThinkURL, enabling me to go directly to the DTP note corresponding to the reference. I had not discovered the $ReferenceRIS attribute or the command to extract data from it, and did not realise that User20 could be transferred by editing the RIS format So I had been copying User20 into the Abstract (which I do not normally use) and then, in TB, using a stamp to copy that into $DevonThinkURL. Now I know can do it using the $ReferenceRIS field, which is much neater.

However, I still have an issue.

The following stamp works perfectly:

$DevonThinkURL=$ReferenceRIS.following("UU - ")

UU is of course the field I defined in the RIS file to hold User20. However, would also like the Text of the note to contain just the BE Notes and not all the RIS information or a formatted reference (I have no idea why I used to get a formatted reference but now I get the RIS data by default), so I added the line,

$Text=$ReferenceRIS.following("N1 - ")

This sometimes does nothing, though sometimes it has cleared made the text blank. Maybe it is better to have a command such as

$Text=$Text.some expression to delete all RIS data except the BE Notes.

A supplementary question, which is still on the topic of the thread is that I currently have several stamps, Rename journal article, Rename book, Rename edited book (and will add Rename chapter in due course) and I apply whichever one is relevant. There will be a way to combine them all, so that if the reference type is Book, it will apply one set of commands, if it is a journal article it will do another, and so on, but I have not worked out how to do this. It was easier just to have several stamps.

[satikusala (Michael Becker)]

It may be tough to zero in on the answer without seeing the data.

@rogerbackhouse

$Text=$ReferenceRIS.following("N1 - ")

If this works sometimes and not other times, I suspect that the problem is in the data itself. It is possible that an unexpected character is sneaking int.

One thing you could try to keep from text getting overwritten is do add a condition, perhaps something like this:

var:string vText=$ReferenceRIS.following("N1 - ");
//only overwrite $Text if vText has an actual value.
if(vText!=""){
   $Text=vText;
};

Again, hard to test without data.

@rogerbackhouse

f the reference type is Book,

I don’t know the name of the reference attribute so for this example I’ll assume it is $RefType.

Create a stamp named ‘Consolidated Reference Renaming’, or something like that.

if($RefType=="Book"){stamp{"Rename Book");};
if($RefType=="Journal"){stamp{"Rename Journal");};
if($RefType=="Article"){stamp{"Rename Article");};

This way, you can keep the individual stamp as well as have the consolidated one.

[mwra (Mark Anderson)]

The RIS getting into $Text is user error at the Bookends end. Tinderbox gets the Bookends’s styled reference info from the drag payload and writes it to $text. If you are getting RIS info in $Text it suggests either of:

• Your Bookends export format is set to RIS
• You are running not-default code (i.e. code you added) that is setting $Text to the contents of $ReferenceRIS.

Stepping back, I sense you are building on sand by trying to fix broken past experiments without cleaning out old code. At this point, to help see the problem clearly, I’d build a test doc (so no code for past tests). Check your Bookends has the correct export format set (as I’ve explained in past posts or see the Bookends manual).

Forgetting the failed past experiments, can you list what you want to happen that differs from Tinderbox defaults and taking into account that $Text is a result of settings you make in Bookends (if the Bookends default isn’t what you want, choose accordingly). That way you have a test doc and clear pass/fail state to achieve. Once you have that working you can go back to your main doc and purge all the failed experiments, if necessary deleting and re-adding prototypes you may have messed with during pat experiments. Otherwise you will continue to get odd errors as abandoned code makes its presence felt. You need a clean experiment to help figure out what in your main doc is a process fail and what is damage from past experiments. Fixing damage to existing records may require some close attention, but let’s leave that until we know it is even necessary.

[rogerbackhouse]

Thank you for these responses, Michael and Mark. Between you, you have solved all three problems.

Yes, I had changed the default reference type to RIS, mistakenly thinking that this was needed for export to TB. Problem solved.

The solution for putting separate stamps into a stamp with if statements is so obvious I should have thought of it. Blindness on my part.

And setting the default bibliography format to the “Just notes” format that you posted earlier solves the problem of getting the Text as I want it. Given that the content of the BE Notes is imported into $ReferenceRIS, it is still a puzzle why I cannot overwrite whatever is in $Text by copying the notes from $ReferenceRIS, so that the default export format in BE is irrelevant and can be anything. But as I rarely use the default format for anything else, and it is easy to change it, that is not really an issue.

Incidentally, there is a simpler way to get formats into the right place for Bookends to use them. Just double click on the file and it is automatically put in the right place and made the default.

Thanks again.

[mwra (Mark Anderson)]

@rogerbackhouse

it is still a puzzle why I cannot overwrite whatever is in $Text by copying the notes from $ReferenceRIS

OK, Tinderbox works on the dropped-data note once at creation. Running the stamp, edict (i.e. action code) to modify $Text needs to occur after Tinderbox has done its on-note-creation work or it might over-write your action code’s result.

So your conjecture is wrong. You can set $Text using data read from $ReferenceRIS, but do it after the note is initially created. If your code isn’t populating $Text properly, likely it’s an error in your code (even if not deliberate.

The latter possibility is why, now I’ve discovered the possibilities at the Bookends, it makes no sense to do srting spelunking at the Tinderbox end when you can far more easily configure the data you want in $Text in Bookends and not have to worry about action code in Tinderbox. Sure you need to open the manual, but the process isn’t hard, Bookends has a support forum (better to discuss the app in its ‘home’ context) and Jon, the app’s, dev is helpful and responsive—and knows a lot about reference wrangling as Bookends started on Mac System 7 OS. We can use action code in Tinderbox post import, but why make life harder for ourselves?

[rogerbackhouse]

I was running the Stamp after the import was complete. But you have persuaded me that it is easier to change the default format in Bookends on the few occasions when I want to drop a formatted reference, and not the code, into a word processor or some other app.