Modifying the RIS file export from Zotero

The linked file part is answered by the last but one answer in the thread you cite and confirmed as working by the next answer there.

As this appears to relate to customisation inside Zotero, you might get more answers on the Zotero forum. Why not PM the thread’s originator on that forum? @lyndondrake does seem to have an account here but hasn’t posted here since 2018.

I’m still lurking :slight_smile:

Hi! Can you shed any light on what @mpom needs to do to replicate what you’ve done importing Zotero data into Tinderbox (as hinted at in the Zotero forum thread linked at the start of the thread)? TIA. :slight_smile:

I hope to - I’m swamped today but will have a look tomorrow

Hey there, great question. I have so much success with this, it is an unbelievable boon to my workflow. Here is a link on the Zotero forum to the RIS translator that I’ve found works great (just one line code change to the standard RIS translator), emilianoeheyns - Zotero Forums.

As for the URI, to the best of my knowledge, this requires a special export translator. So if you use that you can use your RIS translator. I have a post in the zotero forum on this. I’ll let you know when I hear back

Thanks very much to you all for the responses!
Yes, this would be very helpful.
Yours,
Maurice

RIS LD.js.zip (33.0 KB)

Here’s my modified exporter. The diff to the canonical one in Zotero is now quite cluttered as their one keeps being updated. I’m not really up to speed with Git otherwise no doubt I could integrate their changes, but hopefully this is still of some use.

1 Like

Hey there, I see what you’ve done. I was able to get the uri as well, but in a slightly different way. But, when I click on the uri it will open Zotero, but it does to actually take me to the citation. Does it work for you? If so, what did you do? Once I get mine working, I’ll show you what else I’m doing, e.g. I have a killer integration between zotero, tinderbox, pandoc, and keyboard maestro. It makes citation management a breeze.

BTW, here is some other code you might want to look at. I have an Edict that processes an RIS reference when dragged in.

You’ll note how I’m parsing $ReferenceRIS with the .following attribute. I’m also doing few other things that work for my workflow.

if($CitationKey==""&$ReferenceRIS!=""){$CitationKey="@"+$ReferenceRIS.following("LB - ");};

if($CitationKey!=""){$Name=$CitationKey};

if($ItemType==""&$ReferenceRIS!=""){$ItemType=$ReferenceRIS.following("TY - ");};

$Tags=$Tags+$ReferenceRIS.following("KW - ");

$ZoteroURI=$ReferenceRIS.following("LC - ");

$Name=$CitationKey;

$Text="Abstract: "+$Abstract;

$Text=$Text+"\n"+"\n"+

"Q: How does this answers my research question?

Q: What’s the Author’s question?

Q: What’s the Author’s answer to their question?

Q: Are there additional helpful details?"+;

$DatePublished=$ReferenceRIS.following("DA - ");

$DateAccessed=$ReferenceRIS.following("Y2 - ");

$PublicationYear=$ReferenceRIS.following("PY - ");

$Tags=$Tags+$ReferenceRIS.following("KW - ");

$EdictDisabled=true;
1 Like

Hi Again,
Thanks very much to @lyndondrake. This works very well!

And @satikusala thanks very much for your further ideas on importing the data into TB.

I wanted to share a way that I’ve been importing the Notes section from Zotero (now very easily populated with notes/highlights on Mac and ioS courtesy of Zotero’s new pdf reading pane). The notes are exportable from the RIS Export in the N1 field. I convert the HTML to Markdown using pandoc thusly:

$MyString=runCommand("/usr/local/bin/pandoc -t markdown -f html --lua-filter=remove-attr.lua", $MyString)

As you will see, the pandoc command includes a lua filter to get rid of attributes that I don’t need see HTML to Markdown Remove Attributes Lua Filter

I find this even more straightforward than using an app like Highlights to convert my highlights to MD. I think there are some further refinements I could make by using more of the HTML code which I believe indicates the very location on the pdf that the highlight comes from. But that’s for another post…

Yours, Maurice

1 Like

Cool!!! Any chance you can share a copy of your filter?

Also, for those that may not understand the details of this thread, what @mpom is suggesting is quite cool. The current beta version of Zotero (pdf_reader [Zotero Documentation]) lets you preview and highlight PDFs in Zotero.

These highlights can then be pulled over using Zotero’s RIS export. They show up in the $ReferenceRIS attribute. In the image below I’ve pulled the values in my not into text to show what is there.

.

As noted, the highlighted text from the PDF is populated in the Zotero N1 attribute.

From here you can use the aforementioned LUA filter (more on LUA here: Lua: getting started; LUA with Pandoc here: with pandoc Pandoc - Pandoc Lua Filters) to pull out what you want.

This is a rather neat option if you want your workflow to be contained to Zotero and Tinderbox. One consideration that comes to mind, however, is this workflow does not take advantage of Devonthinks natural language processing for later research and search. Nonetheless, this is very cool idea.

1 Like

This is next-level cool. If it could pick up the link back to Zotero to view that would be the perfect workflow tbh…

The other thing about this workflow is the Beta version of Zotero for iOS has the highlighting capability. (I’d have to purchase some Zotero storage to have enough capacity for all my PDFs, to be fair, but still potentially feasible. At the moment I have them all as links into a separate folder.)

In my experience, with thousands of highlights in DevonTHINK, I don’t actually use DT’s search that much on the highlights/notes. I find DT search has its biggest bang for buck at finding things in PDFs I hadn’t thought to look in for a particular subject.

This starts to make the combination of just the two tools — Zotero and Tinderbox — highly attractive for the classic challenge of sense-making.

Hi Michael,
Sure. It is mentioned in the thread I linked to:
Save the script below as remove-attr.lua in your text editor and place it in the pandoc data directory (mine is in ~/.local/share/pandoc/filters):
remove-attr.lua.zip (650 Bytes)
NB: You will have to unzip it.

BTW these lua filters are very powerful and useful to use alongside of pandoc.

@lyndondrake I think the syntax for linking back to the files may have changed in Zotero 5, but linking back to the pdf (or local Zotero item) should be possible. One refinement which would be great would be linking back to the actual location in the pdf. I think you might be able to do this from the html code that is exported in the RIS file under notes which the script I posted currently strips away. I’m just not sure how to handle this piece yet. If anyone has any ideas, I’d be very appreciative.

Just to endorse. I’m an academic, working on a book, been working with Michael Becker @satikusala on my system, same one he recommends here. It is absolutely fantastic. We (by which I mean he :wink: have stripped out a lot of the baggage in a Zettlkasten system to make it more efficient, cleaner, integrated with Zotero, Devonthink, Pandocs (I’m not using yet) and of course Tinderbox. I can’t recommend Michael’s system highly enough.

2 Likes

Thanks! BTW, I’m not sure about Zotero, but the integration I have with Highlights and/or Devonthink lets me link back to the spot in the doc.

Can you help? How do I find this directory, not sure. Thx.

The giveaway here is the full stop at the beginning of a file or folder name. On Unix systems (and via that means macOS), a full stop at the beginning of file or folder name denotes it as hidden. IOW, it does not show up in normal Finder (or Terminal) file listings.

OK so how do we to ‘see’ this location in Finder? We could unhide all hidden files but that would simply make you realise we some stuff is hidden ‘behind the curtain’: it is not because it is ‘secret’ but is simply organisational stuff that needs to exist but most of us rarely, if ever, need to see.

Another general point to bear in mind is although the full stop in .local hides it from normal view, its contents themselves are not hidden except for the fact you cannot visually navigate to them. There would give a different experience:

~/.local/share/pandoc/filters

and

~/.local/.share/.pandoc/.filters

In the latter case everything is hidden in its own right, in the former only .local is hidden.

So, how do we show a hidden OS folder? Open a new Finder window and from the Finder Go menu, choose Go to Folder… or use shortcut ⇧+⌘+G. A drop-down pane opens. If the feature has been used previously, the last used path (as seen here) is inserted and selected so you can easily type in a new value:

Here you can see the results of a previous test of showing the ~/.local/share folder. As it happens I don’t have pandoc data in this location, but look what happens when I use the above path and click the Go button:

Note that I don’t have a pandoc folder there but I do have one for an app called RStudio. notice too that how the ‘local’ folder (in the second column) is slightly faded out. This indicates the folder is OS-hidden (whilst other hidden folders/files are simply not shown). The above also shows that the default Finder view of icons might be less helpful. This that view, the last looks like:

To me, that is a far less informative Finder view choice in this scenario. With column view, the path we are trying to find is made much clearer. I note this in passing as I think we all tend to get stuck in the rut of re-using the same finder view and not using other views when more pertinent. Actually the same could be said of some Tinderbox use. :slight_smile:

†. Aside, I I’ve noticed an unhelpful behaviour creep in whereby the utility named ‘pandoc’ is being mis-described as ‘pandocs’: there is no spurious ‘s’ on the end, see here. My point isn’t pedantry to to disparate others (or a side-swipe at Michael), it is simply about helping the less informed by clarity of communication. Conversationally it doesn’t matter, and Google probably has enough training material to see through the labelling error: but, in a scenario like above, Finder or Terminal won’t find a mislabelled pandocs folder when what is wanted is a pandoc folder (blame computers for being over-precise, if you must!). So folks, it is ‘pandoc’ not ‘pandocs’. Such slang is much harder spot than say, referring to Tinderbox as ‘TB’ or ‘TBX’ where the difference is more self-evident. Indeed, I just wasted a lot of time looking for an under-the-hood Unix utility that was misdescribed to me; had I been able to ‘see’ all content I’d likely have found the right thing, but it was under a hidden folder!

Reading your discussion, I’m trying to reproduce what I saw in last Tinderbox meetup, when Michael Becker showed us how to use both Tinderbox and Zotero, dragging an item from Zotero to Tinderbox. I’m very close to succeed in unifying my note-taking. But, completely new to coding, I have some questions:

  1. I have the Better BibTex Zotero add-on, but I don’t know where to put the code above Modifying the RIS file export from Zotero - #8 by lyndondrake Do I need to insert it into Zotero or into Tinderbox Action Inspector?

At the moment, when I export a Zotero item into Tinderbox, this is what I get:

b

No Tags, no CitationKey, no ReferenceURL and a confusion between ArticleTitle and BookTitle.

  1. How to modify the ris file export in order to see those three attributes filled in Tinderbox as illustrated in this picture:

I hope my questions are not too much off Tinderbox topic.
Thanks for your help.

Dear Dominique,
You insert the file RIS LD.js file into the ~/Zotero/translators location (the ~ indicates your home folder). This should normally be /Users/your-user-name-here/Zotero/translators according to the standard Zotero install.

Then restart Zotero if it is open. You can now select the format RIS LD by heading to Zotero Preferences/Export and setting the Quick Copy to RIS LD.

I’m sorry that I wasn’t in attendance at the meeting to see exactly what Michael Becker explained. But this should get you an RIS file that looks something like this, which you can export (and save to your local drive by right-clicking in Zotero and selecting “Export item…”

The RIS file (adjusted by Lyndon Drake) should look something near to this:

TY - JOUR
TI - La negation d’Eros ou le 'isq d’apres deux epitres d’al-Gahiz
AU - Cheikh-Moussa, Abdallah
T2 - Studia Islamica
DA - 1990///
PY - 1990
DO - 10.2307/1595776
DP - DOI.org (Crossref)
IS - 72
SP - 71
J2 - Studia Islamica
LA - fr
SN - 05855292
UR - La négation d'Éros ou le 'išq d'après deux épîtres d'al-Ǧâḥiẓ on JSTOR
Y2 - 2021/06/21/07:48:40
Z1 - Cheikh-Moussa1990a
Z2 - zotero://select/items/2478
Z3 - http://zotero.org/users/7020248/items/2K8IE3AD
ER -

This file can be accessed in Tinderbox in several ways. I think the easiest way is to utilize the ReferenceDictionary newly available in 9. I’m sure others could explain this far better than I can.

As for the link types, I’ll hopefully be back in touch with more about this. I think I’ve found a way to get a link back to Zotero that will open the local copy of the file with a zotero:// link. As for keywords, I’ll need to do some experimenting, but I think it shouldn’t be a problem so long as they are being exported by the RIS file. It would be remarkable to have a way of gathering notes/links/keywords straight from Zotero’s new pdf viewer especially because it is now working on ios/Ipad and makes bringing the notes over to the Mac very easy.

Good luck,
Maurice

Thank you Maurice for the time you spent to explain me the process. It’s very kind of you. Your explanations are precisely what I was looking for. Michael Becker uploaded a new video about it and also a new ris.js: Tinderbox Training Video 57 - Working with Tinderbox and Zotero
As you can see, now I am able to export my references into Tinderbox. Thanks a lot!

ref

1 Like

BTW, I’ve figured out how to pass additional custom fields from Zotero to TBX. Add a tag in the EXT field, like so:

Modify the Zotero RIS translator, like so:
image

Modify your Reference $Edict, like so:

Create user attributes, e.g. $DT3URL, drag over your RIS export to TBX and you get this:

1 Like