File-crunching, Importing, Notes-crunching


(Andrew McDowell) #21

Thanks Mark, Paul and all others. You’ve boosted me the right direction. I think maybe I glimpse the path ahead dimly now. It appears to me that what I need to do is import some ‘stuff’, play with it, get familiar with Tinderbox and the kinds of things you can create with it, the tools available (you’ve given me some hints). Begin building ‘my system’, then as I understand more and gain familiarity with the tools and methods, work (play) at how best to approach prepping/importing additional material and how best to work it into the system. The first step on this path is to install Mac OS, Tinderbox, probably DEVONthink and BBEdit, which I hope to accomplish this week.

In a broad way, what I’m hearing here is, “Just start, and have faith that with Tinderbox, a way will be found”. From this thread and the modest amount of studying I’ve done, looking in from the outside, I begin to believe that a way will be found.

Not trying to cut this short. I’d certainly welcome more comments and advice, especially (at this point) regarding DEVONthink and how it’s best used in conjunction with a TBx project. But, I no longer feel lost and alone in the dark here, and I think I can go forward with first steps. I’m sure I’ll be back with more questions as I proceed. I’m VERY impressed with this community. Very professional, intelligent and helpful.

Thanks again,
Andy


(Paul Walters) #22

One parting comment as you head off, Andrew – add to your wish list a copy of Mark Bernstein’s 3rd edition of The Tinderbox Way, recently released. Your most recent post echoed a lot of Mark’s advice about learning and using Tinderbox.

Please come back in the future and let us all know about your progress.


(James Fallows) #23

The suggestions in this thread are rich and wise; I think you’ll find them useful guidance.

Also, I endorse the recommendation of the third edition of The Tinderbox Way. It’s part practical advice and part philosophy-of-info-management, each of which sounds relevant in your situation.


(Andrew McDowell) #24

Thanks Paul and James,
I’m on page 18 of The Tinderbox Way as we speak, and even in the early pages I see the echo of my own yearning for a better way, and each page, as well as this thread, make me more relaxed that I’ve made a good investment (already downloaded TBx even though I don’t have Mac OS yet). The way Mark B. talks about notes/thoughts smacks exactly of how I want to use them.
I’m sure I’ll need more help, so I’ll be back with more queries and reports as I progress.
Andy


(Rick) #25

I would like to echo PaulWalters’ suggestion to download DevonThink. I would suggest you do it without (much) further ado once you have macOS installed for a few reasons. Firstly, as mentioned earlier in the thread, there is a generous trial period (which I believe can even be extended on request). Secondly, also as mentioned earlier, whether you just index files in DevonThink or import them, they are not converted to proprietary formats; even importing just means saving them in Finder folders organized by DevonThink, so they can be accessed even if DevonThink doesn’t work for some reason.

The above two reasons are not really positive reasons to go with DevonThink, just neutralisations of possible disadvantages. But the third one is the real reason to go for it: At present, you don’t have any experience with macOS, and you need to get the files onto your new computer or virtual machine. Of course, you could just keep them in the Finder until you’ve figured out what to do, but I think you need them somewhere where you can start organizing them, if only in a rudimentary way, before bringing them into Tinderbox. You also need to be able to search them before you have your system going in Tinderbox, and it is preferable that the search functions should be sophisticated. DevonThink uses latent semantic analysis, or something similar, to analyse the content of documents, and one of its most prized commands is “See Related Text”; using that, you can select a passage of text that particularly interests you in one document, and DevonThink will show you similar passages in other documents. If nothing else, it seems to me that that will be useful in choosing what documents to import into Tinderbox first.

When importing into DevonThink, you can set up a basic folder structure first (I’m traditional, so I would recommend this), or you can Command-Select multiple documents and click Group Items to create new folders and put the documents in them. You can even set up a folder structure with an initial set of documents, then rely on DevonThink’s auto-classify feature when importing subsequent sets of documents.


(Martin Boycott-Brown) #26

When I read the original post, my immediate thought was “this is a job for DEVONthink, not Tinderbox”. I have a similar problem, in that I have notes collected over a period of more than twenty years, in various formats. And I mean “various”! Mind maps, outlines, Word, rtf, pdf – I have everything. The solution is to stick them all in DEVONthink and sort them out later. DEVONthink acts as the major repository for my research material, and Tinderbox can come at a later stage when I want to “do something” with material selected from it. DEVONthink is easily my most-used program after Safari, and the fact that it has versions for iPad and iPhone is useful. All my research material is available on my phone, wherever I am. Something I would emphasise is that DEVONthink is enormously helpful when it comes to the painful process of sorting through and categorising old notes. The search function is particularly good, and helps in finding associated material. You can, for example, set up a seach that will look for a certain word that appears in the text NEAR another word. So, I can look for “social” appearing within ten words before or after “constructivism”, or something of the sort. There are many more possibilites for searching and organising. DEVONthink databases will deal with huge amounts of data without blinking. It is quite normal for people to have databases containing five million words and not notice any hesitation in performance. I have a database on my phone that is 2.5 GB in size, and there are twelve others of various sizes. I have just done a search of the whole collection on my phone and got 342 hits in less than a second. Wonderful (and mysterious) as Tindrbox is, I wouldn’t use it for the first phase of the process you describe. If I had to state the difference between the two programs rather crudely (and this is only how I see them – others may have very different views) I would say that DEVONthink is a collecting, organising and finding tool, and Tinderbox is an analysing tool. However, I suspect that which tool you use might also depend to some extent on what field you work in. Most of my research material is from history, and there are lots of transcriptions from original sources. Clearly, I don’t want to change these in any way – they have to stay the same. Tinderbox becomes interesting for me when I want to cluster items in ways that may not be possible in DEVONthink – for example placing things in groups of themes on a map with linking arrows between them to show relationships. I can certainly see why people working in other fields (philosophy, perhaps?) might find Tinderbox much more suitable for what they do, particularly if their work is text-only. My work isn’t text-only, and I find that a limitation in Tinderbox. My DEVONthink databases have lots of pictures, geographical maps, diagrams, pdfs, outlines, mind maps and the like, happily co-existing together with text, and still searchable. I use iThoughts mind maps a lot, and I use Scrivener for writing, not to mention OmniOutliner, and to have these accessible and searchable inside DEVONthink is one of the bedrocks of my work. I think Tinderbox is a wondrous program, but it is not the tool I would use for this particular scenario. But if you are working in a different field, it might be exactly what you want. Best of luck with it – I’ve been there …


(Martin Boycott-Brown) #27

At the risk of flogging the subject to death, I thought I would do a quick check of my DEVONthink databases and post the information. There seem to be fourteen of them, containing a total of 65.75 million words. The largest is over 47 million words, the next is only 13 million. I thought I would do a search of the whole collection for Napoleon NEAR Josephine (just for a laugh). I got this result, which you can see includes rtfs, emails, pdfs, and an iThoughts mind map – all in half a second (not bad for a search of 65.75 million words, bearing in mind that it was looking for one word only where it appeared within ten words of another word):


(Andrew McDowell) #28

Thanks much, I was hoping to get more input about DEVONthink. My thought is to use both DT and Tinderbox, much as you suggest: collect, find and organize in DT, then do further work in TBx, which will include breaking up files that have multiple notes on different topics; classing, splicing, dicing, combining, organizing them. It will be a big project for me.


(Martin Boycott-Brown) #29

You will probably be able to do much of what you describe in DEVONthink. It comes with numerous AppleScripts for manipulating files and carrying out other operations. I have split and renamed files, reformatted text, and added ID numbers and headers to notes using the scripts and I am a dunderhead at scripting. The DT forum is a wonderfully helpful and generous group of people. I have known cases when a person has described a problem they wished to solve, and within hours someone has written them a script to achieve it. You should go over to the forum and start looking at what is available there. There may be a ready-made solution for you. Look out for a user who goes by the name of korm – a scripting powerhouse, as is houthakker. You can also convert files from one type to another inside DT (e.g. rtf to plain text, etc). In short, you could easily find yourself doing a lot of the preparatory work in DT.

It can take a while to get to grips with DT, so consider buying the Take Control book for the app.

Also, you might find it interesting to read some of the discussions at www.zettelkasten.de, which are all about knowledge management.

Cheers.


(Martin Boycott-Brown) #30

This is a VERY good place to start learning about DT – a big collection of links to useful sites and so forth:

http://www.christopher-mayo.com/?p=2237


(Andrew McDowell) #31

Good stuff! Thank you a lot! I needed someone to give me a little more glimpse of what DT can do. I’ve been looking at Zettelkasten off and on too for awhile, and it certainly gets at some of what I need. Now I’m working on getting Mac OS on my PC. I think I will not attempt this myself, so I’m trying to get it scheduled with my PC tech. Then I can get started.


(Martin Boycott-Brown) #32

Errr, slight misconception, I think. If you want MacOS, I think you are going to have to buy an Apple Mac. I wouldn’t be surprised to learn that there is some way to hack a PC and install another OS, but it sounds dodgy to me, and I don’t know if it is strictly legal. MacOS runs on Intel chips these days, but as far as I know, that is not the only requirement. But your tech will know.

Glad the other stuff was useful. Make sure you have a look at the usage scenarios.


(Martin Boycott-Brown) #33

Hmmm – I just looked at https://hackintosh.com/, but sooner you than me!


(Andrew McDowell) #34

Yes, it can be done, but not real simple and I’m hoping I don’t have to do it myself; waiting to hear from my tech to see if he will tackle it.


(Andrew McDowell) #35

Whether it’s legal is indeed a gray area, but I’m told that Apple chooses to look the other way, as it apparently isn’t done enough to hurt their sales.


(Simon Smailus) #36

A thoughtbase sounds to me like a ZettelKasten?

If this is what you are after there are some tools listed here: https://zettelkasten.de/tools/

The description of the method can be found here at the following link, you want to especially take note of the material under, “The Bare Minimum of a Zettel Note”, which will give you an idea of what a note (german: zettel) consists of. Here’s the link: https://zettelkasten.de/posts/baseline-zettelkasten-software-reviews/

I have been revisiting this whole subject myself never having got it fully off the ground. I actually prefer a physical zettelkasten as I believe that writing and memory creates a firmer connection. However, the man who started the ZettelKasten idea; Niklas Luhmann; ended up with 20,000 notes. That becomes to unwieldy in the physical realm. He used those notes to write 70 books and 40o articles. He called the ZettelKasten his “second brain”, hence my thinking that it is similar to your “Thoughtbase”.

I started mine in DevonTHINK(DTPO) and it went reasonably well. The main caveat is that DTPO doesn’t flinch at thousands of notes. I currently have 65,000 emails in one database alone. DTPO also allows you to create templates, which is extremely helpful. You can switch on wiki-links which will link the text you write that is identical to another note name. Tagging is great as is searching with full boolean operators.

There are however a few challenges. The text editor is not great (although the same as TB7; not sure if this has changed in more recent versions). Styles are not easy to setup and use. It shows that the strength of DTPO is in storing and searching data rather than creating it. If you’re using Markdown it is no better. Notes must be toggled to, “Best Alternative” to either edit or see the converted markup. Finally, all your notes are lists. This is the biggest negative for DTPO being a ZettelKasten. You really want more than one way to view your data. This is where TB comes in.

The attraction to TB is it’s six views (please be aware that I can only comment on version 7.0, as I stopped using TB since then). With a growing volume of notes they can be viewed as a mindmap, outline, chart, by attribute (ie metadata you attach to your notes), as a timeline and Treemap. I’m not too sure of the difference between the chart and treemap, but am sure someone will fill you in on this forum. This really gives you the flexibility to view your notes from different places and see different connections. TB does link to other notes and another killer feature is that you can set your own metadata for your ZettelKasten. Because of the connectivity between TB and DTPO you can also place all your reference material in DTPO and place a link to it in TB.

There are some questions though that come to mind in using TB.

  1. With a growing body of notes how many can the map view handle comfortably? The map view would be a crucial element for me as I can look at all the notes with their connections.
  2. I did think of using adornments in the map view to have notes tagged with the same tag automatically landing on the same adornment. However, I don’t know how you would solve the problem of multiple tags per note? Could you use a note as a switch so that if you selected one tag it turned off the others? This would allow you to see all the notes with one tag grouped together and then you could select another tag and so forth. An alternative to this is whether it is possible to have an adornment grow to cover the number of notes. The whole point to this is to get an idea of note clusters. To see which tags have more connected notes. If the adornment could be set to automatically grow or shrink then the map view could be used to see which adornments (and thereby tags) are larger note clusters because of the size of the adornment.

I am looking at using TB for a ZettelKasten and would be interested to see if the latest version adds more features into the mix to make this use case easier.


(Mark Anderson) #37

Two thoughts:

  1. Map size/usability really is partly a factor of screen size, i.e. what you can see or easily pan to, and the complexity of the map. If you have lots of notes and links, that is making everything work harder. I don’t see this as something against the app, at some point scale will begin to have effects.

  2. You are right that a note can be ‘on’ more than one smart adornments (see more). However, you might consider using actions/agent to do things like change a badge or border colour to mark a common tags. I don’t suggest lots of different colours, but one giving sufficient contrast, and which you can toggle on off to highlight a particular relationship.


(James Vornov) #38

As well, if you use TBX simply as a flat map, you’re not making use of its real power to abstract.

For me, the ability to automate collection of individual notes into meta-notes is where I get the biggest advantage. I learned this from the Zettelkasten folks. You can easily create lists of sources or meta notes that narrate ideas with links to the individual notes or PDFs stored in folders or a database like Devonthink.

I think of Tinderbox as an idea processor. Notes go in, sources are linked and I map out complex ideas.


(Desalegn) #39

Interesting. So, do you drag/link your pdf files or notes from Devonthink and write notes about them in TB?