Help needed to diagnose poor performance of TB file

[enricoscarpella]

Dear all,

I wonder whether anyone could be so kind to help me diagnose the poor performance of a TB file of mine. I am not sure whether the forum is the right place to ask for this kind of help or whether I should email Eastgate about it, but I thought I would start here given that Eastgate often frequents the forum.

The file in question is ~300 MB in size and contains ~400 notes. The notes are all at root level, are heavily interlinked, belong to one of three main prototypes, and — thanks to an Edict @PaulWalters and @mwra kindly help me put together some time ago — change their color based on their degree of connectedness. The notes that belong to one of the three main prototypes also contain links to Bookends entries. The file contains @pat’s Publish to DEVONthink Stamp and watches a DEVONthink folder. I mainly work in map view — the position of the notes is meaningful to me, so I tend to remember it.

I have barely started transferring years of notes in that file, yet it takes ~20 sec to open the file and ~30 sec to change magnification or take any action. I therefore wonder whether my hardware is insufficiently powerful; I have reached TB’s limit of number of notes or their connectedness; or there is something wrong with my file. There is nothing private in those notes — most of the content is published — so I have shared below a copy of the file. If it helps, I have also attached a copy of the file that has been stripped of all the content notes and only has structure notes.

I know in this forum it has frequently been advised to group notes in containers and not to have many notes at root level, so I could transfer all the notes in my file into a container that is at root level if you think it will make a difference; however, I thought I would first ask because I perhaps misunderstood the advice — I fail to understand how performance would improve once I am working in map view inside such container.

I am using TB 8.8.0 and MacOS 10.15.7 on Macs that go from 2016 to 2020 and see no difference in performance. I have not updated TB to 8.9 because I cannot upgrade OS to Big Sur yet, and I was unsure whether 8.9 would be compatible with OS < 11.

Thank you for your consideration.

Best regards,
Enrico

Work - Content - Working Copy - No Notes.tbx (174.7 KB)

Work - Content - Working Copy

[eastgate]

Yes, Tinderbox 8.9 is fine with Catalina and High Sierra.

400 notes is not a lot! 300M with 400 notes suggests that there may be some bulky graphics. I’ll take a look.

[enricoscarpella]

Thank you very much for the prompt reply, the clarification on Tinderbox 8.9 and MacOSs — I will update to 8.9 right away — and for kindly offering to take a look at my file!

Best regards,
Enrico

[eastgate]

For example, the uncompressed images in the note named “q20190507170713” has an image that takes up 12M. If you dispensed with these embedded images, you’re file would be much smaller.

[enricoscarpella]

Thank you very much. I am not sure i can eliminate images altogether, but I certainly can reduce their sizes. On a related note, how did you zero in on that note? Is note size, for example, an attribute that can be used to order note according to decreasing size in the Attribute Browser? I am trying to learn from this experience… Thank you for your consideration.

[enricoscarpella]

My apologies for the second message, but I was checking the images I had pasted in that note, and they add up to 2.1 MB, not 12 MB. Could you please clarify? Thank you.

[eastgate]

Tinderbox has to store the images in an uncompressed format, so they take up more space in a Tinderbox file than they do when compressed.

I simply opened the document in BBEdit and looked for large encoded blocks; I actually was working on a different note, but fumbled that on and got this one instead.

That explains the file size. The map performance is likely a function of your using such a big map at such low scale; that’s a lot of links and a lot of views to draw! But that is just a guess; an actual answer will require some study with instrumentation.

[enricoscarpella]

Thank you for the prompt reply!

@eastgate

I simply opened the document in BBEdit and looked for large encoded blocks; I actually was working on a different note, but fumbled that on and got this one instead.

Thank you for the clarification.

@eastgate

Tinderbox has to store the images in an uncompressed format, so they take up more space in a Tinderbox file than they do when compressed.

So if I understand you correctly, it would be better for me to give up images altogether if I want to add to this file all the notes I accumulated over the years.

@eastgate

That explains the file size. The map performance is likely a function of your using such a big map at such low scale; that’s a lot of links and a lot of views to draw! But that is just a guess; an actual answer will require some study with instrumentation.

Just to give you some feedback, if I work in map view at a magnification that allows me to see between 10 and 15 notes and I select a note by clicking on it, it takes ~5 sec for the note to appear as selected on the map and to show its text in the text pane.

Thank you very much again for your help.
Enrico

[mdavidson (Malcolm)]

You can also try to track down the notes taking up the most space in your document within TinderBox. To do this you need to find and display this notes with the highest $EstimatedNoteSize.

[mwra (Mark Anderson)]

Looking in Outline view, I notice you have Column view ‘on’. It’s a favourite of mine, but ideally turn it off when not needed - that tab will remember the last used column settings. As regards the latter, the first column is always $Name (actually $DisplayName) so unless you’re trying to tell those two apart your first user assigned column (‘Name’) is just duplication.

On the theme of views, I’d close the doc in outline with no columns on as it I think it would make it quicker to open. You can always have a second (non-selected) tab in Map view as the map won’t try to render until you select that tab.

This TBX is a good example of why I advise people not to use root level map view for big projects. More user-friendly is to make a container in the root map and make your main data ‘map’ inside that container. Now, regardless of what view you use you’re not trying to render every note when you load the document.

Sadly, either using a Tab stroke, drag/drop, or a stamp to demote all the data notes into a container, trashes their $Xpos/$Ypos but—just tested in v8.9.0—you can use a stamp like this:

$MyXpos=$Xpos;
$MyYpos=$Ypos;
$Container="/Map";
$Xpos=$MyXpos;
$Ypos=$MyYpos;

This should move all selected notes (adornments/agents, etc) to the target container and retain their original layout. The method uses two new Number-type user attribute that you will need to add to the document before using the stamp^†.

In may test, I had 3 notes and an adornment at root level. I then added a note ‘Map’ that I wished to be the new container, I selected the original 4 items and used the above stamp. Note, if trying this, I’d first set $NeverComposite=true on the new container to avoid accidental composites^‡ and ensure the target container is empty or existing content is moved to a part of the map that doesn’t intersect with the new content.

^†. Incidentally, I used two new user attributes as likely you’ll do this only once in the document’s lifetime and so once the map is moved you can delete these two attributes. Otherwise all your map items are still storing now-pointless data and adding (minimally) to the file size. So, once happy the map is moved, just delete the two user attributes and they and their data are all removed from the document. You can also delete the stamp as well, if you want to tidy up completely.

^‡. But—if your selection includes composites I’d do small tests first as I don’t know what happens.

Now to try the above on your data…

[mwra (Mark Anderson)]

Another suggestion, do explore hyperbolic view, it might be useful here. If you get on with it, one upside os the view doesn’t need all notes to be in the same container (i.e. map). The view only shows linked notes.

I’m not sure the community has quite got the the power of this view yet (though it is ‘young’ in age compared to views like map and outline).

[mwra (Mark Anderson)]

As an experiment, I’ve used the above stamp to move (what I understand as map ‘data’ items) to a new root level container “MAP”. I’ve also put all the prototypes in the /Prototypes container. The root outline now looks like this (columns off for better performance):

Making a new tab and switching to Map view, the root map now looks like this:

Select ‘MAP’ and drill down to the original map, and we see it now moved safely out of the root:

When closing, I switch from the map view (above)—the right-hand tab below— back to the outline:

…and close. Now, the TBX now opens much more rapidly. The big data map doesn’t render until I select that tab although it of course takes a while for reasons already covered above (zoom-level & images, etc.).

In summary:

• Don’t make a big working map at root level
• Putting data into a container also removes back-of house stuff like prototype and template containers of your ‘work’ map.
• Turn off outline column view when not needed - scrolling big/long outlines is much faster. Toggle the columns on/off as needed.
• Embedded Images - already covered above
• Another way to use images, store them externally, link to them via HTML <img> tags in your $Text and view the $Text-with-images via the preview tab of the text pane.

A few ideas. HTH!

[satikusala (Michael Becker)]

RE dealing with the image sizes, here are some includes you can put in your notes. You want to store the images and videos on your local drive and point to them.

WARNING: You want to make sure you’re using SmartQuotes.

Take a look at last Saturday’s meeting Meetup for Dec 05, 9:00a PST and 17:00 CEST | Dec 06, 04:00 AEDT were we discuss these and show examples.

Fore Images:
<img src="file:////DIRECTORY/ProdImages/FILENAME" width="300" height="600">

e.g.

<img src="file:////Users/michaelbecker/Google Drive/DevonThinkTBXDB/ProdImages/screenshot.png" width="300" height="600">

Video:

 <video width="300" height="600" controls>
    <source src="file:////Users/michaelbecker/Google Drive/DevonThinkTBXDB/ProdImages/VIDEOFILE.mp4"  type="video/mp4"
    ></Video>

[enricoscarpella]

Thank you so very much, @mdavidson, @mwra, and @satikusala for so generously donating energy and time to help me solve my problem!

Thanks to @mdavidson’s tip, I displayed the $EstimatedNoteSize attribute as a column in Otuline View, found all the notes with an unusually large size, and — thanks to @satikusala’s tip — I replaced the images in those notes with links to the images. This brought down the file size from ~300 MB to <2 MB!

Next — as per @mwra’s suggestion — I turned off Use Columns in Outline View, and grouped Templates, Prototypes, Stamps, Applescripts, Color Palettes, Reference Notes, Version Control, and watched folder in their respective containers. If closed in Outline View, opening this new file takes <2 sec vs. the ~20 sec it takes to open its original version.

Then, in the inspector I set the system attribute $NeverComposite to true — I did it for the whole document because I never want notes to combine into composites — I created a container named Map, and tried the stamp @mwra kindly provided. When I apply the stamp to an adornment and the notes on top of it, the notes and the adornment move inside the right container, but all the notes are collapsed on top of one another at the top-left corner of the adornment. Could you please help me figure out why? In case it helps, I have attached a copt of the file. Thank you for your consideration.

Best regards,
Enrico

Work - Content - Working Copy 2.tbx (1.7 MB)

[mwra (Mark Anderson)]

This had me scratching my head until I checked the $MyXpos and $MyYpos values to find they were missing. These are user (Number) attributes and you’ve not created them. Adding them, all works as expected. I’ll amend my earlier answer to make plain you need to create these attributes before using the stamp.

Incidentally, I used two new user attributes as likely you’ll do this once in the file’s lifetime and so once the map is moved you can delete the attributes or all your map items are storing now-pointless data and adding (minimally) to the file size. so, just delete the two user attributes and they and their data are all removed from the document

[mwra (Mark Anderson)]

I should perhaps add that if you have smart adornments, i.e. those with a query drawing in notes, the layout after paste may vary (I’ve not tested). But, as such adornments control their layout there is not much you can do. But, if you don’t need the query all the time, you could delete the query (AgentPriority isn’t used by smart adornments) and you can then arrange the items better within the adornment. Of course, if you re-apply a query the now-smart adornment may undo your manual placement.

[enricoscarpella]

Thank you for your patience, @mwra. The thought that $MyXpos and $MyYpos were user attributes did indeed cross my mind — as it should have, give the “My” prefix… — but because $Xpos anf $Ypos are intrinsic systems attributes and in the stamp $MyXpos and $MyYpos seem to be defined in term of $Xpos and $Ypos, I decided I was mistaken; I should have just given it a try; my apologies.

Once I created the $MyXpos and $MyYpos number-type user attributes, everything worked perfectly! I have now deleted those user attributes and a couple of other ones I had created with your help but have now been superseded by the Links pane. I am documenting everything I have done, so I can apply the same strategy to all my other Tinderbox documents.

I have the strong feeling that if my problematic file had been created with a different application, I would have been left with no choice but redo everything from scratch. With Tinderbox, instead, it seems there is always the possibility to change the form of the document after the fact — evidence of the software’s power but also of its forgiveness: I am never afraid of starting a new document without knowledge of its form, sure that I can always let it emerge spontaneously (“by its colorful tunes the lark blissfully climbs up into the air”), so I am really grateful for that. Nevertheless, even with all that software power and forgiveness, I would not have been able to figure out how to do what I needed to do without your help and that of the other users in this forum. I think we are fortunate to have a software like Tinderbox, but I also think Eastgate and all the inexperienced users like me are fortunate to have a community of experienced users like you: so generous with their time and energy; thank you very much for that!

Best regards,
Enrico

[mwra (Mark Anderson)]

No—np apologies needed! Truly, I should have thought to state that you must create the extra attributes. Your ‘mistake’ was perfectly understandable without that instructions.

Amidst all the unhappiness right now about how ‘hard’ some think the app, it’s nice to have a positive counter example.

[satikusala (Michael Becker)]

@enricoscarpella

I have the strong feeling that if my problematic file had been created with a different application, I would have been left with no choice but redo everything from scratch.

Yes! I’ve found this to be the case with mine too. There is a beauty to Tinderbox.

[satikusala (Michael Becker)]

@mwra

Amidst all the unhappiness right now about how ‘hard’ some think the app

Mark, we’re here for you. The new recruits (we’ll not so new or young) have picked up the standard and are working on help spread the word and provide examples to help scaffold people from where they are at to where they can or want to be.