- Massive optimization of data layer -- with ~11,000-item test library on a Mac Pro, decreased initial Zotero pane loading from several minutes to ~10 seconds. This included some small API changes and new methods (e.g. Items.cacheFiles()) in the data layer, but most of it was changing the way loading and caching of data worked internally.
- Moved unique itemData values out to separate itemDataValues table for better normalization
- Updated itemTreeView.sort() to be able to sort a single row into the items list for performance reasons -- itemTreeView.notify() now only sorts a single row when possible (and sometimes doesn't need to sort anything). This should make general interface use dramatically less sluggish with large libraries.
- Consolidated purging on item deletes, which should speed up multi-item deletes quite a bit -- clients should use Items.erase() instead of Item.erase(), since the former calls the new Items.purge() method (which calls the various other purge() methods) automatically
- Notifier no longer throws errors in notify() callbacks and instead just logs them to the Error Console -- this way a misbehaving utility (or Zotero itself) won't keep other observers from receiving change notifications
- Better handling of database corruption -- if an SQL query throws a file corruption error, Zotero adds a marker file to the storage directory and displays a message prompting the user to restart to attempt auto-repair--and, most importantly, no longer copies the corrupt file over the last backup.
- A "Loading items list..." message appears over the items list (at least, sometimes) while data is loading -- useful for large libraries, but may need to be fine-tuned to not be annoying for smaller ones.
- Note titles are now cached in itemNoteTitles table
- orderIndex values are no longer consolidated when removing items from collections -- it just leaves gaps
- Fixed shameful bug in getRandomID() that could result in an item with itemID 0, which wouldn't display correctly and would be impossible to remove
- Fixed autocomplete and search for new location of 'title' field
- Added proper multipart date support for type-specific 'date' fields
- Made the pre-modification array passed to Notifier observers on item updates actually be pre-modification
- New method Zotero.ItemFields.isFieldOfBase(field, baseField) -- for example, isFieldOfBase('label', 'publisher') returns true, as does isFieldOfBase('publisher', 'publisher')
- Restored ability to drag child items in collections into top-level items in those collections
- Disabled unresponsive script message when opening Zotero pane (necessary for large libraries, or at least was before the optimizations)
- Collections in background windows didn't update on item changes
- Modifying an item would cause it to appear incorrectly in other collections in background windows
- Fixed an error when dragging, hovering to open, and dropping a note or attachment on another item
- Removed deprecated Notifier methods registerCollectionObserver(), registerItemObserver(), unregisterCollectionObserver(), and unregisterItemObserver()
- Loading of Zotero core object can be cancelled on error with Zotero.skipLoading
- Removed old disabled DebugLogger code
- New method Zotero.log(message, type, sourceName, sourceLine, lineNumber, columnNumber, category) to log to Error Console -- wrapper for nsIConsoleService.logMessage(nsIScriptError)
- New method Zotero.getErrors(), currently unused, to return array of error strings that have occurred since startup, excluding CSS and content JS errors -- will enable an upcoming Talkback-like feature
- Fixed some JS strict warnings in Zotero.Date.strToMultipart()
- All remaining fields in the items table exists in all items and are non-user-editable
- Simplified some data access code (e.g. removed Item.isEditableField())
- 'title' is now a base field used in case (Case Name), statute (nameOfAct) and e-mail (Subject)
Reengineered parts of the data layer for better performance
- Various recent changes, including the 'title' change above and base field mapping in Item.getField(), had a negative effect on performance. This should help. In particular, itemData values are now loaded in in bulk by Items._load() (via Items.get()) rather than on-demand, the sort process in itemTreeView caches values while sorting, and ItemFields.getFieldIDFromTypeAndBase() is faster.
Addresses #346, mapping for new item types
Localization changes:
- Removed periods on ingester.scrapeComplete and ingester.scrapeError
- Removed changed ingester.scrapeErrorDescription from existing locales
- Added ingester.scrapeErrorDescription.linkText
Related progress window changes:
- Translators now use Zotero.ProgressWindow rather than Zotero_Browser.Progress (removed) -- browser.js currently creates a single instance of ProgressWindow per browser window for its use
- Progress boxes now stay open while the mouse is over them and close when you click on them
- New method Zotero.Utilities.parseMarkup() -- parses a text string for HTML/XUL markup and returns an array of parts (currently only <a>)
- Using parseMarkup, ProgressWindow.addDescription() now supports adding clickable links to notifications
Other progress window changes:
- Fixed progress window offsets on Mac (on OS X outerHeight doesn't include 22px title bar and moveTo() positions popups 22px below the specified location) -- need to test this on other platforms
- Added support for displaying progress window notifications relative to screen edges when there's no browser window
Unrelated:
- Fixed warning when calling getSelectedCollections() before the Z pane had been opened
- Display items with empty titles at beginning of list (since having new notes and items appear at the bottom of the list is distracting)
- On item modify, select item if in active window (which hopefully doesn't have any side effects, though I'm not sure why I ever changed this)
- Focus quicksearch bar on Z pane open
Refresher:
var progress = new Zotero.ProgressWindow();
progress.changeHeadline('Indexing item...');
progress.addLines("My First Book", "chrome://zotero/skin/treeitem-book.png");
progress.addDescription('This is my first book.');
progress.show();
progress.fade();
var progress2 = new Zotero.ProgressWindow();
progress2.changeHeadline('Indexing item 2...');
progress2.addLines("My Second Book", "chrome://zotero/skin/treeitem-book.png");
progress2.addDescription('This is my second book.');
progress2.show();
progress2.fade();
- Tags matching JS Array object methods ("every", "map", "splice") would appear selected in tag selector after clicking Deselect All
- Tag selector wasn't notified correctly on tag renames
- Don't use LIKE if no text string in Tags.search()
- Iterator variable in for loop in Notifier.commit() wasn't local
- Removed extra debug info in pdftotext run
Type-specific fields now display for 'publisher' in the items list
Added caching to ItemFields.isBaseField() and ItemFields.getFieldIDFromTypeAndBase()
Customizable in the preferences
Also improves handling of Zotero startup errors, adding the ability to customize the tooltiptext of the status bar error icon and to have ZoteroPane.toggleDisplay() run a customizable error function (it no longer opens an empty and broken Zotero pane). For example, a missing Z directory now prompts the user to locate the directory.
Various code parts now check for the Zotero object and Zotero.initialized before trying to do stuff, which should cut down on redundant error lines in the console and generally make things cleaner.
Refs #542, Better icons for preferences window -- need icon for Advanced prefpane
'number' data in B3 'patent' items should become 'applicationNumber', since there already was a 'patentNumber', even if going forward 'patentNumber' will map to the 'number' base field
Simon, if you still have a copy of the problematic data, please check that this fixes the upgrade path.
This is still mostly proof-of-concept stage, but it seems to work on my Mac. It requires the pdftotext utility from the Xpdf project, which parses PDFs into plain text files. The Zotero fulltext indexer calls pdftotext on the PDF file and saves the plaintext version as .zotero-ft-cache in the attachment item's storage directory. It runs the fulltext word indexer on the plaintext file and also scans the plaintext file when doing a phrase search.
To try it out, install a copy of Xpdf (or just pdftotext) and either place pdftotext into the Zotero data directory or create a symlink. Either way, the file must be named pdftotext-{platform}[.exe], where {platform} is navigator.platform, with spaces replaced by hyphens (e.g. "Win32", "Linux-i686", "MacPPC", "MacIntel", etc.). On my Mac, with Xpdf installed via Darwin Ports, I create a symlink to /opt/local/bin/pdftotext named pdftotext-MacIntel. This setup will allow users to sync their Firefox profiles and still have Zotero use the appropriate platform-specific binary.
Assuming we go this pdftotext route, I think we'll instruct users to download and install Xpdf/pdftotext, possibly even providing binaries ourselves. The binaries are too big to include in the XPI. I'm going to look into creating a GUI to make linking Zotero to pdftotext easier. I also need to finish some of the other tickets related to indexer feedback and control.
There are also two new hidden prefs, fulltext.pdfMaxPages and fulltext.textMaxLength, currently set to 100 and 500K, respectively. The first determines how many pages of each PDF pdftotext processes, and the second determines how many characters and/or bytes of text files (the PDF cache files included) Zotero indexes and scans. These defaults may want to be adjusted higher or lower.
Closes#315, Hidden pref to set maximum file size to index/scan
Works for regular items and notes, not attachments (and doesn't clone child items when duplicating parent)
New method Item.clone()
Unrelated changes:
- Fix note/attachment dragging, broken by notifier changes (r1131) a while back
- Item.save() now triggers Notifier even if a transaction is in progress, which I hopefully no longer had a reason not to be doing
Better idea: Add expand/collapse keypress listener in treeview setTree() so that it works automatically in all trees
More of the functionality in the overlay could probably be moved into the treeviews like this
Hit + to expand all tree rows, - to collapse. Works in both collections tree and items tree.
Probably could've thought of that 8 months ago when we created the ticket...
Can drag links, images, or the favicon of the current page
Also allows dragging over collections and item pane whitespace to create standalone attachments, but if we want to allow that at all, it would probably be better to create new webpage items with snapshot items (like Create New Item from Current Page) instead.
Shortcut key defaults to "C"
Abstracted clipboard logic in Zotero_File_Interface._doBibliographyOptions() to separate function, Z_F_I.copyItemsToClipboard(items, style), which ZoteroPane.copySelectedItemsToClipboard() calls
Currently limited to citation styles, but there's no reason it couldn't support export formats, etc.
Closes#227, Indent nested collections in search drop-down
Addresses #528, Make search condition drop-down menu less unwieldy
- Created new distinct fields for differently labeled fields
- Mapped lots of fields to base fields
- Made base field search conditions search type-specific fields as well
- Removed type-specific fields that are based on base fields not show up in search conditions drop-down
- Added a tooltip when hovering over a condition in the search conditions drop-down that shows the fields it searches (when there's more than one)
- Moved search dialog CSS to separate file
