Implemented auto-renaming of PDFs and other files based on the parent item's metadata. Format is customizable via 'attachmentRenameFormatString' pref, though options are currently rather limited. Default is '%c - %y - %t{50}', meaning "FirstCreator - 2007 - Title Truncated to 50 Characters"
Also:
- Removed |forceFileName| from Attachments.importFromDocument() -- I don't think it'll be necessary?
- Modified WebPageDump not to save HTML wrapper files around non-HTML documents
Switched importFromURL() from native Mozilla saveDocument() to WebPageDump code (originally from Scrapbook), which fixes CSS saving problems and other issues -- viewing a WPD snapshot of the New York Times front page causes a crash in the Flash plugin in all Mac browsers, however, unless the Flash security settings are changed to whitelist the Zotero storage directory for accessing the network (but this should be done anyway, since the warning is annoying)
Also:
- Fix broken items list in Related and Advanced Search windows from r1231
- Fix several _getItemAtRow() errors from r1231
- Adjusted Zotero.Progress API: fade()->startCloseTimer(), kill()->close()
- Added forceTitle parameter to importFromURL(), currently unimplemented -- note that order of the last parameter changed
- Added progress windows for "Create New Item from Current Page", snapshot, and link actions
- Added "Miscellaneous" caption in General pane of preferences
- Clarify Mac rich-text warnings
- Switched small Export prefpane to generic icon
- Massive optimization of data layer -- with ~11,000-item test library on a Mac Pro, decreased initial Zotero pane loading from several minutes to ~10 seconds. This included some small API changes and new methods (e.g. Items.cacheFiles()) in the data layer, but most of it was changing the way loading and caching of data worked internally.
- Moved unique itemData values out to separate itemDataValues table for better normalization
- Updated itemTreeView.sort() to be able to sort a single row into the items list for performance reasons -- itemTreeView.notify() now only sorts a single row when possible (and sometimes doesn't need to sort anything). This should make general interface use dramatically less sluggish with large libraries.
- Consolidated purging on item deletes, which should speed up multi-item deletes quite a bit -- clients should use Items.erase() instead of Item.erase(), since the former calls the new Items.purge() method (which calls the various other purge() methods) automatically
- Notifier no longer throws errors in notify() callbacks and instead just logs them to the Error Console -- this way a misbehaving utility (or Zotero itself) won't keep other observers from receiving change notifications
- Better handling of database corruption -- if an SQL query throws a file corruption error, Zotero adds a marker file to the storage directory and displays a message prompting the user to restart to attempt auto-repair--and, most importantly, no longer copies the corrupt file over the last backup.
- A "Loading items list..." message appears over the items list (at least, sometimes) while data is loading -- useful for large libraries, but may need to be fine-tuned to not be annoying for smaller ones.
- Note titles are now cached in itemNoteTitles table
- orderIndex values are no longer consolidated when removing items from collections -- it just leaves gaps
- Fixed shameful bug in getRandomID() that could result in an item with itemID 0, which wouldn't display correctly and would be impossible to remove
- Fixed autocomplete and search for new location of 'title' field
- Added proper multipart date support for type-specific 'date' fields
- Made the pre-modification array passed to Notifier observers on item updates actually be pre-modification
- New method Zotero.ItemFields.isFieldOfBase(field, baseField) -- for example, isFieldOfBase('label', 'publisher') returns true, as does isFieldOfBase('publisher', 'publisher')
- Restored ability to drag child items in collections into top-level items in those collections
- Disabled unresponsive script message when opening Zotero pane (necessary for large libraries, or at least was before the optimizations)
- Collections in background windows didn't update on item changes
- Modifying an item would cause it to appear incorrectly in other collections in background windows
- Fixed an error when dragging, hovering to open, and dropping a note or attachment on another item
- Removed deprecated Notifier methods registerCollectionObserver(), registerItemObserver(), unregisterCollectionObserver(), and unregisterItemObserver()
- Loading of Zotero core object can be cancelled on error with Zotero.skipLoading
- Removed old disabled DebugLogger code
- New method Zotero.log(message, type, sourceName, sourceLine, lineNumber, columnNumber, category) to log to Error Console -- wrapper for nsIConsoleService.logMessage(nsIScriptError)
- New method Zotero.getErrors(), currently unused, to return array of error strings that have occurred since startup, excluding CSS and content JS errors -- will enable an upcoming Talkback-like feature
- Fixed some JS strict warnings in Zotero.Date.strToMultipart()
- All remaining fields in the items table exists in all items and are non-user-editable
- Simplified some data access code (e.g. removed Item.isEditableField())
- 'title' is now a base field used in case (Case Name), statute (nameOfAct) and e-mail (Subject)
Reengineered parts of the data layer for better performance
- Various recent changes, including the 'title' change above and base field mapping in Item.getField(), had a negative effect on performance. This should help. In particular, itemData values are now loaded in in bulk by Items._load() (via Items.get()) rather than on-demand, the sort process in itemTreeView caches values while sorting, and ItemFields.getFieldIDFromTypeAndBase() is faster.
Addresses #346, mapping for new item types
Localization changes:
- Removed periods on ingester.scrapeComplete and ingester.scrapeError
- Removed changed ingester.scrapeErrorDescription from existing locales
- Added ingester.scrapeErrorDescription.linkText
Related progress window changes:
- Translators now use Zotero.ProgressWindow rather than Zotero_Browser.Progress (removed) -- browser.js currently creates a single instance of ProgressWindow per browser window for its use
- Progress boxes now stay open while the mouse is over them and close when you click on them
- New method Zotero.Utilities.parseMarkup() -- parses a text string for HTML/XUL markup and returns an array of parts (currently only <a>)
- Using parseMarkup, ProgressWindow.addDescription() now supports adding clickable links to notifications
Other progress window changes:
- Fixed progress window offsets on Mac (on OS X outerHeight doesn't include 22px title bar and moveTo() positions popups 22px below the specified location) -- need to test this on other platforms
- Added support for displaying progress window notifications relative to screen edges when there's no browser window
Unrelated:
- Fixed warning when calling getSelectedCollections() before the Z pane had been opened
- Display items with empty titles at beginning of list (since having new notes and items appear at the bottom of the list is distracting)
- On item modify, select item if in active window (which hopefully doesn't have any side effects, though I'm not sure why I ever changed this)
- Focus quicksearch bar on Z pane open
Refresher:
var progress = new Zotero.ProgressWindow();
progress.changeHeadline('Indexing item...');
progress.addLines("My First Book", "chrome://zotero/skin/treeitem-book.png");
progress.addDescription('This is my first book.');
progress.show();
progress.fade();
var progress2 = new Zotero.ProgressWindow();
progress2.changeHeadline('Indexing item 2...');
progress2.addLines("My Second Book", "chrome://zotero/skin/treeitem-book.png");
progress2.addDescription('This is my second book.');
progress2.show();
progress2.fade();
- Tags matching JS Array object methods ("every", "map", "splice") would appear selected in tag selector after clicking Deselect All
- Tag selector wasn't notified correctly on tag renames
- Don't use LIKE if no text string in Tags.search()
- Iterator variable in for loop in Notifier.commit() wasn't local
- Removed extra debug info in pdftotext run
Type-specific fields now display for 'publisher' in the items list
Added caching to ItemFields.isBaseField() and ItemFields.getFieldIDFromTypeAndBase()
Customizable in the preferences
Also improves handling of Zotero startup errors, adding the ability to customize the tooltiptext of the status bar error icon and to have ZoteroPane.toggleDisplay() run a customizable error function (it no longer opens an empty and broken Zotero pane). For example, a missing Z directory now prompts the user to locate the directory.
Various code parts now check for the Zotero object and Zotero.initialized before trying to do stuff, which should cut down on redundant error lines in the console and generally make things cleaner.
Refs #542, Better icons for preferences window -- need icon for Advanced prefpane
'number' data in B3 'patent' items should become 'applicationNumber', since there already was a 'patentNumber', even if going forward 'patentNumber' will map to the 'number' base field
Simon, if you still have a copy of the problematic data, please check that this fixes the upgrade path.
This is still mostly proof-of-concept stage, but it seems to work on my Mac. It requires the pdftotext utility from the Xpdf project, which parses PDFs into plain text files. The Zotero fulltext indexer calls pdftotext on the PDF file and saves the plaintext version as .zotero-ft-cache in the attachment item's storage directory. It runs the fulltext word indexer on the plaintext file and also scans the plaintext file when doing a phrase search.
To try it out, install a copy of Xpdf (or just pdftotext) and either place pdftotext into the Zotero data directory or create a symlink. Either way, the file must be named pdftotext-{platform}[.exe], where {platform} is navigator.platform, with spaces replaced by hyphens (e.g. "Win32", "Linux-i686", "MacPPC", "MacIntel", etc.). On my Mac, with Xpdf installed via Darwin Ports, I create a symlink to /opt/local/bin/pdftotext named pdftotext-MacIntel. This setup will allow users to sync their Firefox profiles and still have Zotero use the appropriate platform-specific binary.
Assuming we go this pdftotext route, I think we'll instruct users to download and install Xpdf/pdftotext, possibly even providing binaries ourselves. The binaries are too big to include in the XPI. I'm going to look into creating a GUI to make linking Zotero to pdftotext easier. I also need to finish some of the other tickets related to indexer feedback and control.
There are also two new hidden prefs, fulltext.pdfMaxPages and fulltext.textMaxLength, currently set to 100 and 500K, respectively. The first determines how many pages of each PDF pdftotext processes, and the second determines how many characters and/or bytes of text files (the PDF cache files included) Zotero indexes and scans. These defaults may want to be adjusted higher or lower.
Closes#315, Hidden pref to set maximum file size to index/scan
Works for regular items and notes, not attachments (and doesn't clone child items when duplicating parent)
New method Item.clone()
Unrelated changes:
- Fix note/attachment dragging, broken by notifier changes (r1131) a while back
- Item.save() now triggers Notifier even if a transaction is in progress, which I hopefully no longer had a reason not to be doing
Better idea: Add expand/collapse keypress listener in treeview setTree() so that it works automatically in all trees
More of the functionality in the overlay could probably be moved into the treeviews like this
Hit + to expand all tree rows, - to collapse. Works in both collections tree and items tree.
Probably could've thought of that 8 months ago when we created the ticket...
