Added proceedingsTitle with base field publicationTitle for now, since the hierarchical item type structure will require both (the conference itself will be an event)
Copied any conferenceName value to proceedingsTitle
- Massive optimization of data layer -- with ~11,000-item test library on a Mac Pro, decreased initial Zotero pane loading from several minutes to ~10 seconds. This included some small API changes and new methods (e.g. Items.cacheFiles()) in the data layer, but most of it was changing the way loading and caching of data worked internally.
- Moved unique itemData values out to separate itemDataValues table for better normalization
- Updated itemTreeView.sort() to be able to sort a single row into the items list for performance reasons -- itemTreeView.notify() now only sorts a single row when possible (and sometimes doesn't need to sort anything). This should make general interface use dramatically less sluggish with large libraries.
- Consolidated purging on item deletes, which should speed up multi-item deletes quite a bit -- clients should use Items.erase() instead of Item.erase(), since the former calls the new Items.purge() method (which calls the various other purge() methods) automatically
- Notifier no longer throws errors in notify() callbacks and instead just logs them to the Error Console -- this way a misbehaving utility (or Zotero itself) won't keep other observers from receiving change notifications
- Better handling of database corruption -- if an SQL query throws a file corruption error, Zotero adds a marker file to the storage directory and displays a message prompting the user to restart to attempt auto-repair--and, most importantly, no longer copies the corrupt file over the last backup.
- A "Loading items list..." message appears over the items list (at least, sometimes) while data is loading -- useful for large libraries, but may need to be fine-tuned to not be annoying for smaller ones.
- Note titles are now cached in itemNoteTitles table
- orderIndex values are no longer consolidated when removing items from collections -- it just leaves gaps
- Fixed shameful bug in getRandomID() that could result in an item with itemID 0, which wouldn't display correctly and would be impossible to remove
- Fixed autocomplete and search for new location of 'title' field
- Added proper multipart date support for type-specific 'date' fields
- Made the pre-modification array passed to Notifier observers on item updates actually be pre-modification
- New method Zotero.ItemFields.isFieldOfBase(field, baseField) -- for example, isFieldOfBase('label', 'publisher') returns true, as does isFieldOfBase('publisher', 'publisher')
- Restored ability to drag child items in collections into top-level items in those collections
- Disabled unresponsive script message when opening Zotero pane (necessary for large libraries, or at least was before the optimizations)
- Collections in background windows didn't update on item changes
- Modifying an item would cause it to appear incorrectly in other collections in background windows
- Fixed an error when dragging, hovering to open, and dropping a note or attachment on another item
- Removed deprecated Notifier methods registerCollectionObserver(), registerItemObserver(), unregisterCollectionObserver(), and unregisterItemObserver()
- Loading of Zotero core object can be cancelled on error with Zotero.skipLoading
- Removed old disabled DebugLogger code
- New method Zotero.log(message, type, sourceName, sourceLine, lineNumber, columnNumber, category) to log to Error Console -- wrapper for nsIConsoleService.logMessage(nsIScriptError)
- New method Zotero.getErrors(), currently unused, to return array of error strings that have occurred since startup, excluding CSS and content JS errors -- will enable an upcoming Talkback-like feature
- Fixed some JS strict warnings in Zotero.Date.strToMultipart()
- All remaining fields in the items table exists in all items and are non-user-editable
- Simplified some data access code (e.g. removed Item.isEditableField())
- 'title' is now a base field used in case (Case Name), statute (nameOfAct) and e-mail (Subject)
Reengineered parts of the data layer for better performance
- Various recent changes, including the 'title' change above and base field mapping in Item.getField(), had a negative effect on performance. This should help. In particular, itemData values are now loaded in in bulk by Items._load() (via Items.get()) rather than on-demand, the sort process in itemTreeView caches values while sorting, and ItemFields.getFieldIDFromTypeAndBase() is faster.
Addresses #346, mapping for new item types
'number' data in B3 'patent' items should become 'applicationNumber', since there already was a 'patentNumber', even if going forward 'patentNumber' will map to the 'number' base field
Simon, if you still have a copy of the problematic data, please check that this fixes the upgrade path.
Closes#227, Indent nested collections in search drop-down
Addresses #528, Make search condition drop-down menu less unwieldy
- Created new distinct fields for differently labeled fields
- Mapped lots of fields to base fields
- Made base field search conditions search type-specific fields as well
- Removed type-specific fields that are based on base fields not show up in search conditions drop-down
- Added a tooltip when hovering over a condition in the search conditions drop-down that shows the fields it searches (when there's more than one)
- Moved search dialog CSS to separate file
Abstract displays in metadata pane as a cropped one-line field by default; clicking the 'Abstract' label toggles between the cropped field and an expanded view
Some problems with import/export: https://www.zotero.org/trac/ticket/537
Refs #537
Addresss #352, Make sure data layer doesn't allow bad data via the API
Access date field is now human-friendly. Also enforcing SQL date form for the field in the DB and discarding bad data passed via setField().
Closes#453, Check if any fields will actually be discarded on item type change before giving warning
Refs #530, Add base field conversion to translation level
Added mechanism for linking item type fields via base fields, e.g. publisher => label in audioRecording
New methods:
Item.getFieldsNotInType(itemTypeID, allowBaseConversion)
ItemFields.getLocalizedString(itemTypeID, field)
ItemFields.isBaseField(fieldID)
ItemFields.getFieldIDFromTypeAndBase(itemType, baseField)
ItemFields.getBaseIDFromTypeAndField(itemType, typeField)
ItemFields.getTypeFieldsFromBase(baseField)
Currently only the publisher fields are mapped -- I need more feedback on #346 before I implement the others (specifically on whether or not all these sorts of fields should be done as distinct fields or whether some should just be localized strings (in which case they'll autocomplete but not show up separately as search conditions))
Also added 'university' as distinct publisher field for thesis
Values of equivalent fields are now preserved when switching between item types (e.g. the 'studio' value becomes the 'label' value when switching between videoRecording and audioRecording), and the pop-up is much smarter--it will only prompt you if fields will in fact be lost, and it will list the fields that would be deleted.
Not finished:
- Searching for base fields doesn't yet search the type-specific fields, as Elena requested
- import/export/bib should be updated to use the ItemFields base conversion methods where appropriate -- data coming from the 'publisher' field from translators, for example, should be put into the appropriate type-specific field.
- Automatic tags now appear in orange; tooltip says either "User-added tag" or "Automatically added tag"
- New menu in tag selector to toggle automatic tags
- User and automatic tags are combined in tag selector, so renaming/deleting a tag will affect both user and automatic, regardless of view mode
- Editing a tag makes it a user tag, as does adding an identical user tag to an item (rather than creating a second one)
- ingester/export will need to be adjusted to add automatic tags
Changed:
Item.addTag(tag) => addTag(tag, type)
Item.getTags() - now returns 'id', 'tag', 'type'
Item.toArray() - tags now include 'type' property (from Item.getTags())
Tags.getID(tag) => getID(tag, type)
Tags.getAll() => getAll([types]) - types is an optional array of tagTypes to fetch; now returns objects with 'tag' and 'type' properties
Tags.getAllWithinSearch(search) => Tags.getAllWithinSearch(search, [types]) - now returns object with 'tag'/'type'
Added:
Tags.get(tagID) - returns object with 'tag' and 'type' properties
Tags.getIDs(tag) - returns all tagIDs for this tag (of all types)
Tags.getType(tag) - returns array of tag types matching given tag
For type property, 0 == user, 1 == automatic
Closes#318, Use word index if available to narrow search scope of fulltext scanner
Closes#420, Delete local version of remote scrapers with priority 0 on update
- Changed ZoteroPane.buildItemContextMenu() to make updates easier
- Changed inTransaction repository update delay to 10 minutes
- Fix fulltext search of text files with MIME types that don't begin with 'text/'
- ZoteroPane.setItemsPaneMessage(msg) and ZoteroPane.clearItemsPaneMessage()
- New textbox binding conditional-timed-textbox to run a test (e.g. on the value) before triggering command
- Fulltext content search now supports ANY/ALL instead of just AND (I think)
N.B.: Some changes from plan on ticket
New methods:
Item.setAbstract(true|false) -- make a note an abstract (and clear existing abstract if there is one for source item) or clear abstract status
Item.isAbstract() -- returns true if note is an abstract, false if not
Item.getAbstract() - get itemID of child abstract note or false if none
ZoteroPane.toggleAbstractForSelectedItem()
Changed methods:
Item.updateNoteCache(text, isAbstract)
Notes.add(note, sourceItemID, isAbstract)
Item.setSource() -- moving abstract note to another source with an existing abstract or setting as an independent note will make note not abstract
Other changes:
- Context menu options in items pane: "Set note as abstract" and "Unset note as abstract"
- Child notes are now displayed before child attachments so that abstract will be first
Added minVersion and maxVersion times to existing scrapers, setting 1.0.0b3.r1 as minVersion for any >4096 characters; these could theoretically now be added back to the repository without problems, but there's not really much reason to test that theory at the moment
Fixes#381, SIRSI scraper no longer working at William & Mary
And new Amazon scraper. And a few COinS errors. And possibly some others.
It turns out Firefox has a bug in which DOM nodeValues greater than 4096 characters are split into multiple nodes, and so any scrapers pulled from the repository with 'code' fields greater than 4K were being truncated. We didn't see it during testing of repo code because most are smaller.
Calling normalize() on the node combines the nodes, so future releases won't have the problem regardless of when it's fixed in Firefox. For existing installs, I managed to get PubMed, COinS, SIRSI 2003+, and, with quite a lot of effort, Amazon, under 4096 characters, hopefully without breaking anything. I removed all other scrapers from the repository for now.
- Item.setField() stores dates in a multipart format beginning with an SQL date followed by the user's entry, so "November 3, 2006" becomes "2006-11-03 November 3, 2006" -- date field entries are parsed with Zotero.Date.strToDate() if not already in multipart format
- Item.getField() returns just the user part unless passed the new second parameter, _unformatted_, which returns the field directly from DB without processing (e.g. the full multipart string)
- Added SQLite triggers on the itemData table to enforce multipart format even if the table is modified outside the API
- Migration step to update existing dates
- Indicator next to date field to show what we've parsed and a tooltip over the date field to show the SQL date -- though I'm not sure how well the abbreviation part will localize (i.e. can you abbreviate 'month' in Chinese?)
One obvious problem is how to handle date ranges when sorting or searching, which may end up rendering this whole method fairly useless (though I guess the multipart format could begin with two SQL dates instead of just one, at the cost of some storage space...).
Other changes:
- Utilities.lpad() handling for undefined value parameter
- new Zotero.Date methods: strToMultipart(), isMultipart(), multipartToSQL(), multipartToStr(), isSQLDate(), sqlHasYear(), sqlHasMonth, sqlHasDay getLocaleDateOrder() (the last one unused for now)
- try/catch around manual itemData INSERT execute() statements in Item.save()
Client-side updates
(Note that translator updates were broken anyway, since the translator table was changed and the update mechanism wasn't, so that's fixed now too, and <=1.0.0b2.r1 won't be served updates via the repository.)
- Catch errors trying to display missing files and display message to user
- Switch to persistent descriptors rather than relative paths for attachment paths -- this will fix attachments on networked drives (which, at least on Windows, were not working and apparently in some cases breaking entire Zotero installs), but since WebBrowserPersist.saveDocument() is asynchronous and file.persistentDescriptor can't be set on Macs before the file exists, Attachments.importFromDocument() no longer returns the id of the new attachment, so translate.js had to be changed accordingly
- Try to convert relative descriptors to persistent ones with migration step (and later on access, if persistent fails)
- Added Zotero.WebProgressFinishListener(onDone)
Next step would be to throw up a file dialog to let the user locate the missing file
This can't be enabled on databases with existing tables, so for existing users we can either a) add a FAQ with instructions to recreate and let the advanced users do it and let everyone else have larger tables or b) add a migration step to dump to a new file and then replace the old with the new, which may be a risky and not all that easy proposition (given the lack of .dump) but might work...
The steps, by the way, with Firefox closed:
$ sqlite3 zotero.sqlite
sqlite> .out dump.sql
sqlite> .quit
$ sqlite3 db.sqlite
sqlite> PRAGMA auto_vacuum = 1;
sqlite> .read dump.sql
sqlite> .quit
$ mv db.sqlite zotero.sqlite
If anyone's out there, now would be the time to test this.
Missing icons for the new types -- I could try to get some of those now, or we could just launch. Maybe we'll just launch. But really, might want to test this. Just sayin'.
Including in the DB, which it turns out isn't really all that bad (thanks, among other things, to SQLite's ability to DROP tables within transactions without autocommitting (which MySQL can't do))
