While trying to get translation and citing working with asynchronously
generated data, we realized that drag-and-drop support was going to
be...problematic. Firefox only supports synchronous methods for
providing drag data (unlike, it seems, the DataTransferItem interface
supported by Chrome), which means that we'd need to preload all relevant
data on item selection (bounded by export.quickCopy.dragLimit) and keep
the translate/cite methods synchronous (or maintain two separate
versions).
What we're trying instead is doing what I said in #518 we weren't going
to do: loading most object data on startup and leaving many more
functions synchronous. Essentially, this takes the various load*()
methods described in #518, moves them to startup, and makes them operate
on entire libraries rather than individual objects.
The obvious downside here (other than undoing much of the work of the
last many months) is that it increases startup time, potentially quite a
lot for larger libraries. On my laptop, with a 3,000-item library, this
adds about 3 seconds to startup time. I haven't yet tested with larger
libraries. But I'm hoping that we can optimize this further to reduce
that delay. Among other things, this is loading data for all libraries,
when it should be able to load data only for the library being viewed.
But this is also fundamentally just doing some SELECT queries and
storing the results, so it really shouldn't need to be that slow (though
performance may be bounded a bit here by XPCOM overhead).
If we can make this fast enough, it means that third-party plugins
should be able to remain much closer to their current designs. (Some
things, including saving, will still need to be made asynchronous.)
Absolute paths have been stored as strings on all platforms for a while,
but old Mac persistent descriptors (Base64-encoded opaque alias records)
could still exist in the DB. Additionally, relative paths for stored
files were stored as Mozilla-specific opaque strings rather than UTF-8
strings.
This adds a schema step to convert those to strings paths in the DB.
Since Mac persistent descriptors aren't converted if the file isn't
found, we still handle and (convert) old-style persistent descriptors if
necessary when reading paths from the DB.
This also moves path string handling -- converting a path to a prefixed
string for stored or base-dir-relative files -- to the
Zotero.Item#attachmentPath setter instead of save() so that reading it
back immediately returns the correct value. One consequence is that the
attachment link mode must now be set before setting the path.
Zotero.Item#getFile() is now deprecated in favor of getFilePath() and
getFilePathAsync() (which checks file existence).
Zotero.File.directoryContains() now takes string paths instead of files.
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
- When relinking a missing stored file, copy it into the attachment's
storage directory automatically
- Previously, selecting a file outside the attachment subdir would
just result in a missing attachment, since it only looks for stored
files within the subdir
- Display an error message if a Windows shortcut (.lnk) is added via
drag-and-drop or via a file dialog on non-Windows systems, until we
can figure out how to determine the original file
- Shortcuts can cause errors during syncing, for unclear reasons
- Neither nsIFile::copyToFollowingLinks() nor nsIFile::target work for
me to get the original file, even when nsIFile::isSymlink() returns
true
- Windows file dialogs seem to automatically resolve shortcuts, so
it's only an issue there for drag-and-drop
- Disallow hidden files from being selected in relink dialog
- I think some people on Windows with hidden files shown relink the
.zotero* files that show up when they click Locate, which causes
file sync errors. Which brings us to...
- Fix file sync errors for *.lnk and .zotero* files
- Ignore existing .zotero* attachment files, treating the files as
missing instead to encourage relinking
- Strip leading period in getValidFileName() to prevent added files from
being hidden
- This allows hidden files to be added explicitly; they just won't
stay that way in the storage directory
(These things should have tests, but that will have to happen on the 5.0
branch.)
Restore prepopulated charset table, but this time with just the
encodings from the WHATWG Encoding Standard. Assigning a charset to
Zotero.Item::attachmentCharset runs the value through
Zotero.CharacterSets.toCanonical() automatically.
This migrates attachment charsets to the new canonical values, clearing any
that are unsupported.
Other legacy mappings could still be added back, as disussed in #760.
These previously returned an itemID, but now that new saved items can be edited
without a refetch, they should just return the new item.
(Possible I missed a few spots where these are called.)
Change all attachment functions to take parameter objects, including a
'collections' property to assign collections. (Previously, calling code
assigned collections separately, which required a nested transaction,
which is no longer possible.)
Fixes#723, Can't attach files by dragging
Object contains 'libraryID' and 'key' properties
This is due to changed array destructuring behavior in Firefox. Previously,
`var [foo, bar] = maybeArrayMaybeFalse()` always worked, leaving foo and bar
undefined if the function returned false. Now (with ES6, I assume), if the
function returns false it results in a "false[Symbol.iterator] is not a
function" error. But `var {libraryID, key} = false` works as expected, leaving
both values undefined, so instead we can just return an object with those
properties from getLibraryAndKeyFromID(). To assign to different variables, use
`var {libraryID, key: parentItemKey} = ...`.
Note that this loses conflicting changes to translate_item.js from 849803473a,
so those will need to be reapplied if applicable. /cc @aurimasv, @mtd91429
Adds a "My Publications" source after "My Library", implemented as a
separate library. Top-level items can be dragged in and removed.
(This doesn't currently work without disabling Quick Copy.)
Also:
- Make "Group Libraries" an unselectable header instead of a container,
and don't indent group libraries
- Fix relation purging, which maybe never worked
- Pass only libraryID/key on deletes (which should speed them up)
- Fix async item cloning/copying
- Fix miscellaneous other bugs
To-do:
- Confirmation dialog on drag
- API support
Instead of limiting charsets to a fixed list, dynamically populate it
with any charset name of less than 50 ASCII characters. Previously,
unknown charsets were discarded.
Zotero.Item.prototype.attachmentCharset now always returns a charset
name. It can be set with either a name or a charsetID.
Also:
- Remove the unused 'originalPath' column in itemAttachments
Generated files for a more robust attached-link-dialog and localized strings
* AttachLink.js
* AttachLink.xul
zotero/xpcom/attachments.js
* created function cleanAttachmentURI
* in function linkFromURL, removed the regex constraints and the comment list of valid protocols
* removed outdated function declaration from beginning of script
* Improved automatic title generation mechanism
It's way too slow, though, since the whole list is regenerated after
merging.
Fixes#519
Also:
- The arguments to Zotero.Item.prototype.clone() have changed, and it no
longer takes an existing item or copies primary data. To create an
in-memory copy of an item, use the new Zotero.Item.prototype.copy().
- Zotero.Item.prototype.getUsedFields() now gets in-memory fields rather
than the fields in the database
This required doing additional caching at startup (e.g., item types and fields)
so that various methods can remain synchronous.
This lets us switch back to using the current Sqlite.jsm. Previously we were
bundling the Fx24 version, which avoided freezes with locking_mode=EXCLUSIVE
with both sync and async queries.
Known broken things:
- Autocomplete
- Database backup
- UDFs (e.g., REGEXP function used in Zotero.DB.getNextName())
- Zotero.Item.prototype.getFilePath() is now synchronous, with a separate async getFilePathAsync()
- getFile() no longer takes a skipExistsCheck parameter, since that shouldn't happen synchronously
- Zotero.Items.getByLibraryAndKey() is now synchronous again, with a
separate Zotero.Items.getByLibraryAndKeyAsync() - I haven't fully
tested this, so I'm not sure if there will need to be any async
calls.
- Some of the full-text indexing functions now take file paths instead of nsIFile objects
- Zotero.File.getContentsAsync() can now take a string path as well
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
Full-text content is now synced to the server and other clients. Up to
~500K of content is transferred with each sync, so multiple syncs may be
necessary. Downloaded content is written to temporary files and
processed on idle (currently 5 seconds), so synced content may not be
immediately available.
Performance will improve with API syncing and async DB in 4.1.
This also includes some other full-text tweaks, such as storing
text converted from HTML in cache files, which may speed up phrase
searching.
