A queue can be created and passed as an option to data layer methods, which
will then queue events on that queue instead of the main internal queue. A
queue or an array of queues can then be passed to Zotero.Notifier.commit() to
commit those events.
Some auxiliary functions don't yet take a queue, so those events will still get
run on DB transaction commit.
Sync data processing now processes notifier events in batches to reduce
repaints, even though individual objects are processed within their own
transactions (so that failures don't roll back other objects' data).
Also remove some unused notifier code
I'm not sure this matters -- since it was broken, itemTreeView notify()
code probably accounts for situations where, say, a 'modify' comes in
before an 'add' -- but it might avoid some edge cases in the future.
It's generally not necessary to call this, but it can be useful if
specific data needs to be loaded in an unloaded library (e.g., an item's
creators for RTF Scan)
`nArcs` is undefined in `this.ItemProgress.prototype.setProgress()`. This has been the case for a long time but never came up because the progress indicator isn't used in Zotero, I believe. I think it's a nice UI element though and could be used for non-disruptive progress indicators.
- Only show first pane (without Sharing pane) if no files or files
aren't included
- Update authorship checkbox to reflect file include setting to avoid
confusion
- Clarify in intro text that license applies to files, not notes --
notes here are no different than notes in any public library, so it's
sort of up to the user to clarify those if they're substantial enough
for it to matter
- Adjust alignment of authorship checkbox if more than one line
Otherwise, when right-clicking on a collection that's not currently
selected, some of the menu items appear gray at first, and a second
right-click is necessary after the items have loaded. This way the menu
items turn black once the items have loaded.
Items in a library are now loaded only when a library is clicked on and
at sync time. There might be some other areas where they need to be
loaded or where this causes problems (e.g., drag and drop, word
processor integration).
I think this only happens if the Zotero pane hasn't yet been opened, which also
means an initial refresh() hasn't been done, which means that updates aren't
necessary.
Otherwise the test could run scrapeThisPage() before translators were ready. It
would be good to make scrapeThisPage wait for detection to complete so that an
early press still uses a translator for saving, but this way tests can also
test for the proper icon (though they don't now).
Instead of requiring a click on the menuitem at the top of the submenu, allow
a click on the menu itself.
This is a hack that, among other things, replicates the flash effect on
menuitems on OS X. Unfortunately, <menu> elements can't have checkboxes, so
only the menuitem in the submenu will be checked. (Otherwise I'd remove the
redundant menuitem in the submenu.)
- Allow clicking on non-editable abstract field to expand/collapse it
- Change cursor when hovering over abstract to show it can be toggled
- Default abstracts to expanded
- Fix persistence of last translation target
- Add checkbox to menuitem of selected target
- Remove unnecessary flex attributes
Also:
- Move collectionTreeView row id (e.g., "L1", "C123") and image
generation to Zotero.Library and Zotero.Collection properaties,
.collectionTreeViewID and .collectionTreeViewImage -- currently used
only for the feed add-to button, but could be expanded for use in
collectionTreeView
And without extra init() call
This fixes the button being initialized before Zotero.hiDPI is set, resulting
in a low-res icon until the target is changed.
The on-change auto-sync now syncs only the modified library, and does so
quite efficiently (and should be able to be made more efficient), so we
might be able to reduce the timeout below 15 seconds.
Instead of getting batches of unused primary key ids, even if they're lower
than other ids, which for some reason seemed like a good idea in 2008, just do
a `MAX()` on the table at startup and return the next available id on each call
to `Zotero.ID.get()`. This is much simpler, and not reusing ids allows them to
be used as a chronological sort field.
While SQLite's `SELECT last_insert_rowid()` could return auto-increment values,
it's unsafe with async DB access, since a second `INSERT` can come in before
the first `last_insert_rowid()` is called. This is true even in a transaction
unless a function that calls it is never called in parallel (e.g., with
`Zotero.Promise.all()`, which can be faster than sequential `yield`s).
Note that the next id is always initialized as MAX() + 1, so if an object is
added and then deleted, after a restart the same id will be given. (This is
equivalent to (though unrelated to) SQLite's `INTEGER PRIMARY KEY` behavior,
as opposed to its `INTEGER PRIMARY KEY AUTOINCREMENT` behavior.)
Closes#993, Feed items out of order
Previously, objects were first downloaded and saved to the sync cache,
which was then processed separately to create/update local objects. This
meant that a server bug could result in invalid data in the sync cache
that would never be processed. Now, objects are saved as they're
downloaded and only added to the sync cache after being successfully
saved. The keys of objects that fail are added to a queue, and those
objects are refetched and retried on a backoff schedule or when a new
client version is installed (in case of a client bug or a client with
outdated data model support).
An alternative would be to save to the sync cache first and evict
objects that fail and add them to the queue, but that requires more
complicated logic, and it probably makes more sense just to buffer a few
downloads ahead so that processing is never waiting for downloads to
finish.
- Hide notes, tags and related for feed items in itembox
- Add feed support for <enclosure> elements
- Add feed syncing methods for synced settings (additional work is
needed on the sync architecture to download synced settings from the
server)
- Change feed item clear policy to be less aggressive
- Adjust for deasyncification
- Disable translate-on-select
- Closeadomasven/zotero#7, Remove context menu items from feeds
This reverts Zotero.Translate.ItemGetter.prototype.nextItem() to being
synchronous post-deasyncification. This will need to be made to work
asynchronously in the future if _attachmentToArray(), which is called by
nextItem, is changed to use async file access (which might be required
at some point).
Addresses #734, [Async DB] Import/export fails
notificationCallbacks no longer QIs to webNavigation in Firefox 46, so
this gets the relevant browser object by getting the content window id
from channel.loadInfo and finding the window via nsIWindowMediator.
Unfortunately the window id provided under e10s is invalid (or something
-- it's a very high number that can't be found via nsIWindowMediator),
so something else will need to be done for that.
This allows Zotero.Relations.getByPredicateAndObject()/getByObject() and
Zotero.Item::getLinkedItem()/Zotero.Collection::getLinkedCollection() to
be synchronous, which is necessary for word processor integration.
Instead of creating a duplicate copy of the item with the same primary
data and saving that, it's safer just to use clone() (which doesn't
preserve ids) and apply changes to the main object.
While trying to get translation and citing working with asynchronously
generated data, we realized that drag-and-drop support was going to
be...problematic. Firefox only supports synchronous methods for
providing drag data (unlike, it seems, the DataTransferItem interface
supported by Chrome), which means that we'd need to preload all relevant
data on item selection (bounded by export.quickCopy.dragLimit) and keep
the translate/cite methods synchronous (or maintain two separate
versions).
What we're trying instead is doing what I said in #518 we weren't going
to do: loading most object data on startup and leaving many more
functions synchronous. Essentially, this takes the various load*()
methods described in #518, moves them to startup, and makes them operate
on entire libraries rather than individual objects.
The obvious downside here (other than undoing much of the work of the
last many months) is that it increases startup time, potentially quite a
lot for larger libraries. On my laptop, with a 3,000-item library, this
adds about 3 seconds to startup time. I haven't yet tested with larger
libraries. But I'm hoping that we can optimize this further to reduce
that delay. Among other things, this is loading data for all libraries,
when it should be able to load data only for the library being viewed.
But this is also fundamentally just doing some SELECT queries and
storing the results, so it really shouldn't need to be that slow (though
performance may be bounded a bit here by XPCOM overhead).
If we can make this fast enough, it means that third-party plugins
should be able to remain much closer to their current designs. (Some
things, including saving, will still need to be made asynchronous.)
If 'pdf' flag is included in object POSTed to saveSnapshot, import the
PDF directly and save as top-level item. Currently the PDF is
redownloaded -- there might be a better way to get the PDF data over
without redownloading. (It uses passed cookies, though, so gated PDFs
should still work.)
I guess the idea that switching to PNGs obviated the need for this was wishful
thinking (though it doesn't seem to be necessary for the single buttons anymore
on Linux, and it's no longer necessary on OS X, which has generally saner
styling in Firefox).
- Fix spacing on Windows and Linux in latest Firefox versions
- Tweak icon colors on Windows and OS X
- Adjust Z SVG to take up full height, so Z is a full 16px instead of
14px with slight anti-aliasing
- Use generated PNGs instead of SVG for Z toolbar icons, to remove the
need for complicated size rules
- Add separate platform-specific .svg files that are used by a
zotero-build script, make-z-icons, to generate the Z PNGs; the main
SVG is still used directly in the menu panel and customization
palette, with platform media queries to determine the coloring
Unfortunately this will need to be partly redone, since retrieveItem(), and
therefore itemToCSLJSON(), and therefore itemToExportFormat(), need to be
synchronous. The item data load statements in itemToExportFormat() will
probably need to be performed earlier, when they can be async, and made
available to the session for retrieval by retrieveItem(), but I'll let someone
more familiar with the citation infrastructure do that.
This restores some code in retrieveItem() that may have been accidentally
removed in a merge, though it probably won't be useful anymore anyway.
Addresses #529
Now that 500 errors are retried in file downloads (ec28c5a3), we have to
override the default backoff schedule in order to get expected failures.
This also fixes an error that occurred on a retried download.
In Fx44, SQL queries must use '?' with LIKE and cannot concatenate a
placeholder string (e.g., 'foo%'). This is for Sqlite.jsm only, so it
doesn't affect 4.0.
For now, delete it and clear it from the cache so it's updated properly going
forward, but really we want to reinstall the correct file automatically
(#903).
This apparently has been happening for a year, since #617.
Might be better to show open/save on cancel, as implemented for the
RIS/BibTeX prompt in 4ecdd55 (which caused this), but that would have to
wait for the better promise handling in 5.0. Doing "Save Link As..." for
a CSL file also seems a bit more straightforward than for RIS/BibTeX,
which can be served in complicated ways.
Instead, pass ids directly to SQLite. This seems to take about the same
amount of time or a little less (by avoiding the time it takes to start
a transaction) and avoids blocking other transactions when switching
views.
Most checks are now just foreign key checks and can be checked with
"PRAGMA foreign_key_checK".
The DB Repair Tool will need to be updated to handle the new schema (but
still accept the old one).
Fixes the incompatible-version dialog and adds info on the version used
to upgrade the database, which may be helpful for troubleshooting.
Also fixes showing of the Zotero toolbar icons even in case of a startup
error.
Also:
- Remove last-sync-time mechanism for both WebDAV and ZFS, since it can
be determined by storage properties (mtime/md5) in data sync
- Add option to include synced storage properties in item toJSON()
instead of local file properties
- Set "Fake-Server-Match" header in setHTTPResponse() test support
function, which can be used for request count assertions -- see
resetRequestCount() and assertRequestCount() in webdavTest.js
- Allow string (e.g., 'to_download') instead of constant in
Zotero.Sync.Data.Local.setSyncState()
- Misc storage tweaks
Improves UX of sync authentication.
The account is now linked and unlinked and an API key related to
the client is generated transparently in the background.
The API key is deleted on unlinking.
No sync options are allowed before linking an account.
Closes#864
This adds a 'channel' property to Zotero.HTTP.UnexpectedStatusException,
because the 'channel' property of the XHR can be garbage-collected
before handling, and the channel's 'securityInfo' property is necessary
to detect certificate errors.
This should be refactored further, but this is a start at swapping in
Bluebird. Unfortunately the process doesn't work correctly because of
issues in the translation framework, which will need to be addressed
separately -- right now a parent item is successfully created but the UI
still shows an error and the PDF isn't placed under the parent.
Closes#852
Currently shows a *+* icon for adding a tag and
a *link* icon for removing it.
There does not appear to be any way to have a custom mouse cursor,
which makes this interaction extremely unintuitive. It might be
possible to hack this in, but it does not really warrant the effort.
E.g. `window.setCursor` does not respond whilst dragging.
This changes Zotero.Translate.Base.translate() to take an options object (in
order to take a 'collections' parameter, which is passed to the
Zotero.Translate.ItemSaver constructor). The old parameters are still supported
with a deprecation warning, and there may be other places that still need to be
updated.
Display a warning when choosing data directory
Or on opening Zotero Pane for existing users with data dir in dropbox
Also:
Fix a bug where it won't use custom path if "Choose..." button is
pressed instead of radio button.
Change filepicker to show current data directory on display
Disallow everything other than adding a child note in My Publications,
since the wizard only shows on drag (currently, at least). Need to
disable translation separately.
Also show "Delete Item[s]..." in the trash context menu, and hide rather
than disable various menu options.
Addresses #703 -- still need to disable saved search saving
This mostly gets ZFS file syncing and file conflict resolution working
with the API sync process. WebDAV will need to be updated separately.
Known issues:
- File sync progress is temporarily gone
- File uploads can result in an unnecessary 412 loop on the next data
sync
- This causes Firefox to crash on one of my computers during tests,
which would be easier to debug if it produced a crash log.
Also:
- Adds httpd.js for use in tests when FakeXMLHttpRequest can't be used
(e.g., saveURI()).
- Adds some additional test data files for attachment tests
When called on an identified object (i.e., one with an id or
library/key), loadAllData() must be called first. When called on a new
object (which is more common anyway), fromJSON() can be called
immediately.
If Standalone is closed in full-screen mode in 10.11, 'sizemode' on the main
window is persisted as 'fullscreen'. On reopen, the window doesn't go into
full-screen mode, but it still lacks menubar or titlebar. This patch forces the
window into 'normal' mode if it was left in 'fullscreen'.
(Firefox appears to ignore 'fullscreen' without actually changing the value,
but I didn't find the code that handles that.)
Absolute paths have been stored as strings on all platforms for a while,
but old Mac persistent descriptors (Base64-encoded opaque alias records)
could still exist in the DB. Additionally, relative paths for stored
files were stored as Mozilla-specific opaque strings rather than UTF-8
strings.
This adds a schema step to convert those to strings paths in the DB.
Since Mac persistent descriptors aren't converted if the file isn't
found, we still handle and (convert) old-style persistent descriptors if
necessary when reading paths from the DB.
This also moves path string handling -- converting a path to a prefixed
string for stored or base-dir-relative files -- to the
Zotero.Item#attachmentPath setter instead of save() so that reading it
back immediately returns the correct value. One consequence is that the
attachment link mode must now be set before setting the path.
Zotero.Item#getFile() is now deprecated in favor of getFilePath() and
getFilePathAsync() (which checks file existence).
Zotero.File.directoryContains() now takes string paths instead of files.
In Firefox 41, file: URIs can no longer be loaded (at least via Image)
from the hidden window on Windows/Linux, but chrome: URIs still work.
Not sure why I was using a file: URI to begin with.
If multiple collections are highlighted and none are in view, scroll to
the first one in the list.
The logic could be improved here a little more to scroll to the closest
collection instead of the first one, and also to scroll to a few rows
above or below the target.
This also fixes what was probably an incorrect highlight if there were
multiple collections and some had to be expanded first.
Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
Firefox no longer supports transparent windows because it made them do
extra preprocessing on a file to restore the drop shadow when the
Developer Edition theme was active:
https://bugzilla.mozilla.org/show_bug.cgi?id=1162649
This will appear much less frequently, since non-conflicting field changes on
both sides can be resolved automatically, but genuine field conflicts still
require manual conflict resolution.
The merge pane is no longer editable, since the itembox code to do that is
async and can't run in a modal window, but it's not really necessary,
particularly with conflicts happening less frequently.
TODO:
- Remote item deletions
- File conflicts
- Maybe handle some edge cases where the conflicted items fail to save
Save uploaded data to cache, and update local object if necessary (which
it mostly shouldn't be except for invalid characters and HTML filtering
in notes)
Also add some upload and JSON tests
If a file existed locally but somehow ended up marked as to-download without
existing on the server, it was never uploaded. I'm not sure when this can
happen, but I saw it while messing around. Maybe switching between ZFS and
WebDAV?
This will still only check and upload if there's another computer syncing files
to the same library, but we'll check all files in 5.0.
There's a lot more to do, and this isn't ready for actual usage, but the
basic functionality is mostly in place and has decent test coverage. It
can successfully upgrade a library last used with classic syncing and
pull down changes via the API. Uploading mostly works but is currently
disabled for safety until it has better test coverage.
Downloaded JSON is first saved to a cache table, which is then used to
populate other tables and later for generating PATCH requests and
automatically resolving conflicts (since it shows what was changed
locally and what was changed remotely). Objects with unmet dependencies
or unknown fields are skipped for now but don't block the rest of the
sync.
Some of the bigger remaining to-dos:
- Tests for uploading
- Re-do the preferences to get an API key
- File sync integration
- Full-text syncing integration
- Manual conflict resolution (though this already includes much smarter
conflict handling that automatically resolves many conflicts)
Previously, if .synced was already true, setting it to true and saving
would result in .synced == false unless skipSyncedUpdate was passed. Now
the value assigned to .synced is always used on the next save. If the
value hasn't changed and no other values have changed, a save will be a
no-op.
The default items cause problems with conflict resolution for existing
users (and not syncing them or ignoring conflicts for them is kind of
weird), and they require remote changes for new databases. I do like
there not being a completely empty library, but I think it's probably
better just to display a virtual welcome message with a link to the
Quick Start Guide somewhere else, such as in the right-hand pane. (A new
installation also opens the start page on zotero.org.)
Allow saving of CSS links over chrome://. We can scope this to
chrome://global/skin/aboutReader.css if anyone thinks of a reason why
this was disabled to begin with, but I'm not sure in what other
situations CSS it would apply.
- Restore About Zotero item
- Fix Quit item
- Close Standalone when Standalone window is closed (we did this before,
but apparently having our own hidden window breaks that mechanism)
Technically rather than waiting for the schema update we should wait for
translator initialization, which should wait for the schema update
itself, but the schema update also needs to initialize the translators,
so avoiding a hang is tricky, particularly with the use of Zotero.lazy()
for Zotero.Translators.init(). For now, just wait for the schema update.
Clicking the separate buttons would trigger actions in the first window
opened, and other things might not have been updating properly across
multiple windows.
Previously, clicking directly on a noautohide guidance panel closed it, even if
there was a forward nav button. (In the case of the main and save button
guidance panels, the save button would appear on the next Firefox restart, but
this ensures that the save guidance panel will be shown to new users even if
they click on the panel instead of the not-particularly-noticeable forward
button.)
In practice, this appears to be both unnecessary and harmful. Although
the first write happens effectively instantaneously, we can take
several hundred milliseconds to read the second write, and Standalone
can decide we are dead.
This apparently fixes#783 for @dstillman, although I was never able to
reproduce locally.
(I don't see any obvious reason I can't do this. We should probably clean this
up later, not call the endpoint "saveSnapshot", etc. Also, is there a reason it
doesn't follow the snapshot pref by default?)
And fix save button hover height on Windows, which somehow got out of
whack again.
Might need further tweaking on Linux
Need to find a better way to do these buttons...
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
For new installations, a panel will show for the Z icon with a ">"
button to go to a second panel for the save icon. For existing
installations, a panel will show just for the save icon. In both cases,
the panels are non-auto-hiding but are dismissed if clicked on or when a
new content page is loaded.
There's also a new close button, but it's not enabled currently. Might
be useful for panels with embedded links where an accidental click on
the panel alone shouldn't close the panel (like the Firefox customize
mode wizard).
- When relinking a missing stored file, copy it into the attachment's
storage directory automatically
- Previously, selecting a file outside the attachment subdir would
just result in a missing attachment, since it only looks for stored
files within the subdir
- Display an error message if a Windows shortcut (.lnk) is added via
drag-and-drop or via a file dialog on non-Windows systems, until we
can figure out how to determine the original file
- Shortcuts can cause errors during syncing, for unclear reasons
- Neither nsIFile::copyToFollowingLinks() nor nsIFile::target work for
me to get the original file, even when nsIFile::isSymlink() returns
true
- Windows file dialogs seem to automatically resolve shortcuts, so
it's only an issue there for drag-and-drop
- Disallow hidden files from being selected in relink dialog
- I think some people on Windows with hidden files shown relink the
.zotero* files that show up when they click Locate, which causes
file sync errors. Which brings us to...
- Fix file sync errors for *.lnk and .zotero* files
- Ignore existing .zotero* attachment files, treating the files as
missing instead to encourage relinking
- Strip leading period in getValidFileName() to prevent added files from
being hidden
- This allows hidden files to be added explicitly; they just won't
stay that way in the storage directory
(These things should have tests, but that will have to happen on the 5.0
branch.)
- Simplified schema
- Tags are now added without reloading entire tag selector
- On my system, adding 400 tags to an item (separately, with the tag
selector updating each time) went from 59 seconds to 42. (Given that
it takes only 13 seconds with the tag selector closed, though,
there's clearly more work to be done.)
- Tag selector now uses HTML flexbox (in identical fashion, for now, but
with the possibility of fancier changes later, and with streamlined
logic thanks to the flexbox 'order' property)
- Various async fixes
- Tests
To be used by updated deletion listener in new sync code
Also adds explicit Zotero.SyncedSettings.clear() in place of
Zotero.SyncSettings.set() without a value
Still need to make the progress indicator work again. Also there may be
some performance to be gained by pooling item saves into a transaction
if one is already open.
Restore prepopulated charset table, but this time with just the
encodings from the WHATWG Encoding Standard. Assigning a charset to
Zotero.Item::attachmentCharset runs the value through
Zotero.CharacterSets.toCanonical() automatically.
This migrates attachment charsets to the new canonical values, clearing any
that are unsupported.
Other legacy mappings could still be added back, as disussed in #760.
This output is basically only useful when something hangs, so it can
stay off all other times.
For now, 6 can be the new only-use-when-something-is-actively-broken
level. At some point we may want to move DB activity to 4 and make this
sort of thing 5, because we don't have much that's 4 right now.
This uses ISO 8601 dates for generateAllTypesAndFieldsData (and
changes populateDBWithSampleData to use Item#fromJSON), and makes
translators expect ISO 8601 accessDates, although SQL accessDates are
still supported with a deprecation warning. Canonicalization happens in
Zotero.Translate, so I need to remember to update connectors as well.
And add group.fromJSON(json, userID), which sets editable and
filesEditable properties based on the group JSON (libraryReading, role
lists, etc.) and the given user
sinon.useFakeXMLHttpRequest() doesn't work in our case, but if
Zotero.HTTP.mock is set, Zotero.HTTP.request() will create new instances of
that object instead of the built-in XMLHttpRequest, so it can be set to
FakeXMLHttpRequest:
Zotero.HTTP.mock = sinon.FakeXMLHttpRequest;
var server = sinon.fakeServer.create();
server.autoRespond = true;
server.respondWith("GET", "/users/1",
[200, {"Content-Type": "application/json"}, '{"userID": 1}']);
var userInfo = yield getUser();
Zotero.HTTP.mock = null;
server.restore(); // probably not necessary
Also adds Zotero.DataObjects::getLoaded(), which returns an array of all
loaded objects of the given type. This is useful for selective
reloading, for example with item.reload(['relations'], true).
In particular, 0 is kept as a value, and passing undefined to setField
now throws an error.
I'm not sure if we actually want to return an empty string in all cases
for missing/invalid fields, but that's what we do currently.
And simplify tree view load event handling, which may or may not have
been contributing to intermittent test failures, but is cleaner this way
regardless.
Relations are now properties of collections and items rather than
first-class objects, stored in separate collectionRelations and
itemRelations tables with ids for subjects, with foreign keys to the
associated data objects.
Related items now use dc:relation relations rather than a separate table
(among other reasons, because API syncing won't necessarily sync both
items at the same time, so they can't be stored by id).
The UI assigns related-item relations bidirectionally, and checks for
related-item and linked-object relations are done unidirectionally by
default.
dc:isReplacedBy is now dc:replaces, so that the subject is an existing
object, and the predicate is now named
Zotero.Attachments.replacedItemPredicate.
Some additional work is still needed, notably around following
replaced-item relations, and migration needs to be tested more fully,
but this seems to mostly work.
The test runner now downloads and caches the PDF tools for the current
platform within the test data directory and only redownloads them when
out of date, and it updates the download URL so that the full-text code
pulls from the cache directory via a file:// URL.
The installPDFTools() support function now installs the files directly
instead of going through the prefs, and a new uninstallPDFTools()
function removes the tools. Since the presence of the PDF tools can
affect other tests, tests that need the tools should install them in a
before() and uninstall them in an after(), leaving most tests to run
without PDF indexing.
This also adds a callback to the waitForWindow() support function. If a
modal dialog is opened, it blocks the next promise handler from running,
so a callback has to be used to interact with and close the dialog
immediately.
Character sets are now populated on demand, so they can't be run through
Zotero.CharacterSets.getName() in getContentsAsync(), since they might
not exist. (I'm also not sure why this was being done anyway.)
For now, this just calls Zotero.Utilities.generateObjectKey(), but this
function makes more sense in DataObjectUtilities. It does need to be
accessible to the connectors, but if it's possible to add an alias in
Zotero.Utilities just for the connectors, it'd probably be better to do
that and use Zotero.DataObjectUtilities.generateKey() elsewhere.
These previously returned an itemID, but now that new saved items can be edited
without a refetch, they should just return the new item.
(Possible I missed a few spots where these are called.)
Notifier.trigger() needs to be async, since if it actually runs it waits for
promises returned from observers. But the vast majority of trigger() calls are
in transactions where they just queue and can therefore be synchronous. This
replaces all such calls with Notifier.queue().
This should fix a race condition that was causing the emptyTrash() test to fail
intermittently.
citeproc-js relies on this in several locations. Seems that Zotero passes these IDs to citeproc from the item picker. We also need to consider existing embedded items in Word/LO documents, but they do have embedded URIs, so it shouldn't be a problem.
CC @fbennett
* Enable legacy mode for export translators compatible with pre-4.0.27:
* Add compatibility mappings, so that current translators don't break if they specify minVersion lower than 4.0.27. This does introduce non-compatible changes, specifically, "version" field in legacy mode is "versionNumber" in the new format. "version" in the new format corresponds to the "version" as specified for Zotero API JSON format. New translators should expect Zotero web API JSON format and should specify minVersion 4.0.27.
* Update CSL mappings to comply with new itemToExportFormat
* CSL JSON export translator needs to be updated to be compatible with 4.0.27 to export correct CSL JSON
* Use item URI for id in CSL JSON instead of item ID
* Fix note and attachment handling in itemToCSLJSON
After saving a new object and reloading primary data and any changed
data (which we can maybe reconsider at some point), mark all other data
types as loaded, since there's no other data we don't have. For example,
this allows for item.save() to be followed by item.setField() without
needing to call item.loadItemData() first.
updateBundledStyles() already has the contents of a new/updated
translator when it copies it into the data dir, so there's no need for
Zotero.Translators.reinit() to read it again from the just-copied file.
This passes the metadata from updateBundledStyles() to reinit() to avoid
the extra file read.
(Alas, this appears to make essentially zero difference on an OS X
system with an SSD, but maybe it will help elsewhere.)
Get rid of data_access.js, at long last. Existing calls to
Zotero.getCollections() will need to be replaced with
Zotero.Collections.getByLibrary() or .getByParent().
Also removes Zotero.Collection::getCollections(), which is redundant
with Zotero.Collections.getByLibrary(), and Zotero.Collections.add().
The latter didn't didn't include a libraryID anyway, so code might as
well just use 'new Zotero.Collection' instead.
If a view or other resources are destroyed while a promise is being
resolved, subsequent code can fail. This is generally harmless, but it
results in unnecessary errors being logged to the console.
To address this, promises can use a new function,
Zotero.Promise.check(), to test whether a value is truthy or 0 and
automatically throw a specific error that's ignored by the unhandled
rejection handler if not.
Example usage:
getAsync().tap(() => Zotero.Promise.check(this.win));
If this.win is cleaned up while getAsync() is being resolved, subsequent
lines won't be run, and nothing will be logged to the console.
Change all attachment functions to take parameter objects, including a
'collections' property to assign collections. (Previously, calling code
assigned collections separately, which required a nested transaction,
which is no longer possible.)
Fixes#723, Can't attach files by dragging
Wait for the pane's collectionSelected() to finish before returning from
collectionTreeView select methods (e.g., selectLibrary()), and wait for
previous items view to finish loading before creating a new one in
collectionSelected(). This ensures that the items view has been created (though
not loaded) before returning from a select. The tree can still get a bit
confused switching between collections, but I think we're getting closer to
fixing that.
Also switch the items tree to use the same pattern.
This also fixes dragging items to collections (#731).
Groups were already being loaded for the collections list, so we might
as well just store them initially and let Zotero.Libraries.getName() be
a synchronous call.
- Moved ::_get() and _set() from Collection/Search into DataObject, and
disabled in Item
- Don't disable new items after save. We now put new objects into the
DataObjects cache from save() so that changes made post-save are
picked up by other code using .get().
- Added 'skipCache' save() option to avoid reloading data on new objects
and adding them to the cache. (This will be used in syncing, where
objects might be in another library where they're not needed right
away.) Objects created with this option are instead disabled to
prevent reuse.
- Modified some tests to try to make sure we're reloading everything properly
after a save.
- Documented save() options
- Fixes some saving and erasing issues with collections and searches
- Adds Zotero.DataObject::eraseTx() to automatically start transaction,
and have .erase() log a warning like .save()
- Adds createUnsavedDataObject() and createDataObject() helper functions
for tests
The sync cache will have pristine copies of the existing versions of
local objects for better conflict resolution, but downloads will get
saved to the sync cache first before processing, so the cache needs to
be able to hold more than one version.
Replace Z.DataObjects::diff() with Z.DataObjectUtilities.diff(). Instead
of just returning two objects with the differing fields, the new diff()
generates a changeset with operations to apply with applyChanges(),
including at the array member level for collections and tags. This,
combined with cached pristine copies of objects, will allow for vastly
better conflict resolution, with automatic merging of non-conflicting
changes.
Creators currently don't show granular changes, and ordering might make
it too tough to do so. Relations diffing isn't yet implemented.
Also:
- Make .mode == 'patch' optional if .patchBase is provided.
- Remove requirement for item to be unchanged, which hopefully wasn't there for
a good reason
- Add a few tests, though more are needed
This allows calling code to do something other than call Zotero.debug()
on errors (like, say, nothing, in order to avoid logging certain
expected errors) before throwing.
Store and check the last selected items in ZoteroPane.itemSelected() to
see if it's necessary to refresh the item pane. This prevents loss of
textbox focus if another write occurs while editing a field.
Also optimize row adding/removing in itemTreeView.js
Fix a couple cases of lost text field focus after an edit, including
focusing of the Title field after using New Item when a field is already
being edited and has a changed value.
Also, in tests, select My Library and wait for items to load when using
the loadZoteroPane() support function. We could add a parameter to skip
that or move it to a separate function, but the code to detect it is a
bit convoluted and it's a prerequisite for many tests, so it's handy to
have a function for it.
If a new item is created manually when a text field is already open, the
hideEditor is run first on the html:textarea before bubbling up to the
textbox. During a normal blur, the event only happens on the textbox,
though I haven't looked into the reason for the difference.
- Add an 'exclusive' option to transactions that causes them to block other
transactions and wait for other transactions to finish before starting,
instead of nesting
- Resolve Zotero.DB.waitForTransaction() promise before returning from
executeTransaction()
- A side effect of the above: wait for a newly created item to be selected in
the middle pane and rendered in the right-hand pane before returning from
executeTransaction()
- Don't save items multiple times when adding/removing a non-final creator in
the Info pane
- Use a simpler, non-recursive method for focusing the next field in the Info
pane; this prevents "too much recursion" errors if something causes the
right-hand pane not to be rendered when expected
This fixes an issue where two transactions started around the same time
could run separately instead of nesting, causing the statements from one
to end up running not within a transaction
And use 'version' instead of 'itemVersion' for object version for items
Also add deferred foreign key checking to system.sql so that DROP TABLE
commands don't fail mid-transaction
Show a wizard after items are dragged to My Publications choosing
whether to include files and notes and choosing sharing settings for the
items. Sharing options are Creative Commons licenses, CC0, "All rights
reserved", or keeping the existing Rights field if available.
Also blocks collections, searches, linked file attachments, and
top-level attachments/notes from My Publications at the data layer, but
not yet from the UI in all places (so it can crash if you try).
Todo:
- Block certain UI actions with nice messages
- Show a nice scrollable list of items in the wizard to allow selecting
specific files/notes, instead of just having checkboxes for files and
notes that apply to all dragged items
- Show an explanation of My Publications in the right-hand pane when no
items are selected
- Maybe adjust handling when no attached files/notes, since it might be
a bit alarming at the moment to see sharing options for metadata
entries
Previously, object identifiers were registered in a commit callback, but that
meant that they wouldn't be available to getLibraryAndKeyFromID/
getIDFromLibraryAndKey within a transaction. Instead, register them before
saving and clear them in a tranasction rollback if necessary.
This changes Zotero.DataObject to call its own _finalizeSave() in addition to a
descendent one.
Object contains 'libraryID' and 'key' properties
This is due to changed array destructuring behavior in Firefox. Previously,
`var [foo, bar] = maybeArrayMaybeFalse()` always worked, leaving foo and bar
undefined if the function returned false. Now (with ES6, I assume), if the
function returns false it results in a "false[Symbol.iterator] is not a
function" error. But `var {libraryID, key} = false` works as expected, leaving
both values undefined, so instead we can just return an object with those
properties from getLibraryAndKeyFromID(). To assign to different variables, use
`var {libraryID, key: parentItemKey} = ...`.
Search condition ids are now indexed from 0, and always saved
contiguously (no more 'fixGaps' option), since they're just in an array
in the API. (They're still returned as an object from
Zotero.Search.prototype.getConditions() because it's easier for the
advanced search window to not have to deal with shifting ids between
saves.)
Also:
- Return an object from `Zotero.Search.prototype.getConditions()`
instead of an array.
- Add support function `getPromiseError(promise)` to return the error
thrown from a chain of promises, or false if none. (We could make an
`assert.throwsAsync()`, but this allows testing of various properties
such as `.name`, which even the built-in `assert.throws()` can't
test.)
- Clarify some search save errors
'b' for *b*undled files
Translators and styles take a long time to install and initialize in
source installations, and they're unnecessary for many tests.
This shaves about 10 seconds off each test run for me on one system (and
that's with some help from filesystem caching).
This is just a wrapper around nsIPromptService.alert() that takes
Zotero.noUserInput into consideration, which avoids long timeouts during
testing (e.g., for lookup failures).
Previously, getItemURI and getCollectionURI had to check the libraryID
to determine whether to return a local URI, and were doing so using an
obsolete check.
To deal with issues running .vbs scripts on some systems. If the 3.02
binaries haven't been reinstalled since this version, reinstall them as
3.02a automatically or on manual upgrade to fix corrupted binaries from
previous gzip issue.
Other platforms unchanged
Previously, if an editor was entered before an author, it would sort by
the editor. Now, as long as there's an author, it will sort by that
first (and, with 2f3d865f, favor left-bound matches).
Addresses the second issue on
https://forums.zotero.org/discussion/48047/
Courtesy of Pastel SVG (2x famfamfam)
I didn't add 2x versions for any composite icons or any Fugue icons.
Also:
- Removed some unused/redundant images
- Switched to shadowless versions for a couple Fugue icons
Use stdout redirection scripts for pdfinfo and, on Windows, a script to
run pdftotext hidden, which together allow for all unmodified binaries
(including, probably, symlinked system ones, though I didn't test that).
On Windows, using a .vbs does cause a brief wait cursor. The stock
pdfinfo needs the redirection script anyway, so that's unavoidable, but
on the async branch I think we'll be able to switch to pdf.js for the
page count, at which point maybe I'll try to remember how I modified the
Windows binaries to be hidden and use a modified version of pdftotext to
avoid VBScript. (We use the stock pdftotext elsewhere already.)
If automatic translator/style updates are enabled, at least one of the
PDF tools is installed, and the repo returns a more recent version
number than what's installed, automatically upgrade the tools. (Version
3.02 counts as lower, since Poppler's version numbers are lower.)
If an error occurs, wait increasing amounts of time to try the downloads
again, up to one week.
Fixes regression from 37921b0910
cleanISBN/ISSN now return first valid ISBN/ISSN from a list of valid/invalid ISBNs/ISSNs, even if invalid ISBN/ISSN comes first
toISBN13 now corrects the check digit for ISBN13 input
It turns out that the Cygwin console, unlike -console, is actually
usable, so developers on Windows can use that. Since we sometimes need
real-time debug output from end users (who won't have Cygwin installed),
keep logging to the Browser Console if only the -ZoteroDebug flag is
passed.
- Use async DB and OS.File for bundled file updates
- Remove support for translator/style ZIP files -- the two options are
now non-unpacked XPIs with subfolders or unpacked source installations
- Now that we have async file access, don't store translator code in
database cache -- just store metadata so that it's available without
reading each translator file
- Change the (previously partially asyncified) Zotero.Styles/Translators
APIs a bit -- while the getAll/getVisible methods are asynchronous and
will wait for loading, the get() methods are synchronous and require
styles/translators to be initialized before they're called. Most
places that end up calling get() probably call getAll/getVisible first
and should therefore be async, but if there's any way to trigger a
get() first, that will need to be adjusted.
- Asyncify various other style/translator-related code
XPI support is untested, as is style/translator usage, so there are
almost certainly bugs. The latter depends on updated export format
support (#659), since toArray() no longer exists on this branch.
Addresses #529 and #520
I don't think there's any way for this to happen short of manual
preference editing, and the pane width would have to be exactly 250px,
and it's pretty harmless anyway, but might as well be safe.
There are hundreds more, but these are all the ones that generate warnings in
the console at startup. XPCOM/XBL ones don't seem to do so, so we can ignore
those for now (and hopefully not bother with them on 4.0). Instances in
translators do generate warnings.
Addresses #656
Note that this loses conflicting changes to translate_item.js from 849803473a,
so those will need to be reapplied if applicable. /cc @aurimasv, @mtd91429
Adds a "My Publications" source after "My Library", implemented as a
separate library. Top-level items can be dragged in and removed.
(This doesn't currently work without disabling Quick Copy.)
Also:
- Make "Group Libraries" an unselectable header instead of a container,
and don't indent group libraries
- Fix relation purging, which maybe never worked
- Pass only libraryID/key on deletes (which should speed them up)
- Fix async item cloning/copying
- Fix miscellaneous other bugs
To-do:
- Confirmation dialog on drag
- API support
0 allowed for some accidental behavior due to old code expecting NULL,
and it prevented easy checks (``if (!libraryID)``) for a passed
libraryID. Code now uses Zotero.Libraries.userLibraryID instead of a
hard-coded value (except in schema.js). Functions can still make
libraryID optional, but they should then use
Zotero.Libraries.userLibraryID if that's to mean the user library.
There might be some code that still expects 0 that I missed.
Fixes#426: this allows the localization of the Word Processors Add-in
button and messages displayed in the Prefs, under the "Word Processors"
tab. These strings were previously hardcoded in the integration plugins.
(Unfortunately this wipes out "Shortcuts" in all locales, but pushing
non-English locales to Transifex scares me too much. Sorry, localizers!
I think Transifex makes it easy to use previously used translations, at
least.)
[and delete unused "#zotero-prefpane-keys checkbox" (there used to be a checkbox to "Try to override conflicting shortcuts", which has been definely removed by 5b34dce40f )]
- New dedicated Zotero button, available in the Customize palette
- Remove "16px" Z SVG and just render "32px" one at the smaller size
(there's a slight difference at the edges, but we should probably redo
it anyway so that dark lines reach all the way to the bottom)
- Change "Open Zotero" back to just "Zotero", since 1) it's for closing
too, 2) it can also just bring Standalone to the front, and 3) "Open
Zotero" looks weird and inconsistent in the menu panel next to things
like "Preferences"
- For now, show single large Z for combo buttons in customization
palette with "Zotero (Combo)", though we can probably do something
better.
- Fix some HiDPI bugs when moving items between areas
Still need a layout for the panel for the combo buttons
Previously, if you dragged a standalone attachment to a collection in another
library where the item already existed as a child item, it would crash Zotero
on a collection-item constraint.
Known issues:
- Hover doesn't work on the dropmarker alone, so instead of showing the
hover effect for each segment individually, the menu-button parts are
highlighted together, which at least looks like a choice
- When the menu-button is disabled, it still has some hover/active
effects.
- Tested only on Windows 7
The address bar icon now lives in a new combo buttonset containing the main Z, "Save to Zotero", and a dropmarker for the former save-icon right-click menu (which we could conceivably use more heavily going forward now that it's more accessible). There's also a separate dedicated Save to Zotero (+ dropmarker) button, not shown by default, that can be swapped in for people who don't want the Z.
The tooltip for the save icon also now shows the keyboard shortcut (though that unfortunately makes for a lot of parentheses).
Known issues:
- Untested on ESR
- Untested on Linux
- Might need refinement on Windows
- Weird 1px horizontal area at bottom of save button that highlights dropmarker (at least on OS X)
- Probably needs a third button option with just the Z icon so that the main button and the save button can be placed separately (e.g., save button in toolbar, Z in panel)
- Combo buttonset needs an inactive single-icon state for the palette and either needs a state for the panel (which might need to span all three columns?) or if possible should just move the other two icons in and put itself back in the palette
- The absurd amount of time and CSS it took to get the toolbar icons looking right on OS X, since apparently no one has put a menu-button inside a combined toolbar button before
- Add high-res webpage icon
- Show webpage icon in grayscale when no translator (except on hover, for fun)
- Remove pre-Australis icons
- Switch to CustomizableUI API for toolbar icon
- Move icon generation code to separate file
- Add Zotero.hiRes flag for Retina/etc. displays (available only after a window
has loaded)
Known issues:
- While the gray is mostly to be less distracting, the gray/color distinction
will probably be lost on most people. A separate guidance panel for the gray
icon might help.
- On pages with frames, the webpage icon appears first and then is replaced
with a translator icon.
The Firefox theme changed at some point since Australis to use darker,
non-gradient icons. (Sorry ESR users.)
Not sure if this needs to change on Win/Linux too.
The icon now will save using a translator if possible and otherwise fall back
to creating a web page item. This also removes the "Create Web Page Item from
Current Page" button.
Let's see how this feels. (Pushing it to the beta so more people can try it.) I
think we ultimately should do this, but my main concern with this
implementation is that it's just too distracting, since the icon disappears and
reappears on every page. A persistent, possibly monochrome icon that was just
sometimes disabled (as is the case for the Firefox bookmark toolbar icon) might
be better.
Regardless of the approach, there are some follow-up tweaks that should be made:
- The same thing in the connectors
- Context-menu options
- Different icons and descriptions for different file types (PDF, image)?
- Adjust guidance text? Have separate guidance panels for web vs. translation?
itemType comes from the column name in the tree view, so it makes most sense to do it there. Otherwise, Zotero was complaining about invalid field somewhere
Instead of limiting charsets to a fixed list, dynamically populate it
with any charset name of less than 50 ASCII characters. Previously,
unknown charsets were discarded.
Zotero.Item.prototype.attachmentCharset now always returns a charset
name. It can be set with either a name or a charsetID.
Also:
- Remove the unused 'originalPath' column in itemAttachments
* Easier monitoring of preference changes
* Takes a preference name and a handler function that will be passed the new value of the preference
* Unregister observer via Zotero.Prefs.unregisterObserver with the same parameters
.save calls ._initSave(), _saveData(), _finalizeSave() internally passing `env` object to each to act as an environment for passing around variables
* _initSave should determine if the save is possible and return a promise for either `true` or `false`. It should also set up the environment, e.g. determine if this `isNew`
* _saveData performs the actual saving to the database, but should not do any terminal steps in the save process so that any extending classes could extend this method to write additional data to the database
* _finalizeSave should perform any finalization before the data is committed to the database.
_recoverFromSaveError is called with `env` and an error that occurred. This method should perform any recovery steps, e.g. discarding the save and reloading the item from the database into the cache.
checkIsOnline isn't called before the connector gets used except in IE.
The mixed content blocker in modern IE combined with the https
redirect breaks communication with ZSA on all pages anyway, so we may
just want to give up on making the bookmarklet talk to ZSA.
This belongs to issue #509 and was asked several times in the forum.
* Add attribute `collapse="after"` to the splitter and add a grippy element.
* To remember its state after restart I copied `zotero-persist="state"`.
* Handle zotero-items-splitter the same as zotero-collections-splitter in zotero-platform/mac/overlay.css
* Change min-width to 250px for #zotero-item-pane in zotero/overlay.css.
* Update the function `updateToolbarPosition` in zotero/zoteroPane.js:
* The width of the items-toolbar is corrected if the left pane is collapsed (and the icons are grouped on the left margin together).
* If the right pane is collapsed, then the items-toolbar is made flexible while making the item-toolbar unflexible. As a result the search box and locate icon are flushed right to the other icons.
Generated files for a more robust attached-link-dialog and localized strings
* AttachLink.js
* AttachLink.xul
zotero/xpcom/attachments.js
* created function cleanAttachmentURI
* in function linkFromURL, removed the regex constraints and the comment list of valid protocols
* removed outdated function declaration from beginning of script
* Improved automatic title generation mechanism
* Remove splitter
* Add explanation of how Zotero Preview pane works
* Limit filtering to citation format
Also add support for “citation-format” attribute of CSL 1.0
* Remove redundant warning
(already happens on this.refresh)
* Use "items" instead of "references"
* Clean up HTML a little
* Fix some JSHint warnings
Passing sandboxes between translators became difficult. This change
loads all translators in the same sandbox, using the same ugly hack as
in the connectors to get us close enough to being able to load
translators into separate scopes for things to work.
Conflicts:
chrome/content/zotero/xpcom/translation/translate.js
And in the default mode, use the whole string up through 'and' or 'et al.'
before falling back to full creator sorting, which could speed things up
slightly.
This includes storing values into DB and performing searches. Note that export/display of existing data is not normalized. We can try to capture all access points to the database and normalize on output, but that seems like a lot of unnecessary normalization happening all the time. Would probably be best to just normalize existing data.
With Hola enabled, uploads were failing with NS_BASE_STREAM CLOSED -- it
seems to do something that causes the stream to be read more than once.
Adding REOPEN_ON_REWIND fixes this.
* getDomDocument: returns a detached DOMDocument object
* dom2text (TODO): Currently just returns Node.textContent, but is intended to return Zotero-formatted string based on text formatting in the DOM and the Zotero.Item field that the text is meant for
This simply means that detection code will be run first.
Attempting this with Export translators will fail, because trying to detect a translator does not make sense in this case.
I understood the path limit to mean >260, but a user is seeing the error
with a 260-character path [1], so let's try this.
[1] https://forums.zotero.org/discussion/41410
Yosemite introduces a new search textbox shape, but Firefox still uses
the old focus ring shape until 34, so we fake the old searchbox shape
for earlier versions.
Our strategy: put arguments into a property of the function, and then
get them out unwrapped. This avoids security checks on arguments passed
to the function.
It's way too slow, though, since the whole list is regenerated after
merging.
Fixes#519
Also:
- The arguments to Zotero.Item.prototype.clone() have changed, and it no
longer takes an existing item or copies primary data. To create an
in-memory copy of an item, use the new Zotero.Item.prototype.copy().
- Zotero.Item.prototype.getUsedFields() now gets in-memory fields rather
than the fields in the database
Our modifications no longer include a custom yield handler to
automatically call all() on yielded arrays (which maintained Bluebird
1.x behavior). It's now necessary to call all() or similar explicitly.
Also fixed a few incorrect yields hidden by that behavior.
The state check errors were caused by 4812ab6f, which was a fix for
"Q.async(...)(...) is undefined" errors caused by ad8b81f4c, which was a
fix for "too much recursion" errors related to Task.spawn() on Windows
with JIT enabled.
This will hopefully fix some remaining issues with long filenames during
syncing, particularly on Linux with encrypted filenames (which have a
filename length of 143).
(This may have reintroduced some edge case bugs, so it needs some
testing.)
Previously, 'collection' and 'savedSearch' conditions used
"[libraryID]_[key]" format. Now, the condition should contain only a
key, and the libraryID will be drawn from either the search itself or a
libraryID search condition. Old-style conditions are still parsed if
provided.
- Protocol handler extensions can now handle promises and can also make
data available as it's ready instead of all at once (e.g., reports now
output one entry at a time)
- zotero:// URL syntaxes are now more consistent and closer to the web
API (old URLs should work, but some may currently be broken)
Also:
- Code to generate server API, currently available for testing via
zotero://data URLs but eventually moving to HTTP -- zotero://data URLs match
web API URLs, with a different prefix for the personal library (/library vs.
/users/12345)
- Miscellaneous fixes to data objects
Under the hood:
- Extensions now return an AsyncChannel, which is an nsIChannel implementation
that takes a promise-yielding generator that returns a string,
nsIAsyncInputStream, or file that will be used for the channel's data
- New function Zotero.Utilities.Internal.getAsyncInputStream() takes a
generator that yields either promises or strings and returns an async input
stream filled with the yielded strings
- Zotero.Router parsers URLs and extract parameters
- Zotero.Item.toResponseJSON()
And remove UDF setup code
Duplicates view also used REGEXP, so we'll need to figure out another approach
for the async rewrite of that (#519). There are no other current UDF consumers,
so this closes#528.
And catch other errors and throw StopIteration so that they stop the
search. (Non-StopIteration errors in onRow don't stop Sqlite.jsm
queries. We were logging them but then re-throwing them, which didn't do
anything.)
This required doing additional caching at startup (e.g., item types and fields)
so that various methods can remain synchronous.
This lets us switch back to using the current Sqlite.jsm. Previously we were
bundling the Fx24 version, which avoided freezes with locking_mode=EXCLUSIVE
with both sync and async queries.
Known broken things:
- Autocomplete
- Database backup
- UDFs (e.g., REGEXP function used in Zotero.DB.getNextName())
- Zotero.Item.prototype.getFilePath() is now synchronous, with a separate async getFilePathAsync()
- getFile() no longer takes a skipExistsCheck parameter, since that shouldn't happen synchronously
- Zotero.Items.getByLibraryAndKey() is now synchronous again, with a
separate Zotero.Items.getByLibraryAndKeyAsync() - I haven't fully
tested this, so I'm not sure if there will need to be any async
calls.
- Some of the full-text indexing functions now take file paths instead of nsIFile objects
- Zotero.File.getContentsAsync() can now take a string path as well
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
* Accept detailed cookie information (including host, path, secureOnly, and hostOnly) from connectors so we can send correct cookies when fetching pages from different hosts. This way we also don't have to worry about exposing cookies to different hosts.
* Don't drop cookies that we receive from other hosts. Some pages (e.g. PDF URLs) result in redirects to other hosts (and even domains) that then set cookies, which are required to retrieve the PDF. (e.g. Cell Press stores PDFs on ScienceDirect, but their PDF links initially point to cell.com).
* Send detailed cookies where possible. Currently that's only for Chrome/Opera and Firefox in Connector mode. Does not seem to be possible in Safari.
Possibly to other places as well (but not Notepad, which doesn't work
from Firefox or Chrome either)
Unfortunately this requires going back to 'copy' cursor feedback when
dragging, even when Shift is used. We can only choose one on Windows (as
far as I can tell), and we were previously using the unadorned 'move'.
nsICollation broke for some locales. (Testing requires changing the
language setting in Language & Region and then restarting the computer.
The change seems to not fully go into effect until then, even though the
UI changes.) This is fixed in Nightly, but we can work around it by
using the new Intl.Collator.
With Task.spawn, regular expressions in Zotero.DB were causing "too much
recursion" errors on Windows with JIT enabled.
This requires a change to Q to allow async() to take a generator instead
of a generator-maker (which is the reason it was using Task.spawn to
begin with).
S3 upload timeouts were retried already with an exponential backoff, but
this adds retrying for other kinds of upload failures as well as failed
downloads. If 5 consecutive failures occur a file sync error is
thrown.
Failed file sync requests to the Zotero API are not currently retried,
but S3 accounts for the majority.
The download portion of this still needs further testing.
When starting in Connector mode (i.e. Standalone is open), Zotero first starts in Full mode, looks for Standalone, then "shuts down" and restarts in Connector mode. `Zotero.shutdown()` returns a promise which is then followed up by a `Zotero.init` call. Thus, when starting in Connector mode, Zotero initialization is asynchronous and makes it possible for `Zotero_Browser.init()` to be called before `Zotero.initialized` is true, which prevents `Zotero_Browser` from initializing. Additionally, even if `Zotero_Browser.init()` is called after Zotero is initialized in Connector mode, it is possible that `Zotero_Browser.init()` will be called _after_ the "load" event for browser.xul has already fired, so `chromeLoad` is never called. This patch ensures that both of these race conditions are taken into account.
Can be triggered by double-clicking or Return and also by F2 on
Windows/Linux
This does mean double-clicking no longer toggles the collection open and
closed. If we wanted to preserve that we could probably capture the
double-click.
Closes Trac ticket 231, only 8 years later
Previously, position/size was persisted for each item's note
individually, but that meant that there was no default position/size for
the note window and an entry was created in localstore.rdf for
every note opened in a new window. There's also a good chance people had
no idea what was going on.
- Always allow "Report Errors...", even when no errors
- Show submitted diagnostic info in report
- Use white background and unitalicized text for report
- Make window larger by default
That was easy. (@simonster, since you disabled this originally, any reason not
to do this, other than the extra tab stop? At least in current Firefox
versions, this is the default behavior.)
When opening the advanced search window, the current library is
selected, and a different library can be selected to change the search
scope. If a library is read-only, the saved search button is disabled.
For saved searches, the appropriate library is selected and the
drop-down is disabled.
Also:
- Close the advanced search window after a search is saved
- The default name for saved searches ("Untitled 2", etc.) was based on
collections rather than searches
- Once an initial search has been performed, the drop-downs and
checkboxes now update the results
- More consistent spacing in advanced search window
- (dev) Zotero.DB.getNextName() now takes a libraryID as its first
parameter instead of always using My Library; the old parameters are
deprecated but still work
- Each column in the middle pane can now have its own persistent
secondary sort column, configurable from a new submenu in the column
picker menu (top right of items list). The settings are stored in
extensions.zotero.secondarySort.[primaryField]. The submenu title
includes the current primary field (e.g., "Secondary Sort (Creator)"),
which is pretty weird, and I'm not sure I want to keep it, but it does
convey that the setting is specific to the selected column.
- The fallback sort fields (firstCreator, date, title, dateAdded) are
now configurable via the extensions.zotero.fallbackSort. Setting that
pref to an empty string avoids all fallback sorts, which
allows reverse-order clicking to set the order, as requested by
@aurimasv in #275.
- The previous behavior of sorting based on the exact Creator string
(rather than the actual creators) can now be restored with the
extensions.zotero.sortCreatorAsString pref. (It simply circumvents all
the newer code, so it's pretty safe.) This setting should result in
faster sorting in large libraries that have many items with the same
Creator string.
- Some of the lesser fields in the column picker menu are now in the
More Columns submenu (which is now alphabetical)
- The "Type" column is now the less-ambiguous "Item Type".
- This uses a different method to modify the column picker menu that is
simultaneously less and more hacky. (It no longer has to duplicate
Mozilla code in a custom XBL binding that wouldn't reflect future
upstream changes, and instead it bushwhacks its way through various
boxObject properties to get to the underlying menupopup.)
The previous version would keep only the last instance.
This version requires the array to contain only primitives of a single
data type, but I think that's OK for all of our uses. (This version
should also be faster.)
Cmd on OS X, Shift on Windows/Linux
How do I not get to close a ticket for this?
Unfortunately on Windows it doesn't seem possible to set the cursor
effect to arbitrary states (see note in libraryTreeView.js::
_setDropEffect() for the gory details), so this just uses the default
cursor there. On OS X and Linux the cursor reflects the requested
action.
Previously Cmd-Shift-Z worked for redo in text areas (Abstract, Extra)
but would toggle the Zotero pane in regular text fields. This change
allows Redo to work as expected.
The search bar gets focus when you first open the Zotero pane, though,
so we don't redo there, since it would be annoying if you couldn't close
the pane immediately with the same shortcut.
I think requiring everyone with Zotero installed to manually close the panel is
just too annoying. (Not allowing the auto-hide for a couple seconds after
opening to prevent accidental closes would be nice, but preventDefault() in the
panel's onpopuphiding() doesn't seem to work, nor does changing noautohide
after opening.)
Increase the delay to 2 seconds to try to show on top of the Firefox 28->29
upgrade panel on slower computers.
Using separate arrows from treetwisty.svg from Mozilla, since it doesn't seem
to be possible to use list-style-position in the tree
The reason we still have to do this at all is that coloring of the twisties in
the tree still gets messed up as you move around the tree. (This still happens
in Places and Thunderbird as well.)
Network errors (where the connection itself failed, rather than failed
HTTP requests) were being thrown directly in the stream listener, which
prevented the Zotero.Sync.Storage.Request -- and therefore the file sync
-- from ever completing.
"Your socket connection to the server was not read from or written to
within the timeout period."
I can't reproduce these reliably, so this is fairly untested, but it
seems to work. It backs off exponentially when the error occurs, and
halves the delay on successful requests.
Eventually all 50x file sync errors should also be retried
automatically, but that can't really happen in 4.0.
At least for me (in a VM), the text console has always been unusable on
Windows. Logging to the Browser Console is slower than dump() on OS X
and Linux, but on Windows it's much faster.
Sometime in the past SQL datetimes saved to the server could be
interpreted as multipart dates, causing the date portion to be stripped
from the visible field. I fixed this previously on the server, but a
full sync could result in conflicts for any such values that synced
down. Instead, overwrite the local version (keeping Date Modified the
same) if there are no other changes.
- Send modifier keys through to loadURI() when clicking Open Link in notes
- Open link in parent window from external note window
- Don't show both menus on right-click
Follow-up from #450
Follow up from #440
- Convert curly single quotes to straight quotes before inserting
- Add General and Supplemental Unicode punctuation ranges to getClass()
(so that fancy punctuation doesn't end up in words)
- Move single-quote test from getClass() to semanticSplitter(), and
consider it a letter only if in the middle of a word
- Add comments to semanticSplitter()
This might be ever-so-slightly slower, but it's neglible. (War and Peace
seems to now take ~1570ms instead of ~1500ms for me.)
