*As always, but in particular this time, do not test this commit with valuable data -- but please do test.*

- Massive optimization of data layer -- with ~11,000-item test library on a Mac Pro, decreased initial Zotero pane loading from several minutes to ~10 seconds. This included some small API changes and new methods (e.g. Items.cacheFiles()) in the data layer, but most of it was changing the way loading and caching of data worked internally.

- Moved unique itemData values out to separate itemDataValues table for better normalization

- Updated itemTreeView.sort() to be able to sort a single row into the items list for performance reasons -- itemTreeView.notify() now only sorts a single row when possible (and sometimes doesn't need to sort anything). This should make general interface use dramatically less sluggish with large libraries.

- Consolidated purging on item deletes, which should speed up multi-item deletes quite a bit -- clients should use Items.erase() instead of Item.erase(), since the former calls the new Items.purge() method (which calls the various other purge() methods) automatically

- Notifier no longer throws errors in notify() callbacks and instead just logs them to the Error Console -- this way a misbehaving utility (or Zotero itself) won't keep other observers from receiving change notifications

- Better handling of database corruption -- if an SQL query throws a file corruption error, Zotero adds a marker file to the storage directory and displays a message prompting the user to restart to attempt auto-repair--and, most importantly, no longer copies the corrupt file over the last backup.

- A "Loading items list..." message appears over the items list (at least, sometimes) while data is loading -- useful for large libraries, but may need to be fine-tuned to not be annoying for smaller ones.

- Note titles are now cached in itemNoteTitles table

- orderIndex values are no longer consolidated when removing items from collections -- it just leaves gaps

- Fixed shameful bug in getRandomID() that could result in an item with itemID 0, which wouldn't display correctly and would be impossible to remove

- Fixed autocomplete and search for new location of 'title' field

- Added proper multipart date support for type-specific 'date' fields

- Made the pre-modification array passed to Notifier observers on item updates actually be pre-modification

- New method Zotero.ItemFields.isFieldOfBase(field, baseField) -- for example, isFieldOfBase('label', 'publisher') returns true, as does isFieldOfBase('publisher', 'publisher')

- Restored ability to drag child items in collections into top-level items in those collections

- Disabled unresponsive script message when opening Zotero pane (necessary for large libraries, or at least was before the optimizations)

- Collections in background windows didn't update on item changes

- Modifying an item would cause it to appear incorrectly in other collections in background windows

- Fixed an error when dragging, hovering to open, and dropping a note or attachment on another item

- Removed deprecated Notifier methods registerCollectionObserver(), registerItemObserver(), unregisterCollectionObserver(), and unregisterItemObserver()

- Loading of Zotero core object can be cancelled on error with Zotero.skipLoading

- Removed old disabled DebugLogger code

- New method Zotero.log(message, type, sourceName, sourceLine, lineNumber, columnNumber, category) to log to Error Console -- wrapper for nsIConsoleService.logMessage(nsIScriptError)

- New method Zotero.getErrors(), currently unused, to return array of error strings that have occurred since startup, excluding CSS and content JS errors -- will enable an upcoming Talkback-like feature

- Fixed some JS strict warnings in Zotero.Date.strToMultipart()

chrome/content/zotero/itemPane.js

@@ -288,15 +288,15 @@ var ZoteroItemPane = new function()
 			for(var i = 0; i<fieldNames.length; i++)
 			{
 				var editable = !_itemBeingEdited.isPrimaryField(fieldNames[i]);
-				
+				var fieldID = Zotero.ItemFields.getID(fieldNames[i])
 				var val = _itemBeingEdited.getField(fieldNames[i]);
 				
 				// Start tabindex at 1000 after creators
 				var tabindex = editable ? (i>0 ? _tabIndexMinFields + i : 1) : 0;
 				_tabIndexMaxInfoFields = Math.max(_tabIndexMaxInfoFields, tabindex);
 				
-				if (fieldNames[i]=='date'){
+				if (editable && Zotero.ItemFields.isFieldOfBase(fieldID, 'date')) {
-					addDateRow(_itemBeingEdited.getField('date', true), tabindex);
+					addDateRow(fieldNames[i], _itemBeingEdited.getField(fieldNames[i], true), tabindex);
 					continue;
 				}
 				
@@ -695,15 +695,15 @@ var ZoteroItemPane = new function()
 	/**
 	 * Add a date row with a label editor and a ymd indicator to show date parsing
 	 */
-	function addDateRow(value, tabindex)
+	function addDateRow(field, value, tabindex)
 	{
 		var label = document.createElement("label");
-		label.setAttribute("value", Zotero.getString("itemFields.date") + ':');
+		label.setAttribute("value", Zotero.getString("itemFields." + field) + ':');
-		label.setAttribute("fieldname",'date');
+		label.setAttribute("fieldname", field);
 		label.setAttribute("onclick", "this.nextSibling.firstChild.blur()");
 		
 		var hbox = document.createElement("hbox");
-		var elem = createValueElement(Zotero.Date.multipartToStr(value), 'date', tabindex);
+		var elem = createValueElement(Zotero.Date.multipartToStr(value), field, tabindex);
 		
 		// y-m-d status indicator
 		var datebox = document.createElement('hbox');
@@ -897,12 +897,6 @@ var ZoteroItemPane = new function()
 				_tabIndexMaxTagsFields = Math.max(_tabIndexMaxTagsFields, tabindex);
 				break;
 			
-			// Display the SQL date as a tooltip for the date field
-			case 'date':
-				valueElement.setAttribute('tooltiptext',
-					Zotero.Date.multipartToSQL(_itemBeingEdited.getField('date', true)));
-				break;
-			
 			// Convert dates from UTC
 			case 'dateAdded':
 			case 'dateModified':
@@ -919,6 +913,13 @@ var ZoteroItemPane = new function()
 				break;
 		}
 		
+		// Display the SQL date as a tooltip for date fields
+		var fieldID = Zotero.ItemFields.getID(fieldName);
+		if (fieldID && Zotero.ItemFields.isFieldOfBase(fieldID, 'date')) {
+			valueElement.setAttribute('tooltiptext',
+				Zotero.Date.multipartToSQL(_itemBeingEdited.getField(fieldName, true)));
+		}
+		
 		if (fieldName.indexOf('firstName')!=-1){
 			valueElement.setAttribute('flex', '1');
 		}

chrome/content/zotero/overlay.js

@@ -569,10 +569,12 @@ var ZoteroPane = new function()
 			itemgroup.setSearch('');
 			itemgroup.setTags(getTagSelection());
 			
+			Zotero.UnresponsiveScriptIndicator.disable();
 			this.itemsView = new Zotero.ItemTreeView(itemgroup);
 			this.itemsView.addCallback(_setTagScope);
 			document.getElementById('zotero-items-tree').view = this.itemsView;
 			this.itemsView.selection.clearSelection();
+			Zotero.UnresponsiveScriptIndicator.enable();
 		}
 		else
 		{

chrome/content/zotero/overlay.xul

@@ -261,7 +261,7 @@
 					
 					<!-- Label for displaying messages when items pane is hidden
 					     (e.g. "Advanced search mode — press Enter to search.")-->
-					<box pack="center" align="center">
+					<box id="zotero-items-pane-message-box" pack="center" align="center">
 						<label id="zotero-items-pane-message" />
 					</box>
 				</deck>

chrome/content/zotero/xpcom/db.js

@@ -32,6 +32,7 @@ Zotero.DBConnection = function(dbName) {
 	this._transactionRollback = null;
 	this._transactionNestingLevel = 0;
 	this._callbacks = { begin: [], commit: [], rollback: [] };
+	this._skipBackup = false;
 	this._self = this;
 }
 	
@@ -103,6 +104,8 @@ Zotero.DBConnection.prototype.query = function (sql,params) {
 		}
 	}
 	catch (e) {
+		this.checkException(e);
+		
 		var dberr = (db.lastErrorString!='not an error')
 			? ' [ERROR: ' + db.lastErrorString + ']' : '';
 		throw(e + ' [QUERY: ' + sql + ']' + dberr);
@@ -539,12 +542,58 @@ Zotero.DBConnection.prototype.observe = function(subject, topic, data) {
 }
 
 
+Zotero.DBConnection.prototype.checkException = function (e) {
+	if (e.name == 'NS_ERROR_FILE_CORRUPTED') {
+		var file = Zotero.getZoteroDatabase(this._dbName, 'is.corrupt');
+		var foStream = Components.classes["@mozilla.org/network/file-output-stream;1"]
+						 .createInstance(Components.interfaces.nsIFileOutputStream);
+		foStream.init(file, 0x02 | 0x08 | 0x20, 0664, 0); // write, create, truncate
+		foStream.write('', 0);
+		foStream.close();
+		
+		this._skipBackup = true;
+		
+		var ps = Components.classes["@mozilla.org/embedcomp/prompt-service;1"]
+								.getService(Components.interfaces.nsIPromptService);
+		
+		var buttonFlags = (ps.BUTTON_POS_0) * (ps.BUTTON_TITLE_IS_STRING)
+			+ (ps.BUTTON_POS_1) * (ps.BUTTON_TITLE_IS_STRING);
+		
+		var index = ps.confirmEx(null,
+			Zotero.getString('db.dbCorrupted', this._dbName),
+			Zotero.getString('db.dbCorrupted.restart'),
+			buttonFlags,
+			Zotero.getString('general.restartNow'),
+			Zotero.getString('general.restartLater'),
+			null, null, {});
+		
+		if (index == 0) {
+			var appStartup = Components.classes["@mozilla.org/toolkit/app-startup;1"]
+					.getService(Components.interfaces.nsIAppStartup);
+			appStartup.quit(Components.interfaces.nsIAppStartup.eRestart);
+			appStartup.quit(Components.interfaces.nsIAppStartup.eAttemptQuit);
+		}
+		
+		Zotero.skipLoading = true;
+		return false;
+	}
+	return true;
+}
+
+
 Zotero.DBConnection.prototype.backupDatabase = function () {
 	if (this.transactionInProgress()) {
 		this._debug("Transaction in progress--skipping backup of DB '" + this._dbName + "'", 2);
 		return false;
 	}
 	
+	var corruptMarker = Zotero.getZoteroDatabase(this._dbName, 'is.corrupt').exists();
+	
+	if (this._skipBackup || corruptMarker) {
+		this._debug("Database '" + this._dbName + "' is marked as corrupt--skipping backup", 1);
+		return false;
+	}
+	
 	this._debug("Backing up database '" + this._dbName + "'");
 	
 	var file = Zotero.getZoteroDatabase(this._dbName);
@@ -640,6 +689,10 @@ Zotero.DBConnection.prototype._getDBConnection = function () {
 	}
 	
 	catchBlock: try {
+		var corruptMarker = Zotero.getZoteroDatabase(this._dbName, 'is.corrupt');
+		if (corruptMarker.exists()) {
+			throw({ name: 'NS_ERROR_FILE_CORRUPTED' })
+		}
 		this._connection = store.openDatabase(file);
 	}
 	catch (e) {
@@ -706,6 +759,10 @@ Zotero.DBConnection.prototype._getDBConnection = function () {
 			]);
 			alert(msg);
 			
+			if (corruptMarker.exists()) {
+				corruptMarker.remove(null);
+			}
+			
 			break catchBlock;
 		}
 		
@@ -713,7 +770,6 @@ Zotero.DBConnection.prototype._getDBConnection = function () {
 		throw (e);
 	}
 	
-	
 	// Register shutdown handler to call this.onShutdown() for DB backup
 	var observerService = Components.classes["@mozilla.org/observer-service;1"]
 		.getService(Components.interfaces.nsIObserverService);

chrome/content/zotero/xpcom/itemTreeView.js

@@ -33,6 +33,8 @@
  */
 Zotero.ItemTreeView = function(itemGroup, sourcesOnly)
 {
+	this._initialized = false;
+	
 	this._itemGroup = itemGroup;
 	this._sourcesOnly = sourcesOnly;
 	
@@ -42,7 +44,8 @@ Zotero.ItemTreeView = function(itemGroup, sourcesOnly)
 	this._ownerDocument = null;
 	this._needsSort = false;
 	
-	this.refresh();
+	this._dataItems = [];
+	this.rowCount = 0;
 	
 	this._unregisterID = Zotero.Notifier.registerObserver(this, ['item', 'collection-item']);
 }
@@ -77,15 +80,21 @@ Zotero.ItemTreeView.prototype.setTree = function(treebox)
 		if (this._needsSort) {
 			this.sort();
 		}
-		
 		return;
 	}
+	
 	this._treebox = treebox;
 	
+	this._ownerDocument.defaultView.ZoteroPane.setItemsPaneMessage(Zotero.getString('pane.items.loading'));
+	
+	// Generate the tree contents in a timer to allow message above to display
+	var paneLoader = function(obj) {
+		obj.refresh();
+		
 		// Add a keypress listener for expand/collapse
-	var expandAllRows = this.expandAllRows;
+		var expandAllRows = obj.expandAllRows;
-	var collapseAllRows = this.collapseAllRows;
+		var collapseAllRows = obj.collapseAllRows;
-	var tree = this._treebox.treeBody.parentNode;
+		var tree = obj._treebox.treeBody.parentNode;
 		tree.addEventListener('keypress', function(event) {
 			var key = String.fromCharCode(event.which);
 			
@@ -100,10 +109,15 @@ Zotero.ItemTreeView.prototype.setTree = function(treebox)
 			}
 		}, false);
 		
-	this.sort();
+		obj.sort();
 		
 		//Zotero.debug('Running callbacks in itemTreeView.setTree()', 4);
-	this._runCallbacks();
+		obj._runCallbacks();
+		
+		obj._ownerDocument.defaultView.ZoteroPane.clearItemsPaneMessage();
+	}
+	
+	this._ownerDocument.defaultView.setTimeout(paneLoader, 50, this);
 }
 
 
@@ -115,37 +129,45 @@ Zotero.ItemTreeView.prototype.refresh = function()
 {
 	var oldRows = this.rowCount;
 	
-	this._dataItems = new Array();
+	this._dataItems = [];
 	this.rowCount = 0;
 	
-	var newRows = this._itemGroup.getChildItems();
+	var cacheFields = ['title', 'date'];
-	if (newRows.length)
+	// Cache the visible fields so they don't load individually
-	{
+	var visibleFields = this.getVisibleFields();
-		for(var i = 0, len = newRows.length; i < len; i++)
+	for each(var field in visibleFields) {
-		{
+		if (cacheFields.indexOf(field) == -1) {
-			if(newRows[i] &&
+			cacheFields = cacheFields.concat(field);
-			  (!this._sourcesOnly || (!newRows[i].isAttachment() && !newRows[i].isNote())))
-			{
-				this._showItem(new Zotero.ItemTreeView.TreeRow(newRows[i],0,false), i+1); //item ref, before row
 		}
 	}
+	Zotero.Items.cacheFields(cacheFields);
+	
+	var newRows = this._itemGroup.getChildItems();
+	for (var i=0, len=newRows.length; i < len; i++) {
+		if (newRows[i] &&
+		  (!this._sourcesOnly || (!newRows[i].isAttachment() && !newRows[i].isNote()))) {
+			this._showItem(new Zotero.ItemTreeView.TreeRow(newRows[i], 0, false), i+1); //item ref, before row
+		}
 	}
 	
 	this._refreshHashMap();
 	
 	// Update the treebox's row count
 	var diff = this.rowCount - oldRows;
-	if (this._treebox && diff != 0) {
+	if (diff != 0) {
 		this._treebox.rowCountChanged(0, diff);
 	}
 }
 
+
 /*
  *  Called by Zotero.Notifier on any changes to items in the data layer
  */
 Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 {
 	var madeChanges = false;
+	var sort = false;
+	
 	this.selection.selectEventsSuppressed = true;
 	var savedSelection = this.saveSelection();
 	
@@ -206,6 +228,7 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 			}
 			
 			madeChanges = true;
+			sort = true;
 		}
 		
 	}
@@ -216,6 +239,7 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 		{
 			this.refresh();
 			madeChanges = true;
+			sort = true;
 		}
 		
 		// If no quicksearch, process modifications manually
@@ -224,6 +248,8 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 			for(var i=0, len=ids.length; i<len; i++)
 			{
 				var row = this._itemRowMap[ids[i]];
+				var sourceItemID = this._getItemAtRow(row).ref.getSource();
+				var parentIndex = this.getParentIndex(row);
 				// Item already exists in this view
 				if( row != null)
 				{
@@ -234,29 +260,30 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 					}
 					// If item moved from top-level to under another item,
 					// remove the old row
-					else if (!this.isContainer(row) && this.getParentIndex(row)==-1
+					else if (!this.isContainer(row) && parentIndex == -1
-						&& this._getItemAtRow(row).ref.getSource())
+						&& sourceItemID)
 					{
 						this._hideItem(row);
 						this._treebox.rowCountChanged(row+1, -1)
 					}
-					else if (!this.isContainer(row) && this.getParentIndex(row)!=-1
+					// If moved from under another item to top level, add row
-						&& !this._getItemAtRow(row).ref.getSource())
+					else if (!this.isContainer(row) && parentIndex != -1
+						&& !sourceItemID)
 					{
 						var item = Zotero.Items.get(ids[i]);
 						this._showItem(new Zotero.ItemTreeView.TreeRow(item, 0, false), this.rowCount);
 						this._treebox.rowCountChanged(this.rowCount-1, 1);
+						sort = ids[i];
 					}
-					else
+					// If not moved from under one item to another
-					{
+					else if (parentIndex == -1 || !sourceItemID) {
-						this._treebox.invalidateRow(row);
+						Zotero.debug('here');
+						sort = ids[i];
 					}
 					madeChanges = true;
 				}
 				
-				//else if(this._itemGroup.isLibrary() || this._itemGroup.ref.hasItem(ids[i]))
+				else if (this._itemGroup.isLibrary() || this._itemGroup.ref.hasItem(ids[i])) {
-				else
-				{
 					var item = Zotero.Items.get(ids[i]);
 					if (!item) {
 						// DEBUG: this shouldn't really happen but could if a
@@ -269,9 +296,14 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 						this._showItem(new Zotero.ItemTreeView.TreeRow(item,0,false),this.rowCount);
 						this._treebox.rowCountChanged(this.rowCount-1,1);
 						madeChanges = true;
+						sort = true;
 					}
 				}
 			}
+			
+			if (sort && ids.length != 1) {
+				sort = true;
+			}
 		}
 		
 		// If quicksearch, re-run it, since the results may have changed
@@ -279,6 +311,7 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 		{
 			quicksearch.doCommand();
 			madeChanges = true;
+			sort = true;
 		}
 	}
 	else if(action == 'add')
@@ -288,6 +321,7 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 		{
 			this.refresh();
 			madeChanges = true;
+			sort = true;
 		}
 		
 		// If not a quicksearch and not background window saved search,
@@ -310,6 +344,10 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 					madeChanges = true;
 				}
 			}
+			
+			if (madeChanges) {
+				sort = (ids.length == 1) ? ids[0] : true;
+			}
 		}
 		// Otherwise re-run the search, which refreshes the item list
 		else
@@ -319,17 +357,22 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 			}
 			quicksearch.doCommand();
 			madeChanges = true;
+			sort = true;
 		}
 	}
 	
 	if(madeChanges)
 	{
-		this.sort();				//this also refreshes the hash map
-		this._treebox.invalidate();
-		
 		// If adding and this is the active window, select the item
 		if(action == 'add' && ids.length===1 && activeWindow)
 		{
+			if (sort) {
+				this.sort(typeof sort == 'number' ? sort : false);
+			}
+			else {
+				this._refreshHashMap();
+			}
+			
 			// Reset to Info tab
 			this._ownerDocument.getElementById('zotero-view-tabs').selectedIndex = 0;
 			this.selectItem(ids[0]);
@@ -342,8 +385,13 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 				Zotero.debug('Selected item no longer matches quicksearch -- clearing');
 				quicksearch.value = '';
 				quicksearch.doCommand();
-				this.sort();
+			}
-				this._treebox.invalidate();
+			
+			if (sort) {
+				this.sort(typeof sort == 'number' ? sort : false);
+			}
+			else {
+				this._refreshHashMap();
 			}
 			
 			this.rememberSelection(savedSelection);
@@ -354,8 +402,17 @@ Zotero.ItemTreeView.prototype.notify = function(action, type, ids)
 		}
 		else
 		{
+			if (sort) {
+				this.sort(typeof sort == 'number' ? sort : false);
+			}
+			else {
+				this._refreshHashMap();
+			}
+			
 			this.rememberSelection(savedSelection);
 		}
+		
+		this._treebox.invalidate();
 	}
 	this.selection.selectEventsSuppressed = false;
 }
@@ -439,7 +496,7 @@ Zotero.ItemTreeView.prototype.getLevel = function(row)
 	return this._getItemAtRow(row).level;
 }
 
-// Gets the index of the row's container, or -1 if none (container itself or top-level)
+// Gets the index of the row's container, or -1 if none (top-level)
 Zotero.ItemTreeView.prototype.getParentIndex = function(row)
 {
 	if (row==-1)
@@ -467,6 +524,13 @@ Zotero.ItemTreeView.prototype.hasNextSibling = function(row,afterIndex)
 
 Zotero.ItemTreeView.prototype.toggleOpenState = function(row)
 {
+	// Shouldn't happen but does if an item is dragged over a closed
+	// container until it opens and then released, since the container
+	// is no longer in the same place when the spring-load closes
+	if (!this.isContainer(row)) {
+		return;
+	}
+	
 	var count = 0;		//used to tell the tree how many rows were added/removed
 	var thisLevel = this.getLevel(row);
 	
@@ -549,7 +613,7 @@ Zotero.ItemTreeView.prototype.cycleHeader = function(column)
  *  Sort the items by the currently sorted column.
  *  Simply uses Array.sort() function, and refreshes the hash map.
  */
-Zotero.ItemTreeView.prototype.sort = function()
+Zotero.ItemTreeView.prototype.sort = function(itemID)
 {
 	// If Zotero pane is hidden, mark tree for sorting later in setTree()
 	if (!this._treebox.columns) {
@@ -560,12 +624,8 @@ Zotero.ItemTreeView.prototype.sort = function()
 		this._needsSort = false;
 	}
 	
-	var column = this._treebox.columns.getSortedColumn();
+	var columnField = this.getSortField();
-	if (!column){
+	var order = this.getSortDirection() == 'ascending';
-		column = this._treebox.columns.getFirstColumn();
-	}
-	var order = column.element.getAttribute('sortDirection') == 'ascending';
-	var columnField = column.id.substring(20);
 	
 	// Year is really the date field truncated
 	if (columnField == 'year') {
@@ -697,14 +757,14 @@ Zotero.ItemTreeView.prototype.sort = function()
 		return columnSort(a,b);
 	}
 	
-	function oppositeSort(a,b)
+	function reverseSort(a,b)
 	{
-		return(doSort(a,b) * -1);
+		return columnSort(a,b) * -1;
 	}
 	
+	// Need to close all containers before sorting
 	var openRows = new Array();
-	for(var i = 0; i < this._dataItems.length; i++)
+	for (var i=0; i<this._dataItems.length; i++) {
-	{
 		if(this.isContainer(i) && this.isContainerOpen(i))
 		{
 			openRows.push(this._getItemAtRow(i).ref.getID());
@@ -712,13 +772,53 @@ Zotero.ItemTreeView.prototype.sort = function()
 		}
 	}
 	
-	if(order)
+	// Single-row sort
+	if (itemID) {
+		this._refreshHashMap();
+		var row = this._itemRowMap[itemID];
+		Zotero.debug(row);
+		for (var i=0, len=this._dataItems.length; i<len; i++) {
+			if (i == row) {
+				continue;
+			}
+			
+			if (order) {
+				var cmp = reverseSort(this._dataItems[i], this._dataItems[row]);
+			}
+			else {
+				var cmp = doSort(this._dataItems[i], this._dataItems[row]);
+			}
+			
+			// As soon as we find a value greater (or smaller if reverse sort),
+			// insert row at that position
+			if (cmp < 0) {
+				var rowItem = this._dataItems.splice(row, 1);
+				this._dataItems.splice(row < i ? i-1 : i, 0, rowItem[0]);
+				this._treebox.invalidate();
+				break;
+			}
+			
+			// If greater than last row, move to end
+			if (i == len-1) {
+				var rowItem = this._dataItems.splice(row, 1);
+				this._dataItems.splice(i, 0, rowItem[0]);
+				this._treebox.invalidate();
+			}
+		}
+	}
+	// Full sort
+	else {
+		if (order) {
 			this._dataItems.sort(doSort);
-	else
+		}
-		this._dataItems.sort(oppositeSort);
+		else {
+			this._dataItems.sort(reverseSort);
+		}
+	}
 		
 	this._refreshHashMap();
 	
+	// Reopen closed containers
 	for(var i = 0; i < openRows.length; i++)
 		this.toggleOpenState(this._itemRowMap[openRows[i]]);
 	
@@ -976,6 +1076,9 @@ Zotero.ItemTreeView.prototype.rememberSelection = function(selection)
 			}
 			
 			if (this._itemRowMap[parent] != null) {
+				if (this.isContainerOpen(this._itemRowMap[parent])) {
+					this.toggleOpenState(this._itemRowMap[parent]);
+				}
 				this.toggleOpenState(this._itemRowMap[parent]);
 				this.selection.toggleSelect(this._itemRowMap[selection[i]]);
 			}
@@ -1046,6 +1149,18 @@ Zotero.ItemTreeView.prototype.collapseAllRows = function(treebox) {
 }
 
 
+Zotero.ItemTreeView.prototype.getVisibleFields = function() {
+	var columns = [];
+	for (var i=0, len=this._treebox.columns.count; i<len; i++) {
+		var col = this._treebox.columns.getColumnAt(i);
+		if (col.element.getAttribute('hidden') != 'true') {
+			columns.push(col.id.substring(20));
+		}
+	}
+	return columns;
+}
+
+
 /*
  * Returns an array of item ids of visible items in current sort order
  */
@@ -1069,11 +1184,13 @@ Zotero.ItemTreeView.prototype.getSortField = function() {
 
 /*
  * Returns 'ascending' or 'descending'
+ *
+ * A-Z == 'descending'
  */
 Zotero.ItemTreeView.prototype.getSortDirection = function() {
 	var column = this._treebox.columns.getSortedColumn();
 	if (!column) {
-		return 'ascending';
+		return 'descending';
 	}
 	return column.element.getAttribute('sortDirection');
 }
@@ -1237,7 +1354,7 @@ Zotero.ItemTreeView.prototype.canDrop = function(row, orient)
 		}
 		
 		// In library, allow children to be dragged out of parent
-		else if (this._itemGroup.isLibrary())
+		else if (this._itemGroup.isLibrary() || this._itemGroup.isCollection())
 		{
 			for each(var id in ids)
 			{
@@ -1250,7 +1367,6 @@ Zotero.ItemTreeView.prototype.canDrop = function(row, orient)
 			}
 			return true;
 		}
-		
 		return false;
 	}
 	else if (dataType == "text/x-moz-url") {
@@ -1337,17 +1453,11 @@ Zotero.ItemTreeView.prototype.drop = function(row, orient)
 					var source = item.isRegularItem() ? false : item.getSource();
 					
 					// Top-level item
-					if (!source)
+					if (source) {
-					{
+						item.setSource();
+					}
 					this._itemGroup.ref.addItem(id);
 				}
-					// Non-top-level item - currently unused
-					else
-					{
-						// TODO: Prompt before moving source item to collection
-						this._itemGroup.ref.addItem(source);
-					}
-				}
 			}
 		}
 	}

chrome/content/zotero/xpcom/notifier.js

@@ -30,10 +30,6 @@ Zotero.Notifier = new function(){
 	
 	this.registerObserver = registerObserver;
 	this.unregisterObserver = unregisterObserver;
-	this.registerCollectionObserver = registerCollectionObserver;
-	this.registerItemObserver = registerItemObserver;
-	this.unregisterCollectionObserver = unregisterCollectionObserver;
-	this.unregisterItemObserver = unregisterItemObserver;
 	this.trigger = trigger;
 	this.begin = begin;
 	this.commit = commit;
@@ -80,30 +76,6 @@ Zotero.Notifier = new function(){
 		_observers.remove(hash);
 	}
 	
-	// Deprecated
-	function registerCollectionObserver(ref){
-		Zotero.debug('registerCollectionObserver is deprecated and will be removed in a future release -- use registerObserver() instead', 2);
-		return registerObserver(ref, 'collection');
-	}
-	
-	// Deprecated
-	function registerItemObserver(ref){
-		Zotero.debug('registerItemObserver is deprecated and will be removed in a future release -- use registerObserver() instead', 2);
-		return registerObserver(ref, 'item');
-	}
-	
-	// Deprecated
-	function unregisterCollectionObserver(hash){
-		Zotero.debug('unregisterCollectionObserver is deprecated and will be removed in a future release -- use unregisterObserver() instead', 2);
-		unregisterObserver(hash);
-	}
-	
-	// Deprecated
-	function unregisterItemObserver(hash){
-		Zotero.debug('unregisterItemObserver is deprecated and will be removed in a future release -- use unregisterObserver() instead', 2);
-		unregisterObserver(hash);
-	}
-	
 	/**
 	* Trigger a notification to the appropriate observers
 	*
@@ -170,13 +142,14 @@ Zotero.Notifier = new function(){
 			Zotero.debug("Calling notify() on observer with hash '" + i + "'", 4);
 			// Find observers that handle notifications for this type (or all types)
 			if (!_observers.get(i).types || _observers.get(i).types.indexOf(type)!=-1){
+				// Catch exceptions so all observers get notified even if
+				// one throws an error
+				try {
 					_observers.get(i).ref.notify(event, type, ids, extraData);
-				
-				/*
-				if (extraData) {
-					Zotero.debug(extraData);
 				}
-				*/
+				catch (e) {
+					Components.utils.reportError(e);
+				}
 			}
 		}
 		

chrome/content/zotero/xpcom/schema.js

@@ -353,16 +353,21 @@ Zotero.Schema = new function(){
 			Zotero.DB.query(sql);
 			var sql = "INSERT INTO itemAttachments VALUES(123456789, NULL, 3, 'text/html', 25, NULL, NULL)";
 			Zotero.DB.query(sql);
-			var sql = "INSERT INTO itemData VALUES(123456789, 110, 'Zotero - Quick Start Guide')";
+			var sql = "INSERT INTO itemDataValues VALUES (1, 'Zotero - Quick Start Guide')";
 			Zotero.DB.query(sql);
-			var sql = "INSERT INTO itemData VALUES(123456789, 1, 'http://www.zotero.org/documentation/quick_start_guide')";
+			var sql = "INSERT INTO itemData VALUES(123456789, 110, 1)";
 			Zotero.DB.query(sql);
-			var sql = "INSERT INTO itemData VALUES(123456789, 27, '2006-10-05 14:00:00')";
+			var sql = "INSERT INTO itemDataValues VALUES (2, 'http://www.zotero.org/documentation/quick_start_guide')";
+			Zotero.DB.query(sql);
+			var sql = "INSERT INTO itemData VALUES(123456789, 1, 2)";
+			Zotero.DB.query(sql);
+			var sql = "INSERT INTO itemDataValues VALUES (3, '2006-10-05 14:00:00')";
+			Zotero.DB.query(sql);
+			var sql = "INSERT INTO itemData VALUES(123456789, 27, 3)";
 			Zotero.DB.query(sql);
 			var sql = "INSERT INTO itemNotes (itemID, sourceItemID, note) VALUES(123456789, NULL, ?)";
 			var msg = "Welcome to Zotero! Click the \"View Page\" button above to visit our Quick Start Guide and learn how to get started collecting, managing, and citing your research.\n\nThanks for trying Zotero, and be sure to tell your friends about it.";
 			Zotero.DB.query(sql, msg);
-			
 			Zotero.DB.commitTransaction();
 		}
 		catch(e){
@@ -798,6 +803,61 @@ Zotero.Schema = new function(){
 					Zotero.DB.query("INSERT INTO items SELECT * FROM itemsTemp");
 					Zotero.DB.query("DROP TABLE itemsTemp");
 				}
+				
+				if (i==22) {
+					if (Zotero.DB.valueQuery("SELECT COUNT(*) FROM items WHERE itemID=0")) {
+						var itemID = Zotero.getRandomID('items', 'itemID');
+						Zotero.DB.query("UPDATE items SET itemID=? WHERE itemID=?", [itemID, 0]);
+						Zotero.DB.query("UPDATE itemData SET itemID=? WHERE itemID=?", [itemID, 0]);
+						Zotero.DB.query("UPDATE itemNotes SET itemID=? WHERE itemID=?", [itemID, 0]);
+						Zotero.DB.query("UPDATE itemAttachments SET itemID=? WHERE itemID=?", [itemID, 0]);
+					}
+					if (Zotero.DB.valueQuery("SELECT COUNT(*) FROM collections WHERE collectionID=0")) {
+						var collectionID = Zotero.getRandomID('collections', 'collectionID');
+						Zotero.DB.query("UPDATE collections SET collectionID=? WHERE collectionID=0", [collectionID]);
+						Zotero.DB.query("UPDATE collectionItems SET collectionID=? WHERE collectionID=0", [collectionID]);
+					}
+					Zotero.DB.query("DELETE FROM tags WHERE tagID=0");
+					Zotero.DB.query("DELETE FROM itemTags WHERE tagID=0");
+					Zotero.DB.query("DELETE FROM savedSearches WHERE savedSearchID=0");
+				}
+				
+				if (i==23) {
+					Zotero.DB.query("CREATE TABLE IF NOT EXISTS itemNoteTitles (\n    itemID INT,\n    title TEXT,\n    PRIMARY KEY (itemID),\n    FOREIGN KEY (itemID) REFERENCES itemNotes(itemID)\n);");
+					var notes = Zotero.DB.query("SELECT itemID, note FROM itemNotes WHERE itemID IN (SELECT itemID FROM items WHERE itemTypeID=1)");
+					var f = function(text) { var t = text.substring(0, 80); var ln = t.indexOf("\n"); if (ln>-1 && ln<80) { t = t.substring(0, ln); } return t; }
+					for each(var note in notes) {
+						Zotero.DB.query("INSERT INTO itemNoteTitles VALUES (?,?)", [note['itemID'], f(note['note'])]);
+					}
+					
+					Zotero.DB.query("CREATE TABLE IF NOT EXISTS itemDataValues (\n    valueID INT,\n    value,\n    PRIMARY KEY (valueID)\n);");
+					var values = Zotero.DB.columnQuery("SELECT DISTINCT value FROM itemData");
+					for each(var value in values) {
+						var valueID = Zotero.getRandomID('itemDataValues', 'valueID', 2097152); // Stored in 3 bytes
+						Zotero.DB.query("INSERT INTO itemDataValues VALUES (?,?)", [valueID, value]);
+					}
+					
+					Zotero.DB.query("CREATE TEMPORARY TABLE itemDataTemp AS SELECT itemID, fieldID, (SELECT valueID FROM itemDataValues WHERE value=ID.value) AS valueID FROM itemData ID");
+					Zotero.DB.query("DROP TABLE itemData");
+					Zotero.DB.query("CREATE TABLE itemData (\n    itemID INT,\n    fieldID INT,\n    valueID INT,\n    PRIMARY KEY (itemID, fieldID),\n    FOREIGN KEY (itemID) REFERENCES items(itemID),\n    FOREIGN KEY (fieldID) REFERENCES fields(fieldID)\n    FOREIGN KEY (valueID) REFERENCES itemDataValues(valueID)\n);");
+					Zotero.DB.query("INSERT INTO itemData SELECT * FROM itemDataTemp");
+					Zotero.DB.query("DROP TABLE itemDataTemp");
+				}
+				
+				if (i==24) {
+					var rows = Zotero.DB.query("SELECT * FROM itemData NATURAL JOIN itemDataValues WHERE fieldID IN (52,96,100)");
+					for each(var row in rows) {
+						if (!Zotero.Date.isMultipart(row['value'])) {
+							var value = Zotero.Date.strToMultipart(row['value']);
+							var valueID = Zotero.DB.valueQuery("SELECT valueID FROM itemDataValues WHERE value=?", value);
+							if (!valueID) {
+								var valueID = Zotero.getRandomID('itemDataValues', 'valueID', 2097152);
+								Zotero.DB.query("INSERT INTO itemDataValues VALUES (?,?)", [valueID, value]);
+							}
+							Zotero.DB.query("UPDATE itemData SET valueID=? WHERE itemID=? AND fieldID=?", [valueID, row['itemID'], row['fieldID']]);
+						}
+					}
+				}
 			}
 			
 			_updateSchema('userdata');
@@ -824,14 +884,14 @@ Zotero.Schema = new function(){
 		try { Zotero.DB.query("DROP TRIGGER insert_date_field"); } catch (e) {}
 		try { Zotero.DB.query("DROP TRIGGER update_date_field"); } catch (e) {}
 		
-		var itemDataTrigger = "  FOR EACH ROW WHEN NEW.fieldID IN (14, 27)\n"
+		var itemDataTrigger = "  FOR EACH ROW WHEN NEW.fieldID IN (14, 27, 52, 96, 100)\n"
 			+ "  BEGIN\n"
 			+ "    SELECT CASE\n"
-			+ "        CAST(SUBSTR(NEW.value, 1, 4) AS INT) BETWEEN 0 AND 9999 AND\n"
+			+ "        CAST(SUBSTR((SELECT value FROM itemDataValues WHERE valueID=NEW.valueID), 1, 4) AS INT) BETWEEN 0 AND 9999 AND\n"
-			+ "        SUBSTR(NEW.value, 5, 1) = '-' AND\n"
+			+ "        SUBSTR((SELECT value FROM itemDataValues WHERE valueID=NEW.valueID), 5, 1) = '-' AND\n"
-			+ "        CAST(SUBSTR(NEW.value, 6, 2) AS INT) BETWEEN 0 AND 12 AND\n"
+			+ "        CAST(SUBSTR((SELECT value FROM itemDataValues WHERE valueID=NEW.valueID), 6, 2) AS INT) BETWEEN 0 AND 12 AND\n"
-			+ "        SUBSTR(NEW.value, 8, 1) = '-' AND\n"
+			+ "        SUBSTR((SELECT value FROM itemDataValues WHERE valueID=NEW.valueID), 8, 1) = '-' AND\n"
-			+ "        CAST(SUBSTR(NEW.value, 9, 2) AS INT) BETWEEN 0 AND 31\n"
+			+ "        CAST(SUBSTR((SELECT value FROM itemDataValues WHERE valueID=NEW.valueID), 9, 2) AS INT) BETWEEN 0 AND 31\n"
 			+ "      WHEN 0 THEN RAISE (ABORT, 'Date field must begin with SQL date') END;\n"
 			+ "  END;\n";
 		

chrome/content/zotero/xpcom/search.js

@@ -192,9 +192,7 @@ Zotero.Search.prototype.addCondition = function(condition, operator, value, requ
 		
 		for each(var part in parts) {
 			this.addCondition('blockStart');
-			this.addCondition('title', operator, part.text, false);
 			this.addCondition('field', operator, part.text, false);
-			this.addCondition('numberfield', operator, part.text, false);
 			this.addCondition('creator', operator, part.text, false);
 			this.addCondition('tag', operator, part.text, false);
 			this.addCondition('note', operator, part.text, false);
@@ -578,10 +576,7 @@ Zotero.Search.prototype._buildQuery = function(){
 					case 'field':
 					case 'datefield':
 					case 'numberfield':
-						if (!condition['alias']){
+						if (condition['alias']) {
-							break;
-						}
-						
 							// Add base field
 							condSQLParams.push(
 								Zotero.ItemFields.getID(condition['alias'])
@@ -600,6 +595,11 @@ Zotero.Search.prototype._buildQuery = function(){
 							else {
 								condSQL += 'fieldID=? AND ';
 							}
+						}
+						
+						condSQL += "valueID IN (SELECT valueID FROM "
+							+ "itemDataValues WHERE ";
+						openParens++;
 						break;
 					
 					case 'collectionID':
@@ -1104,16 +1104,6 @@ Zotero.SearchConditions = new function(){
 				field: 'collectionID'
 			},
 			
-			{
-				name: 'title',
-				operators: {
-					contains: true,
-					doesNotContain: true
-				},
-				table: 'items',
-				field: 'title'
-			},
-			
 			{
 				name: 'dateAdded',
 				operators: {

chrome/content/zotero/xpcom/zotero.js

@@ -41,6 +41,8 @@ var Zotero = new function(){
 	this.getZoteroDatabase = getZoteroDatabase;
 	this.chooseZoteroDirectory = chooseZoteroDirectory;
 	this.debug = debug;
+	this.log = log;
+	this.getErrors = getErrors;
 	this.varDump = varDump;
 	this.safeDebug = safeDebug;
 	this.getString = getString;
@@ -58,6 +60,7 @@ var Zotero = new function(){
 	
 	// Public properties
 	this.initialized = false;
+	this.skipLoading = false;
 	this.__defineGetter__("startupError", function() { return _startupError; });
 	this.__defineGetter__("startupErrorHandler", function() { return _startupErrorHandler; });
 	this.version;
@@ -79,7 +82,7 @@ var Zotero = new function(){
 	 * Initialize the extension
 	 */
 	function init(){
-		if (this.initialized){
+		if (this.initialized || this.skipLoading) {
 			return false;
 		}
 		
@@ -361,25 +364,65 @@ var Zotero = new function(){
 			return false;
 		}
 		
-		// Note: DebugLogger extension is old and no longer supported -- if there's
+		dump('zotero(' + level + '): ' + message + "\n\n");
-		// a better extension we could switch to that as an option
+		return true;
-		if (false && ZOTERO_CONFIG['DEBUG_TO_CONSOLE']){
-			try {
-				var logManager =
-				Components.classes["@mozmonkey.com/debuglogger/manager;1"]
-				.getService(Components.interfaces.nsIDebugLoggerManager);
-				var logger = logManager.registerLogger("Zotero");
-			}
-			catch (e){}
 	}
 	
-		if (logger){
+	
-			logger.log(level, message);
+	/*
+	 * Log a message to the Mozilla JS error console
+	 *
+	 * |type| is a string with one of the flag types in nsIScriptError:
+	 *    'error', 'warning', 'exception', 'strict'
+	 */
+	function log(message, type, sourceName, sourceLine, lineNumber,
+			columnNumber, category) {
+		var consoleService = Components.classes["@mozilla.org/consoleservice;1"]
+			.getService(Components.interfaces.nsIConsoleService);
+		var scriptError = Components.classes["@mozilla.org/scripterror;1"]
+			.createInstance(Components.interfaces.nsIScriptError);
+		
+		if (!type) {
+			type = 'warning';
 		}
-		else {
+		var flags = scriptError[type + 'Flag'];
-			dump('zotero(' + level + '): ' + message + "\n\n");
+		
+		scriptError.init(
+			message,
+			sourceName ? sourceName : null,
+			sourceLine != undefined ? sourceLine : null,
+			lineNumber != undefined ? lineNumber : null, 
+			columnNumber != undefined ? columnNumber : null,
+			flags,
+			category
+		);
+		consoleService.logMessage(scriptError);
 	}
-		return true;
+	
+	
+	function getErrors() {
+		var errors = [];
+		var cs = Components.classes["@mozilla.org/consoleservice;1"].
+			getService(Components.interfaces.nsIConsoleService);
+		var messages = {};
+		cs.getMessageArray(messages, {})
+		
+		var skip = ['CSS Parser', 'content javascript'];
+		
+		for each(var msg in messages) {
+			Zotero.debug(msg);
+			try {
+				msg.QueryInterface(Components.interfaces.nsIScriptError);
+				if (skip.indexOf(msg.category) != -1) {
+					continue;
+				}
+				errors.push(msg.errorMessage);
+			}
+			catch(e) {
+				errors.push(msg.message);
+			}
+		}
+		return errors;
 	}
 	
 	
@@ -622,18 +665,18 @@ var Zotero = new function(){
 			max = 16383;
 		}
 		
+		max--; // since we use ceil(), decrement max by 1
 		var tries = 3; // # of tries to find a unique id
 		do {
 			// If no luck after number of tries, try a larger range
 			if (!tries){
 				max = max * 128;
 			}
-			var rnd = Math.floor(Math.random()*max);
+			var rnd = Math.ceil(Math.random()*max);
 			var exists = Zotero.DB.valueQuery(sql + rnd);
 			tries--;
 		}
 		while (exists);
-		
 		return rnd;
 	}
 	
@@ -1226,14 +1269,13 @@ Zotero.Date = new function(){
 		var utils = new Zotero.Utilities();
 		
 		var parts = strToDate(str);
-		parts.month = typeof parts.month != undefined ? parts.month + 1 : '';
+		parts.month = typeof parts.month != "undefined" ? parts.month + 1 : '';
 		
-		var multi = utils.lpad(parts.year, '0', 4) + '-'
+		var multi = (parts.year ? utils.lpad(parts.year, '0', 4) : '0000') + '-'
 			+ utils.lpad(parts.month, '0', 2) + '-'
-			+ utils.lpad(parts.day, '0', 2)
+			+ (parts.day ? utils.lpad(parts.day, '0', 2) : '00')
 			+ ' '
 			+ str;
-			
 		return multi;
 	}
 	
@@ -1455,7 +1497,6 @@ Zotero.WebProgressFinishListener = function(onFinish) {
 				//Zotero.debug('onFinish() called before STATE_STOP in WebProgressFinishListener.onStateChange()');
 				return;
 			}
-			
 			onFinish();
 		}
 	}

chrome/locale/en-US/zotero/zotero.properties

@@ -40,6 +40,7 @@ pane.tagSelector.numSelected.none				= 0 tags selected
 pane.tagSelector.numSelected.singular			= %S tag selected
 pane.tagSelector.numSelected.plural				= %S tags selected
 
+pane.items.loading				= Loading items list...
 pane.items.delete				= Are you sure you want to delete the selected item?
 pane.items.delete.multiple		= Are you sure you want to delete the selected items?
 pane.items.delete.title			= Delete
@@ -274,6 +275,8 @@ ingester.scrapeError			= Could Not Save Item
 ingester.scrapeErrorDescription	= An error occurred while saving this item. Check %S for more information.
 ingester.scrapeErrorDescription.linkText = Known Translator Issues
 
+db.dbCorrupted					= The Zotero database '%S' appears to have become corrupted.
+db.dbCorrupted.restart			= Please restart Firefox to attempt an automatic restore from the last backup.
 db.dbCorruptedNoBackup			= The Zotero database '%S' appears to have become corrupted, and no automatic backup is available.\n\nA new database file has been created. The damaged file was saved in your Zotero directory.
 db.dbRestored					= The Zotero database '%1$S' appears to have become corrupted.\n\nYour data was restored from the last automatic backup made on %2$S at %3$S. The damaged file was saved in your Zotero directory.
 db.dbRestoreFailed				= The Zotero database '%S' appears to have become corrupted, and an attempt to restore from the last automatic backup failed.\n\nA new database file has been created. The damaged file was saved in your Zotero directory.

chrome/skin/default/zotero/overlay.css

@@ -248,6 +248,11 @@
 	background-image: none;
 }
 
+#zotero-items-pane-message-box
+{
+	-moz-appearance: listbox;
+}
+
 
 #zotero-annotate-tb-add
 {

components/zotero-autocomplete.js

@@ -208,12 +208,6 @@ ZoteroAutoComplete.prototype.startSearch = function(searchString, searchParam,
 			}
 			break;
 		
-		case 'title':
-			var sql = "SELECT DISTINCT " + searchParam + " FROM items "
-				+ "WHERE " + searchParam + " LIKE ? ORDER BY " + searchParam;
-			var results = this._zotero.DB.columnQuery(sql, searchString + '%');
-			break;
-		
 		case 'dateModified':
 		case 'dateAdded':
 			var sql = "SELECT DISTINCT DATE(" + searchParam + ", 'localtime') FROM items "
@@ -245,14 +239,15 @@ ZoteroAutoComplete.prototype.startSearch = function(searchString, searchParam,
 			// use the user part of the multipart field
 			var valueField = searchParam=='date' ? 'SUBSTR(value, 12, 100)' : 'value';
 			
-			var sql = "SELECT DISTINCT " + valueField;
+			var sql = "SELECT DISTINCT " + valueField
-			sql += " FROM itemData WHERE fieldID=?1 AND " + valueField;
+				+ " FROM itemData NATURAL JOIN itemDataValues "
-			sql += " LIKE ?2 "
+				+ "WHERE fieldID=?1 AND " + valueField
+				+ " LIKE ?2 "
 			
 			var sqlParams = [fieldID, searchString + '%'];
 			if (extra){
 				sql += "AND value NOT IN (SELECT value FROM itemData "
-				+ "WHERE fieldID=?1 AND itemID=?3) ";
+					+ "NATURAL JOIN itemDataValues WHERE fieldID=?1 AND itemID=?3) ";
 				sqlParams.push(extra);
 			}
 			sql += "ORDER BY value";

userdata.sql

@@ -1,4 +1,4 @@
--- 21
+-- 24
 
 -- This file creates tables containing user-specific data -- any changes
 -- to existing tables made here must be mirrored in transition steps in
@@ -67,12 +67,18 @@ CREATE TABLE IF NOT EXISTS items (
 CREATE TABLE IF NOT EXISTS itemData (
     itemID INT,
     fieldID INT,
-    value,
+    valueID,
     PRIMARY KEY (itemID, fieldID),
     FOREIGN KEY (itemID) REFERENCES items(itemID),
     FOREIGN KEY (fieldID) REFERENCES fields(fieldID)
+    FOREIGN KEY (valueID) REFERENCES itemDataValues(valueID)
+);
+
+CREATE TABLE IF NOT EXISTS itemDataValues (
+    valueID INT,
+    value,
+    PRIMARY KEY (itemID)
 );
-CREATE INDEX IF NOT EXISTS value ON itemData(value);
 
 -- Note data for note items
 CREATE TABLE IF NOT EXISTS itemNotes (
@@ -85,6 +91,13 @@ CREATE TABLE IF NOT EXISTS itemNotes (
 );
 CREATE INDEX IF NOT EXISTS itemNotes_sourceItemID ON itemNotes(sourceItemID);
 
+CREATE TABLE IF NOT EXISTS itemNoteTitles (
+    itemID INT,
+    title TEXT,
+    PRIMARY KEY (itemID),
+    FOREIGN KEY (itemID) REFERENCES itemNotes(itemID)
+);
+
 -- Metadata for attachment items
 CREATE TABLE IF NOT EXISTS itemAttachments (
     itemID INT,