Sqlite stumped across multiple threads with different files
I have an OS X (Yosemite, 10.10) application that does long running jobs involving heavy sqlite usage across multiple threads. And I got stumped across 8 threads, all of which got stuck in sqlite code connecting to various database files. There is no visible resource link between them. I am debugging it on a new Mac Pro (late 2013).
Four of them are on this stack. Of these, three work on the same table (again, different database files); three updates and one request.
__psynch_mutexwait
_pthread_mutex_lock
unixLock
sqlite3PagerSharedLock
sqlite3BtreeBeginTrans
sqlite3VdbeExec
sqlite3_step
One of them is on this stack, updating the same table as the three on the stack above.
guarded_close_np
nolockClose
pager_end_transaction
sqlite3BtreeCommitPhaseTwo
sqlite3VdbeHalt
sqlite3VdbeExec
sqlite3_step
Two are on this stack, opening database files with the same name in different locations. Open mode SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX
.
__psynch_mutexwait
_pthread_mutex_lock
sqlite3ParseUri
openDatabase
One is on this stack, ending the transaction.
__psynch_mutexwait
_pthread_mutex_lock
unixLock
sqlite3VdbeHalt
sqlite3VdbeExec
sqlite3_step
So the question is: what can cause sqlite to get locked without any shared resources involved?
Update: I now have seven threads blocking the call sqlite3_open_v2
and one on sqlite3_close
, all in different database files (several with the same name but in different folders). Stack:
__psynch_mutexwait
_pthread_mutex_lock
sqlite3ParseUri
openDatabase
Private stack:
guarded_close_np
unixClose
sqlite3PagerClose
sqlite3BtreeClose
sqlite3LeaveMutexAndCloseZombie
sqlite3Close
I managed to get it to work longer before blocking by fixing some memory leaks (this doesn't work with ARC) and removing some of the transaction statements.
Update 2: I connected SQLITE_LOG
via sqlite3_config
( documentation ) and I "I see a lot of code 28 logging ( sqlite_warning ) with the message" file renamed when opened: ".
Update 3: I wiped my machine and did a fresh install of Yosemite trying to rule out filesystem issues. I still block the same way. It will run for a few minutes and then block threads one by one. There's one in guarded_close_np
, stuck in an assembly with an instruction jae <address here>
, where the address it was knocked into has an instruction retq
. I asked a separate question about the sqlite log message about renaming files in the hope that it is related.
source to share
My application was using sqlite, linking it with a dynamic link library. I downloaded the latest raw SQLite merge from here (3.8.7.4 at the time of writing) and compiled it straight into the application and everything was up and running. So it might have been a bug in 3.8.5. Apparently compiling the source in the application directly is the recommended way to use sqlite anyway.
I still don't know exactly what caused the problem. The only thing I can think of is that it has to do with how I create the database files: I create an empty file with NSFileManager createFileAtPath
and then pass it sqlite3_open_v2
with SQLITE_OPEN_CREATE
as part of flags. Therefore, writing the database to an existing file instead of creating the database file at the specified location.
source to share
It looks like you are stuck on a UNIX master mutex that needs to be acquired before closing the file:
/*
** Close a file.
*/
static int unixClose(sqlite3_file *id){
int rc = SQLITE_OK;
unixFile *pFile = (unixFile *)id;
verifyDbFile(pFile);
unixUnlock(id, NO_LOCK);
unixEnterMutex(); <- HERE
...
This mutex is mainly executed during low-level file operations. You will need to find the thread containing the mutex and see what it does. It might be running on a slow or broken filesystem.
source to share