Db-Wal And Log File Sizes

[AnthorNet]

Hi,

I had two questions.. On debian I'm syncing a folder with approximatly 100K files for 1.6GB

 

My main problem is that the db-wal file associated with that folder is constantly growing and is now at 13GB, wich is very huge for so few files ^^

Had the same problem with sync.log growing at 10GB

 

My question is : Is there a way to flush those files or desactivate them ?

I already clear the log file, but have to flush it every week, and don't know if I can touch the db-wal one.

 

Thanks by advance.

Edited January 11, 2014 by AnthorNet

[RomanZ]

Hi AnthorNet,

 

SQlite stores a write-ahead log in this file. It is growing rapidly most likely due to extensive file operations in Synced folders. We are working to optimise it's size in future.

 

As for sync.log - it grows as you have extended logging turned on. To turn off logging go to .sync folder (located bar btsync binary) and find a file debug.txt. Delete it, your sync.log should not get growing (or growing extremely slowly).

[AnthorNet]

Hi RomanZ,

Thanks for your answer..

 

I saw the debub.txt and forgot I turned it on, it's now better !

As for the db-wal, I restarted and changed the sync on debian to read-only as I only use it as a backup storage.

I now have a new db-wal, but, is it safe to delete it sometimes ?

 

News problem, since then, my sync cannot be terminated ! I have 9kb unsynced and can't find wich file is causing it !

Is there any method to know wich file cause that ?

 

I already try rebooting all the machines, without success.

Edited January 15, 2014 by AnthorNet

[RomanZ]

db-wal files should be deleted automatically. I'll check with developers if it safe to delete db-wal when btsync is down, but you definitely shouldn't do it while it is up and running.

 

As for the sync showing 9kb - give it several minutes and it should re-evaluate data volume necessary to sync. Current UI does not show which files are actually going to be synced, only those that are in process.

[AnthorNet]

It's been 4 days since I had those 9kb.

I also have another folder with 1.6kb.

 

I tried to add those folder to another of my computer wich also run Windows 7, but the issue is the same !

This doesn't seems to be a no go issue as new/edited and deleted files get synced properly.

[RomanZ]

4 days are definitely enough. Looks like BTSync wants to sync some file and fails to do it by some reason. Can you provide logs from both machines for analysis? Here goes the instruction.

 

Also, it would be safe to remove db-wal files when BTSync is down.

[capi]

Hi,

 

if you have WAL-mode enabled for SQLite and delete the WAL file, you'll corrupt the database file. DON'T DELETE IT! (This is not specific to btsync, but to any app using sqlite with WAL enabled.)

 

Additonal information and how one could deal with it is available from http://www.sqlite.org/draft/wal.html and other ressources on http://www.sqlite.org/

 

Best regards,

Capi

[AnthorNet]

Actually, what I saw while debugging, is that stopping or restarting btsync just reset the db-wal files,

they're deleted automatically and recreated on restart.

 

I'll send the debugging logs.

[capi]

It depends on how an application closes the DB connection, e.g. if it resets to REDO-log mode, it will remove the WAL files, for instance. But still, removing an existing WAL file will cause corruption, as they are part of the DB.

[RomanZ]

Capi,

 

according to official SQLite documentation db-wal files are temp files. Until everything is going fine with the database itself - they are safe to be deleted. Also, they are still deleted automatically when all connections to DB are closed.

 

They could be useful when say your app crashes and db gets corrupt.

[capi]

RomanZ,

 

you lose data, if no checkpoint occured, as data has then not yet been written back to database. If you delete it during checkpointing or the PC/app crashed while performing a checkpoint, you still need the WAL to recover/repair the data file. So no, it is not safe to remove the WAL file ever, unless sqlite removes it itself (which it does, when the last connection to the database is closed, as you mentioned). If the file still exists, the shutdown was not clean and the WAL will be required on next connection to the database.

 

It is a temporary file in the sense that it will be removed once everything is fine and applied, but it is not temp in the sense that you should/could remove it on will.

 

I have done my fair share of development with SQLite, and used WAL-mode a lot as it is a huge performance improvement in case of multiple writers and the advantage that a writer transaction won't block reader transactions.