AnthorNet

Db-Wal And Log File Sizes

Recommended Posts

Hi,

I had two questions.. On debian I'm syncing a folder with approximatly 100K files for 1.6GB

 

My main problem is that the db-wal file associated with that folder is constantly growing and is now at 13GB, wich is very huge for so few files ^^

Had the same problem with sync.log growing at 10GB

 

My question is : Is there a way to flush those files or desactivate them ?

I already clear the log file, but have to flush it every week, and don't know if I can touch the db-wal one.

 

Thanks by advance.

Edited by AnthorNet

Share this post


Link to post
Share on other sites

Hi AnthorNet,

 

SQlite stores a write-ahead log in this file. It is growing rapidly most likely due to extensive file operations in Synced folders. We are working to optimise it's size in future.

 

As for sync.log - it grows as you have extended logging turned on. To turn off logging go to .sync folder (located bar btsync binary) and find a file debug.txt. Delete it, your sync.log should not get growing (or growing extremely slowly).

Share this post


Link to post
Share on other sites

Hi RomanZ,

Thanks for your answer..

 

I saw the debub.txt and forgot I turned it on, it's now better !

As for the db-wal, I restarted and changed the sync on debian to read-only as I only use it as a backup storage.

I now have a new db-wal, but, is it safe to delete it sometimes ?

 

News problem, since then, my sync cannot be terminated ! I have 9kb unsynced and can't find wich file is causing it !

Is there any method to know wich file cause that ?

 

I already try rebooting all the machines, without success.

Edited by AnthorNet

Share this post


Link to post
Share on other sites

db-wal files should be deleted automatically. I'll check with developers if it safe to delete db-wal when btsync is down, but you definitely shouldn't do it while it is up and running.

 

As for the sync showing 9kb - give it several minutes and it should re-evaluate data volume necessary to sync. Current UI does not show which files are actually going to be synced, only those that are in process.

Share this post


Link to post
Share on other sites

It's been 4 days since I had those 9kb.

I also have another folder with 1.6kb.

 

I tried to add those folder to another of my computer wich also run Windows 7, but the issue is the same !

This doesn't seems to be a no go issue as new/edited and deleted files get synced properly.

Share this post


Link to post
Share on other sites

4 days are definitely enough. Looks like BTSync wants to sync some file and fails to do it by some reason. Can you provide logs from both machines for analysis? Here goes the instruction.

 

Also, it would be safe to remove db-wal files when BTSync is down.

Share this post


Link to post
Share on other sites

Hi,

 

if you have WAL-mode enabled for SQLite and delete the WAL file, you'll corrupt the database file. DON'T DELETE IT! (This is not specific to btsync, but to any app using sqlite with WAL enabled.)

 

Additonal information and how one could deal with it is available from http://www.sqlite.org/draft/wal.html and other ressources on http://www.sqlite.org/

 

Best regards,

Capi

Share this post


Link to post
Share on other sites

Actually, what I saw while debugging, is that stopping or restarting btsync just reset the db-wal files,

they're deleted automatically and recreated on restart.

 

I'll send the debugging logs.

Share this post


Link to post
Share on other sites

It depends on how an application closes the DB connection, e.g. if it resets to REDO-log mode, it will remove the WAL files, for instance. But still, removing an existing WAL file will cause corruption, as they are part of the DB.

Share this post


Link to post
Share on other sites

Capi,

 

according to official SQLite documentation db-wal files are temp files. Until everything is going fine with the database itself - they are safe to be deleted. Also, they are still deleted automatically when all connections to DB are closed.

 

They could be useful when say your app crashes and db gets corrupt.

Share this post


Link to post
Share on other sites

RomanZ,

 

you lose data, if no checkpoint occured, as data has then not yet been written back to database. If you delete it during checkpointing or the PC/app crashed while performing a checkpoint, you still need the WAL to recover/repair the data file. So no, it is not safe to remove the WAL file ever, unless sqlite removes it itself (which it does, when the last connection to the database is closed, as you mentioned). If the file still exists, the shutdown was not clean and the WAL will be required on next connection to the database.

 

It is a temporary file in the sense that it will be removed once everything is fine and applied, but it is not temp in the sense that you should/could remove it on will.

 

I have done my fair share of development with SQLite, and used WAL-mode a lot as it is a huge performance improvement in case of multiple writers and the advantage that a writer transaction won't block reader transactions.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.