.syncignore List Generator

[null null null null]

As you might know the .SyncIgnore file is a very powerful tool to selectively sync files/folders from a folder populated with many subfolders and other files.

 

Due to data caps I have been relying on btsync to keep my steam folder synced between different devices, rather than updating the games on each device. In a normal case btsync would copy all the games in my steam folder, however there is no point in having an AAA-Title on an ultrabook. To overcome this I played with the .SyncIgnore file and managed to select what should be synced to the netbook. This made the overall syncing process quicker and wasted less harddrive space.

 

The process of adding files manually works fine when you only want to exclude a few files but is rather time consuming when you only want 3 games from the folder. To overcome this problem I wrote a script in python that can take inputs as to what you do not want to ignore. The script then delivers a file with all the files that should be ignored. 

 

The script scans the directory it is placed in and generates a list of those directories. It then removes all the directories from the list related to the keywords in the white list. Once it is done it delivers a text file containing the directories which should be ignored. You can then just copy the list into the .SyncIgnore file.

 

If anyone is interested in this script I will look into converting it to a exe file and post it on the forum.

 

 NB :1. The script is still being refined and error handling is still an ongoing task.

        2. I am an engineer not a full time programmer. IT WONT BE PERFECT

        3. The steam case is one of many in which I have tested this successfully  

[null null null null]

Basic Instruction:

1. Save it as an .py file

2. Put the script in the folder you want to index

3. Make a text file WL.txt -> White list

4. Add the folder names of folders you want to allow to sync, but make sure there are no blank lines in the textfile!

 

When you run the script it will prompt how deep it should look in your sub folders. By default it is 7 layers

If  you do not make a White list and say no when prompted to use a white list, it will index everything in the directory

 

Known problems

1. The filtering system looks for keywords in a path. This can lead to folders containing part of the keyword being filtered although you did not want it to happen.

example: The folder contains 3 folders with the names Data, Private Data and Public Data. If the keyword were Data all three of the items would be removed from the generated black list.

2. UNIX directories:  I wrote the script for my windows pc's and as such the directory separator is an "\" and not a "/" as in UNIX based operating systems. The script currently does not support being run on a UNIX OS, however I hope to add it later on. 

 

 

Without a white list: looking 2layers deep

 

*Guild Wars 2

*STEAMGAMES_Common

 

 

*STEAMGAMES_Common\BioShock Infinite

*STEAMGAMES_Common\Borderlands 2

*STEAMGAMES_Common\CastleCrashers

*STEAMGAMES_Common\company of heroes

*STEAMGAMES_Common\Company of Heroes Relaunch

*STEAMGAMES_Common\darksiders

*STEAMGAMES_Common\DisplayFusion

*STEAMGAMES_Common\Far Cry 3

*STEAMGAMES_Common\GarrysMod

*STEAMGAMES_Common\magicka

*STEAMGAMES_Common\metro 2033

*STEAMGAMES_Common\Natural Selection 2

*STEAMGAMES_Common\Orcs Must Die 2

*STEAMGAMES_Common\red faction armageddon

*STEAMGAMES_Common\saints row the third

*STEAMGAMES_Common\Sanctum2

*STEAMGAMES_Common\Serious Sam 3

*STEAMGAMES_Common\Terraria

*STEAMGAMES_Common\Titan Quest

*STEAMGAMES_Common\total war shogun 2

*STEAMGAMES_Common\War Thunder

*STEAMGAMES_Common\Warhammer 40,000 Space Marine

 

 

*STEAMGAMES_Common\BioShock Infinite\Binaries

*STEAMGAMES_Common\BioShock Infinite\Engine

*STEAMGAMES_Common\BioShock Infinite\XGame

*STEAMGAMES_Common\Borderlands 2\Binaries

*STEAMGAMES_Common\Borderlands 2\DLC

*STEAMGAMES_Common\Borderlands 2\Documents

*STEAMGAMES_Common\Borderlands 2\Engine

*STEAMGAMES_Common\Borderlands 2\Gearbox

*STEAMGAMES_Common\Borderlands 2\WillowGame

*STEAMGAMES_Common\CastleCrashers\data

*STEAMGAMES_Common\CastleCrashers\directx

*STEAMGAMES_Common\company of heroes\CoH

*STEAMGAMES_Common\company of heroes\DLC1

*STEAMGAMES_Common\company of heroes\DLC2

*STEAMGAMES_Common\company of heroes\DLC3

*STEAMGAMES_Common\company of heroes\DX_redist_install

*STEAMGAMES_Common\company of heroes\Engine

*STEAMGAMES_Common\company of heroes\MilesRedist

*STEAMGAMES_Common\company of heroes\RelicDownloader

*STEAMGAMES_Common\company of heroes\RelicOnline

*STEAMGAMES_Common\company of heroes\tovredist

*STEAMGAMES_Common\company of heroes\validators

*STEAMGAMES_Common\company of heroes\VCRedist

*STEAMGAMES_Common\company of heroes\WorldBuilder

*STEAMGAMES_Common\company of heroes\WW2

*STEAMGAMES_Common\Company of Heroes Relaunch\CoH

*STEAMGAMES_Common\Company of Heroes Relaunch\DLC1

*STEAMGAMES_Common\Company of Heroes Relaunch\DLC2

*STEAMGAMES_Common\Company of Heroes Relaunch\DLC3

*STEAMGAMES_Common\Company of Heroes Relaunch\Drivers

*STEAMGAMES_Common\Company of Heroes Relaunch\Engine

*STEAMGAMES_Common\Company of Heroes Relaunch\MilesRedist

*STEAMGAMES_Common\Company of Heroes Relaunch\RelicOnline

*STEAMGAMES_Common\Company of Heroes Relaunch\WorldBuilder

*STEAMGAMES_Common\Company of Heroes Relaunch\WW2

*STEAMGAMES_Common\Company of Heroes Relaunch\_CommonRedist

*STEAMGAMES_Common\darksiders\media

*STEAMGAMES_Common\Far Cry 3\bin

*STEAMGAMES_Common\Far Cry 3\data_win32

*STEAMGAMES_Common\Far Cry 3\Support

*STEAMGAMES_Common\GarrysMod\bin

*STEAMGAMES_Common\GarrysMod\garrysmod

*STEAMGAMES_Common\GarrysMod\platform

*STEAMGAMES_Common\GarrysMod\sourceengine

*STEAMGAMES_Common\magicka\Content

*STEAMGAMES_Common\magicka\Cursors

*STEAMGAMES_Common\magicka\Dependencies

*STEAMGAMES_Common\magicka\SaveData

*STEAMGAMES_Common\metro 2033\install

*STEAMGAMES_Common\Natural Selection 2\assets

*STEAMGAMES_Common\Natural Selection 2\builds

*STEAMGAMES_Common\Natural Selection 2\core

*STEAMGAMES_Common\Natural Selection 2\docs

*STEAMGAMES_Common\Natural Selection 2\explore

*STEAMGAMES_Common\Natural Selection 2\lua_test

*STEAMGAMES_Common\Natural Selection 2\ns2

*STEAMGAMES_Common\Natural Selection 2\system

*STEAMGAMES_Common\Natural Selection 2\tutorial

*STEAMGAMES_Common\Natural Selection 2\utils

*STEAMGAMES_Common\Natural Selection 2\_CommonRedist

*STEAMGAMES_Common\Orcs Must Die 2\build

*STEAMGAMES_Common\Orcs Must Die 2\data

*STEAMGAMES_Common\Orcs Must Die 2\installer

*STEAMGAMES_Common\red faction armageddon\build

*STEAMGAMES_Common\red faction armageddon\DirectX

*STEAMGAMES_Common\red faction armageddon\reslists

*STEAMGAMES_Common\saints row the third\DirectX

*STEAMGAMES_Common\saints row the third\packfiles

*STEAMGAMES_Common\saints row the third\reslists

*STEAMGAMES_Common\saints row the third\videos

*STEAMGAMES_Common\Sanctum2\Binaries

*STEAMGAMES_Common\Sanctum2\Engine

*STEAMGAMES_Common\Sanctum2\Install

*STEAMGAMES_Common\Sanctum2\SanctumGame

*STEAMGAMES_Common\Serious Sam 3\Bin

*STEAMGAMES_Common\Serious Sam 3\Content

*STEAMGAMES_Common\Serious Sam 3\Log

*STEAMGAMES_Common\Serious Sam 3\Redist

*STEAMGAMES_Common\Serious Sam 3\Temp

*STEAMGAMES_Common\Terraria\Content

*STEAMGAMES_Common\Titan Quest\Audio

*STEAMGAMES_Common\Titan Quest\Database

*STEAMGAMES_Common\Titan Quest\EULA

*STEAMGAMES_Common\Titan Quest\Manual

*STEAMGAMES_Common\Titan Quest\MilesRedist

*STEAMGAMES_Common\Titan Quest\Resources

*STEAMGAMES_Common\Titan Quest\Settings

*STEAMGAMES_Common\Titan Quest\Text

*STEAMGAMES_Common\Titan Quest\Toolset

*STEAMGAMES_Common\Titan Quest\Videos

*STEAMGAMES_Common\total war shogun 2\awesomium_cache

*STEAMGAMES_Common\total war shogun 2\benchmarks

*STEAMGAMES_Common\total war shogun 2\data

*STEAMGAMES_Common\total war shogun 2\locales

*STEAMGAMES_Common\total war shogun 2\miles

*STEAMGAMES_Common\total war shogun 2\redist

*STEAMGAMES_Common\War Thunder\.launcher_log

*STEAMGAMES_Common\War Thunder\cache

*STEAMGAMES_Common\War Thunder\compiledShaders

*STEAMGAMES_Common\War Thunder\content

*STEAMGAMES_Common\War Thunder\launcher_upd

*STEAMGAMES_Common\War Thunder\levels

*STEAMGAMES_Common\War Thunder\redist

*STEAMGAMES_Common\War Thunder\Replays

*STEAMGAMES_Common\War Thunder\res

*STEAMGAMES_Common\War Thunder\Screenshots

*STEAMGAMES_Common\War Thunder\sound

*STEAMGAMES_Common\War Thunder\ui

*STEAMGAMES_Common\War Thunder\video

*STEAMGAMES_Common\War Thunder\_CommonRedist

*STEAMGAMES_Common\War Thunder\_debuginfo

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\config

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\data

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\logs

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\redist

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\Screenshots

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\streamingaudio

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\system

*STEAMGAMES_Common\Warhammer 40,000 Space Marine\videos

 

WL.txt 

Far Cry 3

 

With a white list looking 7 layers deep for "Far Cry 3"

 

#########################################

#Space required:10.891GB

#########################################

 

*Guild Wars 2

 

 

*STEAMGAMES_Common\BioShock Infinite

*STEAMGAMES_Common\Borderlands 2

*STEAMGAMES_Common\CastleCrashers

*STEAMGAMES_Common\company of heroes

*STEAMGAMES_Common\Company of Heroes Relaunch

*STEAMGAMES_Common\darksiders

*STEAMGAMES_Common\DisplayFusion

*STEAMGAMES_Common\GarrysMod

*STEAMGAMES_Common\magicka

*STEAMGAMES_Common\metro 2033

*STEAMGAMES_Common\Natural Selection 2

*STEAMGAMES_Common\Orcs Must Die 2

*STEAMGAMES_Common\red faction armageddon

*STEAMGAMES_Common\saints row the third

*STEAMGAMES_Common\Sanctum2

*STEAMGAMES_Common\Serious Sam 3

*STEAMGAMES_Common\Terraria

*STEAMGAMES_Common\Titan Quest

*STEAMGAMES_Common\total war shogun 2

*STEAMGAMES_Common\War Thunder

*STEAMGAMES_Common\Warhammer 40,000 Space Marine

 

 

SyncIgnore Blacklist GeneratorV3.8.py

-----------------------------------------------------------------------------------------------------------------------------------------

 

#######################################################################  

def estimated_size(White_list):

    from os import chdir,walk,path,listdir

    los=[]

    for root,dirs,files in walk('.'):  

        for skp in White_list:

            if skp in root:

                los.append(root)

    size = 0    

    for dd in los:

        dirr = listdir(dd)

        for d in dirr:

            if path.isfile( dd+'\\'+d):

                size += path.getsize(dd+'\\'+d)

 

    if size > 1024**4:

        siz = str(round((size/(1024.)**4),3)) +'TB'        

    elif size > 1024**3:

        siz = str(round((size/(1024.)**3),3)) +'GB'

    elif size > 1024**2:

        siz = str(round((size/(1024.)**2),3)) +'MB'       

    elif size > 1024:

        siz = str(size/1024.) +'KB'

    return(siz)

#######################################################################  

def remove_eksta(lys):

    a = lys    

    b = lys

    while '' in a:

        a.remove('')

    verwyder = []

     

    for i in a:

        for j in b:

            x = j.split('\\')

            y = ('\\').join(x[:-1])

            if (i in y) and (i != j) and (len(i) >1):

                verwyder.append(j)                

 

    for skip in verwyder:

        for lyn in b:

            if skip == lyn:

                 b.remove(skip) 

    return( d)

#######################################################################        

#add new line

def nl_add(lys):

    for lyn in enumerate(lys):

        lys[lyn[0]] = lyn[1]+'\n'

    return(lys)

 

#######################################################################        

#clean_dir_trail

def clc_dir_trail(lys,skip):

    add_skip =[]

    for lyn in enumerate(lys):

        #kyk na alles

        for i in enumerate(skip):

            #Is element in die lyn            

            if i[1] in lyn[1]:  

                #remove parents

                temp = []

                for split_lyn in enumerate(lyn[1].split('\\')):

                    temp.append(split_lyn[1])                 

                    if temp[split_lyn[0]] not in add_skip:

                        add_skip.append(('\\').join(temp))

    for skip in add_skip:

        for lyn in lys:

            if skip == lyn:

                 lys.remove(skip) 

    return(lys)

#######################################################################        

#filter basiese 

def skip_goed(lyn_str,skip):

    los=[]

    for skp in skip:

        for lyn in lyn_str:

            if skp in lyn:

                los.append(lyn)

    

    for skip in los:

        for lyn in lyn_str:       

            if skip in lyn:

                lyn_str.remove(lyn)    

    return(lyn_str)

 

#######################################################################    

# soek vir dubbels in die lys

def Soek_dubbels(lyn_str):

    tel = 0

    new_list_sondub=[]

    while tel < (len(lyn_str)-1):

        if lyn_str[tel] <> lyn_str[tel+1]:

            new_list_sondub.append(lyn_str[tel])

        tel+=1

    new_list_sondub.append(lyn_str[tel]) 

    return(new_list_sondub)

 

#######################################################################    

#sorteer die directories volgens diepte

 

def sort_dirs(dir_array,layers):

    full_list = []

    level = 1

    while level<=int(layers):

        for index in range(0,(len(dir_array))):

            if len(dir_array[index].split('\\')) == level:

                full_list.append(dir_array[index])

        level +=1

        full_list.append('\n')

    return(full_list)

    

#######################################################################    

#Kry al die dir inligting en verwerk na n lys wat in n txt file 

# gebruik kan word

def fix(dir,layers):

    from os import chdir,walk,path

    chdir(dir)

    tel = 0

    lyn_str=[]

    for root,dirs ,files in walk(dir):

        dirr = root.split('\\')

        layer = int(layers)

        lengte = len(dirr)

        if (lengte<=layer):

            lyn = dirr[1:layer+1]

        else:

            lyn = dirr[1:layer+1]         

        lyn_str.append('*'+'\\'.join(lyn))

        tel+=1

    return(lyn_str)

 

 

#######################################################################

#script

#######################################################################    

from os import curdir, startfile

#maak die txt file

new_file = open('SyncIgnore.txt','w')

new_file.close()

 

#Hoeveel subdirs moet na gekyk word

layers = raw_input("Subdir depth: ")

whitelist = raw_input('Do you want to use your white list (y/n): ')

#is die input valid as nie default na 3

import string

if layers == '':

    layers = 7

elif (layers[0] not in string.digits):

    layers = 7

   

 

Final_list = []

#Filter lys

Final_list = fix(curdir,layers)

Final_list = sort_dirs(Final_list,layers)

Final_list = Soek_dubbels(Final_list)

#Filrer skip uit

skip = ['SyncArchive','$RECYCLE','MSOCache']

Final_list = skip_goed(Final_list,skip)

 

basic =['# .SyncIgnore is a UTF-8 encoded .txt file that helps you specify single files, paths and rules\n', 

'# for ignoring during the synchronization job. It supports "?" and "*" wildcard symbols.\n',

'#\n',

'#\n',

'# OS generated files #\n',

'.DS_Store\n',

'.DS_Store?\n',

'._*\n',

'.Spotlight-V100\n',

'.Trashes\n',

'ehthumbs.db\n',

'desktop.ini\n',

'Thumbs.db\n',

'~$*\n',

'*.laccdb\n',

'\n']

new_file = open('SyncIgnore.txt','a')

new_file.writelines(basic)

new_file.close() 

 

 

# If whitelist exists

if whitelist in ['y','Y']:   

    skip =[]

    p = open('WL.txt','r')

    skip_raw = [line for line in p.readlines()]

    for index in range(0,len(skip_raw)-1):

        skip.append(skip_raw[index][:-1])

    skip.append(skip_raw[len(skip_raw)-1]) 

 

    while '' in skip:        

        skip.remove('')

        

    while '\n' in skip:

        skip.remove('\n')

    p.close()

    Final_list=clc_dir_trail(Final_list,skip)

    Final_list = skip_goed(Final_list,skip)

    size = estimated_size(skip)

    

    

    

    new_file = open('SyncIgnore.txt','a')

    new_file.writelines(['#########################################\n',

                        '#Space required:'+size+'\n',

                        '#########################################\n','\n'])

    new_file.close() 

    Final_list= remove_eksta(Final_list)

 

################    

Final_list=nl_add(Final_list)

#print na txt file

new_file = open('SyncIgnore.txt','a')

new_file.writelines(Final_list[1:])

new_file.close() 

startfile('SyncIgnore.txt')

#######################################################################