Watch for changed files in SyncThing

For syncing files between my devices I’m using SyncThing. This tool is reliable, available on Linux, Android, Mac and iOS. And it encrypts the communication.

But sometimes I want to know when files in certain directories have changed - as example in my Obsidian vault. This allows me to post-process the files.

Arrows

Some of the use cases I have in Obsidian:

Resolve links in Daily Notes: when I share a URL from my RSS reader or from other sources into Obsidian, the URL sometimes is just a link to a URL shortener. I then later need to resolve the link - or let a script do it right away for known shortlinks, and update the daily note.
Remove tracking information from URLs: many shared links include campaign and tracking information, and this can be removed straight away.
Extract content from certain Toots: I follow a couple interesting accounts on Mastodon, and when I share the Toot link into Obsidian, the script extracts the Toot content and adds it, along with the original link, to a pre-defined note.
Extract links from Toots: many news websites include a link (sometimes again with tracking information) in their Toots. When I share such a Toot into Obsidian, a script picks up the link, extract the target link and updates the daily note.

All of this is not very complicated, and a couple of lines in Python do the job. The main parts for the script are:

extract the API key from the SyncThing configuration
Open a connection to the local SyncThing instance
Watch for certain events

Extract the API key

One could pre-configure the API key in a script configuration file, but why not extract it from the SyncThing instance config? After all, both the script and SyncThing are running locally. The SyncThing configuration file is in XML format, which makes it necessary to load the xml.etree.ElementTree module in Python.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import os
import sys
import xml.etree.ElementTree as ET

def get_syncthing_apikey():
    configfile = os.path.join(os.environ.get('HOME'), ".config", "syncthing", "config.xml")
    tree = ET.parse(configfile)
    root = tree.getroot()
    apikey = root.findall("./gui/apikey")[0].text

    if (len(apikey) < 2):
        print("Can't find API key!")

    return apikey

apikey = get_syncthing_apikey()

SyncThing Connection

This part requires the Syncthing Python module, which you may or may not need to install using pip.

1
2
3
4
5
6
7
8
from syncthing import Syncthing, SyncthingError

def open_syncthing_connection(apikey):
    s = Syncthing(apikey, host = '127.0.0.1', timeout = 30.0)

    return s

conn = open_syncthing_connection(apikey)

Listen for events

Now that the connection is open, it gets a bit tricky. The connection will timeout eventually, and this needs to be handled in Python. The following code will do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
event_stream = conn.events(limit = 50)
while (True):
    try:
        for event in event_stream:
            handle_item(event)
    except SyncthingError as e:
        pass
    except KeyboardInterrupt as e:
        print("Exit ...")
        sys.exit(0)

SyncThing generates quite a few different events, but the only one which is necessary to watch out for is LocalIndexUpdated. This one is triggered once files are synced locally. That’s the right time to check the file out and see if there is something to process.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def handle_item(item):
    type = item['type']

    if (type == "LocalIndexUpdated"):
        data = item['data']
        if (data['folder'] == "<your folder ID>"):
            handle_syncdir_obsidian(item)
            return

    return

In my case I only watch for files in my Obsidian vault, and work on the daily notes. From here it’s basically whatever you want to to with the synced files. My example code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
def handle_syncdir_obsidian(item):
    obsidiandir = os.path.join(os.environ.get('HOME'), "obsidian-vault")
    id = item['id']
    data = item['data']

    filenames = data['filenames']
    for f in filenames:
        if ('/.obsidian/' in f):
            continue
        if ('Default/DailyNotes/' in f):
            if (pathlib.Path(f).suffix == ".md"):
                handle_obsidian_daily_notes(f, obsidiandir)

The function handle_obsidian_daily_notes() gets passed the modified file and the path to the vault directory. It’s up to this function to do whatever it wants, but that is beyond the scope of this blog posting.

Summary

It’s easy to attach a listener to a local SyncThing instance and watch for changes. The script is triggered once file changes are synved, which allows for post-processing of files in an Obsidian vault.

Full example code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
import os
import sys
import xml.etree.ElementTree as ET
from syncthing import Syncthing, SyncthingError

def get_syncthing_apikey():
    configfile = os.path.join(os.environ.get('HOME'), ".config", "syncthing", "config.xml")
    tree = ET.parse(configfile)
    root = tree.getroot()
    apikey = root.findall("./gui/apikey")[0].text

    if (len(apikey) < 2):
        print("Can't find API key!")

    return apikey

def open_syncthing_connection(apikey):
    s = Syncthing(apikey, host = '127.0.0.1', timeout = 30.0)

    return s

def handle_item(item):
    type = item['type']

    if (type == "LocalIndexUpdated"):
        data = item['data']
        if (data['folder'] == "<your folder ID>"):
            handle_syncdir_obsidian(item)
            return

    return

def handle_syncdir_obsidian(item):
    obsidiandir = os.path.join(os.environ.get('HOME'), "obsidian-vault")
    id = item['id']
    data = item['data']

    filenames = data['filenames']
    for f in filenames:
        if ('/.obsidian/' in f):
            continue
        if ('Default/DailyNotes/' in f):
            if (pathlib.Path(f).suffix == ".md"):
                handle_obsidian_daily_notes(f, obsidiandir)

def handle_obsidian_daily_notes(f, dir):
    # do something with the daily note here
    pass

# main code
apikey = get_syncthing_apikey()
conn = open_syncthing_connection(apikey)

event_stream = conn.events(limit = 50)
while (True):
    try:
        for event in event_stream:
            handle_item(event)
    except SyncthingError as e:
        pass
    except KeyboardInterrupt as e:
        print("Exit ...")
        sys.exit(0)

Image

The image in this posting is from OpenClipart-Vectors on Pixabay.

Categories: [Obsidian] [Python] [Syncthing]

Tags: [Obsidian] [Python] [Syncthing] [Xml]