Huginn: Monitor changes in websites

Posted by ads' corner on Sunday, 2020-11-22
Posted in [Huginn][Linux]

The Huginn software is not only good for monitoring Twitter feeds, it can also be used to monitor websites for changes.

That’s actually easy and needs only two agents:

In my example I’m monitoring a certain Play Store entry for price changes. Because, why not. So I need to configure the Website Agent with the following details:

Website Agent

  • expected_update_period_in_days: 2 (since I run this agent every day, it should produce an update every day)
  • url: <the Play Store URL>
  • type: html
  • modeon_change

So far, so easy. The complicated part is figuring out the CSS selector for the price. It helps to open the website in any modern browser in inspection mode, and hover over the interesting area. The CSS selector is listed there. You also need to name the selector in Huginn, in my case I name it price.

  • extract: add price

Under price add:

  • css#fcxH9b > div.WpDbMd > c-wiz > div > div.ZfcPIb > div > div > main > c-wiz:nth-child(1) > c-wiz:nth-child(1) > div > div.D0ZKYe > div > div.wE7q7b > div > div.hfWwZc > div > c-wiz > c-wiz > div > span > button
  • valuenormalize-space(.)

If the content is parsed as html (or xml), then the selector is an XPath function. If parsing as JSON, the selector is a JSONPath function.

The complete option field, for reference:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "expected_update_period_in_days": "2",
  "url": "https://play.google.com/store/apps/details?id=com.flashlight.ultra.gps.logger",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "price": {
      "css": "#fcxH9b > div.WpDbMd > c-wiz > div > div.ZfcPIb > div > div > main > c-wiz:nth-child(1) > c-wiz:nth-child(1) > div > div.D0ZKYe > div > div.wE7q7b > div > div.hfWwZc > div > c-wiz > c-wiz > div > span > button",
      "value": "normalize-space(.)"
    }
  }
}

Email Agent

All what’s left is sending a notification when something changes. I’m using the Email Agent for this, but any notification agent can be used.

  • expected_receive_period_in_days: 3650 (since I don’t expect changes anytime soon, I just set this to a very high number - otherwise Huginn will disable the agent because it’s not functioning)
  • subject: set the subject of the email
  • content_type: text/html
  • recipients: specify add email addresses for all recipients
  • body: text for the email body

As body text I configured this text:

1
Updates for the Ultra GPS Logger App!<br/><br/><a href="https://play.google.com/store/apps/details?id=com.flashlight.ultra.gps.logger">Visit App Store</a>

Categories: [Huginn] [Linux]