Skip to content

Blog website crawlers and bots in Apache2

Found a couple more bots crawling my website, and from the look at online resources it seems I catched a few of the bad guys. Crawlers which ignore the robots.txt standard, and just crawl a website for content.

Decided to do something against it, and added a filter in Apache2.

The way I have my webserver setup is that I have templates for every website (they all have different configs), and deploy them using Ansible. Parts of the website configuration which are the same, or at least similar, are handled by includes.

 

Continue reading "Blog website crawlers and bots in Apache2"
  • Twitter
  • Bookmark Blog website crawlers and bots in Apache2 at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Blog website crawlers and bots in Apache2
  • Bloglines Blog website crawlers and bots in Apache2
  • Technorati Blog website crawlers and bots in Apache2
  • Fark this: Blog website crawlers and bots in Apache2
  • Bookmark Blog website crawlers and bots in Apache2 at YahooMyWeb
  • Bookmark Blog website crawlers and bots in Apache2 at Furl.net
  • Bookmark Blog website crawlers and bots in Apache2 at reddit.com
  • Bookmark Blog website crawlers and bots in Apache2 at blinklist.com
  • Bookmark Blog website crawlers and bots in Apache2 at Spurl.net
  • Bookmark Blog website crawlers and bots in Apache2 at Simpy.com
  • Bookmark Blog website crawlers and bots in Apache2 at blogmarks
  • Bookmark Blog website crawlers and bots in Apache2 with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Monitor website status with Huginn

After setting up Huginn, and implementing the actions on my todo list, I had a look at the available agents and started thinking what else they can be useful for.

One of the ideas I came up with is monitoring if a website is available, or has some trouble. I already have a monitoring system in place, but it's a nice exercise to learn more about the other agents.

 

Continue reading "Monitor website status with Huginn"
  • Twitter
  • Bookmark Monitor website status with Huginn at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Monitor website status with Huginn
  • Bloglines Monitor website status with Huginn
  • Technorati Monitor website status with Huginn
  • Fark this: Monitor website status with Huginn
  • Bookmark Monitor website status with Huginn at YahooMyWeb
  • Bookmark Monitor website status with Huginn at Furl.net
  • Bookmark Monitor website status with Huginn at reddit.com
  • Bookmark Monitor website status with Huginn at blinklist.com
  • Bookmark Monitor website status with Huginn at Spurl.net
  • Bookmark Monitor website status with Huginn at Simpy.com
  • Bookmark Monitor website status with Huginn at blogmarks
  • Bookmark Monitor website status with Huginn with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Huginn: Filter Retweets

A while ago I started using Huginn, as a replacement for IFTTT. That's going quite well. Huginn offers more features, integrations, and especially your chains (scenarios) can be as complex as you wish. IFTTT is quite limited in this area.

I use the Twitter integration to find certain Tweets. Now this does not only find native Tweets, but also finds every Retweet made for a native Tweet. Obviously I am not interested in duplicate content.

Huginn offers a way to filter out Retweets.

 

Continue reading "Huginn: Filter Retweets"
  • Twitter
  • Bookmark Huginn: Filter Retweets at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Huginn: Filter Retweets
  • Bloglines Huginn: Filter Retweets
  • Technorati Huginn: Filter Retweets
  • Fark this: Huginn: Filter Retweets
  • Bookmark Huginn: Filter Retweets at YahooMyWeb
  • Bookmark Huginn: Filter Retweets at Furl.net
  • Bookmark Huginn: Filter Retweets at reddit.com
  • Bookmark Huginn: Filter Retweets at blinklist.com
  • Bookmark Huginn: Filter Retweets at Spurl.net
  • Bookmark Huginn: Filter Retweets at Simpy.com
  • Bookmark Huginn: Filter Retweets at blogmarks
  • Bookmark Huginn: Filter Retweets with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Public previews in Hugo

Hugo is a static templating system. It is (mainly) used to deploy websites/blogs which don't have and need dynamic content. The content of all pages is pre-generated, and the webserver delivers files from disk (or rather from cache, once files are loaded into memory). This approach allows for extremely fast websites, as no dynamic content is generated on every request.

While I know Hugo from work, I haven't really used it for private projects - until recently. I have started a new project where I present interviews with people behind the PostgreSQL Project - and this is perfect for a static website. Interviews don't change, once published.


There was just a little problem: every interview must be approved by the interviewed person. This requires a full preview, but one which does not show up on the main website, or the Sitemap, or the RSS feed. By default, even drafts show up in Sitemap and the RSS feed in Hugo.

 

Continue reading "Public previews in Hugo"
  • Twitter
  • Bookmark Public previews in Hugo at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Public previews in Hugo
  • Bloglines Public previews in Hugo
  • Technorati Public previews in Hugo
  • Fark this: Public previews in Hugo
  • Bookmark Public previews in Hugo at YahooMyWeb
  • Bookmark Public previews in Hugo at Furl.net
  • Bookmark Public previews in Hugo at reddit.com
  • Bookmark Public previews in Hugo at blinklist.com
  • Bookmark Public previews in Hugo at Spurl.net
  • Bookmark Public previews in Hugo at Simpy.com
  • Bookmark Public previews in Hugo at blogmarks
  • Bookmark Public previews in Hugo with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Pushover app on Huawei Android phones

While diving deeper into my openHAB installation, the need for notifications on mobile phones came up. After some research, I settled with Pushover, which provides Android and iOS apps, as well as Desktop notifications. openHAB Rules have support for Pushover, and everything works well together.

 

Except when Android decides to kill apps, because they sit idle and do nothing. It so happens on my Huawei phone that notifications are delivered to the device, but no popup shows up. Only when I open the up, suddenly all the messages are there. The Pushover FAQ has an entry for this, even for Huawei phones, but it is outdated.

 

Continue reading "Pushover app on Huawei Android phones"
  • Twitter
  • Bookmark Pushover app on Huawei Android phones at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Pushover app on Huawei Android phones
  • Bloglines Pushover app on Huawei Android phones
  • Technorati Pushover app on Huawei Android phones
  • Fark this: Pushover app on Huawei Android phones
  • Bookmark Pushover app on Huawei Android phones at YahooMyWeb
  • Bookmark Pushover app on Huawei Android phones at Furl.net
  • Bookmark Pushover app on Huawei Android phones at reddit.com
  • Bookmark Pushover app on Huawei Android phones at blinklist.com
  • Bookmark Pushover app on Huawei Android phones at Spurl.net
  • Bookmark Pushover app on Huawei Android phones at Simpy.com
  • Bookmark Pushover app on Huawei Android phones at blogmarks
  • Bookmark Pushover app on Huawei Android phones with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

OpenVPN on iPad

Vacation time ahead, and the iPad in the house (who ever had the idea of buying one?) needs a proper VPN connection, in order to be able to access Netflix in Germany.

On Android it's fairly easy: install the app, drop CA cert, private cert, and key on the device, add everything - done. Three devices ready in less than 10 minutes.

And then I tried the iPad ...

 

Continue reading "OpenVPN on iPad"
  • Twitter
  • Bookmark OpenVPN on iPad at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg OpenVPN on iPad
  • Bloglines OpenVPN on iPad
  • Technorati OpenVPN on iPad
  • Fark this: OpenVPN on iPad
  • Bookmark OpenVPN on iPad at YahooMyWeb
  • Bookmark OpenVPN on iPad at Furl.net
  • Bookmark OpenVPN on iPad at reddit.com
  • Bookmark OpenVPN on iPad at blinklist.com
  • Bookmark OpenVPN on iPad at Spurl.net
  • Bookmark OpenVPN on iPad at Simpy.com
  • Bookmark OpenVPN on iPad at blogmarks
  • Bookmark OpenVPN on iPad with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

How to prioritize WiFi connections on Android

At home I have two different WiFi networks, both known to all my mobile devices. One is our internal network, password protected. The other one is an open Freifunk network. It's nice to walk around in the village and automatically connect to other Freifunk WiFi routers, instead of using expensive mobile data.

But when at home I really want my devices connected to our internal network. Makes it easy to copy files between devices, or access services only available in our internal network. Plus the native WiFi connection is much faster than the Freifunk connection, which is limited by the speed of the VPN.

However Android connects to the last used WiFi network, and when the last one was another Freifunk router, it will connect to this network at home as well. By default, Android does not allow to prioritize networks. That's where "WiFi Prioritizer" comes to the rescue!

 

Continue reading "How to prioritize WiFi connections on Android"
  • Twitter
  • Bookmark How to prioritize WiFi connections on Android at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg How to prioritize WiFi connections on Android
  • Bloglines How to prioritize WiFi connections on Android
  • Technorati How to prioritize WiFi connections on Android
  • Fark this: How to prioritize WiFi connections on Android
  • Bookmark How to prioritize WiFi connections on Android at YahooMyWeb
  • Bookmark How to prioritize WiFi connections on Android at Furl.net
  • Bookmark How to prioritize WiFi connections on Android at reddit.com
  • Bookmark How to prioritize WiFi connections on Android at blinklist.com
  • Bookmark How to prioritize WiFi connections on Android at Spurl.net
  • Bookmark How to prioritize WiFi connections on Android at Simpy.com
  • Bookmark How to prioritize WiFi connections on Android at blogmarks
  • Bookmark How to prioritize WiFi connections on Android with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

How I improve OpenStreetMap

OpenStreetMap is an awesome project. Contributors constantly improve a online map and add new data, fix items in the map which did change in the real world, or improve the data for a number of projects (1, 2, 3, 4, 5, 6, List) building on top of the map. Being outside a lot, I figured that I can contribute as well. It's just a question of how to transport the data from "out there" back to the laptop in my office. After probing around and trying a few tools, I came up with 3 different ways, depending on the situation and amount of changes.

 

Continue reading "How I improve OpenStreetMap"
  • Twitter
  • Bookmark How I improve OpenStreetMap at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg How I improve OpenStreetMap
  • Bloglines How I improve OpenStreetMap
  • Technorati How I improve OpenStreetMap
  • Fark this: How I improve OpenStreetMap
  • Bookmark How I improve OpenStreetMap at YahooMyWeb
  • Bookmark How I improve OpenStreetMap at Furl.net
  • Bookmark How I improve OpenStreetMap at reddit.com
  • Bookmark How I improve OpenStreetMap at blinklist.com
  • Bookmark How I improve OpenStreetMap at Spurl.net
  • Bookmark How I improve OpenStreetMap at Simpy.com
  • Bookmark How I improve OpenStreetMap at blogmarks
  • Bookmark How I improve OpenStreetMap with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Reduce Spotify cache size under Linux

The Spotify client for Linux, by default, will eat away a GB of your precious disk space. Since I mostly listen to random playlists, and rarely ever hear a song twice, there is no need to store all the songs on my hard disk.

 

Continue reading "Reduce Spotify cache size under Linux"
  • Twitter
  • Bookmark Reduce Spotify cache size under Linux at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Reduce Spotify cache size under Linux
  • Bloglines Reduce Spotify cache size under Linux
  • Technorati Reduce Spotify cache size under Linux
  • Fark this: Reduce Spotify cache size under Linux
  • Bookmark Reduce Spotify cache size under Linux at YahooMyWeb
  • Bookmark Reduce Spotify cache size under Linux at Furl.net
  • Bookmark Reduce Spotify cache size under Linux at reddit.com
  • Bookmark Reduce Spotify cache size under Linux at blinklist.com
  • Bookmark Reduce Spotify cache size under Linux at Spurl.net
  • Bookmark Reduce Spotify cache size under Linux at Simpy.com
  • Bookmark Reduce Spotify cache size under Linux at blogmarks
  • Bookmark Reduce Spotify cache size under Linux with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca

Good-bye Trello, Hello Wunderlist

For quite a while now, my wife and I were using Trello to keep things organized. We both had our private lists and boards, and also a few shared boards for joint tasks. Over time we realized that this tool is more annoying than helpful, and we decided to try something else. After a lengthy discovery phase we settled with Wunderlist.

 

There are a few requirements we have for a to-do list tool:

  • Shared boards/lists
  • Managed using a Website, obviously
  • Android app, possibly iOS app
  • Proper notifications, as in configurable alarms and separate notification per task
  • Subtasks and checklists
  • Attachments
  • Option to assign a task to someone else
  • A bit more fancy and modern interface than Trello has


All in all not very advanced features, all basic stuff.

 

Continue reading "Good-bye Trello, Hello Wunderlist"
  • Twitter
  • Bookmark Good-bye Trello, Hello Wunderlist at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg Good-bye Trello, Hello Wunderlist
  • Bloglines Good-bye Trello, Hello Wunderlist
  • Technorati Good-bye Trello, Hello Wunderlist
  • Fark this: Good-bye Trello, Hello Wunderlist
  • Bookmark Good-bye Trello, Hello Wunderlist at YahooMyWeb
  • Bookmark Good-bye Trello, Hello Wunderlist at Furl.net
  • Bookmark Good-bye Trello, Hello Wunderlist at reddit.com
  • Bookmark Good-bye Trello, Hello Wunderlist at blinklist.com
  • Bookmark Good-bye Trello, Hello Wunderlist at Spurl.net
  • Bookmark Good-bye Trello, Hello Wunderlist at Simpy.com
  • Bookmark Good-bye Trello, Hello Wunderlist at blogmarks
  • Bookmark Good-bye Trello, Hello Wunderlist with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Identi.ca