Add canonical headers for files in Apache2

Search engines will complain if websites and files do not have a canonical name, and appear under multiple different URLs with the same content. Especially for search engine optimization this is a big no no.

How does that happen

Someone might link to your resource with a tracking link, as example this entry in the webserver log:

xyz.xyz.xyz.xyz - - [20/Dec/2024:01:04:57 +0100] "GET /writings/Add_Geocoordinates_to_your_addresses_in_PostgreSQL_-_Meetup_HH_April_2024.pdf?fbclid=<some random id> HTTP/1.1" 200 1032842 "-" "Mozilla/<some random browser>"

The attached fbclid header makes it obvious that this PDF was posted on Facebook, and then Facebook attaches a semi-random string, it’s called the Facebook Click Identifier. It is used to circumvent Ad-blockers and Third-Party-Cookie Protection and allows Facebook to track the performance of the link.

It also creates two unique URLs for the same content, at least. One without fbclid, one with every single fbclid attached.

Add to that problem the gclid Google Click Identifier and the li_fat_id LinkedIn Click Identifier and it becomes obvious how this is a problem for SEO.

A search engine might stumble over any of those links, read the content of the link, receive the PDF and identify it as “the same content as the other URLs”.

How to prevent this

The HTML5 standard knows the link rel="canonical" header with the canonical path for a resource. Every time a search engine fetches the website, it uses this header as original path and can apply the content of the path with tracking id to the canonical path, as defined by the website creator.

As example:

1

<link rel="canonical" href="https://andreas.scherbaum.la/content/post/2024-12-26_add-canonical-headers-for-files-in-apache2/">

This is the canonical URL for this blog posting.

But that does not work with files. It’s not possible to add a HTML5 header to a PDF. However there is a second way: add the canonical name as HTTP header.

This is easy to do in Apache2, using the mod_rewrite and mod_headers modules.

1
2
3
4
5
6
7
8

	<FilesMatch "\.(pdf|bz2|xml|zip|sql|odt|ods|odp|odg|pptm|pptx)">
		RewriteEngine On
		RewriteCond %{REQUEST_URI} (.*)
		RewriteRule .* - [E=RU:%1]
		RewriteCond %{SERVER_NAME} (.*)
		RewriteRule .* - [E=SN:%1]
		Header always set Link "<https://%{SN}e%{RU}e>; rel=\"canonical\""
	</FilesMatch>

The piece of config above will store the Server Name and the Request URI (the path, without parameters) in environment variables RU and SN, and then add the Link header with the canonical name.

The following example uses curl to fetch a PDF from the website with a parameter attached:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

curl --verbose --output /dev/null --silent "https://andreas.scherbaum.la/writings/Add_Geocoordinates_to_your_addresses_in_PostgreSQL_-_Meetup_HH_April_2024.pdf?test"

* Host andreas.scherbaum.la:443 was resolved.
...
> GET /writings/Add_Geocoordinates_to_your_addresses_in_PostgreSQL_-_Meetup_HH_April_2024.pdf?test HTTP/1.1
> Host: andreas.scherbaum.la
> User-Agent: curl/8.11.1
> Accept: */*

...

< HTTP/1.1 200 OK
< Date: Wed, 25 Dec 2024 23:13:32 GMT
< Server: Apache
< X-Frame-Options: SAMEORIGIN
< Link: <https://andreas.scherbaum.la/writings/Add_Geocoordinates_to_your_addresses_in_PostgreSQL_-_Meetup_HH_April_2024.pdf>; rel="canonical"
< Last-Modified: Mon, 29 Apr 2024 12:28:50 GMT
< ETag: "fc28a-6173b654e155b"
< Accept-Ranges: bytes
< Content-Length: 1032842
< Cache-Control: max-age=2419200, public
< Content-Type: application/pdf

Along with the content of the file (Content-Length: 1032842), the webserver also returns the canonical name:

Link: <https://andreas.scherbaum.la/writings/Add_Geocoordinates_to_your_addresses_in_PostgreSQL_-_Meetup_HH_April_2024.pdf>; rel="canonical"

That’s neat!

Other options

One possible other option to solve this problem is to 301 (permanent) redirect every request with a parameter to the canonical URL. However this approach is problematic: many websites use parameters for valid website operations, think CMS. This only works if you are sure not to destroy the link while removing parameters.

It also doesn’t work well with POST request. However in the example above a file will never receive valid POST requests, therefore I don’t need to handle this case.

Comments

With an account on the Fediverse or Mastodon, you can respond to this post. Known replies are displayed below:

Load Comments from Fediverse

Note: This will load data from mastodon.social.

Learn how this is implemented here.