Webmaster recipes - Revision history

Shawn: Added hits per minute

2025-03-16T02:39:42Z

Added hits per minute

← Older revision		Revision as of 19:39, 15 March 2025
Line 5:		Line 5:
	===Count access from an IP prefix===		===Count access from an IP prefix===
	grep -c "^xxx.xxx." access.log		grep -c "^xxx.xxx." access.log
			===Count hits per minute===
			cut -d" " -f4 access.log\|cut -d":" -f2,3\|sort\|uniq -c
	===Dealing with hotlinking===		===Dealing with hotlinking===
	Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.		Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.

Shawn: Added logs one-liners

2025-03-16T02:21:19Z

Added logs one-liners

← Older revision		Revision as of 19:21, 15 March 2025
Line 1:		Line 1:
			===Detect heavy users in logs===
			cat access.log \| cut -d" " -f1\|sort \| uniq -c \| sort -r\|head
			===Count access to each file, ignoring URL parameters===
			cat access.log \|cut -d" " -f7\|cut -d"?" -f1\|sort\|uniq -c
			===Count access from an IP prefix===
			grep -c "^xxx.xxx." access.log
	===Dealing with hotlinking===		===Dealing with hotlinking===
	Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.		Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.

Shawn: /* Converting IP abbreviations to CIDR */

2015-02-03T21:03:21Z

Converting IP abbreviations to CIDR

← Older revision		Revision as of 14:03, 3 February 2015
Line 98:		Line 98:
	;Convert three-byte addresses		;Convert three-byte addresses
	:Here, the first 24 bits are constant.		:Here, the first 24 bits are constant.
			'''WARNING: IF COPYING, DON"T FORGET THE SPACES, ESPECIALLY AT THE END OF EACH EXPRESSION!'''
	([0-9]+\.[0-9]+\.[0-9]+)		([0-9]+\.[0-9]+\.[0-9]+)
	$1.0/24		$1.0/24
	;Convert four-byte (complete) addresses		;Convert four-byte (complete) addresses
	:Here, all 32 bits are constant.		:Here, all 32 bits are constant.
			'''WARNING: IF COPYING, DON"T FORGET THE SPACES, ESPECIALLY AT THE END OF EACH EXPRESSION!'''
	([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)		([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)
	$1/32		$1/32

Shawn: /* Making IP abbreviations normal CIDR */

2015-02-03T21:02:54Z

Making IP abbreviations normal CIDR

← Older revision		Revision as of 14:02, 3 February 2015
Line 87:		Line 87:
	#> pipes the output to a file		#> pipes the output to a file

	===~~Making~~ IP abbreviations ~~normal~~ CIDR===		===Converting IP abbreviations to CIDR===
	in Apache directives you might block a range of IP addresses with an abbreviated domain like 198.183 , meaning all the addresses between 198.183.0.0 and 198.183.255.255 . My block lists used to contain such abbreviations, but also CIDR addresses, such as 198.182.0.0/16, which blocks the range 198.182.0.0 to 198.182.255.255 . CIDR is useful because we can block not just 256 or 65536 addresses, but any power of two. For example, we can block 198.184.0.0 to 198.184.16.255 with this CIDR-formatted address: 198.184.0.0/20. Internet addresses are allotted in large swaths to regional authorities, who then allot smaller parts to internet providers, who may even break it down into smaller lots. Most of these lots can be addressed with a single CIDR address.		in Apache directives you might block a range of IP addresses with an abbreviated domain like 198.183 , meaning all the addresses between 198.183.0.0 and 198.183.255.255 . My block lists used to contain such abbreviations, but also CIDR addresses, such as 198.182.0.0/16, which blocks the range 198.182.0.0 to 198.182.255.255 . CIDR is useful because we can block not just 256 or 65536 addresses, but any power of two. For example, we can block 198.184.0.0 to 198.184.16.255 with this CIDR-formatted address: 198.184.0.0/20. Internet addresses are allotted in large swaths to regional authorities, who then allot smaller parts to internet providers, who may even break it down into smaller lots. Most of these lots can be addressed with a single CIDR address.

	To convert an abbreviated two or three byte address to the full CIDR notation, use these handy regular expressions and replacements. Please note that there are spaces ~~around~~ the regular expressions as well as the replacements, to prevent converting partial addresses:		To convert an abbreviated two or three byte address to the full CIDR notation, use these handy regular expressions and replacements. Please note that there are spaces before and after the regular expressions as well as the replacements, to prevent converting partial addresses:
	;Convert two-byte addresses		;Convert two-byte addresses
	:The parentheses mark the group to be captured, in between spaces—Two sets of any number (+) of numerical digits ([0-9]) separated by a period (the backslash lets the regex engine know we want a period, not the special meaning of period in regex). Whatever was in parentheses will be returned by $1, followed by .0.0/16, which indicates the full range of possibilities in those two digits. We put 16 because the first 16 bits, of the range of 32-bit addresses, are constant.		:The parentheses mark the group to be captured, in between spaces—Two sets of any number (+) of numerical digits ([0-9]) separated by a period (the backslash lets the regex engine know we want a period, not the special meaning of period in regex). Whatever was in parentheses will be returned by $1, followed by .0.0/16, which indicates the full range of possibilities in those two digits. We put 16 because the first 16 bits, of the range of 32-bit addresses, are constant.
			'''WARNING: IF COPYING, DON"T FORGET THE SPACES, ESPECIALLY AT THE END OF EACH EXPRESSION!'''
	([0-9]+\.[0-9]+)		([0-9]+\.[0-9]+)
	$1.0.0/16		$1.0.0/16
Line 99:		Line 100:
	([0-9]+\.[0-9]+\.[0-9]+)		([0-9]+\.[0-9]+\.[0-9]+)
	$1.0/24		$1.0/24
			;Convert four-byte (complete) addresses
			:Here, all 32 bits are constant.
			([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)
			$1/32

	===See also===		===See also===

Shawn at 04:40, 2 February 2015

2015-02-02T04:40:50Z

@@ Line 86: / Line 86: @@
 #Sh runs the echo and grep commands that awk prints
 #> pipes the output to a file
 ===See also===

Shawn: added page-visitor-tracking

2014-02-24T21:11:57Z

added page-visitor-tracking

@@ Line 71: / Line 71: @@
   {print substr($4,5,3)" "substr($4,2,2)}' access* |uniq -c
 This script skips all requests from my own IP, so I don't count my own hits, includes all requests for a certain PAGE_NAME or subset of pages with a certain text in their name, and returns the month and date, then counts how many hits there are each date. Note the backslash at the end of the first line is not part of the awk script but just a way to split this shell command into two lines. I.e., if you use it on one line, remove the backslash.
 ===See also===

Shawn: /* Searching logs for hotlinkers */

2014-02-21T19:47:05Z

Searching logs for hotlinkers

← Older revision		Revision as of 12:47, 21 February 2014
Line 6:		Line 6:
	if ($11 ~ /^"https?:\/\/[^\/]*energyteachers.org/) next;		if ($11 ~ /^"https?:\/\/[^\/]*energyteachers.org/) next;
	if ($11 ~ /"-"/) next;		if ($11 ~ /"-"/) next;
	print;}' access_log*		print $11;}' access_log* \|sort\|uniq -c\|sort
	;Explanation		;Explanation
	*$7 refers to the seventh segment of the log, split by spaces, which contains the request.		*$7 refers to the seventh segment of the log, split by spaces, which contains the request.
Line 13:		Line 13:
	*So, the first line of if...next tell awk to skip lines that aren't requests for image files.		*So, the first line of if...next tell awk to skip lines that aren't requests for image files.
	*The second line skips references from energyteachers.org		*The second line skips references from energyteachers.org
	*The third ~~lines~~ skips empty references, allowing people to bookmark images or type their addresses in their browser.		*The third line skips empty references, allowing people to bookmark images or type their addresses in their browser.
			*The fourth line prints the reference, and after finishing the awk command set, contains the wildcard-appended log-file-name to be processed, followed by a sort, then a count of how many times that reference was used, then sorting by that count.

	====Preventing hot-linking of images====		====Preventing hot-linking of images====

Shawn: /* Dealing with hotlinking */

2014-02-21T19:28:26Z

Dealing with hotlinking

← Older revision		Revision as of 12:28, 21 February 2014
Line 3:		Line 3:

	====Searching logs for hotlinkers====		====Searching logs for hotlinkers====
			awk '{if ($7 !~ /\.(jpe?g\|gif\|png)$/) next;
			if ($11 ~ /^"https?:\/\/[^\/]*energyteachers.org/) next;
			if ($11 ~ /"-"/) next;
			print;}' access_log*
			;Explanation
			*$7 refers to the seventh segment of the log, split by spaces, which contains the request.
			*!~ means "does not contain the following expression"
			*The expression between the slashes finds text that include a dot followed by one of the image extensions. The dollar sign signifies the end of the request.
			*So, the first line of if...next tell awk to skip lines that aren't requests for image files.
			*The second line skips references from energyteachers.org
			*The third lines skips empty references, allowing people to bookmark images or type their addresses in their browser.

	====Preventing hot-linking of images====		====Preventing hot-linking of images====

Shawn at 18:17, 21 February 2014

2014-02-21T18:17:30Z

← Older revision		Revision as of 11:17, 21 February 2014
Line 1:		Line 1:
	===~~Preventing hot-linking of images~~===		===Dealing with hotlinking===
	Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.		Hot-linking is the use of links to content at one site, usually images, by pages on another site. At first, page-designers may have used links to images on other sites because they couldn't create useful images themselves. These days, robots are made to make pages with random content, using both content and images from around the world to create a site in the hopes of fooling search engines into thinking that the site contains useful information on specific topics.

			====Searching logs for hotlinkers====

			====Preventing hot-linking of images====

	There are hundreds of pages telling how to turn away hot-linkers using Apache's Rewrite module, but Apache suggests you use their built-in directives when possible, and this is one of those cases. So, I found a different way, thanks to the [http://wiki.apache.org/httpd/DisableImageHotLinking wiki] at apache.org.		There are hundreds of pages telling how to turn away hot-linkers using Apache's Rewrite module, but Apache suggests you use their built-in directives when possible, and this is one of those cases. So, I found a different way, thanks to the [http://wiki.apache.org/httpd/DisableImageHotLinking wiki] at apache.org.

Shawn: /* Preventing hot-linking of images */

2014-02-21T02:21:29Z

Preventing hot-linking of images

← Older revision		Revision as of 19:21, 20 February 2014
Line 12:		Line 12:
	#The second line sets the local_ref variable to 1 if there is no referrer, such as when someone browses at an image from a bookmark or uses curl or some special tool for the blind.		#The second line sets the local_ref variable to 1 if there is no referrer, such as when someone browses at an image from a bookmark or uses curl or some special tool for the blind.
	#The third through sixth lines apply only if the files requested have image-type extensions.		#The third through sixth lines apply only if the files requested have image-type extensions.
	#The fifth line allows from anyone with the proper reference, ~~then leaves~~ the rest to be denied by the order of the fourth line.		#The fifth line allows such requests from anyone with the proper reference, leaving the rest to be denied by the order of the fourth line.

	===Multiple whois lookups===		===Multiple whois lookups===