Sebastien Vasey's Doc

Organization of sites

The basic model is as follows, when the user requests a file in a domain dom1.dom2.dom3.tld (e.g if the requested domain is foo.bar.svasey.org, dom1 would be foo, dom2 would be bar etc...), it is served the content in /home/www/sites/dom3.tld/sub/dom2/sub/dom1/pages. Most of the time pages is a symbolic link pointing to the site's content somewhere in /home/www/build/.

If the path above does not exist, this most likely means the domain does not exist or no http content is served for that domain. In that case, the content served is in /home/www/sites/default/pages, and will probably just be a simple html page indicating an error.

Of course, if the domain is dom2.dom3.tld, the content served would be at /home/www/sites/dom3.tld/sub/dom2/pages. For completeness, if the user asks for an IP, the same system is used, and the digits of the IP are considered as domain names, i.e if the ip is 1.2.4.8, dom1 is 1, dom2 is 2 etc...

In short, the format is always the same: $domainpath/pages. There are other directories in $domainpath:

• sub/ contains the subdomains
• conf/ contains lighttpd configuration files used only for the given domain. If they are specified in /etc/lighttpd/trusted-conf.conf, they will be used when the specific domain is queried. They are usually used for redirection or authentication. Any conf/ directory must contain at least one empty file named FAKE.

lighttpd-htpasswd

This is used in the lighttpd configuration file to print all the authentication directives: this basically finds all the file with name htpasswd in the hierarchy and prints configuration directives so that lighttpd uses them. This is a very simple program that takes no command line arguments.

lighttpd-addconf

This is used in the lighttpd configuration file to include other configuration files via trusted-conf.conf (using the include_shell command).

The usage is as follows:

lighttpd-addconf [file1 file2 ...]

where the arguments are configuration files to include, in the order they are to be included. The program will display those files to standard output if they are valid lighttpd configuration files.

To check the syntax of your configuration file, the program will need to see the main http configuration file. By default, its path is assumed to be /etc/lighttpd/lighttpd.conf, but you can change this with the --mainfile option.

You can also use the --config=CONFIGFILE option to read the configuration files from CONFIGFILE, which must contain one file path per line (the path must be relative to CONFIGFILE); lines beginning with # are ignored. The files given will be included after those given on lighttpd-addconf's command line, in the order they are given in CONFIGFILE.

Assuming your files are organized in the same way as mine, you can use the --condhost option to make sure the content of a configuration file in /home/www/sites is included only if its domain is asked for by the client.

For example, if your configuration file is in /home/www/sites/svasey.org/conf/redirect.conf, lighttpd-addconf will output something like:

$HTTP["host"] =~ "^(svasey\.org)[.]*$" {
    $contentoffile
}

where $contentoffile is the content of your configuration file.

website-install and website-remove

I use those to install, remove and update website content on my server. They assume everything is configured exactly as said in that documentation.

website-install takes two arguments: the path to the website's content (containing conf/ and pages/ directories), and the domain name on which the website should be installed. Note that this will not install any subdomain.

website-install takes two command line options:

• --to-copy: used to specify additionnal items to be copied into the final website. This is especially useful when you cannot integrate some part of your website into the simplewp hierarchy. Trac is an example: I initialize the environment after installing everything else, directly into the existing website. This means I have to use --to-copy whenever I modify the website; otherwise the trac environment will simply not be considered and get removed. --to-copy can be given multiple times in order to copy multiple directories.
• --wpcompile-opts: used to give additionnal, space-separated, options to wpcompile.

website-remove takes one argument: the domain name of the website to be removed. Not that this will not remove any subdomain.

Internals

I run a script daily to analyze my log files from last month, update the awstats database, and generate static html files summing up all the information.

Here are the main directories:

• /var/lib/awstats/ contains the awstats database
• /home/awstats/ contains all the static html files. It is owned by http:http.
  • pages/ contains the actual html files. The pages.1/ and pages.2/ directories are used internally to provide atomicity.
• /etc/awstats/ contains the awstats configuration files:
  • awstats-www-common.conf contains settings global to all sites
  • awstats.*.svasey.conf is the generic format for a site-specific configuration file. It should begin with Include "awstats-www-common.conf"
• /home/www/sites/svasey.org/sub/secure/sub/stats/pages/ is the web interface to browse my statistics.
  • stats/ is a symbolic link to the /home/awstats pages.

Web interface

A password protected interface can be accessed at stats.secure.svasey.org. It is implemented in the www-stats package.

Updating the stats

The script awstats-update, from the awstats-config package, is run to update the awstats database. The command takes two options:

• --reset can be used to reread all the logs from scratch and remove all the existing data.
• --awstats-args=AWSTATSARGS can be used to pass additionnal space-separated arguments to awstats.

awstats-update is run daily, and also 30 minutes after log rotation.

Monitored sites

The following sites are monitored:

• svasey.org
• documentation.svasey.org
• certs.svasey.org
• download.svasey.org
• public-repo.svasey.org

MIME type test

This is run by the checkmime tool from http-helpers. It checks that a MIME type is defined for all my files. This is done by looking up a list of known extensions in /etc/lighttpd/mimecheck.conf, and making sure no file with another extension hang around.

The /etc/lighttpd/mimecheck.conf file is a simple INI-style configurationb file. Two variables can be given:

• known_extensions = .html .pdf .tar.gz README ...: give a list of space-separated extensions to consider as known. If it does not begin with a dot, the extension is taken to mean "the whole filename".
• exclude = /first/path /second/path2 ...: give a list of domain path to exclude from the search. They should be given as absolute path and be a domain directory, i.e a directory containing the sub and pages subdirectories.

The mimecheck command itself takes no argument and returns 0 if the test was successfull, 1 if it failed, outputting the problematic files to stderr.

Broken links and validity test

This is run by the htmlcheck tool from http-helpers. It checks that there are no dead links in my pages, and that they are all valid (x)html and css. I run them only on public websites, as using it on private websites can cause some security issues, and is not that useful anyway.

Most of the work is done by the excellent linkchecker , htmlcheck is just a wrapper over it.

htmlcheck takes as arguments the name of my websites, as they are declared in the /etc/linkchecker/htmlcheck.conf file, and checks each of them using linkchecker. If no name is given, all of them will be checked.

The syntax of /etc/linkchecker/htmlcheck.conf is INI style, as usual. Each section documents a website. Here is an example:

[documentation]

urls = http://documentation.svasey.org/
opts = --check-html-w3 --check-css-w3

[test]

urls = http://test.svasey.org/
opts = --check-html-w3 --check-css-w3

[certs]

urls = http://certs.svasey.org/
opts = --check-html-w3 --check-css-w3

Each website must be given a urls variable, containing one or more space-separated urls. An additionnal variable, opts is also possible and can be used to give linkchecker specific options for the given site.

For all sites, linkchecker reads its configuration from /etc/linkchecker/linkcheckerrc, whose syntax is documented in its manpage.

You can also give additionnal options to linkchecker using the --opts command line option of htmlcheck.

Plain-HTTP sites

• test.svasey.org: Just a test page to quickly check if my http server is working
• svasey.org: The main public http website. For simplicity reasons it has no www prefix.
• certs.svasey.org: where I publish all my SSL certificates
• documentation.svasey.org: where I publish my (public) documentation
• download.svasey.org: Where I publish "large" files
• public-repo.svasey.org: My public Arch Linux repository. Also contains the latest source tarballs of my software.

SSL (secured) sites

• test.secure.svasey.org: another test page, testing SSL and HTTP authentication.
• trac.secure.svasey.org: My personal private trac.
• sv-documentation.secure.svasey.org: Where I publish documentation that is for my eyes only.
• stats.secure.svasey.org: Containing some of my sites statistics