With current browsers and typical ISP web hosting account http server setups, surfers can view a web site by typing either www.example.com or example.com into their browser’s URL field. Prevailing wisdom in the SEO community is that web administors should convert all visits to one or the other. They suggest that if your site’s visitors split half & half in their choice of URL, Google and other page rankers may see the two variants as separate sites and divide your page visits accordingly (or even drop one set of pages as duplicates). Matt Cutts from Google suggests canonicalization as a good idea, but doesn’t comment on any problems if you don’t. It seems to me that converting all visitors to a canonical URL has no downside and many benefits, so I decided to do it.

Using Apache’s Rewrite Engine to Force the use of one Server Name

We can consolidate all visitors to a single, canonical form of the server name portion of our URL using the Apache rewrite engine. The Apache documentation’s Rewriting Guide presents example code related to this case:

   RewriteCond %{HTTP_HOST}   !^www\\.domain\\.name [NC]
   RewriteCond %{HTTP_HOST}   !^$
   RewriteRule ^/(.*)         http://www.domain.name/$1 [L,R]

I tried this code on two different ISP accounts and could not get it to work. After checking all sorts of blind alleys, I realized that none of my other rewriting rules started with a slash, so I took it (shown in red) out and the rule started working. Either the example in the Apache manual is wrong, or some server installations pass the opening slash to the engine and others (like my two ISPs) do not.

Although the example now worked, it required hand-editing for each new site. So I wrote the following rule set which can be inserted into any root .htaccess file:

   # Force top-level "domain.com" requests to "www.domain.com":
   RewriteCond %{HTTP_HOST}   !^www\\. [NC]
   RewriteCond %{HTTP_HOST}   ^[^.]+\\.(com|edu|net|org)$ [NC]
   RewriteRule (.*)           http://www.%{HTTP_HOST}/$1 [R=permanent,L]
   # NOTE: Other rules must follow the www rule:
   RewriteRule ^sitemap.xml   sitemap.php [NC]

The second rewrite condition restricts the rule to primary domain requests, preventing intranet, subdomain, and/or localhost accesses from being rewritten. I also set the force redirect (R) flag code to permanent (301). The last rule (L) flag instructs the engine to stop rewriting and return the redirect at this point. The browser receives the 301 message and re-requests the page using the canonical form. Canonical page requests are scanned by any other rewriting rules, which should always follow the renaming rules.

To Prefix or Not to Prefix, That is the Question

My final rule set above chooses the “www” variant as the canonical form. However, the WWW is Deprecated folks advocate choosing the non-prefixed form — “use of the www subdomain is redundant and time consuming to communicate. The internet, media, and society are all better off without it.”

Excellent point, but small sites must follow the real-world trendsetters in matters like this. A quick survey shows that yahoo.com, google.com, and msn.com all choose “www” as their canonical URL format.

I now have 3 different Apache hosting environments: my openSUSE 10.2 workstation, GoDaddy, and A2Hosting. Signing up for a month is the only way to determine if a web application will work on an ISP’s service. Their hosting environments are all slightly different. Here are the issues I’ve encountered getting a PHP web application running on each system.

openSUSE 10.2

• Apache2, PHP 5.2, mod_php, mod_rewrite.
• PHP magic_quotes_gpc is set to off in openSUSE (the proper choice, but not the PHP default as I painfully learned when installing at the ISPs).
• PEAR supported in the Apache PHP include path.
• Subdirectories written into by the application (e.g. Smarty template compilation area) need world-write privilege.

GoDaddy Hosting

• Apache1.3, PHP4 (5.1.4 optional), CGI/fastCGI, mod_rewrite.
• Cannot use php_flag because using CGI, must use php5.ini instead.
• Switch from PHP 4 to 5 in .htaccess:
  AddHandler x-httpd-php5 .php
  AddHandler x-httpd-php .php4
• PHP magic_quotes_gpc on by default, turn off with local php5.ini file.
• AllowOverride Options disabled, can’t use in .htaccess.
• Files created by web site users (i.e. by the httpd user) have the same uid/gid as my ftp login user. An excellent configuration approach, as I don’t need to give world write privileges to local data subdirectories.
• ini_set(’session.cache_limiter’, ‘private’) causes server 500 error.

A2 Hosting

• Apache 1.x (server_signature empty), PHP 5.2, mod_php, mod_rewrite.
• PHP magic_quotes_gpc is on by default, turn off using php_flag (here is the Running PHP as an Apache Module PHP man page describing how to use php.ini directives within .htaccess).
• Subdirectories written to by the application need world-write privilege. My ftp area is a public_html directory, so A2 uses mod_userdir. The home directory looks similar to my local workstation.
• ini_set(’session.cache_limiter’, ‘private’) works fine. Moved this parameter to a php_flag directive.
• A2 mentions PEAR support on their web page, but they don’t add the directory to the PHP include path and only offer a handful of modules. Will add modules per customer request, but what will happen if they move my account to a different server?

Web Hosting Requirements

• PHP Extensions: CURL, XMLWriter.
• Apache Modules: rewrite.

How shall I install PmWiki into our web application’s directory structure? I have source code, third-party library, and (user generated) data sub-directories. A wiki spans all three: it should be installed in the library area, the generated wiki pages should go into data, and pages that are to be kept long-term must be backed up. I don’t want a separate backup procedure, thus I wish to place the wiki content under Subversion source code control.

Wiki packages seem to prefer their default directory scheme, so I decided against trying to force PmWiki into my structure and instead will create a new top-level directory just for it. The plan is that only this pmwiki directory and the wiki.d and local sub-directories will be under subversion control. All other files and directories will be unversioned.

Initial installation of the PmWiki software:

1. mkdir -p pmwiki/{wiki.d,local}
2. svn add pmwiki
3. ln -s pmwiki{,-2.1.27} # symbolic link so that the versioned tar file is extracted into the controlled directory.
4. tar -xzf pmwiki.tgz
5. rm pmwiki-2.1.27
6. Edit local/config.php
7. Add the “rename” action to cookbook area, config.php (and optionally to Site/ EditForm).
8. Modify the logo and skins/themes.

Both developers and eventually end-users will be editing Help and Manual pages using the Wiki interface. This editing may occur on the development web server or the public system. I want to capture “approved” changes into subversion, which I believe can be done by committing the wiki.d directory periodically.

Once end-users start editing the public site, developers must either: a) make our edits on the public site also, or b) update our development server to match the public site, make our edits, commit the update, and re-publish to the public site. Option b must be done quickly or by disabling editing on the public site during the update.

Suppose we arrive at a point where there are numerous desired updates made by the user community, a few vandalized pages, plus other edits which we don’t want to adopt into the official documentation. Here’s a procedure:

1. Clone the public site’s wiki.d onto the development server.
2. On the development machine, restore or svn revert the vandalized pages.
3. Restore, modify, or svn revert the updates we wish to discard.
4. svn commit the wiki.d directory.
5. Check-out and publish the updated configuration to the public site.

I’ve always used smbmount (or mount.smbfs) when I need to mount a shared Windows folder, but when I tried it today, no luck:

mount: unknown filesystem type ’smbfs’

Figured I just needed to install the package, but YaST came up empty. Searching on the web, I find that SMB file system support is not compiled into openSUSE 10.2 and that people are using cifs (Common Internet Filesystem) instead. I tried it and it worked, but some people aren’t happy about the removal of smbfs.

mount -t cifs -o username=uuu,password=ppp //winHost/Shared\ Docs /mnt

The mount seems to work just like the smbfs mount. A cursory search turned up no reason why smbfs support was removed by openSUSE. Anyone know the reason?

Update: Upon receiving Nick’s comment (thanks, Nick), I searched for more details and discovered the Linux CIFS Client Guide on the Samba web site. It describes the differences between the Linux smbfs and cifs file system drivers, and continues with an in-depth discussion of cifs driver usage and optimization.

How do small groups do shared document development these days? We had a SharePoint site; it requires heavyweight applications (MS Office) on each client machine. We have Subversion for source code and are also using it for shared scheduling with GNU Project. I had a Writely account; it is a pure Web Service Application, but I sort of lost track of the account after Google bought them.

We are writing a browser-based app, so should be eating our own dogfood when possible during development. That says an HTML-based browser application such as Writely or a Wiki. Well, I decided to go with a Wiki, so now I need to select a Wiki Engine.

When I went searching, I found a Top Ten Wiki Engines page and the WikiMatrix. I put some criteria into the Matrix’s Choice Wizard and then compared its output to the Top Ten list. Here are my finalists:

• MediaWiki - Wikipedia’s engine. It’s the only wiki engine that openSUSE 10.2 includes in the distribution. I was about to go with it by default, but then saw that it is targeted for high-traffic sites.
• PhpWiki – “a WikiWikiWeb clone written in PHP. PhpWiki works right out of the box with zero configuration,” says the Matrix. Things were in some disarray at its SourceForge sites, however.
• TWiki – “a flexible, powerful, and easy to use enterprise collaboration platform. It is a Structured Wiki, typically used to run a project development space, a document management system, or any other groupware tool,” says the Matrix. I was impressed with their web site. TWiki is a high-traffic, high-powered system.
• PmWiki – another engine written in PHP. Their web page also looks nice and lists lots of current users which gives a good idea of what its capabilities are. Many of the sites are small installations.

TWiki and MediaWiki are high-horsepower systems for big sites and PhpWiki and PmWiki are more appropriate for small sites. I want minimal overhead and maintenance at this point, so I’ll start small and move up later.

I selected PhpWiki for my first attempt, swayed by the “runs out of the box” promise. It installed out-of-the-box, as I didn’t have to modify my Apache configuration files or install any other rpm packages. However it didn’t run out-of-the-box, and the configuration was rather tedious. It has a configurator, but it is apparently newer than that installation doc that I read. After a couple of hours playing with it, I decided thumbs-down on the out-of-the-box claim.

Next up, PmWiki. It did run out-of-the-tarball and essentially met all of my initial expectations. Installation configuration is straightforward, well documented, and confined to one file. All of the user-created pages go into the wiki.d sub-directory. Based on the painless installation and quick learning curve, PmWiki is my choice for our wiki engine.

The Amazon Web Services Simple Storage Service (AWS S3) agreement makes it clear that you the user are completely liable for any unauthorized use of your secret key. However, their PHP, Python, and Ruby code examples usually start off with “$secretKey = <insert your key here>” followed by a caution to “only use this example code on a secure server.”

I want to host a PHP application on a shared, remote server where I have only ftp access. I have to assume such a server is not secure, since any sysadmin at the hosting service can browse at will. I’m surely not the first person to host an S3 application on a remote server, yet there are only unanswered threads in the AWS forums regarding best security practices. So I’m guessing that some users are uploading the AWS secret keys and hoping for the best.

Here are the ways I’ve considered for using S3 from a Shared, Remote Web server:

1. Upload Secret Key – either in plain text or nominally concealed by an XOR hash or hex encoding. It would be concealment only, not encryption, since PHP is not compiled.
2. Encrypt and Upload – use another language (Java) or extension (Zend engine) that provides some security. AWS has some sample Java code that does this.
3. Use Public Buckets – from a secure machine, create a public_read_write bucket on S3. The PHP application on the shared server can then use anonymous access to read and write objects without requiring the secret key.
4. User Grantee – the AWS documentation states that access can be granted to anyone who has an account on amazon.com, even those without an AWS account. Obviously, such a person would not have a secret key. So the shared server application could use an amazon account to access a bucket with such a grant. However, the documentation does not explain how to sign a request with a canonical account name without a secret key.

I rejected options 2 & 4 as having too many unknowns at this stage of development. Adding a new language just to encrypt a key is a big hit in complexity. Similarly, with no cookbook example and the rather hastily assembled user documentation, I expect lots of trial and error for option 4. But once documented, option 4 would be my clear first choice.

Options 1 & 3 shift risk between a particular bucket and one’s secret key. With option 3, anyone who discovers the publicly read/writable bucket can use it immediately without even needing to find my PHP source code. But since they don’t have my secret key, I still have control over the bucket and can turn it off (delete it or make it private).

Option 1 exposes no public buckets on the web. But the worst case (someone gaining access to the remote server and stealing the secret key) has a big exposure. Only Amazon customer service can turn off a secret key, and they’re available only by email and only during business hours. If they take a week to shut off the key, I’d be liable for all of the charges.

My choice is option 3. I’m more comfortable with the risk profile being concentrated onto one bucket. Also, option 4 is a straighforward upgrade once more documentation becomes available. One unknown: is there any easy way for hackers to scan AWS S3 searching for public buckets? Let’s hope not.

I’ve been investigating the following code bundles to help me to get underway with the Amazon Web Services Simple Storage Service (AWS S3):

• Test Utility for Amazon S3 in PHP – from the AWS Developer Connection, it provides s3-test-utility-php.zip which contains: s3.php, index.php, and readme.html. Using the browser GUI provided by index.php to control s3.php, I was able to create a bucket and upload an object. Code is documented and lists the derivation history of the code (see Storage3 and Mission Data Blog below).
• Amazon S3 Sample in PHP – an alternative s3.php that has some common heritage with the Test Utility s3.php.
• Storage3 Project – the predecessor of the AWS Test Utility package, provides file Storage3.php plus the required PEAR modules (making it easy to install on a remote server where PEAR can’t be invoked).
• Mission Data Blog – presents the original putObjectStream() method that has now been modified to use a file in the AWS Test Utility.

I want to stream data directly to S3 (without first saving to a file) thus the current form of the AWS Test Utility package won’t work directly. I can either use the Storage3 project or substitute the Mission Data putObjectStream() method back into the AWS Test Utility s3 class.

At some point several years ago, SUSE and Redhat migrated to new multi-file and sub-directory approaches (search anacron) for their system crontabs. A helpful comment posted to my 2 March entry motivated me to learn how these are set up. Here’s how openSUSE 10.2 structures its cron configuration and provides the means of controlling it:

• YaST, System, sysconfig editor, cron – edits various configuration parameters stored in the /etc/sysconfig/cron file (this file can also be edited directly).
• YaST, System, System Services – turn the cron service on/off here.
• /etc/crontab, /etc/cron.d/*, /var/spool/cron/tab/<userid> – these files all use the format described in the crontab 5 man page. My as-installed configuration has one line in /etc/crontab that calls run-crons every 15 minutes (*/15). All of the other directories were empty. Also, the one line starts with a dash, a special feature for root that prevents sending a message to the syslog file.
• /usr/lib/cron/run-crons – shell script called by the default /etc/crontab entry. run-crons runs all pending shell scripts in the /etc/cron.* sub-directories. This is an extensive script containing the logic to handle systems that don’t run 24×7.
• /etc/cron.{hourly, daily, weekly, monthly} – these directories contain bash scripts, not crontab-format files.

As installed, /etc/crontab runs dailies within the first 15 minutes after rebooting and every 24 hours thereafter. If one has the misfortune to get that first run during work hours, cron will continue to run the dailies every day at that time! The daily tasks have a lot of system-wide finds (core files, locate, beagle, etc) that consume 100% of the cpu. This annoyed me frequently in my SUSE 10.0 days.

From reading the run-crons code, I found out about the DAILY_TIME parameter in /etc/sysconfig. If this is set, cron runs the daily tasks only during the hour specified. The trade-off is that the computer must be running at DAILY_TIME (but if not, cron will run daily tasks anyway after MAX_NOT_RUN days). The weekly and monthly tasks aren’t controlled by DAILY_TIME, so I guess if I reboot one minute after noon, weekly and monthly tasks will run at 12:15 pm thereafter.

To try all this out, I set DAILY_TIME to 14:00 and changed the /etc/crontab minutes parameter to 22,52, and will expect to see activity each day at 2:22 pm.

A nice extension would be a WORK_DAY parameter; a time span during which cron would avoid running any of the periodic tasks. This would retain cron’s ability to run tasks within the first 15 minutes after a reboot, so long as it didn’t occur during work hours.

Update, Dec 07: Have successfully removed ZENworks/zmd, solving all problems described herein. See post of 12-Dec-07.

My happiness with openSUSE 10.2 disappears when the Zen Updater runs. Every time it starts, does an update, restarts, etc, update-status (a compiled 64-bit executable, not a Perl or Python script) runs for 5-7 minutes with 99% cpu usage.

Others have complained about this problem, so it’s not just my installation. When I first wrote this, I had a couple of angry paragraphs about the ills of ZENworks and zmd. Since then, I located a more well-reasoned critique of ZENworks (complete with screen shots). It reviews openSUSE 10.1, but most everything still applies.

I started YaST, Software, Online Update Configuration. This turns out to be a canned task that accepts no user input and assigns you randomly to a SUSE update repository each time it runs. So I was moved from utah.edu to somewhere else (with a couple 6 minute update-status runs thrown in).

How shall I deal with ZENworks? First I read an article in SUSEWiki.org on Adding YaST Sources. This discussed how to use a local ISO file instead of the installation CD as the main package catalog, and pointed out a second, YaST update configuration screen: YaST, Software, Installation Source. This is the familiar YaST interface that allows one to configure all package and update repository catalogs.

A log of the various steps I tried (long and tedious, scroll down to the Conclusions):

• YaST, Software, Installation Source.
• Add, Media Type -> Local Directory.
• I checked the ISO image box, browsed to my new /iso directory where I had placed the 10.2 ISO image.
• YaST scanned the image and a new “Configured Software Catalog” appeared.
• To test, I turned the Status Source Setting to Off for the DVD, left the “Synchronize with ZENworks option turned on, and clicked Finish.
• A “synchronizing with ZENworks” info box appeared, so off to the kitchen for a snack during a few minutes of parse-metadata followed by another 6 minutes of update-status.
• Started YaST, Software, Online Update (called YOU in SUSE 10). It suggested an update to Firefox 1.5.0.9, which I new was wrong because openSUSE 10.2 is on 2.0. Where did the problem come from: using YOU, or the ZEN reconfig that switched repositories?
• Installation Source GUI confirms my ISO catalog and the osuosl.org update catalog are both 10.2. Software Management GUI shows the correct version of Firefox currently installed.
• Noticed in the ZEN Software Updater, Configure, that one of the catalogs I had deleted with YaST still appeared. I removed it here also, and the process took only a few seconds! Something is different with update-status.
• Lots of changes through two separate interfaces, so let’s reboot … update-status ran less than 2 min, but both zen-updater and YOU don’t present the correct pending updates.
• Tar file of /var/lib/rpm, then rpm –initdb; rpm –rebuilddb to see if rebuilding the RPM database would fix things. The dates on the DB files were no different upon completion.
• Hid the files, re-ran rpm, no new rpm databases appeared. Apparently rebuilddb is no longer functional. Replaced the files, now YaST Software Management looks normal again. ZEN’s database seems to be in /var/lib/zypp.
• Perhaps the ZEN Management Daemon (zmd) is responsible. Crossing my fingers, I ran Online Update Configuration again. update-status completed in less than a minute. ZEN failed, saying that it had updated YaST (ftp.ale.org this time) but could not update ZEN. Indeed ZEN still retains the old update repository.
• Started YOU, and the correct updates appear!
• Accepted all updates, and they installed correctly, even the Nokia one that had choked ZEN, starting this problem in the first place. Also, new dates have appeared on all of the /var/lib/rpm database files, so they are apparently okay.
• Read some Novell instructions and decided to add a third party repository packman.unixheads.com/suse/10.2 which now appears as a YUM repository. This triggered “synchronizing with ZENworks”; let’s see if the YUM update repository gets fixed.
• The ZEN updater is now showing the exclamation point. It wants to update a bunch of packages that I don’t have installed, packages from packman apparently. I guess it thinks this is the update repository. I turned off packman in the configure screen.
• update-status ran 3 minutes with packman added. So it appears that nothing was wrong with my system: ZEN is just cpu intensive.
• YaST, Software Management now shows ksensor, which I know is from packman. Installation was successful.
• YaST Online Updater is currently working fine with one update catalog and ZEN now presents only the local ISO catalog. Bad news. Options:
  • Keep re-running YaST Online Update Configuration until ZEN installs a catalog that both it and YaST accept. But I’m worried that I’ve broken ZEN somehow and it won’t accept any catalog.
  • Switch the panel updater applet from ZEN to YaST (an earlier post describes the controlling /etc/sysconfig option). Does this also turn off zmd?
  • Make no changes. See if the ZEN applet shows new updates. If not, periodically run YOU to check. Take further action in the event that YOU shows updates and ZEN does not.
• I don’t know what process updates the rpm database and since rpm –rebuilddb was shown to have no effect, I’d better make sure to make a copy of the database before doing anything risky.

Conclusions:

• ZEN processes (zmd, parse-metadata, update-status) are cpu intensive. With no progress bar or feedback of any kind in zen-updater to show concerned users what’s happening, it’s easy to think “infinite loop” when cpu times skyrocket. But it’s probably just a catalog that takes a long time to process.
• ZEN’s configuration of and interaction with YUM update catalogs is fragile. Running YaST, Online Update Configuration to assign a different update catalog is a good option to solve unexplained updater failures and error messages. Sometimes it must be run more than once. Consider it a ZEN reboot.

Current Status:

• Both YaST and ZEN have accepted the switch from the DVD to the local ISO file that I installed. So I won’t ever need to load the DVD again. Excellent – will add this step to my Linux upgrade checklist.
• I now better understand the repositories and DVDs. The retail package DVD (which is double layer) contains noarch, i586, and x86_64 packages for both oss and non-oss. My downloaded ISO DVD image is missing i586 packages. Online repositories have all architectures, but are split between oss and non-oss. I need a source for oss and non-oss i586 packages. Options:
  • Add two online i586 repositories. But update-status run times might return to 6 minutes.
  • Download the i586 DVD ISO image and install that locally.
• Re-ran YaST Online Update Configuration, ZEN installed mirrors.usc.edu successfully such that both zen-updater and opensuseupdater report the same updates.
• Switched the panel updater applet (/etc/sysconfig/sw_management) from the zlm (zen-updater) update manager to opensuse. The SUSE updater now appears (but only in KDE because the updater applet is part of KDE).
• My updating process now looks like it did in SUSE 10.0: the GUI that resembles the package installer with the detailed progress log above and the double progress bars at the bottom. I left the “sync with ZEN” option turned on.

Other information related to package updating:

• Offical openSUSE pages: Using the ZEN Updater, Package Repositories.
• From a LinuxQuestions.org post:
  

  – only common lib is libzypp. For example I deleted all zmd/mono/rug (with exception of single libzypp) and I am using Yast/smart only.
  - rpm db is common to all package managers otherwise eat time you would use different PM different installed/not installed packages would be shown.

• The SMART Package Updater was recommended by a forum post, but a detailed, multi-step installation process and co-existence with ZEN are concerns.
• The Package Management Category at SUSEWiki.org is the source for several of the above links. Other links on Finding RPMs and Package Database Management might be helpful.

Amazon S3 instructions require the following PEAR packages:HMAC, HTTP_Request, Net_Socket, and Net_URL. After my problems with PEAR in SuSE 10.0, I want good notes of what I did here:

• All commands run as root.
• pear remote-list
  WARNING: channel “pear.php.net” has updated its protocols, use “channel-update pear.php.net” to update.
• pear update-channels – successful.
• pear info <pkg> – always responded “No information found for ‘pkg’. I tried both installed packages and the packages I wanted to install.
• pear remote-list – listed available packages and Net_Socket was not listed, even though I could see it on the PEAR web site.
• pear install HTTP_Request -
  install ok: channel://pear.php.net/Net_Socket-1.0.6
  install ok: channel://pear.php.net/Net_URL-1.0.14
  install ok: channel://pear.php.net/HTTP_Request-1.4.0
• pear info HTTP_Request – now it lists information about this package. Something isn’t working; I certainly should be able to get info about a package before installing.
• pear list -
  Installed packages, channel pear.php.net:
Package Version State
Archive_Tar 1.3.1 stable
Console_Getopt 1.2 stable
Crypt_HMAC 1.0.1 stable
HTTP_Request 1.4.0 stable
Net_Socket 1.0.6 stable
Net_URL 1.0.14 stable
PEAR 1.4.11 stable
• This shows another problem. YaST offered a few PEAR packages at install time and I selected a few including PEAR::DB. The DB package appears in /usr/share/php5/PEAR along with Archive Tar and Console Getopt, yet is not listed above. Why not?
• pear run-tests -pr Crypt_HMAC – “running 0 tests”, even though HMAC has a test.php file under the test subdirectory. So I don’t understand this command either.