Changeset 1886

Timestamp:

02/02/07 20:00:11 (1 year ago)

Author:

moseley

Message:

Initial updates for moving website to new machine.
Just a few more tweaks, shell scripts and httpd.conf and
we can have a beer. No, wait, beer first.

Files:

• swish_website/README (modified) (2 diffs)
• swish_website/bin/build (modified) (5 diffs)
• swish_website/bin/index_hypermail.pl (added)
• swish_website/bin/swish-daily.pl (added)
• swish_website/build.sh (modified) (4 diffs)
• swish_website/etc/archive.conf (added)
• swish_website/lib/config/site (modified) (1 diff)

swish_website/README

@@ -1 +1 @@
-Mon Jan 17 07:32:40 PST 2005
-
-
+
-
-Swish-e site.
@@ -8 +7 @@
-Check out the site from cvs, and modify the pages locally.  Rebuild locally
-to review and then check in.
+
+Overview
+--------
+
+The website generation has a few dependencies.  There are scripts to generate
+the various parts.
+
+    - swish-e current release source
+        Used to build the swish-e docs.
+
+    - swish-e cvs checkout
+        Used to build the nightly build, and the docs from that.
+
+    - archive (hypermail)
+        currently, procmail feeds new posts to hypermail to
+        add new posts to the archive.  The posts are also archived
+        in the "mbox" directory by month in mbox format.
+
+    - archive index
+        When the archive is updated a flag is set and then cron
+        can re-index the hypermail archive.
+
+    - swish-daily
+        This is where daily tarballs are placed.  The website
+        creates an index page from the list of files
+
+    - swish-release
+        Similar to the swish-daily directory, but for releases
+
+
+
+Directory Structure
+-------------------
+
+    /<top level dir>
+
+        swish_daily_build/          - where the daily builds are created
+        swish-daily/                - daily tarballs
+
+        swish_release_build/        - where the release is build (for the docs)
+        distribution/               - releases (tarballs, .exe, old stuff)
+            old/                    - this is needed.
+
+        mbox/                       - mailbox archives by month
+        archive/                    - hypermail archive
+
+        indexes/                    - indexes for the archive and website
+
+        swish_website/              - source for building the site (more below)
+
+        pubilc_html                 - bulk of generated website.
+
+
+    Apache's DocumentRoot is pubilc_html, and swish-daily and distribution
+    are Aliased.  See httpd.conf.
+
+
+Notes on building the HTML docs
+-------------------------------
+
+    There's somewhat of a circular dependency.  Before the website can be
+    built the swish-daily needs to be generated.  This makes the pod docs
+    available for generating the documentation.
+
+    But, to build a daily tarball that include the HTML docs the website code
+    needs to be installed.
+
+    The swish-e configure script looks for the program "build-swish-docs" in the
+    path.  This is typically a symlink to the swish_website/bin/build program.
+    This program is used to build the HTML documentation that is placed in the tarball
+    generated by the swish-daily.pl program.
+
+
+
+Creating the above directories
+------------------------------
+
+    The following examples use $ROOT as the top-level directory above.
+    This might be /opt/swish, for example.
+
+
+    swish_website
+    -------------
+        The website isn't generated yet, but the code is needed before
+        building the daily tarballs.
+
+            cvn co http://svn.swish-e.org/swish_website/ $ROOT/swish_website
+            ln -s $ROOT/swish_website/bin/build /usr/local/bin/build-swish-docs
+
+        The actual site will be built later.
+
+
+
+
+    swish-daily_build / swish-daily
+    -------------------------------
+
+        Use the swish-daily.pl script to create the daily builds.
+        The script uses the cwd for the default "topdir" if not specified,
+        and also has the command to check out from trunk by default.
+
+        But, it's probably more explicit to pass in the build directory
+        by specifying "topdir".
+
+            $ROOT/swish_website/bin/swish-daily.pl \
+                --topdir=$ROOT/swish_daily_build \
+                --tardir=$ROOT/swish-daily \
+                || echo "Problem building swish-daily"
+
+
+        The script will exit non-zero if there's a problem and any errors
+        should be in the log files.  This should be run nightly.
+
+        swish_daily_build will be created if doesn't exist,
+        but the tardir directory must exist.
+
+            mkdir $ROOT/swish-daily
+
+
+        This does a fresh checkout.  Obviously, doing a svn update would
+        be much faster, but better to test a fresh checkout.
+
+        After all that there's also a script Make_Daily.sh
+        that does the above.
+
+
+
+    swish_release_build / distribution
+    ----------------------------------
+
+        This is only run when a new release is created.
+        But the current release source needs to be available
+        so the documentation can be generated on the website.
+
+        This is similar to the above, but the source is not fetched
+        via svn, rather from the current release tarball.
+        Also, assuming the tarball is already in the releases directory
+        there's no need to write it to the tardir.
+
+
+            $ROOT/swish_website/bin/swish-daily.pl \
+                --fetchtarurl=http://swish-e.org/distribution/latest.tar.gz \
+                --topdir=$ROOT/swish_release_build \
+                --notimestamp \
+                --noremove \
+                --tardir=$ROOT/distribution \
+                || echo "Problem building release"
+
+        There's also a Make_Release.sh script.  Review it before use.
+
+        running --help will show available options.
+
+
+        Note that this can be used to generate an actual release (by fetching
+        from svn instead), but it's better to generate the release separately
+        and test it.  In that case REMOVE the --tardir option above.
+
+
+
+
+    mbox / archive
+    --------------
+
+        Copy the existing mbox directory from where ever they currently are
+        located.
+
+        Generate the hypermail archive.  This needs to be done from the hypermail
+        directory since that's where the templates are located.
+
+        This step is only needed to create the initial archive.  The archive
+        is added to as new messages arrive (e.g. via procmail).
+
+            cd $ROOT/swish_website/hypermail
+            gzip -dc $ROOT/mbox/*.gz \
+                | hypermail \
+                    -i \
+                    -c hypermailrc \
+                    -d $ROOT/archive.new \
+            && mv $ROOT/archive.new $ROOT/archive
+
+
+        Note, would really be good dynamically display the archive
+        so that the headers and footers can be shared.
+
+
+    indexes
+    -------
+
+        This directory contains two indexes.  One for the archive and
+        another for the website.
+
+        The web site is not created yet, so it cannot be indexed.
+
+            mkdir $ROOT/indexes
+
+            $ROOT/swish_website/bin/index_hypermail.pl $ROOT/archive \
+                | swish-e \
+                    -S prog \
+                    -i stdin \
+                    -c $ROOT/swish_website/etc/archive.conf \
+                    -f $ROOT/indexes/archive.swish-e \
+                    -v0
+
+        This should be a cron task.  Normally set a flag when
+        the archive is updated (new message arrives).
+
+
+    public_html
+    -----------
+        Now the website can be created.  The bin/build script is used
+        to generate the website.  The script will detect when files
+        need to be generated, or passing --all will tell it to generate
+        the complete site.
+
+        There are a lot of options, and running build --help will list them.
+        The same build script is used when building from cvs to generate the
+        html docs for the tarball.
+
+        The site must be told where to find all the directories created above.
+        It determines where the source files are relative to the location of the
+        script, but must be told everything else:
+
+
+            $ROOT/swish_website/bin/build \
+                --dest=$ROOT/public_html \
+                --archive=$ROOT/indexes \
+                --swishsrc=$ROOT/swish_release_build/latest_swish_build/source \
+                --develsrc=$ROOT/swish_daily_build/latest_swish_build/source \
+                --download=$ROOT/distribution \
+                --daily=$ROOT/swish-daily \
+                --all
+
+
+        Luckily, there's a build script that does all that, and also indexes
+        the site.
+
+
+        indexes -- part 2
+        -----------------
+
+            Once the site is it's time to build the index of the web site.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
-

swish_website/bin/build

@@ -68 +68 @@
-pop @src_dirs;
-my $src_default     = File::Spec->catfile( @src_dirs );
-
-
+# 
+# 
-
-
-
-my %default_config = (
-
+.svn', '
-    copy    => [ '\.(pdf|gif|png|jpe?g|htaccess)$' ],
-    include => ['lib' ],
-    src     => $src_default,
-
-
+# 
+# 
-    websitesrc => 'src'  # default website source directory relative to -src
-                          # added to INCLUDE_PATH
@@ -141 +141 @@
-    -develsrc=<src> = Directory of development.
-
-
+
+
+
-
-    -- options you don't need --
@@ -178 +180 @@
-        develsrc=s
-        archive=s
+        download=s
+        daily=s
-        check
-);
@@ -199 +203 @@
-$SIG{__DIE__} = '';
-
+
+
+$config->{poddest} = File::Spec->catfile( $config->{dest}, 'docs' )
+    unless $config->{poddest};
-
-
@@ -327 +335 @@
-    if ( $config->archive ) {
-        my $index = File::Spec->rel2abs( $config->archive );
-normal file\n" unless -f
+directory\n" unless -d
-        $config->{archive} = $index;
-    }

swish_website/build.sh

@@ -2 +2 @@
-
-# Rebuild the site, and if any files are written reindex
+# This is expected to be run as a cron job
+# cd to the swish_website dir then run this script
+
+
+# Directory where everything happens
+WEB_ROOT="$(pwd)/.."
+
+
+# Path to the archive index file -- so search script can find the index.
+ARCHIVE_INDEX="$WEB_ROOT/swish/search/index.swish-e"
+
+# Path to swish-e source to find pods for building the docs
+SWISH_SRC="$WEB_ROOT/swish_src"
+
+# Path to the daily build soruce to build the dev docs
+DEV_SRC="$WEB_ROOT/devel_src"
+
-
-SWISH_SITE=http://swish-e.org
@@ -7 +24 @@
-export SWISH_SITE SPIDER_QUIET
-
-
+
+
+
-
-
-
-
+
+
+
+
-    echo
-    cat $TMP; rm $TMP
@@ -18 +39 @@
-
-# Look out for any local problems
+#    A  Added
+#    D  Deleted
+#    U  Updated
+#    C  Conflict
+#    G  Merged
-
-
+
+
+
+
-    echo "Some files need attention"
-    echo
@@ -25 +54 @@
-    rm  $TMP
-    exit
- 
+
-
-
-
+
+
+
+
-    echo "updating site"
-
-
-
-
+
+
+
+
+
+
-fi
-
+
+
-rm $TMP

swish_website/lib/config/site

@@ -11 +11 @@
-        topdir   = self.topdir
-        webroot  = self.src_dir
-
-
-
-
+
+
+
+
+
-
-   # URL defines (Badger book has this a separate template