Merge pull request #7 from nhr/distro_handling

Doc'ed distro handling and added distro support to builder

Author: nhr

Gemfile

@@ -1,6 +1,7 @@
 source "https://rubygems.org"
 
 gem 'asciidoctor'
+gem 'git'
 gem 'guard'
 gem 'guard-shell'
 gem 'guard-livereload'

README.adoc

@@ -8,15 +8,6 @@ This repo contains the documentation for
 
 The documentation is sourced in http://www.methods.co.nz/asciidoc/[AsciiDoc] and transformed into HTML/CSS and other formats through http://asciidoctor.org/[AsciiDoctor]-based automation.
 
-== Branches
-
-Documentation for different OpenShift distributions is handled on different branches.
-
-* `master` - OpenShift Origin latest code
-* `openshift-origin-release-N` - OpenShift Origin most recent stable release
-* `openshift-online` - OpenShift Online most recent release
-* `enterprise-N.N` - OpenShift Enterprise support releases
-
 == Repo Organization
 
 Each directory of the repo represents a different collection of topics (you can think of directories as books). The exception is the `build_system` directory, which contains the code used to generate the finished documentation. Within each 'book' directory, topics exist as separate asciidoc files and an `images` directory contains any images that are included in the topics.
@@ -33,27 +24,101 @@ Each directory of the repo represents a different collection of topics (you can
 /bookN
 ----
 
-== Metadata
+== Version Management
+The overlap of documentation across OpenShifts Origin, Online and Enterprise is no less than 80%. In many cases, this means that individual topics may need to include or exclude individual paragraphs with respect to a specific OpenShift distribution. While it is _possible_ to accomplish this solely by using git branches to maintain slightly different versions of a given topic, doing so would make the task of maintaining internal consistency extrememely difficult for content contributors.
+
+Git branching is still extremely valuable, and serves the important role of tracking the release versions of documentation for the various OpenShift distributions.
+
+=== Distribution-Specific Conditionals
+OpenShift documentation uses AsciiDoc's `ifdef/endif` macro to conditionalize document segments for specific OpenShift distributions down to the single-line level.
+
+The supported distribution attributes used in the OpenShift document generator are:
+
+* `openshift-origin`
+* `openshift-online`
+* `openshift-enterprise`
+
+These attributes can be used alone or together to conditionalize text within a topic document.
+
+Here is an example of this concept in use:
+
+----
+This first line is unconditionalized, and will appear for all versions.
+
+\ifdef::openshift-online[]
+This line will only appear for OpenShift Online.
+\endif::[]
+
+\ifdef::openshift-enterprise[]
+This line will only appear for OpenShift Enterprise.
+\endif::[]
+
+\ifdef::openshift-origin,openshift-enterprise[]
+This line will appear for OpenShift Origin and Enterprise, but not for OpenShift Online.
+\endif::[]
+----
+
+Two important points to keep in mind:
+
+* The `ifdef/endif` blocks have no size limit, however they should _not_ be used to conditionalize an entire topic. If an entire topic file is specific to a given OpenShift distribution, refer to the link:#document-set-metadata[Document Set Metadata] section for information on how to conditionalize at the whole-topic level.
+
+* The `ifdef/endif` blocks _cannot be nested_. In other words, one conditional block cannot contain other conditional blocks.
+
+=== Release Branches
+Through the use of link:#distribution-specific-conditionals[Distribution-Specific Conditionals] and link:#document-set-metadata[Document Set Metadata], the master branch of this repository always contains a complete set of documentation that includes all of the OpenShift distributions. However, when and as new versions of the OpenShift distros are released, the master branch is merged down to new or existing release branches. Here is the general naming scheme used in the branches:
+
+* `master` - OpenShift Origin latest code
+* `origin-N.N` - OpenShift Origin most recent stable release
+* `online` - OpenShift Online most recent release
+* `enterprise-N.N` - OpenShift Enterprise support releases
+
+On a nightly basis, the documentation is rebuilt for each of these branches. In this manner, documentation for released versions of OpenShift will remain the same even as development continues on master. Additionally, any corrections or additions that are "cherry-picked" into the release branches will show up in the release documentation the next day.
+
+== Document Set Metadata
+In order to construct the documentation site from these sources, the build system looks at the `_build_cfg.yml` metdata file. The build system _only_ looks in this file for information on which files to include, so any new file submissions must be accompanied by an update to this metadata file.
 
-In order to construct the documentation site from these sources, the build system looks at the `_build_cfg.yml` metdata file. The format of this files is as indicated:
+=== File Format
+The format of this file is as indicated:
 
 ----
 --- <1>
 Name: Origin of the Species <2>
 Dir:  origin_of_the_species <3>
+Distros: all <4>
 Topics:
-  - Name: The Majestic Marmoset <4>
-    File: the_majestic_marmoset <5>
+  - Name: The Majestic Marmoset <5>
+    File: the_majestic_marmoset <6>
+    Distros: all
   - Name: The Curious Crocodile
     File: the_curious_crocodile
+    Distros: openshift-online,openshift-enterprise <7>
 ----
 <1> Record separator at the top of each topic group
 <2> Display name of topic group
 <3> Directory name of topic group
-<4> Topic name
-<5> Topic file under the topic group dir withour without '.adoc'
+<4> Which OpenShift versions this topic group is part of
+<5> Topic name
+<6> Topic file under the topic group dir without '.adoc'
+<7> Which OpenShift versions this topic is part of
+
+=== Notes on "Distros"
+
+* The "Distros" setting is optional for topic groups and topic items. When the "Distros" setting is absent, the system treats the topic group or topic as though the user had set "Distros: all".
+* The "all" value for "Distros" is a synonym for "openshift-origin,openshift-enterprise,openshift-online".
+* The "all" value trumps other values, so "openshift-online,all" is treated as "all"
+
+== Understanding the Complete Distribution Condition Chain
+It is important to understand the ordering of distribution conditionals in determining whether or not a specific piece of content appears in the documentation set. The hierarchy is fairly straightforward:
+
+1. Topic group "Distros" setting from `_build_cfg.yml`
+2. Topic item "Distros" setting from `_build_cfg.yml`
+3. Document-level `ifdef/endif` blocks
+
+In this manner:
+
+* If a topic group is configured with "Distros: openshift-online", the entire group will be skipped for OpenShift Enterprise and OpenShift Origin, regardless of the Topic-level and document-level content rules within that group.
 
-The build system only looks in this file for information on which files to include, so any new file submissions must be accompanied by an update to this metadata file.
+* When a topic group is available to all Distros, but a specific topic item is limited, the topic group will appear for all distros and the specific topic item will only appear for the indicated distros.
 
 == Live Editing
 If you would like to work on one of the documentation files in an editing environment that automatically redraws the resulting HTML, follow these steps.

Rakefile

@@ -1,4 +1,6 @@
 require 'asciidoctor'
+require 'git'
+require 'logger'
 require 'pandoc-ruby'
 require 'pathname'
 require 'rake'
@@ -38,6 +40,27 @@ def package_dir
   end
 end
 
+def git
+  @git ||= Git.open(source_dir)
+end
+
+def git_checkout branch_name
+  target_branch = git.branches.local.select{ |b| b.name == branch_name }[0]
+  if not target_branch.current
+    target_branch.checkout
+  end
+end
+
+# Returns the local git branches; current branch is always first
+def local_branches
+  @local_branches ||= begin
+    branches = []
+    branches << git.branches.local.select{ |b| b.current }[0].name
+    branches << git.branches.local.select{ |b| not b.current }.map{ |b| b.name }
+    branches.flatten
+  end
+end
+
 def build_config_file
   @build_config_file ||= File.join(source_dir,BUILD_FILENAME)
 end
@@ -46,6 +69,63 @@ def build_config
   @build_config ||= validate_config(YAML.load_stream(open(build_config_file)))
 end
 
+def distro_map
+  @distro_map ||= begin
+    { 'openshift-origin' => {
+        :name     => 'OpenShift Origin',
+        :branches => {
+          'master'   => {
+            :name => 'Nightly Build',
+            :dir  => 'latest',
+          },
+          'origin-4' => {
+            :name => 'Version 4',
+            :dir  => 'stable',
+          },
+        },
+      },
+      'openshift-online' => {
+        :name     => 'OpenShift Online',
+        :branches => {
+          'online' => { 
+            :name => 'Latest Release',
+            :dir  => 'online',
+          },
+        },
+      },
+      'openshift-enterprise' => {
+        :name     => 'OpenShift Enterprise',
+        :branches => {
+          'enterprise-2.2' => {
+            :name => 'Version 2.2',
+            :dir  => 'enterprise/v2.2',
+          },
+        },
+      }
+    }
+  end
+end
+
+def distro_branches
+  @distro_branches ||= distro_map.map{ |distro,dconfig| dconfig[:branches].keys }.flatten
+end
+
+def parse_distros distros_string, for_validation=false
+  values = distros_string.split(',').map{ |v| v.strip }
+  return values if for_validation
+  return distro_map.keys if values.include?('all')
+  return values.uniq
+end
+
+def validate_distros distros_string
+  return false if not distros_string.is_a?(String)
+  values = parse_distros(distros_string, true)
+  values.each do |v|
+    return false if not v == 'all' or not distro_map.keys.include?(v)
+  end
+  return true
+end
+
 def validate_config config_data
   # Validate/normalize the config file straight away
   if not config_data.is_a?(Array)
@@ -70,6 +150,16 @@ def validate_config config_data
     if not File.exists?(File.join(source_dir,topic_group['Dir']))
       raise "In #{build_config_file}, the directory #{topic_group['Dir']} for topic group #{topic_group['Name']} does not exist under #{source_dir}"
     end
+    # Validate the Distros setting
+    if topic_group.has_key?('Distros')
+      if not validate_distros(topic_group['Distros'])
+        key_list = distro_map.keys.map{ |k| "'#{k.to_s}'" }.sort.join(', ')
+        raise "In #{build_config_file}, the Distros value #{topic_group['Distros'].inspect} for topic group #{topic_group['Name']} is not valid. Legal values are 'all', #{key_list}, or a comma-separated list of legal values."
+      end
+      topic_group['Distros'] = parse_distros(topic_group['Distros'])
+    else
+      topic_group['Distros'] = parse_distros('all')
+    end
     if not topic_group['Topics'].is_a?(Array)
       raise "The #{topic_group['Name']} topic group in #{build_config_file} is malformed; the build system is expecting an array of 'Topic' definitions."
     end
@@ -90,17 +180,29 @@ def validate_config config_data
       if not File.exists?(File.join(source_dir,topic_group['Dir'],"#{topic['File']}.adoc"))
         raise "In #{build_config_file}, could not find file #{topic['File']} under directory #{topic_group['Dir']} for topic #{topic['Name']} in topic group #{topic_group['Name']}."
       end
+      if topic.has_key?('Distros')
+        if not validate_distros(topic['Distros'])
+          key_list = distro_map.keys.map{ |k| "'#{k.to_s}'" }.sort.join(', ')
+          raise "In #{build_config_file}, the Distros value #{topic_group['Distros'].inspect} for topic item #{topic['Name']} in topic group #{topic_group['Name']} is not valid. Legal values are 'all', #{key_list}, or a comma-separated list of legal values."
+        end
+        topic['Distros'] = parse_distros(topic['Distros'])
+      else
+        topic['Distros'] = parse_distros('all')
+      end
     end
   end
   config_data
 end
 
-def nav_tree
+def nav_tree distro
   @nav_tree ||= begin
     navigation = []
     build_config.each do |topic_group|
+      next if not topic_group['Distros'].include?(distro)
+      next if topic_group['Topics'].select{ |t| t['Distros'].include?(distro) }.length == 0
       topic_list = []
       topic_group['Topics'].each do |topic|
+        next if not topic['Distros'].include?(distro)
         topic_list << ["#{topic_group['Dir']}/#{topic['File']}.html",topic['Name']]
       end
       navigation << { :title => topic_group['Name'], :topics => topic_list }
@@ -110,25 +212,63 @@ def nav_tree
 end
 
 task :build do
-  # Copy stylesheets into preview area
-  system("cp -r _stylesheets #{preview_dir}/stylesheets")
-  # Build the topic files
-  build_config.each do |topic_group|
-    src_group_path = File.join(source_dir,topic_group['Dir'])
-    tgt_group_path = File.join(preview_dir,topic_group['Dir'])
-    if not File.exists?(tgt_group_path)
-      Dir.mkdir(tgt_group_path)
+  # First, notify the user of missing local branches
+  missing_branches = []
+  distro_branches.sort.each do |dbranch|
+    next if local_branches.include?(dbranch)
+    missing_branches << dbranch
+  end
+  if missing_branches.length > 0
+    puts "\nNOTE: The following branches do not exist in your local git repo:"
+    missing_branches.each do |mbranch|
+      puts "- #{mbranch}"
     end
-    #if File.exists?(File.join(src_group_path,'images'))
-    #  system("cp -r #{src_group_path}/images #{tgt_group_path}")
-    #end
-    topic_group['Topics'].each do |topic|
-      src_file_path = File.join(src_group_path,"#{topic['File']}.adoc")
-      tgt_file_path = File.join(tgt_group_path,"#{topic['File']}.adoc")
-      system('cp', src_file_path, tgt_file_path)
-      Asciidoctor.render_file tgt_file_path, :in_place => true, :safe => :unsafe, :template_dir => template_dir, :attributes => ['source-highlighter=coderay','coderay-css=style',"stylesdir=#{preview_dir}/stylesheets","imagesdir=#{src_group_path}/images",'stylesheet=origin.css','linkcss!','icons=font','idprefix=','idseparator=-','sectanchors']
-      system('rm', tgt_file_path)
+    puts "The build will proceed but these branches will not be generated."
+  end
+  distro_map.each do |distro,distro_config|
+    puts "\n\nBuilding #{distro_config[:name]}"
+    distro_config[:branches].each do |branch,branch_config|
+      if missing_branches.include?(branch)
+        puts "- skipping #{branch}"
+        next
+      end
+      puts "- building #{branch}"
+
+      # Put us on the correct branch
+      git_checkout(branch)
+
+      # Create the target dir
+      branch_path = "#{preview_dir}/#{branch_config[:dir]}"
+      system("mkdir -p #{branch_path}")
+
+      # Copy stylesheets into preview area
+      system("cp -r _stylesheets #{branch_path}/stylesheets")
+
+      # Build the nav tree
+      navigation = nav_tree(distro)
+
+      # Build the topic files
+      build_config.each do |topic_group|
+        next if not topic_group['Distros'].include?(distro)
+        next if topic_group['Topics'].select{ |t| t['Distros'].include?(distro) }.length == 0
+        src_group_path = File.join(source_dir,topic_group['Dir'])
+        tgt_group_path = File.join(branch_path,topic_group['Dir'])
+        if not File.exists?(tgt_group_path)
+          Dir.mkdir(tgt_group_path)
+        end
+        topic_group['Topics'].each do |topic|
+          next if not topic['Distros'].include?(distro)
+          src_file_path = File.join(src_group_path,"#{topic['File']}.adoc")
+          tgt_file_path = File.join(tgt_group_path,"#{topic['File']}.adoc")
+          system('cp', src_file_path, tgt_file_path)
+          Asciidoctor.render_file tgt_file_path, :in_place => true, :safe => :unsafe, :template_dir => template_dir, :attributes => ['source-highlighter=coderay','coderay-css=style',"stylesdir=#{branch_path}/stylesheets","imagesdir=#{src_group_path}/images",'stylesheet=origin.css','linkcss!','icons=font','idprefix=','idseparator=-','sectanchors', distro, "product-title=#{distro_config[:name]}", "product-version=#{branch_config[:name]}"]
+          system('rm', tgt_file_path)
+        end
+      end
     end
+
+    # Return to the original branch
+    git_checkout(local_branches[0])
   end
 end