From 2438305142ef2365d8a79076d0e94f1b3e23f2af Mon Sep 17 00:00:00 2001 From: "N. Harrison Ripps" Date: Tue, 29 Jul 2014 15:20:00 -0400 Subject: [PATCH] Doc'ed distro handling and added distro support to builder --- Gemfile | 1 + README.adoc | 97 ++++++++++++++++++++++++----- Rakefile | 176 ++++++++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 240 insertions(+), 34 deletions(-) diff --git a/Gemfile b/Gemfile index 14a030ab236d..5c2c6fdd1b2a 100644 --- a/Gemfile +++ b/Gemfile @@ -1,6 +1,7 @@ source "https://rubygems.org" gem 'asciidoctor' +gem 'git' gem 'guard' gem 'guard-shell' gem 'guard-livereload' diff --git a/README.adoc b/README.adoc index a5549364ac4b..570d7635472c 100644 --- a/README.adoc +++ b/README.adoc @@ -8,15 +8,6 @@ This repo contains the documentation for The documentation is sourced in http://www.methods.co.nz/asciidoc/[AsciiDoc] and transformed into HTML/CSS and other formats through http://asciidoctor.org/[AsciiDoctor]-based automation. -== Branches - -Documentation for different OpenShift distributions is handled on different branches. - -* `master` - OpenShift Origin latest code -* `openshift-origin-release-N` - OpenShift Origin most recent stable release -* `openshift-online` - OpenShift Online most recent release -* `enterprise-N.N` - OpenShift Enterprise support releases - == Repo Organization Each directory of the repo represents a different collection of topics (you can think of directories as books). The exception is the `build_system` directory, which contains the code used to generate the finished documentation. Within each 'book' directory, topics exist as separate asciidoc files and an `images` directory contains any images that are included in the topics. @@ -33,27 +24,101 @@ Each directory of the repo represents a different collection of topics (you can /bookN ---- -== Metadata +== Version Management +The overlap of documentation across OpenShifts Origin, Online and Enterprise is no less than 80%. In many cases, this means that individual topics may need to include or exclude individual paragraphs with respect to a specific OpenShift distribution. While it is _possible_ to accomplish this solely by using git branches to maintain slightly different versions of a given topic, doing so would make the task of maintaining internal consistency extrememely difficult for content contributors. + +Git branching is still extremely valuable, and serves the important role of tracking the release versions of documentation for the various OpenShift distributions. + +=== Distribution-Specific Conditionals +OpenShift documentation uses AsciiDoc's `ifdef/endif` macro to conditionalize document segments for specific OpenShift distributions down to the single-line level. + +The supported distribution attributes used in the OpenShift document generator are: + +* `openshift-origin` +* `openshift-online` +* `openshift-enterprise` + +These attributes can be used alone or together to conditionalize text within a topic document. + +Here is an example of this concept in use: + +---- +This first line is unconditionalized, and will appear for all versions. + +\ifdef::openshift-online[] +This line will only appear for OpenShift Online. +\endif::[] + +\ifdef::openshift-enterprise[] +This line will only appear for OpenShift Enterprise. +\endif::[] + +\ifdef::openshift-origin,openshift-enterprise[] +This line will appear for OpenShift Origin and Enterprise, but not for OpenShift Online. +\endif::[] +---- + +Two important points to keep in mind: + +* The `ifdef/endif` blocks have no size limit, however they should _not_ be used to conditionalize an entire topic. If an entire topic file is specific to a given OpenShift distribution, refer to the link:#document-set-metadata[Document Set Metadata] section for information on how to conditionalize at the whole-topic level. + +* The `ifdef/endif` blocks _cannot be nested_. In other words, one conditional block cannot contain other conditional blocks. + +=== Release Branches +Through the use of link:#distribution-specific-conditionals[Distribution-Specific Conditionals] and link:#document-set-metadata[Document Set Metadata], the master branch of this repository always contains a complete set of documentation that includes all of the OpenShift distributions. However, when and as new versions of the OpenShift distros are released, the master branch is merged down to new or existing release branches. Here is the general naming scheme used in the branches: + +* `master` - OpenShift Origin latest code +* `origin-N.N` - OpenShift Origin most recent stable release +* `online` - OpenShift Online most recent release +* `enterprise-N.N` - OpenShift Enterprise support releases + +On a nightly basis, the documentation is rebuilt for each of these branches. In this manner, documentation for released versions of OpenShift will remain the same even as development continues on master. Additionally, any corrections or additions that are "cherry-picked" into the release branches will show up in the release documentation the next day. + +== Document Set Metadata +In order to construct the documentation site from these sources, the build system looks at the `_build_cfg.yml` metdata file. The build system _only_ looks in this file for information on which files to include, so any new file submissions must be accompanied by an update to this metadata file. -In order to construct the documentation site from these sources, the build system looks at the `_build_cfg.yml` metdata file. The format of this files is as indicated: +=== File Format +The format of this file is as indicated: ---- --- <1> Name: Origin of the Species <2> Dir: origin_of_the_species <3> +Distros: all <4> Topics: - - Name: The Majestic Marmoset <4> - File: the_majestic_marmoset <5> + - Name: The Majestic Marmoset <5> + File: the_majestic_marmoset <6> + Distros: all - Name: The Curious Crocodile File: the_curious_crocodile + Distros: openshift-online,openshift-enterprise <7> ---- <1> Record separator at the top of each topic group <2> Display name of topic group <3> Directory name of topic group -<4> Topic name -<5> Topic file under the topic group dir withour without '.adoc' +<4> Which OpenShift versions this topic group is part of +<5> Topic name +<6> Topic file under the topic group dir without '.adoc' +<7> Which OpenShift versions this topic is part of + +=== Notes on "Distros" + +* The "Distros" setting is optional for topic groups and topic items. When the "Distros" setting is absent, the system treats the topic group or topic as though the user had set "Distros: all". +* The "all" value for "Distros" is a synonym for "openshift-origin,openshift-enterprise,openshift-online". +* The "all" value trumps other values, so "openshift-online,all" is treated as "all" + +== Understanding the Complete Distribution Condition Chain +It is important to understand the ordering of distribution conditionals in determining whether or not a specific piece of content appears in the documentation set. The hierarchy is fairly straightforward: + +1. Topic group "Distros" setting from `_build_cfg.yml` +2. Topic item "Distros" setting from `_build_cfg.yml` +3. Document-level `ifdef/endif` blocks + +In this manner: + +* If a topic group is configured with "Distros: openshift-online", the entire group will be skipped for OpenShift Enterprise and OpenShift Origin, regardless of the Topic-level and document-level content rules within that group. -The build system only looks in this file for information on which files to include, so any new file submissions must be accompanied by an update to this metadata file. +* When a topic group is available to all Distros, but a specific topic item is limited, the topic group will appear for all distros and the specific topic item will only appear for the indicated distros. == Live Editing If you would like to work on one of the documentation files in an editing environment that automatically redraws the resulting HTML, follow these steps. diff --git a/Rakefile b/Rakefile index f038e93e2519..d390d534cc32 100644 --- a/Rakefile +++ b/Rakefile @@ -1,4 +1,6 @@ require 'asciidoctor' +require 'git' +require 'logger' require 'pandoc-ruby' require 'pathname' require 'rake' @@ -38,6 +40,27 @@ def package_dir end end +def git + @git ||= Git.open(source_dir) +end + +def git_checkout branch_name + target_branch = git.branches.local.select{ |b| b.name == branch_name }[0] + if not target_branch.current + target_branch.checkout + end +end + +# Returns the local git branches; current branch is always first +def local_branches + @local_branches ||= begin + branches = [] + branches << git.branches.local.select{ |b| b.current }[0].name + branches << git.branches.local.select{ |b| not b.current }.map{ |b| b.name } + branches.flatten + end +end + def build_config_file @build_config_file ||= File.join(source_dir,BUILD_FILENAME) end @@ -46,6 +69,63 @@ def build_config @build_config ||= validate_config(YAML.load_stream(open(build_config_file))) end +def distro_map + @distro_map ||= begin + { 'openshift-origin' => { + :name => 'OpenShift Origin', + :branches => { + 'master' => { + :name => 'Nightly Build', + :dir => 'latest', + }, + 'origin-4' => { + :name => 'Version 4', + :dir => 'stable', + }, + }, + }, + 'openshift-online' => { + :name => 'OpenShift Online', + :branches => { + 'online' => { + :name => 'Latest Release', + :dir => 'online', + }, + }, + }, + 'openshift-enterprise' => { + :name => 'OpenShift Enterprise', + :branches => { + 'enterprise-2.2' => { + :name => 'Version 2.2', + :dir => 'enterprise/v2.2', + }, + }, + } + } + end +end + +def distro_branches + @distro_branches ||= distro_map.map{ |distro,dconfig| dconfig[:branches].keys }.flatten +end + +def parse_distros distros_string, for_validation=false + values = distros_string.split(',').map{ |v| v.strip } + return values if for_validation + return distro_map.keys if values.include?('all') + return values.uniq +end + +def validate_distros distros_string + return false if not distros_string.is_a?(String) + values = parse_distros(distros_string, true) + values.each do |v| + return false if not v == 'all' or not distro_map.keys.include?(v) + end + return true +end + def validate_config config_data # Validate/normalize the config file straight away if not config_data.is_a?(Array) @@ -70,6 +150,16 @@ def validate_config config_data if not File.exists?(File.join(source_dir,topic_group['Dir'])) raise "In #{build_config_file}, the directory #{topic_group['Dir']} for topic group #{topic_group['Name']} does not exist under #{source_dir}" end + # Validate the Distros setting + if topic_group.has_key?('Distros') + if not validate_distros(topic_group['Distros']) + key_list = distro_map.keys.map{ |k| "'#{k.to_s}'" }.sort.join(', ') + raise "In #{build_config_file}, the Distros value #{topic_group['Distros'].inspect} for topic group #{topic_group['Name']} is not valid. Legal values are 'all', #{key_list}, or a comma-separated list of legal values." + end + topic_group['Distros'] = parse_distros(topic_group['Distros']) + else + topic_group['Distros'] = parse_distros('all') + end if not topic_group['Topics'].is_a?(Array) raise "The #{topic_group['Name']} topic group in #{build_config_file} is malformed; the build system is expecting an array of 'Topic' definitions." end @@ -90,17 +180,29 @@ def validate_config config_data if not File.exists?(File.join(source_dir,topic_group['Dir'],"#{topic['File']}.adoc")) raise "In #{build_config_file}, could not find file #{topic['File']} under directory #{topic_group['Dir']} for topic #{topic['Name']} in topic group #{topic_group['Name']}." end + if topic.has_key?('Distros') + if not validate_distros(topic['Distros']) + key_list = distro_map.keys.map{ |k| "'#{k.to_s}'" }.sort.join(', ') + raise "In #{build_config_file}, the Distros value #{topic_group['Distros'].inspect} for topic item #{topic['Name']} in topic group #{topic_group['Name']} is not valid. Legal values are 'all', #{key_list}, or a comma-separated list of legal values." + end + topic['Distros'] = parse_distros(topic['Distros']) + else + topic['Distros'] = parse_distros('all') + end end end config_data end -def nav_tree +def nav_tree distro @nav_tree ||= begin navigation = [] build_config.each do |topic_group| + next if not topic_group['Distros'].include?(distro) + next if topic_group['Topics'].select{ |t| t['Distros'].include?(distro) }.length == 0 topic_list = [] topic_group['Topics'].each do |topic| + next if not topic['Distros'].include?(distro) topic_list << ["#{topic_group['Dir']}/#{topic['File']}.html",topic['Name']] end navigation << { :title => topic_group['Name'], :topics => topic_list } @@ -110,25 +212,63 @@ def nav_tree end task :build do - # Copy stylesheets into preview area - system("cp -r _stylesheets #{preview_dir}/stylesheets") - # Build the topic files - build_config.each do |topic_group| - src_group_path = File.join(source_dir,topic_group['Dir']) - tgt_group_path = File.join(preview_dir,topic_group['Dir']) - if not File.exists?(tgt_group_path) - Dir.mkdir(tgt_group_path) + # First, notify the user of missing local branches + missing_branches = [] + distro_branches.sort.each do |dbranch| + next if local_branches.include?(dbranch) + missing_branches << dbranch + end + if missing_branches.length > 0 + puts "\nNOTE: The following branches do not exist in your local git repo:" + missing_branches.each do |mbranch| + puts "- #{mbranch}" end - #if File.exists?(File.join(src_group_path,'images')) - # system("cp -r #{src_group_path}/images #{tgt_group_path}") - #end - topic_group['Topics'].each do |topic| - src_file_path = File.join(src_group_path,"#{topic['File']}.adoc") - tgt_file_path = File.join(tgt_group_path,"#{topic['File']}.adoc") - system('cp', src_file_path, tgt_file_path) - Asciidoctor.render_file tgt_file_path, :in_place => true, :safe => :unsafe, :template_dir => template_dir, :attributes => ['source-highlighter=coderay','coderay-css=style',"stylesdir=#{preview_dir}/stylesheets","imagesdir=#{src_group_path}/images",'stylesheet=origin.css','linkcss!','icons=font','idprefix=','idseparator=-','sectanchors'] - system('rm', tgt_file_path) + puts "The build will proceed but these branches will not be generated." + end + distro_map.each do |distro,distro_config| + puts "\n\nBuilding #{distro_config[:name]}" + distro_config[:branches].each do |branch,branch_config| + if missing_branches.include?(branch) + puts "- skipping #{branch}" + next + end + puts "- building #{branch}" + + # Put us on the correct branch + git_checkout(branch) + + # Create the target dir + branch_path = "#{preview_dir}/#{branch_config[:dir]}" + system("mkdir -p #{branch_path}") + + # Copy stylesheets into preview area + system("cp -r _stylesheets #{branch_path}/stylesheets") + + # Build the nav tree + navigation = nav_tree(distro) + + # Build the topic files + build_config.each do |topic_group| + next if not topic_group['Distros'].include?(distro) + next if topic_group['Topics'].select{ |t| t['Distros'].include?(distro) }.length == 0 + src_group_path = File.join(source_dir,topic_group['Dir']) + tgt_group_path = File.join(branch_path,topic_group['Dir']) + if not File.exists?(tgt_group_path) + Dir.mkdir(tgt_group_path) + end + topic_group['Topics'].each do |topic| + next if not topic['Distros'].include?(distro) + src_file_path = File.join(src_group_path,"#{topic['File']}.adoc") + tgt_file_path = File.join(tgt_group_path,"#{topic['File']}.adoc") + system('cp', src_file_path, tgt_file_path) + Asciidoctor.render_file tgt_file_path, :in_place => true, :safe => :unsafe, :template_dir => template_dir, :attributes => ['source-highlighter=coderay','coderay-css=style',"stylesdir=#{branch_path}/stylesheets","imagesdir=#{src_group_path}/images",'stylesheet=origin.css','linkcss!','icons=font','idprefix=','idseparator=-','sectanchors', distro, "product-title=#{distro_config[:name]}", "product-version=#{branch_config[:name]}"] + system('rm', tgt_file_path) + end + end end + + # Return to the original branch + git_checkout(local_branches[0]) end end