Producing nine different versions of Elite from a central library repository
The article on generating websites from source code explains how I generate the Aviator, Revs and Lander websites from their corresponding source code repositories. But what about the Elite site?
In terms of the source code web pages that are common to all my disassembly sites, the same approach is used. Just as in the other sites, the mammoth create-disassembly-websites.py script takes each of the Elite source code repositories - one for each version of Elite - and generates HTML pages containing the source code, all marked-up and cross-referenced and ready to deploy to the website along with the static website content from the relevant part of the bbcelite-websites repository. There is one Elite-specific function call that supports the addition of a comparison link to the subroutine headers, but apart from that the process is the same.
There's an awful lot more going on with the Elite site, though, and that's what we're going to look at here.
What the scripts do
-------------------
As discussed in the overview, here's the flowchart for producing the Elite site:
+-- elite-source-code-library ---+ bbcelite-websites | | | | | | create-elite-repositories.py | | | | | | | | v | | elite-source-code-bbc-micro-cassette | | elite-source-code-bbc-micro-disc | | elite-source-code-acorn-electron | | elite-source-code-6502-second-processor | | elite-source-code-commodore-64 | | elite-source-code-apple-ii | | elite-source-code-bbc-master | | elite-source-code-nes | | elite-a-source-code-bbc-micro | | | | | | | | | | | create-disassembly-websites.py create-disassembly-websites.py | | | | | | | | | | +-------------|-- elite.bbcelite.com website --|-------------------|------+ | | | | | | | | v | | v | | | v Homepage | | Code pages About site | | Indexes Code comparisons Deep dives | | Statistics Hacks | | Version info | | Compare info | | | +-------------------------------------------------------------------------+
In the bottom-left corner you can see the same generation process that's used to produce the other sites - the create-disassembly-websites.py script ingests the source code from a repository, and generates the code pages, indexes and statistics pages, just like the Aviator, Revs and Lander sites. The only difference here is that there are nine source code repositories, one for each version of Elite, so for Elite we just run the script nine times, passing a different argument to the script on each run.
On the right side you can see the process of combining the generated content with the static content from the bbcelite-websites repository - that's the same as in the other sites, too, there's just more hand-crafted content for Elite. The idea is the same, though.
There are two new processes in this flowchart, though:
- In the top-left corner, the create-elite-repositories.py script generates the contents of the nine source code repositories from the contents of the elite-source-code-library repository. In the other sites the source code repositories are hand-crafted, but in Elite they are generated by script.
- In the middle column, the create-disassembly-websites.py script that we use to create the source code pages also produces the compare section of the Elite website, but for this process it ingests the content from the elite-source-code-library repository.
We're going to look at the first of these in this article, and you can read about the second in the article on generating code comparisons for Elite. But first let's take a look at the library repository, which is at the heart of the process that generates the Elite website and repositories.
Structure of the library repository
-----------------------------------
The library repository at elite-source-code-library contains the core content of the Elite website; as mentioned in the overview, it's the source for the 7,482 web pages that are generated by the two Python scripts. It is, therefore, a fairly sprawling affair.
On the other hand, it's a relatively simple repository to understand. The library repository is nothing more than a fully buildable BeebAsm-based repository, just like all the other repositories in my disassembly projects. It consists of source code files (*.asm) and a Makefile-based build process, and that's about it. The make command will build all nine versions of Elite, and you can pass parameters to the make command to change which variants are built, just like the source code repositories for the nine individual versions of Elite.
The structure of the library repository is a bit of a giveaway here. At the top level of the repository are two folders: versions and library. We'll talk about the library folder in a moment, but let's start with the versions folder. This contains one subfolder for each version of Elite, and the file structure in each subfolder mirrors that of the generated source code repository.
To explain this structure, let's pick one version - the 6502 Second Processor version. If you look at the generated repository for this version and compare it to the corresponding versions/6502sp folder in the library repository, then you'll notice that they have the same folder and file structure, the same README file, and so on. This is no coincidence - when we generate the source code repository for the 6502 Second Processor version, we use the versions/6502sp folder from the library repository as the source.
The big difference is that instead of containing huge source files with megabytes of text in them, the source files in the library repository are broken down into tiny elements that are composed using BeebAsm's include function, and that's where the library folder comes in. The library is a collection of over 2,600 relatively small hand-crafted source files, with each of them being a valid BeebAsm *.asm file.
The core concept is that there is one file for each element of the Elite source code, where elements are individual subroutines, variables, workspaces or macros. Those elements exist as individual files within the library folder structure, as follows:
- If the routine is shared between all the different versions of the game (i.e. cassette, disc, electron, 6502sp, c64, apple, master, nes and elite-a), then the source file lives in the library/common folder.
- If it's common only to the enhanced versions (i.e. disc, 6502sp, c64, apple, master, nes and elite-a) then it lives in the library/enhanced folder.
- If it's common only to the advanced versions (i.e. 6502sp, c64, apple, master and nes) then it lives in the library/advanced folder.
- If a routine is unique to a specific version, or its code has very little in common with the routine of the same name in the other versions, then it lives in the relevant version's library folder, i.e. library/cassette, library/disc, library/electron, library/6502sp, library/c64, library/apple, library/master, library/nes or library/elite-a.
So the library folder contains a few thousand small source files, and the versions folder contains each of the source code repositories, one for each version of Elite. The key to understanding the library repository is how these two folders work together.
The easiest way to understand how the library repository works is to compare the source files in the library repository with the same files in the generated repository. Again taking the 6502 Second Processor version as an example, take a look at the game's main source file, first in the generated repository for this version, and then in the versions/6502sp folder in the library repository. The generated source file is over 53,000 lines long, while the library source only contains 1,360 lines. But both files assemble the exact same binary, so how is this done?
It's because the library source file is made up of BeebAsm INCLUDE directives that load the relevant files from the library folder - lots of them. Essentially the library repository breaks the code down into its smallest parts, and then combines them using the INCLUDE directive. This allows us to include the same library files in multiple versions of Elite; as an example, you'll find that all versions of Elite construct the first part of the ship-drawing code like this:
INCLUDE "library/common/main/subroutine/shppt.asm" INCLUDE "library/common/main/subroutine/ll5.asm" INCLUDE "library/common/main/subroutine/ll28.asm" INCLUDE "library/common/main/subroutine/ll38.asm" INCLUDE "library/common/main/subroutine/ll51.asm" INCLUDE "library/common/main/subroutine/ll9_part_1_of_12.asm" INCLUDE "library/common/main/subroutine/ll9_part_2_of_12.asm"
This approach allows us to build the full games in the library repository, and it also acts as a structural database for each version - a database that the create-elite-repositories.py script can use to construct the generated repository.
Including library files will work if routines are identical across the different versions, but more often than not the code differs slightly. The final piece of the puzzle is to add conditional logic into each library file to assemble the correct code for each version. As the includes are still BeebAsm files, we can implement this easily enough using the assembler's IF, ELIF and ENDIF conditionals.
The library repository defines an integer build variable called _VERSION that defines which version is being built. Logic at the start of each source file in the versions folder converts this integer variable into a fixed set of Boolean variables that determine whether each individual version is being built. This is the logic:
_CASSETTE_VERSION = (_VERSION = 1) _DISC_VERSION = (_VERSION = 2) _6502SP_VERSION = (_VERSION = 3) _MASTER_VERSION = (_VERSION = 4) _ELECTRON_VERSION = (_VERSION = 5) _ELITE_A_VERSION = (_VERSION = 6) _NES_VERSION = (_VERSION = 7) _C64_VERSION = (_VERSION = 8) _APPLE_VERSION = (_VERSION = 9)
So if this is the disc version then _DISC_VERSION will be true and the others will be false, but if this is the NES version then _NES_VERSION will be the only one that's true. For some versions there are further variables that define the exact part of the code we're building, which allows us to support sharing of different library code within individual versions. For example, if we are building the docked code for the disc version, we would also define the following:
_DISC_DOCKED = TRUE _DISC_FLIGHT = FALSE
while in non-disc versions we would just define both variables as false (because otherwise BeebAsm will complain about undefined variables, even if they are never used).
These variables can then be used to determine which code is built for each version. For example, this code appears in part 2 of the BR1 routine in the library folder:
IF _CASSETTE_VERSION OR _ELECTRON_VERSION
LDA #147 \ Call TITLE to show a rotating Mamba (#3) and token
LDX #3 \ 147 ("PRESS FIRE OR SPACE,COMMANDER.{crlf}{crlf}"),
JSR TITLE \ returning with the internal number of the key pressed
\ in A
ELIF _DISC_DOCKED OR _ELITE_A_VERSION
LDA #7 \ Call TITLE to show a rotating Krait (#KRA) and token
LDX #KRA \ 7 ("PRESS SPACE OR FIRE,{single cap}COMMANDER.{cr}
JSR TITLE \ {cr}"), returning with the internal number of the key
\ pressed in A
ELIF _6502SP_VERSION
LDA #7 \ Call TITLE to show a rotating Asp Mk II (#ASP) and
LDX #ASP \ token 7 ("PRESS SPACE OR FIRE,{single cap}COMMANDER.
JSR TITLE \ {cr}{cr}"), returning with the internal number of the
\ key pressed in A
ELIF _MASTER_VERSION
LDA #7 \ Call TITLE to show a rotating Cougar (#COU) and token
LDX #COU \ 7 ("PRESS SPACE OR FIRE,{single cap}COMMANDER.{cr}
LDY #100 \ {cr}"), with the ship at a distance of 100, returning
JSR TITLE \ with the internal number of the key pressed in A
ELIF _C64_VERSION
LDA #7 \ Call TITLE to show a rotating Adder (#ADA) and token
LDX #ADA \ 7 ("PRESS SPACE OR FIRE,{single cap}COMMANDER.{cr}
LDY #48 \ {cr}"), with the ship at a distance of 48, returning
JSR TITLE \ with the internal number of the key pressed in A
ELIF _APPLE_VERSION
LDA #7 \ Call TITLE to show a rotating Sidewinder (#SH3) and
LDX #SH3 \ token 7 ("PRESS SPACE OR FIRE,{single cap}COMMANDER.
LDY #75 \ {cr}{cr}"), with the ship at a distance of 75,
JSR TITLE \ returning with the internal number of the key pressed
\ in A
ENDIF
This code is responsible for displaying the second ship when starting the game, which varies between versions: the BBC Micro cassette and Acorn Electron versions show a rotating Mamba, the BBC Micro disc version shows a rotating Krait, the 6502 Second Processor version shows a rotating Asp Mk II, the BBC Master version shows a rotating Cougar, the Commodore 64 version shows a rotating Adder, and the Apple II version shows a rotating Sidewinder. It assembles nothing for the NES version or the disc version's flight code, as the NES version doesn't show a second title screen, and the flight code doesn't show either title screen; as you can see in the above snippet, _NES_VERSION and _DISC_FLIGHT aren't mentioned at all.
Note that in the conditional statements that control which code is used in which versions, any ELIFs that solely contain the following versions must be the last ELIFs in the IF block:
ELIF _C64_VERSION ELIF _NES_VERSION ELIF _APPLE_VERSION ELIF _ELITE_A_*
In other words, non-Acornsoft ELIF blocks must come after any Acornsoft ELIF blocks. Also, within the IF statements themselves, if there are any Acornsoft variables in the list of conditionals, then the statement must start with an Acornsoft variable. So this is correct:
IF _DISC_VERSION OR _ELITE_A_VERSION OR _C64_VERSION
while this is incorrect:
IF _C64_VERSION OR _ELITE_A_VERSION OR _DISC_VERSION
because the Commodore 64 and Elite-A versions are not Acornsoft versions, but the disc version is, and the list of conditionals must start with an Acornsoft version where possible (i.e. the disc version in this case).
This ensures that the comparison section - which only compares Acornsoft versions - is correctly generated.
This logic not only controls the way the game binary is assembled, but it's also how we generate the source code repositories from the library, so let's talk about that next.
Generating source code from the library
---------------------------------------
Given that the library repository contains content for all the different versions - content that is marked-up with IF, ELIF and ENDIF conditionals that specify the version in which each bit of code should be assembled - it's probably no surprise to find that the create-elite-repositories.py script uses these conditionals when generating the nine source code repositories for Elite.
The approach is simple enough: to generate the repository for a specific version, the script works through the source files in the relevant versions subfolder, converting each INCLUDE directive to the contents of the included file, and processing any version-based IFs in the process. The result is the full source for that version, with no library INCLUDEs left, which contains only the code for that version.
The script also supports different comment delimiters and hexadecimal prefixes. This is used to generate the source code repositories for the Commodore 64, Apple II and NES, which use a semicolon for comments and $ instead of & for hexadecimal numbers. The source code in the library repository uses \ and & throughout, but for these repositories, the script is configured to convert \ into ; and & into $, to give a source code that is more in-keeping with the non-Acorn style of code. The corresponding sections of the Elite website are generated from these source code repositories, so they continues to use non-Acorn delimiters.
Although this approach of recursively expanding INCLUDE directives is a fairly simple process, the create-elite-repositories.py script does contain some pretty obscure code. This is because it also generates the source code repository for Elite-A, and Elite-A is a special case, so let's look at that next.
Generating mods for Elite-A
---------------------------
You can think of Elite-A as a fork of the BBC Micro disc version, as that's the version that Angus Duggan took and modified to create Elite-A, and I wanted the Elite-A repository and site to reflect this modding process.
As a result, the script treats Elite-A differently to the other versions. When the script comes across version-based IFs that include different code for Elite-A, it includes both the Elite-A code and the original disc version code that it replaces, if applicable. The disc code is included but is commented out, and the whole thing is wrapped in comments that show exactly what Angus changed to create Elite-A from the disc version. These are marked up as "mods" in the source code, and are referred to as "diffs" in the script.
You can see an example of this in part 2 of the main loop in the Elite-A source. This includes two blocks of code that were inserted for Elite-A, and a block of disc-version code at the end of the routine that was removed. The "Mod:" comments show exactly what each of the changes are, and the logic for this is encapsulated in the create-elite-repositories.py script. The code is rather impenetrable, but it helps to know that the this_version variable defines the current version that we are generating (i.e. Elite-A in this case), and the that_version variable refers to code that should be included but commented out (i.e. the disc version). Though these variable names are not optimal, I have to admit...
To ensure that the library code is processed correctly for Elite-A, there are a few rules to understand:
- As mentioned in the previous section, in the conditional statements that control which code is used in which versions, any ELIFs that solely match the _ELITE_A_* versions (or other non-Acornsoft versions) must be the last ELIFs in the IF block, and if an IF contains any Acornsoft variables, then that IF conditional must start with one of those Acornsoft variables.
- In the library source files for Elite-A, a commented-out INCLUDE directive that is commented out with a single \ character denotes a section that was removed by Angus when creating Elite-A:
\INCLUDE "library/disc/main/subroutine/deeor.asm"
- In the library source files for Elite-A, a commented-out INCLUDE directive that is commented out with double \\ characters denotes a section that was moved by Angus when creating Elite-A:
\\INCLUDE "library/enhanced/main/subroutine/detok3.asm"
If these rules are followed, then the script produces labelled mods for Elite-A. It also processes them to make them as succinct as possible within the source, so we can all see exactly what Angus did to create his masterpiece without distraction.
A deeper look at the script
---------------------------
Here's a summary of what the create-elite-repositories.py script does:
- Print "Generating source files" to the terminal.
- Call create_folder() to create the folders we need to hold the generated repositories.
- Work through each version of Elite in the library and call process_file() to process each source code file.
- If this is Elite-A, tidy the diff blocks by removing duplicated code, merging consecutive diffs and spacing diffs correctly, and by moving code from each end of a diff where that code matches the original (and is therefore not actually part of a diff).
- If configured, convert each line to use a different comment delimiter and hexadecimal prefix.
- Write the results into the folder we created above.
The rather convoluted Elite-A step in the middle ensures that the diffs in the source start and end with actual differences. This is necessary because of the way the markup in the library has to share the same source across all the other versions, not just Elite-A and the disc version, and without trimming the diff blocks, we would end up with a fair number of unnecessary lines in the diffs that don't actually represent differences.
Here's a call hierarchy of the above processes, which will help you orientate yourself if you want to look through the script. This is not a breakdown of each routine's actions, it's just a list of function usage in the script, so it's more of a map for your own investigations rather than a full explanation.
In the following, a + indicates a routine that is called from multiple places, while a - indicates this subroutine is only called once in the whole program.
| Routine | Details |
|---|---|
+ process_file() | Process each source file from the library, converting includes into the correct code for the version we are generating |
+ start_header() | Output a "Mod:" header for Elite-A to start a modification block |
+ end_header() | Output a "Mod:" header for Elite-A to end a modification block |
- should_we_show_this() | Work out if we should show the code in the current IF block, depending on the current version and the IF condition |
- process_line() | Process a line from a library source file, expanding any includes as required |
+ write_line() | Output a line of code, ensuring that multiple blank lines are reduced to single blank lines |
+ comment_out_line() | Insert a comment before a line of code |
- do_replacements() | Apply any configured replacements to a line of code |
+ write_line() | Output a line of code, ensuring that multiple blank lines are reduced to single blank lines |
+ comment_out_line() | Insert a comment before a line of code |
- do_replacements() | Apply any configured replacements to a line of code |
- tidy_diff_blocks() | Process diffs in a processed file using a buffer, removing duplicated code, merging consecutive diffs and spacing diffs correctly |
- remove_commentary() | Strip a trailing comment from a code line |
+ comment_out_line() | Insert a comment before a line of code |
- add_line_to_buffer() | Add a line of code to the buffer |
- shrink_diffs() | Remove lines from each end of a diff where those lines match the original (and therefore aren't actually different) |
- shrink_from_start() | Remove lines from the start of a diff where those lines match the original |
- move_lines_out_of_top() | Move lines out of the top of a diff block into the code above |
- code_style_6502() | Convert a line of code into 6502 style for the Commodore 64, Apple II and NES, as opposed to BBC Micro style |
To finish off, let's take a look at the shell script that joins everything in the flowchart together.
A deeper look at the Elite site generation process
--------------------------------------------------
The Elite site is updated by running the generate-elite.sh script. As with the other sites, if you want to have a go at running this process yourself, then the bbcelite-scripts repository contains step-by-step instructions on setting up and running the scripts yourself. The process has been built on a Mac, but it wouldn't take much effort to get it working on Linux or Windows.
This shell script is quite a bit bigger than the equivalent scripts for Aviator, Revs and Lander, but the structure is pretty similar - it just looks bigger because the Elite site is effectively nine websites rolled into one.
It starts by running the create-elite-repositories.py script to generate the nine source code repositories from the library repository (see the previous section for more details). It then syncs the results to each of the nine repositories themselves.
It then generates the nine source code sections in the website, by calling the create-disassembly-websites.py script nine times, each time with a different argument:
python3 create-disassembly-websites.py cassette python3 create-disassembly-websites.py disc python3 create-disassembly-websites.py electron python3 create-disassembly-websites.py 6502sp python3 create-disassembly-websites.py c64 python3 create-disassembly-websites.py apple python3 create-disassembly-websites.py master python3 create-disassembly-websites.py nes python3 create-disassembly-websites.py elite-a
(Note that the Apple generation script is still in progress.)
This generates the nine different source code sections of the Elite website, using the exact same process as for Aviator, Revs and Lander (see the article on generating websites from source code for details).
The shell script then runs the script once more, but this time with an argument of "compare":
python3 create-disassembly-websites.py compare
This generates the comparison section for the Elite site, and the results are then synced to the website folders. If you want to read more about the comparison process, see the article on generating code comparisons for Elite.