Jump to content

Sonic Adventure Autodemo/Restoration: Difference between revisions

From SA Docs
Speeps (talk | contribs)
It's a start.
 
mNo edit summary
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
(WIP)
''This page is dedicated to projects that restore content and functionality in the Autodemo prototype of Sonic Adventure (Dreamcast). For Sonic Adventure DX (PC) mods that implement Autodemo features, see [[SADX Autodemo Mods|this page]] instead.''


When the Sonic Adventure Autodemo was first dumped in 2013, it was quickly found that only some stages could be loaded in-game. While the non-functional stages could have their assets extracted using SA Tools and ported to the PC version, being able to play them in the Autodemo itself was desirable in order to see differences in the game's code, which couldn't be exported by the tools.
When the [[Sonic Adventure Autodemo]] was first dumped in 2013, it was soon discovered that it was a semi-functional build of the full game. The Autodemo was quickly hacked to unlock player controls and load stages manually so that they could be explored freely. However, many stages did not load because the binaries containing them were originally compiled for an older version of the game's main executable.


This restoration was achieved in 2024 through a collaboration of X-Hax users and symbol information acquired from the leaked Xbox 360 SADX prototype.
The Autodemo has been researched ever since it was dumped, and the non-functional stages were eventually extracted using [[SA Tools]] and ported to the PC version. However, being able to play them in the Autodemo itself was desirable in order to see differences in the game's code, which couldn't be exported by the tools.


==Background==
Initial work on restoring Autodemo functionality ingame was done back in 2015, when the user ''evilhamwizard'' posted their findings in the [https://forums.sonicretro.org/posts/846056/ Autodemo discussion thread on Sonic Retro]. The work focused primarily on restoring the original Windy Valley, and a number of GIFs of the level running on the emulator [https://forums.sonicretro.org/posts/867245/ were posted in 2016]. Some updates were posted on [https://forums.sonicretro.org/posts/937406/ booting Sky Deck] as well.
===Initial Research===
Early discussion/findings were posted on the Sonic Retro Forums.  


Only stages that match 1ST_READ's date (October 15, 1998) could load in-game. All others crash immediately due to containing older pointers to 1ST_READ functions and texture lists. Despite this, the pointers in 1ST_READ has to the stage files are still correct.
In 2024, based on ''evilhamwizard''<nowiki/>'s initial research, a new restoration project was started by [[x-hax]] (primarily by ''Speeps''), aiming to make other stages bootable, as well as improving the original Windy Valley restoration. This project is a collaboration between x-hax users, using symbol information acquired from the leaked Xbox 360 SADX prototype. The project is [https://github.com/X-Hax/dreamcast-mods available as a GitHub repository].


There are 3 build dates for stages:
==Background and Method==
The files included on the Autodemo disc come from different builds of the game that were at different stages of development. When the Autodemo was put together, only the stages that were meant to be shown in the demo were recompiled, and the other stages were left over from older builds (for most of these, textures and lighting files were also missing). We can tell which binaries were recompiled by looking at file dates on the Autodemo GD-ROM. Only the files dated October 15 work ingame as-is, and other files cause the game to crash. The crash happens because the functions in the game's updated main binary (1ST_READ.BIN) are at different memory addresses from those expected by the code in stage binaries which were compiled for the older version of the game.
{| class="wikitable mw-collapsible mw-collapsed"
!'''October 15, 1998 (working)'''
!'''October 7, 1998 (broken)'''
!'''September 24, 1998 (broken)'''
|-
|STG01 (Emerald Coast)
|AL_GARDEN01 (Egg Carrier/Beta Windy Valley Chao Garden)
|STG00 ("Practice" Stages)
|-
|STG04 (Speed Highway)
|AL_GARDEN02 (Mystic Ruins Chao Garden)
|STG02 (Windy Valley)
|-
|STG05 (Red Mountain)
|AL_RACE (Chao Race)
|STG03 (Twinkle Park)
|-
|STG08 (Ice Cap)
|
|STG06 (Sky Deck)
|-
|STG10 (Final Egg)
|
|STG07 (Lost World)
|-
|ADV00 (Station Square)
|
|STG09 (Casinopolis)
|-
|ADV00OBJ (Station Square objects)
|
|ADV02 (Mystic Ruins)
|-
|ADV0100 (Egg Carrier Outside)
|
|ADV02OBJ (Mystic Ruins objects)
|-
|ADV0130 (Egg Carries Inside)
|
|SHOOTING (Sky Chase)
|-
|ADV01OBJ (Egg Carrier objects)
|
|MINICART (Twinkle Circuit)
|-
|AL_GARDEN00 (Station Square Chao Garden)
|
|B_E101_R (E-101 MK2)
|-
|AL_MAIN (Chao)
|
|B_ROBO (ZERO)
|-
|B_CHAOS2 (Chaos 2)
|
|B_EGM1 (Egg Hornet)
|-
|
|
|B_EGM3 (Egg Viper)
|-
|
|
|SBOARD (Sand Hill)
|-
|
|
|B_CHAOS0 (Chaos 0)
|-
|
|
|B_CHAOS4 (Chaos 4)
|-
|
|
|B_CHAOS6 (Chaos 6)
|-
|
|
|B_CHAOS7 (Perfect Chaos)
|}
However, as originally demonstrated by ''evilhamwizard'', it’s possible to find the functions and data they were meant to point to by comparing the pointers to known functions and data between these files and the files that boot. In 2024, ''Speeps'' started putting together a new [https://github.com/X-Hax/dreamcast-mods/raw/refs/heads/main/AutoDemoPointers.xlsx spreadsheet] of known broken pointers and their correct counterparts. As the pointer list grew, more levels became functional when known pointers were replaced.


* October 15, 1998: Emerald Coast, Speed Highway, Red Mountain, Icecap, Final Egg, Chaos 2, Station Square, Egg Carrier, SS Chao Garden.
The exact pointer matching procedure involved in stage restoration is as follows:
* October 7, 1998: EC Chao Garden, MR Chao Garden.
#Get the address of all pointers that read the stage's file from 1ST_READ. These are the Round Manager, Background Manager, Object List and Path List.
* September 24, 1998: Everything else.
# Using IDA's graph view, gather all pointers to 1ST_READ (8Cxxxxxx).
# Using Ghidra, identify an object (using its name string) and check if the same code exists in SADX, where its symbol information can be taken from the 360 symbols.
# For all references to broken pointers in the object, attempt to find a working reference in a functional AutoDemo stage, such as Speed Highway.
# Replace the broken pointer with the working one.
All pointers were added to a [https://github.com/X-Hax/dreamcast-mods/raw/refs/heads/main/AutoDemoPointers.xlsx spreadsheet] along with their fixed version.
 
For any code that didn't have a matching reference in the PC version or a functional Autodemo stage file, such as code for objects that didn't exist in the final game, the original byte difference method was used instead. Using Ghidra, the resulting pointer was checked to ensure it led to an appropriate address (The beginning of a function etc.). Any pointers "fixed" with this method were marked in the table as unsure.


Every map (except MR Jungle, which had a geometry bug) could still be viewed and extracted by SA Tools, so a lot of them (Early WV) were quickly ported to PC.  
Stages that were similar enough to their final counterparts to share a lot of code (Twinkle Park, Lost World) were repaired first to build the database of known pointers, as stages with larger changes (Windy Valley) consisted almost entirely of code that didn't exist in the final game.


The first attempt at fixing the broken stages was done by evilhamwizard in 2015. He got Windy Valley 3 to load as well as a few objects by comparing the pointers in E103 and E104 (Red Mountain, a working stage), which are almost identical objects:
Early hacks by ''Speeps'' were using simple patches to the executables applied through a hex editor. To facilitate mass pointer replacement and game testing, ''PkR'' reworked his SA1-DC-HD tool (originally meant to build GDIs of the final version of SA1 with enhancements) and created the [[Dreamcast Image Builder]]. The image builder uses a "mods" system that makes it easier to replace pointers, do all sorts of patching of game binaries and build a modified game image containing all of the selected hacks. In addition, some other tools on the [https://github.com/X-Hax/sa_tools_research SA Tools Research] repository were repurposed to add missing textures to the Autodemo by matching the names in the Autodemo’s texture lists against textures in the final version, the E3 build, the Autodemo Windy Valley mod for the PC version (where missing textures were recreated by its developers) and Dreamcast SDKs (which surprisingly have some early SA1 textures). Apart from making stages bootable and readding textures, the mods include various experiments with controls, framerate, lighting, skipping the intro movie, restoring debug functionality etc.


<pre>(TODO: add his gifs)</pre>
The restoration was primarily discussed in a series of threads on x-hax Discord, which have been archived below:  


At the time there wasn't enough known about the game to get the rest of the correct pointers and it was only possible to go so far with one object as a reference, so gathering info about these broken maps was left to extracting the data with external tools.
[https://dcmods.unreliable.network/owncloud/data/PiKeyAr/files/SA1%20research/autodemo/threads/sa1-modding%20-%20New%20Autodemo%20research.html Thread 1: New Autodemo Research (May 10, 2024)]


===2015-2024===
[https://dcmods.unreliable.network/owncloud/data/PiKeyAr/files/SA1%20research/autodemo/threads/sa1-modding%20-%20Autodemo%20research.html Thread 2: Autodemo Research (Private Version, May 10 - June 18, 2024)]
A few notable things were discovered in the following years:


* In 2018, the MR Jungle map was finally extracted (Thanks to ItsEasyActually) - it was the version from the Tokyo International Announcement video, with lots of paths that were removed.
[https://dcmods.unreliable.network/owncloud/data/PiKeyAr/files/SA1%20research/autodemo/threads/game-research%20-%20Sonic%20Adventure%20Autodemo.html Thread 3: Game Research: Sonic Adventure Autodemo (Public Version, June 7, 2024 - February 28, 2026)]
* In 2020, PkR discovered the pirate ship for Twinkle Park had a different model:
* In 2024, woofmute learned how to skip the intro video, found part of the object editor (Also seen in SADX:Preview and the SA2 May 7 build) and an early results tally.


It turned out Practice 1 could load by just removing its SET File (Object Layout) because it only references 1ST_READ through that.
== Restoration Details ==


===Restoration Begins===
=== Twinkle Park ===
In 2024, Speeps posted that he had loaded Twinkle Park's map without crashing, after removing all the ways running STG03 would encounter a pointer to 1ST_READ. This was partly thanks to ehw's original research, which provided the address of the Round Manager pointer list in 1ST_READ.
Twinkle Park was the first stage restored by ''Speeps''.


<pre>(TODO: Maybe I should write a tutorial instead of this wordy list lol -Speeps)</pre>
=== Stage Select ===
The Stage Select screen is similar to the final version. ''woofmute'' and ''PkR'' found out that the texture error at the bottom right was caused by the game trying to load character pictures, which were added back afterwards.<gallery mode="nolines">
File:Sonic Adventure Autodemo Restoration 1772422970977.png|Autodemo Stage Select with the texture error (''MainJP'')
File:Sonic Adventure Autodemo Restoration 1772422993825.png|Autodemo Stage Select with character pictures (''woofmute'')
</gallery>


The broken pointers were fixed with the following method:
=== Debug Functionality ===
# Get the address of all pointers that read the stage's file from 1ST_READ. These are the Round Manager, Background Manager, Object List and Path List.
The SET Editor was restored by ''woofmute''.
# Using IDA's graph view, gather all pointers to 1ST_READ (8Cxxxxxx).
 
# Using Ghidra, identify an object (using its name string) and check if the same code exists in SADX, where its symbol information can be taken from the 360 symbols.
=== Ninja Lights ===
# For all references to broken pointers in the object, attempt to find a working reference in a functional AutoDemo stage, such as Speed Highway.
 
# Replace the broken pointer with the working one.
=== Practice Stages ===
All pointers were added to a spreadsheet along with their fixed version.
''Speeps'': It turned out Practice 1 could load by just removing its SET File (Object Layout) because it only references 1ST_READ through that.
 
''PkR'': Practice 2 did not load because one of the meshes is too big for the game's [[Sonic Adventure/Lantern Engine|LANTERN lighting system]] to handle. However, it boots if [[Ninja Lights]] are used instead.
{| class="wikitable"
|+Autodemo Practice Stages
![[File:Sonic Adventure Autodemo Restoration 1772347046976.mp4|none|thumb|Gamma in Practice Stage (''Speeps'')]]
![[File:Pinball mode practice.mp4|thumb|Pinball mode in Practice Stage (''MainJP'')|none]]
|}
<gallery mode="nolines" caption="Practice Stages with Ninja Lights (images by ''MainJP'')">
File:Practice.webp|alt=
File:Sonic Adventure Autodemo Restoration 1772203956319.png
File:Screenshot 2024-08-30 15-59-50.webp
File:Screenshot 2024-08-30 16-15-14.webp
</gallery>
 
=== Windy Valley ===
 
=== Sky Deck ===
 
=== Lost World ===


For any code that didn't have a matching reference in the PC version or a functional Autodemo stage file, such as code for objects that didn't exist in the final game, the original byte difference method was used instead. Using Ghidra, the resulting pointer was checked to ensure it led to an appropriate address (The beginning of a function etc.). Any pointers "fixed" with this method were marked in the table as unsure.
=== Casinopolis ===


Stages that were similar enough to their final counterparts to share a lot of code (Twinkle Park, Lost World) were repaired first to build the database of known pointers, as stages with larger changes (Windy Valley) consisted almost entirely of code that didn't exist in the final game.
=== Station Square ===


The restoration was primarily discussed in a Discord thread, which this page will attempt to archive. <pre>(TODO)</pre>
=== Mystic Ruins ===

Latest revision as of 04:51, 2 March 2026

This page is dedicated to projects that restore content and functionality in the Autodemo prototype of Sonic Adventure (Dreamcast). For Sonic Adventure DX (PC) mods that implement Autodemo features, see this page instead.

When the Sonic Adventure Autodemo was first dumped in 2013, it was soon discovered that it was a semi-functional build of the full game. The Autodemo was quickly hacked to unlock player controls and load stages manually so that they could be explored freely. However, many stages did not load because the binaries containing them were originally compiled for an older version of the game's main executable.

The Autodemo has been researched ever since it was dumped, and the non-functional stages were eventually extracted using SA Tools and ported to the PC version. However, being able to play them in the Autodemo itself was desirable in order to see differences in the game's code, which couldn't be exported by the tools.

Initial work on restoring Autodemo functionality ingame was done back in 2015, when the user evilhamwizard posted their findings in the Autodemo discussion thread on Sonic Retro. The work focused primarily on restoring the original Windy Valley, and a number of GIFs of the level running on the emulator were posted in 2016. Some updates were posted on booting Sky Deck as well.

In 2024, based on evilhamwizard's initial research, a new restoration project was started by x-hax (primarily by Speeps), aiming to make other stages bootable, as well as improving the original Windy Valley restoration. This project is a collaboration between x-hax users, using symbol information acquired from the leaked Xbox 360 SADX prototype. The project is available as a GitHub repository.

Background and Method

The files included on the Autodemo disc come from different builds of the game that were at different stages of development. When the Autodemo was put together, only the stages that were meant to be shown in the demo were recompiled, and the other stages were left over from older builds (for most of these, textures and lighting files were also missing). We can tell which binaries were recompiled by looking at file dates on the Autodemo GD-ROM. Only the files dated October 15 work ingame as-is, and other files cause the game to crash. The crash happens because the functions in the game's updated main binary (1ST_READ.BIN) are at different memory addresses from those expected by the code in stage binaries which were compiled for the older version of the game.

October 15, 1998 (working) October 7, 1998 (broken) September 24, 1998 (broken)
STG01 (Emerald Coast) AL_GARDEN01 (Egg Carrier/Beta Windy Valley Chao Garden) STG00 ("Practice" Stages)
STG04 (Speed Highway) AL_GARDEN02 (Mystic Ruins Chao Garden) STG02 (Windy Valley)
STG05 (Red Mountain) AL_RACE (Chao Race) STG03 (Twinkle Park)
STG08 (Ice Cap) STG06 (Sky Deck)
STG10 (Final Egg) STG07 (Lost World)
ADV00 (Station Square) STG09 (Casinopolis)
ADV00OBJ (Station Square objects) ADV02 (Mystic Ruins)
ADV0100 (Egg Carrier Outside) ADV02OBJ (Mystic Ruins objects)
ADV0130 (Egg Carries Inside) SHOOTING (Sky Chase)
ADV01OBJ (Egg Carrier objects) MINICART (Twinkle Circuit)
AL_GARDEN00 (Station Square Chao Garden) B_E101_R (E-101 MK2)
AL_MAIN (Chao) B_ROBO (ZERO)
B_CHAOS2 (Chaos 2) B_EGM1 (Egg Hornet)
B_EGM3 (Egg Viper)
SBOARD (Sand Hill)
B_CHAOS0 (Chaos 0)
B_CHAOS4 (Chaos 4)
B_CHAOS6 (Chaos 6)
B_CHAOS7 (Perfect Chaos)

However, as originally demonstrated by evilhamwizard, it’s possible to find the functions and data they were meant to point to by comparing the pointers to known functions and data between these files and the files that boot. In 2024, Speeps started putting together a new spreadsheet of known broken pointers and their correct counterparts. As the pointer list grew, more levels became functional when known pointers were replaced.

The exact pointer matching procedure involved in stage restoration is as follows:

  1. Get the address of all pointers that read the stage's file from 1ST_READ. These are the Round Manager, Background Manager, Object List and Path List.
  2. Using IDA's graph view, gather all pointers to 1ST_READ (8Cxxxxxx).
  3. Using Ghidra, identify an object (using its name string) and check if the same code exists in SADX, where its symbol information can be taken from the 360 symbols.
  4. For all references to broken pointers in the object, attempt to find a working reference in a functional AutoDemo stage, such as Speed Highway.
  5. Replace the broken pointer with the working one.

All pointers were added to a spreadsheet along with their fixed version.

For any code that didn't have a matching reference in the PC version or a functional Autodemo stage file, such as code for objects that didn't exist in the final game, the original byte difference method was used instead. Using Ghidra, the resulting pointer was checked to ensure it led to an appropriate address (The beginning of a function etc.). Any pointers "fixed" with this method were marked in the table as unsure.

Stages that were similar enough to their final counterparts to share a lot of code (Twinkle Park, Lost World) were repaired first to build the database of known pointers, as stages with larger changes (Windy Valley) consisted almost entirely of code that didn't exist in the final game.

Early hacks by Speeps were using simple patches to the executables applied through a hex editor. To facilitate mass pointer replacement and game testing, PkR reworked his SA1-DC-HD tool (originally meant to build GDIs of the final version of SA1 with enhancements) and created the Dreamcast Image Builder. The image builder uses a "mods" system that makes it easier to replace pointers, do all sorts of patching of game binaries and build a modified game image containing all of the selected hacks. In addition, some other tools on the SA Tools Research repository were repurposed to add missing textures to the Autodemo by matching the names in the Autodemo’s texture lists against textures in the final version, the E3 build, the Autodemo Windy Valley mod for the PC version (where missing textures were recreated by its developers) and Dreamcast SDKs (which surprisingly have some early SA1 textures). Apart from making stages bootable and readding textures, the mods include various experiments with controls, framerate, lighting, skipping the intro movie, restoring debug functionality etc.

The restoration was primarily discussed in a series of threads on x-hax Discord, which have been archived below:

Thread 1: New Autodemo Research (May 10, 2024)

Thread 2: Autodemo Research (Private Version, May 10 - June 18, 2024)

Thread 3: Game Research: Sonic Adventure Autodemo (Public Version, June 7, 2024 - February 28, 2026)

Restoration Details

Twinkle Park

Twinkle Park was the first stage restored by Speeps.

Stage Select

The Stage Select screen is similar to the final version. woofmute and PkR found out that the texture error at the bottom right was caused by the game trying to load character pictures, which were added back afterwards.

Debug Functionality

The SET Editor was restored by woofmute.

Ninja Lights

Practice Stages

Speeps: It turned out Practice 1 could load by just removing its SET File (Object Layout) because it only references 1ST_READ through that.

PkR: Practice 2 did not load because one of the meshes is too big for the game's LANTERN lighting system to handle. However, it boots if Ninja Lights are used instead.

Autodemo Practice Stages
Gamma in Practice Stage (Speeps)
Pinball mode in Practice Stage (MainJP)

Windy Valley

Sky Deck

Lost World

Casinopolis

Station Square

Mystic Ruins