Fixing VPK mounting for older Source SDK Bases

From Valve Developer Community
Jump to navigation Jump to search

Background

When Valve rolled out the SteamPipe update and the existing titles were updated to use the VPK file format, Source SDK BaseSource 2006 and Source SDK BaseSource 2007 were supplied with "depot" VPKs, but were left unable to properly mount them. Old mods using the Bases were often broken, and 2007 Base mods were unable to reference Half-Life 2: Episode Two content, unless it was directly copied into the mod's folder; this is despite the fact that the depot VPKs included those files (but were switched around; see below).

Apparently, this wasn't meant to happen, and both SDK Base 2006 and SDK Base 2007 can mount their VPKs with a specific fix.

Overview

The gmod9.com blog details an extensive research into this issue and provides a working solution.

It explains that the Steam2 wrapper, meant to handle VPK mounting for the old games, requires a missing file, steam_vpks.vdf, which is not shipped in any known product and whose structure needed to be reverse engineered.

Elaboration on the Steam2 wrapper 
The following text is copied from the original blog.
The Steam2 wrapper is a steam.dll replacement that acts kind-of as a proxy between the new post-Steampipe Steam and the game. It's the thing that handles the VPK mounting and makes the VGUI friends dialog still work. Note that the version of Steam2 wrapper inside the main Steam directory isn't the one that the old Source games use, instead they come with an old build of it dated Wed Feb 26 2014. The Steam2 wrapper uses an internal Valve library called vpklib to open VPK files and provide it to the game's virtual file system, the library is also used in newer Source Engine builds for the filesystem module and vpk.exe.

There are three known methods to fix it, starting from the easiest one:

Creating custom steam_vpks.vdf

The file in question needs to list relative paths to the depot VPKs.

Create it in the Base's root (e. g. 🖿<steam_dir>/steamapps/common/Source SDK Base/ for '06, 🖿<steam_dir>/steamapps/common/Source SDK Base 2007/ for '07). You can use Notepad++ (or basic Windows' Notepad) to edit it.

The following example lists depot VPKs for Base 2006: 

"steam_vpks"
{
	"spewfileio" "true"
	"mount"
	{
		"vpk" "vpks/depot_206" // Base Source Shared Materials
		"vpk" "vpks/depot_207" // Base Source Shared Models
		"vpk" "vpks/depot_208" // Base Source Shared Sounds
		"vpk" "vpks/depot_212" // Base Source Engine 2 (Source 2006 assets)
		"vpk" "vpks/depot_213" // HL2: Episode One shared
		"vpk" "vpks/depot_215" // Source Engine Test shared
		"vpk" "vpks/depot_381" // Episode One content + HL2 VCDs (mislabeled as Episode 1 French)
		// other vpks can be mounted here as well
	}
}

The following example lists depot VPKs for Base 2007: 

"steam_vpks"
{
	"spewfileio" "true"
	"mount"
	{
		"vpk" "vpks/depot_206" // Base Source Shared Materials
		"vpk" "vpks/depot_207" // Base Source Shared Models
		"vpk" "vpks/depot_208" // Base Source Shared Sounds
		"vpk" "vpks/depot_212" // Base Source Engine 2 (Source 2006 assets)
		"vpk" "vpks/depot_213" // HL2: Episode One shared
		"vpk" "vpks/depot_215" // Source Engine Test shared
		"vpk" "vpks/depot_381" // Episode One content + HL2 VCDs (mislabeled as Episode 1 French)
		"vpk" "vpks/depot_216" // Source 2007 Binaries
		"vpk" "vpks/depot_305" // Source 2007 Shared Materials
		"vpk" "vpks/depot_306" // Source 2007 Shared Models
		"vpk" "vpks/depot_307" // Source 2007 Shared Sound
		"vpk" "vpks/depot_308" // Episode One - Source 2007 Shared
		"vpk" "vpks/depot_309" // Episode One - Source 2007 Content
		"vpk" "vpks/depot_310" // Source 2007 Dedicated Server Windows
		"vpk" "vpks/depot_311" // Source 2007 Dedicated Server Linux
		"vpk" "vpks/depot_421" // Episode Two Content (mislabeled as Episode 2 French)
		"vpk" "vpks/depot_422" // Episode Two Materials (mislabeled as Episode 2 German)
		"vpk" "vpks/depot_423" // Episode Two Maps (mislabeled as Episode 2 Russian)
		// other vpks can be mounted here as well
	}
}
  • The "vpk" keys point at files relative to the Base's root folder (as you can see the 🖿/vpks/ folder right inside the root).
  • The spewfileio parameter defines if additional debug info is being spewed; usually you can leave it at false.
  • Note.pngNote:Depot 421, 422, 423 will not mount without copying it from SDK Base 2006 (Source SDK Base\vpks), and paste it on Source SDK Base 2007\vpks, due to mistake from Valve. See #Misplaced VPKs.

This file allows the Base to load its depot VPKs and, in turn, allows a mod for that Base to properly initiate and reference content. This works without needing to change the mod's gameinfo.txt or moving it to another Base.

Note.pngNote:Keep in mind that steam_vpks.vdf disables the automatic dependency enumeration and mounting by taking a different code path, so you have to include the base VPKs in the steam_vpks.vdf file.
Todo: Elaborate. Is just listing all the depots enough?
Note.pngNote:You can use steam_vpks.vdf to manually specify any number of additional VPKs inside it (within vpklib's maximum), so long as you use a path respective of the Base's install path. Without this file, Steam.dll automatically determines what IDs to mount, then finds the respective depot VPK. At this point, it won't load any additional VPKs. With the file, the wrapper instead mounts whatever is listed in the file.

Testing

A simple way to check it is to try and load an old mod for this Base without the file present (or renamed to anything other than steam_vpks.vdf). The typical behavior is that either the mod would crash and report being Unable to load manifest file (typically scripts/surfaceproperties_manifest.txt or something similar); another frequent error is Can't find background image materials/console/startup_loading.vtf. Some mods can load, but then will be missing assets; this is frequently the case with Base 2007 mods that need Ep2 content.

Now place the file you created, steam_vpks.vdf, next to the Base's hl2.exe . The mod should now properly initiate.

Using mount_vpk function

A relatively simple alternative in your mod's code. Calling mount_vpk with the path to a depot, relative to the Base's main directory, should get the target VPK at that path mounted.

Todo: Elaborate. Is it the same relative paths as in steam_vpks.vdf? And where is the function supposed to be called?

Using SteamMountFilesystem

According to the blog, this isn't the best way, as it requires using specific paths to the depots - 

<game_path>/vpks/depot_<depot_id>_dir.vpk

Nevertheless, calling this function in the same manner as filesystem_steam.dll does it, should work.

Additional caveats

Difference in paths

The Bases' depot VPKs follow a slightly different filing structure compared to all others. Their tree looks like 🖿<root>/<game name>/<content folders> (materials, models, sound, etc), whereas normal VPKs look like 🖿<root>/<content folders>.

Thus, even with the fix, they won't properly mount "traditional" VPKs shipped with the games - the resulting path to files would be treated as incorrect due to a 'missing' folder in it.

It also means that while you can pack the mod's content into a VPK and reference that VPK in steam_vpks.vdf, you have to add a 'fake' game directory into the path: 

root\fake_game_dir\maps
root\fake_game_dir\models
root_fake_game_dir\materials
etc...

and reference that folder in the mod's gameinfo.txt's SearchPaths: 

Game  fake_game_dir

Misplaced VPKs

To make matters worse, SDK Base 2007 does not include files from Episode Two. Those VPKs - depot_421, depot_422, depot 423 - are all shipped with SDK Base 2006 by mistake, even though 2006 is supposed to "reflect" Episode One.

A crude, but working solution requires having both SDK Base 2006 and 2007 installed at the same directory (same 🖿/<steam_dir>/steamapps/common/) and referencing 2006's depots by their relative paths, i. e.: 

"vpk" "./../Source SDK Base/vpks/depot_421"
"vpk" "./../Source SDK Base/vpks/depot_422"
"vpk" "./../Source SDK Base/vpks/depot_423"

Alternatively, if you don't want to have SDK Base 2006 installed, but still want Episode Two content on SDK Base 2007, first install SDK Base 2006 and 2007, then copy (or cut) the following files from 🖿Source SDK Base\vpks folder: 

depot_421_000.vpk to depot_421_005.vpk
depot_421_dir.vpk
depot_422_000.vpk to depot_422_003.vpk
depot_422_dir.vpk
depot_423_000.vpk to depot_423_003.vpk
depot_423_dir.vpk

Next, paste these files on 🖿Source SDK Base 2007\vpks folder and then uninstall SDK Base 2006. And finally, edit steam_vpks.vdf and add: 

"vpk" "vpks/depot_421"
"vpk" "vpks/depot_422"
"vpk" "vpks/depot_423"
Note.pngNote:The alternative method does also works even if you chose to keep Source SDK Base 2006. Just don't verify SDK Base on Steam, as this will redownload depot 421 - 423 to 2006 Base, wasting storage space.

Suspected corruption of the VPKs

A sidenote about the VPK files coming with the game and the vpk(lib) version used 
The following text is copied from the original blog.
I deem the VPK files and the vpklib version built into the steam.dll a very specific one. The VPK files coming with all Valve games and licensed games that utilize this mounting system are special, specifically it's pretty much a valid VPK2 file except for the checksums at the end. The tree and chunk checksums have been replaced by something that looks like corruption, which might've been caused by the checksums being uninitialized memory aligned next to each other. Here's how it looks like in the depot_215_dir.vpk file from Source SDK Base 2006. This pattern can also be seen in the depot_307_dir.vpk file from Source SDK Base 2007.
    {
        "PersonaName"       "martin

Here is another pattern, this one is found in depot_309_dir.vpk and depot_308_dir.vpk from Source SDK Base 2007.

    {
        "0"     "0100000065a3e27eb

As to what the actual cause of the corruption is, I can only speculate. The VPK files might've been made by some internal vpk.exe build, which licensees would also have, because this can be observed in both Valve and licensed games as I have stated before.

Base 2007 doesn't draw menus

There is a bug that causes Base 2007 and its mods to not display the main menu, console panel, HUD or other interfaces. While they are functional (main menu buttons work and produce button sounds, the console can accept input), they are invisible and have to be navigated 'blindly'.

It appears to be caused by mounting 🖿depot_317.vpk. This was later revealed to be files intended to be used for Source 2009 games (Valve labeled this as Source 2007 Binaries 2 instead of Source 2009 Binaries). However when unpacked, its contents don't appear to be causing the same issues (if any).

Note.pngFix:Excluding 🖿depot_317.vpk from the listing should be enough.

Credits

All info on researching and documenting this is taken from the gmod9.com blog.

The original credits include:

mv - for providing BGT and The Ship VPK files.

Nafrayu - for trying to figure this out with me long ago
DarkOK - for reviewing this

… and everyone else that I've forgotten.

Thanks to Jai "Choccy" Fox for helping me with the article and the subject.

Retrieved from "https://developer.valvesoftware.com/w/index.php?title=Fixing_VPK_mounting_for_older_Source_SDK_Bases&oldid=470672"