When you build a GP (as well as when you install it from the MSI, with some minor differences), a post-build command (that you will need to see by opening the csproj file in notepad, sorry :( ) is issued against the RecipeFramework.dll assembly, that contains the core functionality of GAX. It's an InstallUtil call that will run an installer class in GAX.
When a host is first "bound" with GAX, a class implementing a given interface must be registered with GAX as the one responsible for registering a GP with that host. When the previous command is issued for your GP, the attribute Host in your GP (which defaults to VS) is used to determine which registration class to call to give the host a chance to setup its environment for the new GP. Hence, GAX is not necessarily tied exclusively to VS ;-).
GAX itself does little work at this point: it basically registers the GP in a manifest of installed packages (currently in c:\Documents and Settings\All Users\Application Data\Microsoft\Recipe Framework\RecipeFramework.xml). It then just calls the host registration class.
The class reponsible for all the ctc and resource dll generation lives in the Microsoft.Practices.RecipeFramework.VisualStudio.dll inside %IDE%\PublicAssemblies. The class called from GAX to register a GP in VS is Microsoft.Practices.RecipeFramework.VisualStudio.RecipeFrameworkInstallerl
The process itself is fairly complex, and involves processing custom RegistrationAttributes (from the new Microsoft.VisualStudio.Shell), several classes in the CTC namespace inside our assembly, etc.
And we don't use a registry hive for the new packages. They are registered in the root registry tree for VS, so that you can test it directly anything special.
HTH,
/kzu