RCE Endeavors 😅

This post will cover the topic of hooking a classes’ virtual method table. This is a useful technique that has many applications, but is most commonly seen in developing game hacks. For example, employing VMT hooking of objects in a Direct3D/OpenGL graphics engine is how in-game overlays are displayed.

Virtual Method Tables (or vtables)

Usage of VMTs, in the context of C++ for this post, is how polymorphism is implemented at the language level. Internally, the VMT is represented as an array of function pointers, and typically resides at the beginning or end of the memory layout of the object. Whenever a C++ class declares a virtual function, the compiler will add an entry in to the VMT for it. If a class inherits from a base object and overrides a base virtual function, then the pointer to the overriden function will be present in the derived objects VMT. For example, take the following code, compiled with the VS 2013 compiler on an x86 system:

class Base
{
public:
    Base() { printf("-  Base::Base\n"); }
    virtual ~Base() { printf("-  Base::~Base\n"); }
 
    void A() { printf("-  Base::A\n"); }
    virtual void B() { printf("-  Base::B\n"); }
    virtual void C() { printf("-  Base::C\n"); }
};
 
class Derived final : public Base
{
public:
    Derived() { printf("-  Derived::Derived\n"); }
    ~Derived() { printf("-  Derived::~Derived\n"); }
 
    void B() override { printf("-  Derived::B\n"); }
    void C() override { printf("-  Derived::C\n"); }
};

with the instances of Base and Derived created as follows:

Base base;
Derived derived;
Base *pBase = new Derived;

The class Base has three virtual functions: ~Base, B, and C. The class Derived, which inherits from Base overrides the two virtual functions B and C. In memory, the VMT for Base will contain ~Base, B, and C, as can be inspected with the debugger:

while the VMT for the two Derived instances contain ~Derived, B, and C, but with different addresses for each than the ones in Base (see below).

So how are these actually used? Take, for example, a function that takes a pointer to a Base instance and invokes the functions A, B, and C, on it:

void Invoke(Base * const pBase)
{
    pBase->A();
    pBase->B();
    pBase->C();
}

and is invoked in the following manner:

    Invoke(&base);
    Invoke(&derived);
    Invoke(pBase);

The Invoke function disassembled for x86 is as follows:

    pBase->A();
004012C9 8B 4D 08             mov         ecx,dword ptr [pBase]  
004012CC E8 8F FE FF FF       call        Base::A (0401160h)  
    pBase->B();
004012D1 8B 45 08             mov         eax,dword ptr [pBase]  
004012D4 8B 10                mov         edx,dword ptr [eax]  
004012D6 8B 4D 08             mov         ecx,dword ptr [pBase]  
004012D9 8B 42 04             mov         eax,dword ptr [edx+4]  
004012DC FF D0                call        eax  
    pBase->C();
004012DE 8B 45 08             mov         eax,dword ptr [pBase]  
004012E1 8B 10                mov         edx,dword ptr [eax]  
004012E3 8B 4D 08             mov         ecx,dword ptr [pBase]  
004012E6 8B 42 08             mov         eax,dword ptr [edx+8]  
004012E9 FF D0                call        eax  

This disassembly shows exactly what is going on under the hood with relation to polymorphism. For the invocations to B and C, the compiler moves the address of the object in to the EAX register. This is then dereferenced to get the base of the VMT and stored in the EDX register. The appropriate VMT entry for the function is found by using EDX as an index and storing the address in EAX. This function is then called. Since Base and Derived have different VMTs, this code will call different functions — the appropriate ones — for the appropriate object type. Seeing how it’s done under the hood also allows us to easily write a function to print the VMT.

void PrintVTable(Base * const pBase)
{
    unsigned int *pVTableBase = (unsigned int *)(*(unsigned int *)pBase);
    printf("First: %p\n"
        "Second: %p\n"
        "Third: %p\n",
        *pVTableBase, *(pVTableBase + 1), *(pVTableBase + 2));
}

Hooking the VMT

Knowing the layout of the VMT makes it trivial to hook. To accomplish this, all that is needed is to overwrite the entry in the VMT with the address of the desired hook function. This is done by using the VirtualProtect function to set the appropriate memory permissions alongside with memcpy to write in the desired hook address. Note that memcpy is used since everything resides within the same address space, otherwise WriteProcessMemory would have to be used. A hooking routine might look like the following:

void HookVMT(Base * const pBase)
{
    unsigned int *pVTableBase = (unsigned int *)(*(unsigned int *)pBase);
    unsigned int *pVTableFnc = (unsigned int *)((pVTableBase + 1));
    void *pHookFnc = (void *)VMTHookFnc;
 
    SIZE_T ulOldProtect = 0;
    (void)VirtualProtect(pVTableFnc, sizeof(void *), PAGE_EXECUTE_READWRITE, &ulOldProtect);
    memcpy(pVTableFnc, &pHookFnc, sizeof(void *));
    (void)VirtualProtect(pVTableFnc, sizeof(void *), ulOldProtect, &ulOldProtect);
}

and VMTHook having a simple definition of

void __fastcall VMTHookFnc(void *pEcx, void *pEdx)
{
    Base *pThisPtr = (Base *)pEcx;
 
    printf("In VMTHookFnc\n");
}

Here the fastcall calling convention is used to easily retrieve the this pointer, which is typically stored in the ECX register.

Applications

The application of this technique will show how to hook IDXGISwapChain::Present and allow for rendering/overlaying of text on a Direct3D10 application. This is not the only way to overlay text, nor necessarily the best, but still provides an adequate example to illustrate the point. The target application will be a Direct3D10 sample provided by the June 2010 DirectX SDK. See /Samples/C++/Direct3D10/Tutorials/Tutorial01 in the SDK. The sample application initializes the Direct3D device and swap chain with a call to D3D10CreateDeviceAndSwapChain then simply sets up a view and renders a blue background on the window (screenshot below).

To overlay text on a Direct3D application, the IDXGISwapChain object must be obtained. Then the Present function of the interface must be hooked, since that is the function responsible for showing the rendered image to the user. This is done here by hooking D3D10CreateDeviceAndSwapChain. Once this function is hooked, the hook will call the real D3D10CreateDeviceAndSwapChain function in order to set up the IDXGISwapChain interface. Then the VMT entry for Present will be replaced with a hooked version that renders text. Put into code it looks like the following:

HRESULT WINAPI D3D10CreateDeviceAndSwapChainHook(IDXGIAdapter *pAdapter, D3D10_DRIVER_TYPE DriverType, HMODULE Software,
    UINT Flags, UINT SDKVersion, DXGI_SWAP_CHAIN_DESC *pSwapChainDesc, IDXGISwapChain **ppSwapChain,
    ID3D10Device **ppDevice)
{
 
    printf("In D3D10CreateDeviceAndSwapChainHook\n");
 
    //Create the device and swap chain
    HRESULT hResult = pD3D10CreateDeviceAndSwapChain(pAdapter, DriverType, Software, Flags, SDKVersion,
        pSwapChainDesc, ppSwapChain, ppDevice);
 
    //Save the device and swap chain interface.
    //These aren't used in this example but are generally nice to have addresses to
    if(ppSwapChain == NULL)
    {
        printf("Swap chain is NULL.\n");
        return hResult;
    }
    else
    {
        pSwapChain = *ppSwapChain;
    }
    if(ppDevice == NULL)
    { 
        printf("Device is NULL.\n");
        return hResult;
    }
    else
    {
        pDevice = *ppDevice;
    }
 
    //Get the vtable address of the swap chain's Present function and modify it with our own.
    //Save it to return to later in our Present hook
    if(pSwapChain != NULL)
    {
        DWORD_PTR *SwapChainVTable = (DWORD_PTR *)pSwapChain;
        SwapChainVTable = (DWORD_PTR *)SwapChainVTable[0];
        printf("Swap chain VTable: %X\n", SwapChainVTable);
        PresentAddress = (pPresent)SwapChainVTable[8];
        printf("Present address: %X\n", PresentAddress);
 
        DWORD OldProtections = 0;
        VirtualProtect(&SwapChainVTable[8], sizeof(DWORD_PTR), PAGE_EXECUTE_READWRITE, &OldProtections);
        SwapChainVTable[8] = (DWORD_PTR)PresentHook;
        VirtualProtect(&SwapChainVTable[8], sizeof(DWORD_PTR), OldProtections, &OldProtections);
    }
 
    //Create the font that we will be drawing with
    CreateDrawingFont();
 
    return hResult;
}

CreateDrawingFont simply sets up a ID3DX10Font to draw with. Now since the VMT entry was replaced, PresentHook will be invoked instead of Present. Here is where the drawing can be done.

HRESULT WINAPI PresentHook(IDXGISwapChain *thisAddr, UINT SyncInterval, UINT Flags)
{
 
    //printf("In Present (%X)\n", PresentAddress);
 
    RECT Rect = { 100, 100, 200, 200 };
    pFont->DrawTextW(NULL, L"Hello, World!", -1, &Rect, DT_CENTER | DT_NOCLIP, RED);
    return PresentAddress(thisAddr, SyncInterval, Flags);
}

I chose a different calling convention here than for the earlier example code, but everything still functions the same. The end result shows the Present hook successfully rendering the text:
A few important caveats about doing it this way:

• The hook must be installed prior to the call to D3D10CreateDeviceAndSwapChain. Otherwise handles to the device and swap chain won’t be obtained.
• ID3DX10Font::DrawText can mess with the blend states, shaders, rasterizer, etc. Overlaying text on an application that makes use of these requires the hook developer to account for this and save/restore the states properly.

The source code for the VMT hook example, the slightly modified Direct3D10 sample application, and the Direct3D10 hook can be found here. The hook uses Microsoft Detours as a dependency to perform the initial hooking of D3D10CreateDeviceAndSwapChain.