Plugin reloading for 3dsmax exporters

I revisited recently a dormant project of mine, for which I unfortunately need to write a 3dsmax exporter plugin.

Now, I’m always pissed off from the start when I have to write code on windows and visual studio, but having to deal with 3dsmax on top of that, really just adds insult to injury. It’s not just that maxsdk is a convoluted mess. Or that it needs a very specific version of visual studio to write plugins for it (which is really Microsoft’s fault, to be fair). No, my biggest issue so far is that 3dsmax takes about 3 years to start up, and there is no way to unload, or reload a plugin without restarting it.

Whenever I fix a tiny thing in the exporter plugin I’m writting, and I want to try it out and see if it does the buissiness, I have to shut down 3dsmax, start it up again (which takes forever), load my test scene, then try to export again and see what happens. This is obviously unacceptable, so I really had to do something about it.

3dsmax plugin handling

But first let’s examine how exporter plugins are handled by 3dsmax. The 3dsmax installation directory contains a subdirectory called plugins. At startup, it looks into that directory, and loads all the plugins located there. Plugins are obviously just DLLs, but with a particular naming convention for their suffix, so that 3dsmax will know how to treat them as different plugin classes. For instance, exporter plugins must be named with a .dle suffix.

Exporter plugins are supposed to define a subclass of SceneExport, which overrides the DoExport function. When 3dsmax loads the plugin dll, it pulls from it a function called LibCreate, which is supposed to create and return an instance of this class. Then, whenever the user tries to export a scene of the type registered by this plugin, the appropriate DoExport function is called, to do it.

Solution

So as I mentioned previously, the problem is that plugin DLLs can’t be unloaded and reloaded while 3dsmax is running, making it necessary to shut down 3dsmax and restart it all the time, while writing my exporter.

After cursing Autodesk and their stupid plugin system for a while, I came up with a pretty neat idea to circuimvent that restriction. Instead of letting 3dsmax grab hold of my exporter DLL when it starts, what if I write a second exporter DLL that is just a stub, a proxy if you will. What if that stub gets loaded by 3dsmax at startup, but my actual exporter plugin DLL is separate, 3dsmax doesn’t even know about it, and the stub exporter loads it up on demand, and passes through the DoExport call!

With this scheme, it doesn’t actually matter much that plugins can’t be unloaded by 3dsmax, because my plugin wasn’t loaded by 3dsmax in the first place. I’m loading it on demand when the user requests an export, and releasing it immediately after the export is done.

Turns out, this works beautifully. I created post-build events for both projects (maxgoat, and maxgoat_stub) that copies the DLLs in the 3dsmax plugin directory, but only the maxgoat_stub post-build event changes the suffix to “.dle”. The actual exporter is left with the original “.dll” suffix, and 3dsmax ignores its existence.

prop_maxgoat prop_maxgoat_stub

The only tricky bit the stub DLL has to do in order to load and use the real exporter DLL, is to find the plugin directory where both of them are installed. This little snippet of win32 barf handles that correctly:

static const char *find_dll_dir()
{
    static char path[MAX_PATH];

    HMODULE dll;
    if(!GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS |
            GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
            (LPCSTR)find_dll_dir, &dll)) {
        return 0;
    }
    GetModuleFileNameA(dll, path, sizeof path);

    char *last_slash = strrchr(path, '\\');
    if(last_slash) {
        *last_slash = 0;
    }
    return path;
}

At that point it can happily proceed to load the actual exporter DLL, and call everything it needs to, in order to perform the export. Here’s the whole DoExport function of the stub, with all error handling removed for brevity:

int GoatExporterStub::DoExport(const MCHAR *name, ExpInterface *eiface,
        Interface *iface, BOOL non_interactive, DWORD opt)
{
    static const char *dll_fname = "maxgoat.dll";
    const char *plugdir = find_dll_dir();
    char *dll_path = new char[strlen(dll_fname) + strlen(plugdir) + 2];
    sprintf(dll_path, "%s\\%s", plugdir, dll_fname);

    // load plugin and grab all the functions we need
    HMODULE dll = LoadLibraryA(dll_path);

    ClassDescFunc get_class_desc = (ClassDescFunc)GetProcAddress(dll, "LibClassDesc");
    InitFunc init = (InitFunc)GetProcAddress(dll, "LibInitialize");
    ShutdownFunc shutdown = (ShutdownFunc)GetProcAddress(dll, "LibShutdown");

    init();
    ClassDesc *desc = get_class_desc(0);

    // create the exporter instance and call DoExport
    SceneExport *ex = (SceneExport*)desc->Create();
    int res = ex->DoExport(name, eiface, iface);
    delete ex;

    shutdown();

    delete [] dll_path;
    FreeLibrary(dll);
    return res;
}

With this system in place, iterating on figuring out all the maxsdk bullshit is a breeze. Just keep 3dsmax open with my stub plugin loaded all the time, compile/install the exporter at will, and hit export again from within 3dsmax. The current version of the exporter DLL is automatically loaded, used and shoved out of the way again. The whole process becomes almost enjoyable.

Advertisements

2 Responses to “Plugin reloading for 3dsmax exporters”

  1. mgeorgoulopoulos Says:

    This post revisits my nightmares step-by-step and wakes up deamons from the past! Even though I never came up with this kind of work-around, main reason was – I ENJOYED the break while 3ds max was loading and I could look at something a lot more interesting than the SDK for a while – ie: the still 3ds max logo :D

  2. Zalo (@Zal0) Says:

    This is a wonderful solution. I was thinking on something similar and then googled a bit to check if someone had already done it… et voila! Thanks a lot


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: