win32 application as both GUI and Console

I wanted to have a program run in the command line when invoked from there (maybe also when other factors are involved):

but also run as a regular Windows application when launched from Explorer, without having to flash any unnecessary console windows:

Win32 subsystems

All Linux applications can interact with a console due to the way in which the terminal works and handles the standard IO streams there.

This is not the case on Windows and programs must be linked either for windowing or console operation. There is a Visual Studio linker flag - /SUBSYSTEM - controlling this behavior. GUI applications can call:

• AllocConsole() to create a console window
• AttachConsole() to grab an existing console

MSDN contains lots of info about Console Functions around the area where those two from above are documented.

Console subsystem applications can call any Win32 function, create windows, have message loops and access all Windows functionality. The only special aspect here is that they run from a console, and even if they detach from it and release it, the user will see the terminal window, if only for a limited time.

Attaching GUIs

To avoid unnecessarily showing a terminal window, the solution is to target the GUI subsystem, like any regular windowing application, and attempt to detect and grab a pre-existing console from the parent process.

int APIENTRY wWinMain(_In_ HINSTANCE hInstance,
                     _In_opt_ HINSTANCE hPrevInstance,
                     _In_ LPWSTR    lpCmdLine,
                     _In_ int       nCmdShow)
{
    UNREFERENCED_PARAMETER(hPrevInstance);
    UNREFERENCED_PARAMETER(lpCmdLine);

    // TODO: Place code here.
    if (CanAttachToConsole())
    {
        return RunAsConsole(__argc, __wargv);
    }

    return RunAsGui(hInstance, nCmdShow);
}

The RunAsConsole() function basically works like main() from a console application, while RunAsGui() registers WNDCLASSes, draws windows, etc.

CanAttachConsole() tries to grab a console with an AttachConsole(ATTACH_PARENT_PROCESS) call, makes sure file handle redirects are respected (for stdlib use) and its success means that RunAsConsole() will work as expected.

Console locking

Because the application is GUI, when launching it from the console the parent interpreter process (conhost.exe, pwsh.exe) will not wait for the application to return. bash.exe from MSYS does wait, regardless of subsystem, probably due to the fact that it's a Linux port respecting POSIX conventions.

The key is to wrap the executable in a batch script. The FreeConsole() documentation gives a hint as to why this works:

A console is closed when the last process attached to it terminates or calls FreeConsole.

Batch files run synchronously and create a child terminal which invokes the GUI application and ends. But its console is not freed until the GUI application also frees it.

This is all a launcher called mm.cmd needs to do:

@"%~dp0Metamorph.exe" %*

Code

Head to to the GitHub repository for the project.