Alternative title: On Console Windows, GUI Processes, and Pipe Inheritance in Windows

While working on Overlord Systems I wanted to do something simple: run the Overlord Engine (a native Windows app) and any child processes it creates without any console window showing to the user.

It is simple, kind of. I just haven’t seen the entire behavior and background knowledge in one place, so this is a short write-up in case it helps someone.

In Windows, a process is by default a ‘console’ process where it has a console attached and stdout/stdin/stderr are piped between the process and the console.

To prevent this console from appearing when running your program, you can use certain linker flags (/SUBSYSTEM:windows /ENTRY:mainCRTStartup) to switch from the ‘console’ to the ‘windows’/gui subsystem of the Windows OS.

However, this causes your program to not get assigned any std in/out/err pipes, which means anything you log will just get lost and no one can send you anything on stdin. If you want logs you will have to redirect them to a file or something yourself.

This is about it for the main process. Now lets talk about child processes.

If you create a child process (e.g., using CreateProcessW) that is compiled as a ‘console’ process, and you pass bInheritHandles=true, it will inherit the parent process’s std in/out/err pipes (and any other handles it can inherit, like inheritable file handles), which means it will not show a console window of its own and anything it logs will show mixed with the logs of the parent process, in the console window of the parent. If you disable inheritance the child will display its own console window, visible to the user.

But what if we pass bInheritHandles=true while the parent process is a GUI one with no console? In that case the child process will create its own console window visible to the user.

To avoid that (i.e., to run a child process from a GUI program with neither parent nor child showing a console window), you will need to pass the Process Create Flag CREATE_NO_WINDOW when creating the child process, which will cause Windows to create an invisible console for the child. Practically speaking, stdout/stderr is again lost as it is sent to the invisible console.

It is important to keep in mind that going from ‘console parent and console child’ to ‘gui parent and/or no-window child’ changes behavior.

When the above happens, the behavior of the child inheriting the parent’s std in/out/err pipes is no longer true (even if bInheritHandles=true!). The child will have its own pipes, and you will no longer see the parent and child logs mixed together in the same console.

This might cause bugs in cases where the parent expects to see the child’s logs in its own stdout, which is now no longer be true, or this could simply confuse the developer about why they aren’t seeing the child’s logs anymore.

Note that a parent can access stdout/stderr/stdin of the child even if the child has separate (aka non-inherited) pipes because the parent can create and pass stdout/stderr/stdin pipes to the child to use via the STARTUPINFOW struct when calling, e.g., CreateProcessW, and then read/write those pipes to communicate with the child process.

Of course, this article is not a comprehensive description of the create process API, rather it’s just what I thought is enough information to allow one to create parent and child processes without console windows and without surprises in behavior.

To summarize:

  • If your program shows a console, then you don’t need to do anything to hide consoles of child processes, just pass bInheritHandles=true when creating them. They will use the parent’s console and not create any additional ones.

  • In the above case, logs of parent and children will be mixed together in one console window.

  • If your program was compiled as a GUI, then child processes will create console windows visible to the user. If you don’t want them to do that, pass the CREATE_NO_WINDOW flag (e.g., in CreateProcessW) when creating the child processes, and their console windows won’t be shown to the user. Their logs will no longer show on the parent console window.

  • If the child is not using the parent stdout/stderr/stdin for whatever reason (e.g., the above case), you can still communicate with them by explicitly creating and passing a pipe for each of stdout/stderr/stdin to the child via the STARTUPINFOW struct when calling, for example, CreateProcessW.