Please read the Launchpad bug report where the issue was raised.
At the bottom of /etc/gdm/PreSession/Default change :
# Default value
if [ "x$BACKCOLOR" = "x" ]; then
BACKCOLOR="#dab082"
fi
to:
# Default value
if [ "x$BACKCOLOR" = "x" ]; then
BACKCOLOR="x"
fi
Where #dab082 is the default brown color that appears right after GDM exits and before the desktop files are loaded. Works nicely with Xfce here.
Edit: from the bug report, the bug is fixed in Hardy :)