CFileDialog::DoModal() never returns [MFC]

Prev: CMFCRibbonButton pressed state, how to do that ?
Next: converting vc++2005 project to VC++2008

From: jbreher on 21 Feb 2008 03:27

I am encountering a wierd error, and I can't figure out which way to turn.
Something deep within the call tree of CFileDialog::DoModal() is (apparently)
failing, causing the dialog to never be displayed, and the f() never to
return. Here's the failing code:

void CPvtDlg::OnBnClickedBnFileBd()
{

// specify a file
CFileDialog fileOpenBox ( TRUE,
// BOOL bOpenFileDialog
NULL,
// LPCTSTR lpszDefExt
NULL,
// LPCTSTR lpszFileName
(OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
// DWORD dwFlags
NULL, // LPCTSTR lpszFilter
NULL );
// CWnd* pParentWnd

// extract file name
if( IDOK == fileOpenBox.DoModal() )
{
m_strFileNameBd = fileOpenBox.GetFileName();
m_strPathNameBd = fileOpenBox.GetPathName();
}
....

Single stepping, I am unable to step over the DoModal() call. Stepping into
the call to DoModal(), I find myself inside dlgfile.cpp:

INT_PTR CFileDialog::DoModal()
{
ASSERT_VALID(this);
ASSERT(m_ofn.Flags & OFN_ENABLEHOOK);
ASSERT(m_ofn.lpfnHook != NULL); // can still be a user hook

// zero out the file buffer for consistent parsing later
ASSERT(AfxIsValidAddress(m_ofn.lpstrFile, m_ofn.nMaxFile));
DWORD nOffset = lstrlen(m_ofn.lpstrFile)+1;
ASSERT(nOffset <= m_ofn.nMaxFile);
memset(m_ofn.lpstrFile+nOffset, 0, (m_ofn.nMaxFile-nOffset)*sizeof(TCHAR));

// This is a special case for the file open/save dialog,
// which sometimes pumps while it is coming up but before it has
// disabled the main window.
HWND hWndFocus = ::GetFocus();
BOOL bEnableParent = FALSE;
m_ofn.hwndOwner = PreModal();
AfxUnhookWindowCreate();
if (m_ofn.hwndOwner != NULL && ::IsWindowEnabled(m_ofn.hwndOwner))
{
bEnableParent = TRUE;
::EnableWindow(m_ofn.hwndOwner, FALSE);
}

_AFX_THREAD_STATE* pThreadState = AfxGetThreadState();
ASSERT(pThreadState->m_pAlternateWndInit == NULL);

if (m_ofn.Flags & OFN_EXPLORER)
pThreadState->m_pAlternateWndInit = this;
else
AfxHookWindowCreate(this);

INT_PTR nResult;
if (m_bOpenFileDialog)
nResult = ::AfxCtxGetOpenFileName(&m_ofn);
else
nResult = ::AfxCtxGetSaveFileName(&m_ofn);
....
I am unable to step over the call to ::AfxCtxGetOpenFileName(&m_ofn); in the
if branch above. Stepping into it allows me to step through

OPENFILENAME& CFileDialog::GetOFN()
{
return *m_pOFN;
}

and

COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
unnamed1),(unnamed1))

to

AFX_MODULE_STATE* AFXAPI AfxGetModuleState()
{
_AFX_THREAD_STATE* pState = _afxThreadState;
ENSURE(pState);
AFX_MODULE_STATE* pResult;
if (pState->m_pModuleState != NULL)
{
// thread state's module state serves as override
pResult = pState->m_pModuleState;
}
else
{
// otherwise, use global app state
pResult = _afxBaseModuleState.GetData();
}
ENSURE(pResult != NULL);
return pResult;
}

back through

COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
unnamed1),(unnamed1))

I could probably continue, but it seems to me that I am inside the innards
of an area of the libs that I have no business in. What is it up in the API
level that I am screwing up?

From: Mikel on 21 Feb 2008 04:31

See below:

On 21 feb, 09:27, jbreher <jbre...(a)discussions.microsoft.com> wrote:
> I am encountering a wierd error, and I can't figure out which way to turn.
> Something deep within the call tree of CFileDialog::DoModal() is (apparently)
> failing, causing the dialog to never be displayed, and the f() never to
> return. Here's the failing code:

*************
Well, DoModal returns when you select a file o close the dialog, so if
the dialog is never displayed, it's logical that it doesn't return.
*************
>
> void CPvtDlg::OnBnClickedBnFileBd()
> {
>
> // specify a file
> CFileDialog fileOpenBox ( TRUE,
> // BOOL bOpenFileDialog
> NULL,
> // LPCTSTR lpszDefExt
> NULL,
> // LPCTSTR lpszFileName
> (OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
> // DWORD dwFlags
> NULL, // LPCTSTR lpszFilter
> NULL );
> // CWnd* pParentWnd

************
You are using the defaults here, so why not remove them and use just
CFileDialog fileOpenBox(TRUE); ?
************

>
> // extract file name
> if( IDOK == fileOpenBox.DoModal() )
> {
> m_strFileNameBd = fileOpenBox.GetFileName();
> m_strPathNameBd = fileOpenBox.GetPathName();
> }
> ...
>
> Single stepping, I am unable to step over the DoModal() call. Stepping into
> the call to DoModal(), I find myself inside dlgfile.cpp:
>
> INT_PTR CFileDialog::DoModal()
> {
> ASSERT_VALID(this);
> ASSERT(m_ofn.Flags & OFN_ENABLEHOOK);
> ASSERT(m_ofn.lpfnHook != NULL); // can still be a user hook
>
> // zero out the file buffer for consistent parsing later
> ASSERT(AfxIsValidAddress(m_ofn.lpstrFile, m_ofn.nMaxFile));
> DWORD nOffset = lstrlen(m_ofn.lpstrFile)+1;
> ASSERT(nOffset <= m_ofn.nMaxFile);
> memset(m_ofn.lpstrFile+nOffset, 0, (m_ofn.nMaxFile-nOffset)*sizeof(TCHAR));
>
> // This is a special case for the file open/save dialog,
> // which sometimes pumps while it is coming up but before it has
> // disabled the main window.
> HWND hWndFocus = ::GetFocus();
> BOOL bEnableParent = FALSE;
> m_ofn.hwndOwner = PreModal();
> AfxUnhookWindowCreate();
> if (m_ofn.hwndOwner != NULL && ::IsWindowEnabled(m_ofn.hwndOwner))
> {
> bEnableParent = TRUE;
> ::EnableWindow(m_ofn.hwndOwner, FALSE);
> }
>
> _AFX_THREAD_STATE* pThreadState = AfxGetThreadState();
> ASSERT(pThreadState->m_pAlternateWndInit == NULL);
>
> if (m_ofn.Flags & OFN_EXPLORER)
> pThreadState->m_pAlternateWndInit = this;
> else
> AfxHookWindowCreate(this);
>
> INT_PTR nResult;
> if (m_bOpenFileDialog)
> nResult = ::AfxCtxGetOpenFileName(&m_ofn);
> else
> nResult = ::AfxCtxGetSaveFileName(&m_ofn);
> ...
> I am unable to step over the call to ::AfxCtxGetOpenFileName(&m_ofn); in the
> if branch above. Stepping into it allows me to step through
>
> OPENFILENAME& CFileDialog::GetOFN()
> {
> return *m_pOFN;
>
> }
>
> and
>
> COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
> unnamed1),(unnamed1))
>
> to
>
> AFX_MODULE_STATE* AFXAPI AfxGetModuleState()
> {
> _AFX_THREAD_STATE* pState = _afxThreadState;
> ENSURE(pState);
> AFX_MODULE_STATE* pResult;
> if (pState->m_pModuleState != NULL)
> {
> // thread state's module state serves as override
> pResult = pState->m_pModuleState;
> }
> else
> {
> // otherwise, use global app state
> pResult = _afxBaseModuleState.GetData();
> }
> ENSURE(pResult != NULL);
> return pResult;
>
> }
>
> back through
>
> COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
> unnamed1),(unnamed1))
>
> I could probably continue, but it seems to me that I am inside the innards
> of an area of the libs that I have no business in. What is it up in the API
> level that I am screwing up?

*******
I have tried stepping into all of the functions until it finally
displays the dialog, so I really don't know what is going wrong. Those
are MFC/WIN32 internals and I am not an expert on that...

Have you checked wether the dialog is under other windows, or outside
the screen? It shouldn't, but anyway.

Another thing that comes to mind, try setting breakpoints in all the
functions in CFileDialog and see where it stops...

From: Joseph M. Newcomer on 21 Feb 2008 07:20

This is not a good description. If DoModal "fails", it will return 0. If it succeeds,
you should see a dialog box.

Presumably you are going to fill in all those parameters at some point, because you are
using the defaults, so there is no reason to provide default parameters to the
constructor.

I see no function f() here.

What version of VS are you using, what platform SDK, what OS? Not enough information here
to even make a good guess. There are some known issues with using older Platform SDKs on
modern operating systems because the size of the m_ofn member changed, but they have
typically led to unpleasant crashes, usually due to access faults or heap corruption.
joe

On Thu, 21 Feb 2008 00:27:00 -0800, jbreher <jbreher(a)discussions.microsoft.com> wrote:

>I am encountering a wierd error, and I can't figure out which way to turn.
>Something deep within the call tree of CFileDialog::DoModal() is (apparently)
>failing, causing the dialog to never be displayed, and the f() never to
>return. Here's the failing code:
>
>void CPvtDlg::OnBnClickedBnFileBd()
>{
>
> // specify a file
> CFileDialog fileOpenBox ( TRUE,
>// BOOL bOpenFileDialog
> NULL,
>// LPCTSTR lpszDefExt
> NULL,
>// LPCTSTR lpszFileName
> (OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
>// DWORD dwFlags
> NULL, // LPCTSTR lpszFilter
> NULL );
>// CWnd* pParentWnd
>
> // extract file name
> if( IDOK == fileOpenBox.DoModal() )
> {
> m_strFileNameBd = fileOpenBox.GetFileName();
> m_strPathNameBd = fileOpenBox.GetPathName();
> }
>...
>
>Single stepping, I am unable to step over the DoModal() call. Stepping into
>the call to DoModal(), I find myself inside dlgfile.cpp:
>
>INT_PTR CFileDialog::DoModal()
>{
> ASSERT_VALID(this);
> ASSERT(m_ofn.Flags & OFN_ENABLEHOOK);
> ASSERT(m_ofn.lpfnHook != NULL); // can still be a user hook
>
> // zero out the file buffer for consistent parsing later
> ASSERT(AfxIsValidAddress(m_ofn.lpstrFile, m_ofn.nMaxFile));
> DWORD nOffset = lstrlen(m_ofn.lpstrFile)+1;
> ASSERT(nOffset <= m_ofn.nMaxFile);
> memset(m_ofn.lpstrFile+nOffset, 0, (m_ofn.nMaxFile-nOffset)*sizeof(TCHAR));
>
> // This is a special case for the file open/save dialog,
> // which sometimes pumps while it is coming up but before it has
> // disabled the main window.
> HWND hWndFocus = ::GetFocus();
> BOOL bEnableParent = FALSE;
> m_ofn.hwndOwner = PreModal();
> AfxUnhookWindowCreate();
> if (m_ofn.hwndOwner != NULL && ::IsWindowEnabled(m_ofn.hwndOwner))
> {
> bEnableParent = TRUE;
> ::EnableWindow(m_ofn.hwndOwner, FALSE);
> }
>
> _AFX_THREAD_STATE* pThreadState = AfxGetThreadState();
> ASSERT(pThreadState->m_pAlternateWndInit == NULL);
>
> if (m_ofn.Flags & OFN_EXPLORER)
> pThreadState->m_pAlternateWndInit = this;
> else
> AfxHookWindowCreate(this);
>
> INT_PTR nResult;
> if (m_bOpenFileDialog)
> nResult = ::AfxCtxGetOpenFileName(&m_ofn);
> else
> nResult = ::AfxCtxGetSaveFileName(&m_ofn);
>...
>I am unable to step over the call to ::AfxCtxGetOpenFileName(&m_ofn); in the
>if branch above. Stepping into it allows me to step through
>
>OPENFILENAME& CFileDialog::GetOFN()
>{
> return *m_pOFN;
>}
>
>and
>
>COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
>unnamed1),(unnamed1))
>
>to
>
>AFX_MODULE_STATE* AFXAPI AfxGetModuleState()
>{
> _AFX_THREAD_STATE* pState = _afxThreadState;
> ENSURE(pState);
> AFX_MODULE_STATE* pResult;
> if (pState->m_pModuleState != NULL)
> {
> // thread state's module state serves as override
> pResult = pState->m_pModuleState;
> }
> else
> {
> // otherwise, use global app state
> pResult = _afxBaseModuleState.GetData();
> }
> ENSURE(pResult != NULL);
> return pResult;
>}
>
>back through
>
>COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
>unnamed1),(unnamed1))
>
>I could probably continue, but it seems to me that I am inside the innards
>of an area of the libs that I have no business in. What is it up in the API
>level that I am screwing up?
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

From: jbreher on 21 Feb 2008 19:32

"Mikel" wrote:

> See below:
>

> *************
> Well, DoModal returns when you select a file o close the dialog, so if
> the dialog is never displayed, it's logical that it doesn't return.
> *************

Understood. Hence, my confusion.

> >
> > void CPvtDlg::OnBnClickedBnFileBd()
> > {
> >
> > // specify a file
> > CFileDialog fileOpenBox ( TRUE, // BOOL bOpenFileDialog
> > NULL, // LPCTSTR lpszDefExt
> > NULL, // LPCTSTR lpszFileName
> > (OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
> > // DWORD dwFlags
> > NULL, // LPCTSTR lpszFilter
> > NULL ); // CWnd* pParentWnd
>
> ************
> You are using the defaults here, so why not remove them and use just
> CFileDialog fileOpenBox(TRUE); ?
> ************

OK, I tried that. No apparent change. Still unable to step over the call
nResult = ::AfxCtxGetOpenFileName(&m_ofn);
witihin the implementation of INT_PTR CFileDialog::DoModal().

> > I could probably continue, but it seems to me that I am inside the innards
> > of an area of the libs that I have no business in. What is it up in the API
> > level that I am screwing up?
>
> *******
> I have tried stepping into all of the functions until it finally
> displays the dialog, so I really don't know what is going wrong. Those
> are MFC/WIN32 internals and I am not an expert on that...

Likewise.

> Have you checked wether the dialog is under other windows, or outside
> the screen? It shouldn't, but anyway.

I can't seem to find it. The parent dialog *seems* to retain focus. However,
it refuses to repaint itself. Perhaps that is to be expected, anyway? Also,
all my past experience wit hthe CFileDialog is that it is displayed above all
other windows when DoModal() is invoked. At least that is my recollection.
Should it not have top-most position, and focus, in all cases?

How do I check for it outside the screen?

> Another thing that comes to mind, try setting breakpoints in all the
> functions in CFileDialog and see where it stops...

Well, as reported, I can step into the MFC functions (at least to a point).
However, that should be irrelevant, right? I shouldn't have to reverse
engineer MFC in order to use MFC, should I? I never had to in the past -
although I must admit to noobishness.

If you think it would help, I'm certainly willing to dive into the guts of
MFC, to get more specific about where things go astray. It is rather obtuse
code, however.

From: jbreher on 21 Feb 2008 19:51

"Joseph M. Newcomer" wrote:

> This is not a good description. If DoModal "fails", it will return 0. If it succeeds,
> you should see a dialog box.

Sorry for my imprecise language. What shall we call it when an API
invocation does not do what the documentation purports it to do? You are
right, it did not 'fail' in the sense that it did not return a value
indicating failure. It never returned, period. However, I apparently have no
File Open dialog. At least none that I can find.

> Presumably you are going to fill in all those parameters at some point, because you are
> using the defaults, so there is no reason to provide default parameters to the
> constructor.

As suggested by another poster, I have also tried invoking it as CFileDialog
fileOpenBox(TRUE);, with no change in behavior.

> I see no function f() here.

True. Sorry again for my imprecision. f() here is meant to be shorthand for
'the function we are discussing'. More specifically, f() was meant to denote
CFileDialog::DoModal().

Accordingly, the symptons could be described as follows:
Something deep within the call tree of CFileDialog::DoModal() is (apparently)
failing, causing the dialog to never be displayed, and the
CFileDialog::DoModal() never to return.

> What version of VS are you using,

Help>About has the following to say:
Visual Studio 2005 Professional Edition
Microsoft Visual Studio 2005
Version 8.0.50727.762 (SP.050727-7600)

> what platform SDK,

Quite frankly, I don't know how to check this. Is the SDK version not
inferred from the VS version?

> what OS?

System Information says:
Microsoft Wondows XP Professional
5.1.2600 Service Pack 2 Build 2600

> Not enough information here
> to even make a good guess. There are some known issues with using older > Platform SDKs on
> modern operating systems because the size of the m_ofn member changed, but they have
> typically led to unpleasant crashes, usually due to access faults or heap corruption.

How old is old? I guess I just assumed that VS installed whatever it needed
to build working code. I have certainly used this particular MFC API call in
other projects, built on this machine, using this exact installation of VS.
Unless Windows Update changed something germane out from underneath me.

Is there any more info that may be of use in diagnosing the problem? I was
kind of hoping someone could point to some glaring error in the parameter
list of the DoModal call, but I guess nothing there looks obviously amiss?

| Next | Last
Pages: 1 2 3 4 5
Prev: CMFCRibbonButton pressed state, how to do that ?
Next: converting vc++2005 project to VC++2008