Unable to execute shellcode in stack with no DEP

Unable to execute shellcode in stack with no DEP

Here is the simplest example to show my problem. Everything is in WinXp sp3 with DEP disabled.

char f() {
    // shellcode starting with some NOPs,
    char shellcode[400]= {
            0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 
            0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 
            0x66, ... some basic shellcode here ...   0x30. 
            0xCC
    };

    // Overwriting Return address in stack to return to NOPs in shellcode above
    b[404]=0xf0;
    b[405]=0xfb;
    b[406]=0x12;
    b[407]=0x00;
}

This ended with failure:

(920.e7c): Illegal instruction - code c000001d (!!! second chance !!!)
eax=7c801d7b ebx=7c80262c ecx=7c801bfa edx=00060002 esi=00000000 edi=7c802654
eip=0012fc15 esp=0012ff08 ebp=7e410000 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
0012fc15 d9907ca5e490    fst     dword ptr [eax-6F1B5A84h] ds:0023:0d64c2f7=????????

But if I execute the shellcode directly, it works:

char f() {
    // shellcode starting with some NOPs,
    char shellcode[400]= {
            0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90,  
            0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90,
            0x66, ... some basic shellcode here ...   0x30. 
            0xCC
    };

    void *exec = shellcode;

    ((void(*)())exec)();
}

This will work fine.

I did trace the execution in the above two cases. And in both cases, EIP did successfully landed on the NOPs area and continue execution. But I don't know why the first case failed.

One of guess is that in the first case, when f() returns to 0x0012fbf0, the shellcode in the area outside of the stack frame(ie. at that point, the ESP is below the SHELLCODE area) .

Anyone can give an idea why the first case failed?