When App Exposé shows my browser windows (i.e., 4-finger swipe down or Control-↓), the order of the windows changes from one time to the next, and I cannot find the window I'm looking for easily. (What I'm really after is an efficient way to flip back and forth between the two or three most recently used windows in an application, but that's another story.) Can anyone tell me if there is any logic to the ordering of the application windows in App Exposé, or if it is just random?
I myself was looking for the answer to this this afternoon, but unfortunately it wasn't answered here, so I tried to figure it out myself with some experimentation, and this answer is what I came up with. FYI, I was doing this with OS 10.9 with somewhere between 2-9 windows open.
The application windows, in what I'll call "view mode", are presented, more or less, in a rectangular grid. If I have 8 windows open, view mode is a 2x4 grid. The main preference for the algorithm seems to be to try to approximate the relative positions of the windows on the screen. If the windows are cleanly ordered on the screen, then the position of the windows in view mode will be completely determined by their order on screen (e.g., the top left window will be the top left window), regardless of how they are layered (e.g., which is currently on top (in front)).
However, it often happens, particularly if you have a lot of overlapping windows open, that their positions cannot be cleanly put in a rectangular grid. (Think about the case where you have 2 windows open, one small and one big, where the small window is positions completely in the center of the big window.) In this case, the layering comes into play. I tried with just 2 windows of the same size open, occupying the exact same position on screen. In this case the one in front (on top) appeared on the left. (If they were not in the same position, which one is currently in front does not affect this.)
Now if you have many windows open, and there is a lot of overlapping with ill-defined relative positioning, then these rules combine in a way such that changing which windows are in front may not change the order in the "view mode" at all, or it can mix things up quite a bit, so that the order changes in an apparently random way. My guess is that the order in view mode is completely determined by the "3-d positions" of the windows (2-d positions on screen + positions in layers), but it becomes complicated to understand completely when you have more than a few windows open.