It's still not exactly what you want, but since you can check the
$ITERM_SESSION_ID environment variable to see which panel you are in, by combining this with a saved window arrangement, you can have each of your panel to start with different initial path, or just anything you would like to run by default.
For example, my saved windows arrangment is two tabs with two panels in the first tab. So I will have
$ITERM_SESSION_ID=w0t0p0 for the first panel in the first tab,
$ITERM_SESSION_ID=w0t0p1 for the 2nd panel, and
$ITERM_SESSION_ID=w0t1p0 for the single panel in the 2nd tab. 'w' stands for window, 't' for tab, and 'p' for panel. I check these variables in my
.zshrc, and run Vim in the w0t0p0, iPython in the w0t1p0, respectively.