Ytukyg

Let $H<G$ be finite groups. Must there be a positive integer $n$ such that there exists a homomorphism $f:Gto S_n$ with $f(G)$ transitive and $f(H)$ not transitive?

If $H$ is a normal subgroup of $G$, then we can consider $G/Hhookrightarrow S_{lvert G/Hrvert}$.

Lord Shark the Unknown has already answered the question, but here is another answer:

Let $n = [G:H]$, and define $phi:Grightarrow S_n$ as $G$'s left translation action on the left cosets of $H$. The $G$-action is transitive, but the action restricted to $H$ has the coset $H$ as a fixed point.

A generalization encompassing both answers is: let $K$ be any subgroup of $H$; define $n=[G:K]$, and define $phi:Grightarrow S_n$ as the left-translation action on the left cosets of $K$. As before, the $G$-action is transitive, but the restriction to $H$ cannot move the coset $K$ to any coset of $K$ lying outside $H$.

The above takes $K=H$, and Lord Shark the Unknown's answer takes $K={1}$.

Addendum 6/28/18: Yet another answer. Let $K$ be any proper subgroup of $G$ containing $H$. Just as with $K=H$, the $G$-action is transitive while the $H$-action has $K$ as a fixed point.

This prompts me to ask a more comprehensive version of the question. Let $Hsubset G$ be a proper subgroup and let $Grightarrow S_n$ be a transitive action. Under what circumstances is the restriction to $H$ transitive?

Since any transitive action is isomorphic to a translation action on left cosets, there is a subgroup $K$ such that $n=[G:K]$ and we can identify the elements of the set being acted on with left cosets of $K$. Then the action of $H$ is transitive if and only if $H$ meets each of $K$'s cosets. $H$ containing and being contained in $K$ are both ways to guarantee that $H$ fails to meet all of $K$'s left cosets, but they are not the only ways this can happen.

Addendum 12/10/18: Keeping notation as in the previous addendum ($K$ is any subgroup of $G$, giving a transitive action by $G$ on the left cosets of $K$; $H$ is a specific subgroup and we wish to know if the restriction of the action to $H$ is also transitive), we know the restriction to $H$ is transitive if $H$ meets every coset of $K$.

This is equivalent to $HK = G$. The above results can be explained as $Ksubset H Rightarrow HK = H <G$, and $Hsubset K Rightarrow HK = K < G$. But anything that guarantees $HK < G$ will also do, e.g. if $|H||K|<|G|$. Thus take $K$ to be any subgroup with order $<$ the index of $H$.

Let $n=|G|$. Then the regular representation of $G$ is a homomorphism
$f:Gto S_n$ whose image is simply transitive. If $H$ is a proper subgroup
of $G$ then $f(H)$ is a non-transitive subgroup of $S_n$.