Some admissible gauge groups of N=4 Chern-Simons gauged supergravity in threedimensions with exceptional scalar manifolds $G_{2(2)}/SO(4)$,$F_{4(4)}/USp(6)\times SU(2)$, $E_{6(2)}/SU(6)\times SU(2)$,$E_{7(-5)}/SO(12)\times SU(2)$ and $E_{8(-24)}/E_7\times SU(2)$ are identified.In particular, a complete list of all possible gauge groups is given for thetheory with $G_{2(2)}/SO(4)$ coset space. We also study scalar potentials forall of these gauge groups and find some critical points. In the case of$F_{4(4)}/USp(6)\times SU(2)$ target space, we give some semisimple gaugegroups which are maximal subgroups of $F_{4(4)}$. Most importantly, weconstruct the $SO(4)\ltimes \mathbf{T}^6$ gauged supergravity which isequivalent to N=4 SO(4) Yang-Mills gauged supergravity. The latter is proposedto be obtained from an $S^3$ reduction of $(1,0)$ six dimensional supergravitycoupled to two vector and two tensor multiplets. The scalar potential of thistheory on the scalar fields which are invariant under SO(4) is explicitlycomputed. Depending on the value of the coupling constants, the theory admitsboth dS and AdS vacua when all of the 28 scalars vanish. The maximal N=4supersymmetric $AdS_3$ should correspond to the $AdS_3\times S^3$ solution ofthe $(1,0)$ six dimensional theory. Finally, some gauge groups of the theorieswith $E_{6(2)}/SU(6)\times SU(2)$, $E_{7(-5)}/SO(12)\times SU(2)$ and$E_{8(-24)}/E_7\times SU(2)$ scalar manifolds are identified.