Vjylku

Cayley used the following notation in his original paper
A Memoir on the Theory of Matrices 1

My attempt to reproduce the notation is shown below:

documentclass{article}

usepackage{amsmath,amssymb,graphicx}

begin{document}

scalebox{2.5}{parbox{0.3textwidth}{$$

renewcommandarraystretch{0}setlengtharraycolsep{0pt}

(mathrm{X}, mathrm{Y}, mathrm{Z}){=}lower10pthbox{$begin{array}{r@{}c@{,mkern10mu}c @{,mkern10mu}c@{,mkern10mu}c l}

  (   &a   &b   & c & )\

 lvert& a'  &b' & c' &rvert\

 lvert& a'' & b'' & c'' & rvert\

end{array}$}mkern-6mu(x,y,z),

$$}}

end{document}

As you will observe the delimiters need to be improved. Any suggestions?

Here is an approximation, with basic commands:

documentclass{article}

usepackage{amsmath,amssymb,graphicx}



begin{document}



    [ begin{array}{c@{}ccc@{}c}

  (mkern 4.2mu{} & a &b & c &mkern 4.2mu)\[-0.31ex]

   vline & a' &b' & c' & vline \

   vline & a'' &b'' & c''& vline

end{array} ]%



end{document} 

My proposal:

documentclass{article}

usepackage{amsmath,amssymb,graphicx}



DeclareMathSymbol{A}{mathalpha}{operators}{`A}

DeclareMathSymbol{B}{mathalpha}{operators}{`B}

DeclareMathSymbol{C}{mathalpha}{operators}{`C}

DeclareMathSymbol{D}{mathalpha}{operators}{`D}

DeclareMathSymbol{E}{mathalpha}{operators}{`E}

DeclareMathSymbol{F}{mathalpha}{operators}{`F}

DeclareMathSymbol{G}{mathalpha}{operators}{`G}

DeclareMathSymbol{H}{mathalpha}{operators}{`H}

DeclareMathSymbol{I}{mathalpha}{operators}{`I}

DeclareMathSymbol{J}{mathalpha}{operators}{`J}

DeclareMathSymbol{K}{mathalpha}{operators}{`K}

DeclareMathSymbol{L}{mathalpha}{operators}{`L}

DeclareMathSymbol{M}{mathalpha}{operators}{`M}

DeclareMathSymbol{N}{mathalpha}{operators}{`N}

DeclareMathSymbol{O}{mathalpha}{operators}{`O}

DeclareMathSymbol{P}{mathalpha}{operators}{`P}

DeclareMathSymbol{Q}{mathalpha}{operators}{`Q}

DeclareMathSymbol{R}{mathalpha}{operators}{`R}

DeclareMathSymbol{S}{mathalpha}{operators}{`S}

DeclareMathSymbol{T}{mathalpha}{operators}{`T}

DeclareMathSymbol{U}{mathalpha}{operators}{`U}

DeclareMathSymbol{V}{mathalpha}{operators}{`V}

DeclareMathSymbol{W}{mathalpha}{operators}{`W}

DeclareMathSymbol{X}{mathalpha}{operators}{`X}

DeclareMathSymbol{Y}{mathalpha}{operators}{`Y}

DeclareMathSymbol{Z}{mathalpha}{operators}{`Z}





newsavebox{cayleybox}

newlength{cayleyheight}

newenvironment{cayleymatrix}[1]

 {begin{lrbox}{cayleybox}

  $begin{array}{@{enspace}l*{numexpr#1-1}{@{{,}enspace}l}@{enspace}}}

 {end{array}$end{lrbox}%

  setlength{cayleyheight}{dimexprhtcayleybox+dpcayleybox}%

  kern-0.8pt

  vtop to cayleyheight{hbox{vphantom{$between$}}leadersvrulevfill}%

  raisebox{dimexprhtstrutbox-htcayleybox}{useboxcayleybox}%

  vtop to cayleyheight{hbox{vphantom{$between$}}leadersvrulevfill}%

  kern-0.8pt

 }

newcommand{by}{kern-0.4pt{between}kern-0.4pt}



thinmuskip=6mu

medmuskip=0mu

thickmuskip=0mu



begin{document}



[ % page 17

(X,Y,Z)=

(begin{cayleymatrix}{3}

a   & b   & c \

a'  & b'  & c' \

a'' & b'' & c''

end{cayleymatrix}

by x,y,z)

]



[ % page 32

(begin{cayleymatrix}{2} a & c \ b & d end{cayleymatrix} by

begin{cayleymatrix}{2} a & b \ c & d end{cayleymatrix} by

begin{cayleymatrix}{2} a & c \ b & d end{cayleymatrix} by

begin{cayleymatrix}{2} a & b \ c & d end{cayleymatrix} )

=

bigl(

  (begin{cayleymatrix}{2} a & c \ b & d end{cayleymatrix}by

  begin{cayleymatrix}{2} a & b \ c & d end{cayleymatrix})

bigr)^2

]



end{document}

Extending Bernard's idea slightly:

...but not quite up to the standards of egreg's nice solution:)

documentclass{article}

usepackage{array}

newcounter{cayley}

newcolumntype{L}{>{stepcounter{cayley}%

ifnumvalue{cayley}=1raisebox{-0.4ex}{(}kern0.5exelsevlinefi}c@{}}

  newcolumntype{R}{@{}c<{spaceifnumvalue{cayley}=1kern0.5exraisebox{-0.4ex}{)}elsevlinefi}}

newenvironment{cayley}[1][3]{setcounter{cayley}{0}array{L*{#1}{c}R}}{endarray}



begin{document}



  [

     begin{cayley}

       &a  & b  & c  &\

       &a' & b' & c' &\

       &a''& b''& c''&\

     end{cayley}

     qquad

     begin{cayley}[4]

       &a  & b  & c  & d  &\

       &a' & b' & c' & d' &\

       &a''& b''& c''& d''&\

     end{cayley}

  ]



end{document}

A TikZ proposal. Similarly to Andrew's nice answer you can focus on typing the matrix.

documentclass{article}

usepackage{tikz}

makeatletter% https://tex.stackexchange.com/a/85531/121799

longdefifnodedefined#1#2#3{%

    @ifundefined{pgf@sh@ns@#1}{#3}{#2}%

}

makeatother

newcounter{CM}

usetikzlibrary{matrix}

newcommand{CayleyMatrix}[2][]{stepcounter{CM}%

begin{tikzpicture}[baseline=(mat-numbervalue{CM}-1-1.base),inner sep=2pt,#1]

  matrix[matrix of math nodes,ampersand replacement=&] (mat-numbervalue{CM})

 { #2

  };

 ifnodedefined{mat-numbervalue{CM}-2-1}{%

 draw[semithick] (mat-numbervalue{CM}.south west) -- (mat-numbervalue{CM}.south west|-mat-numbervalue{CM}-1-1.south) to[out=135,in=-135]

 (mat-numbervalue{CM}.south west|-mat-numbervalue{CM}-1-1.north)

 (mat-numbervalue{CM}.south east) -- (mat-numbervalue{CM}.south east|-mat-numbervalue{CM}-1-1.south) to[out=45,in=-45]

 (mat-numbervalue{CM}.south east|-mat-numbervalue{CM}-1-1.north) ;}{

 draw[semithick] (mat-numbervalue{CM}.south west|-mat-numbervalue{CM}-1-1.south) to[out=135,in=-135]

 (mat-numbervalue{CM}.south west|-mat-numbervalue{CM}-1-1.north)

 (mat-numbervalue{CM}.south east|-mat-numbervalue{CM}-1-1.south) to[out=45,in=-45]

 (mat-numbervalue{CM}.south east|-mat-numbervalue{CM}-1-1.north) ;

 }

end{tikzpicture}}

begin{document}

[CayleyMatrix{

X   &Y   & Z \

}=CayleyMatrix{

a   &b   & c \

  a'  &b' & c' \

  a'' & b'' & c'' \

}!!CayleyMatrix{

x   &y   & z \

}]

end{document}