If the matrix $A=\left[ \begin{matrix}
   1 & 2 & 3 & 0 \\
   2 & 4 & 3 & 2 \\
   3 & 2 & 1 & 3 \\
   6 & 8 & 7 & \alpha \\
\end{matrix} \right]$ is of rank 3, then $\alpha $ =
(a) 1
(b) 5
(c) 4
(d) -4

Hint: In order to solve the question we need to understand the rank of the matrix. The rank of a matrix is defined as the maximum number of linearly independent row vectors in the matrix. We can also say that the rank of the matrix is defined as the maximum number of linearly independent column vectors in the matrix. Also, there is a property we need to know is that whenever that rank of the matrix is less than the number of rows or columns then the determinant of that matrix is zero.

Complete step-by-step solution:
We have given a 4 X 4 matrix and we have to find the unknown in the matrix given the condition that the rank of the matrix is 3.
Lets first understand by the rank of the matrix.
The rank of a matrix is defined as the maximum number of linearly independent row vectors in the matrix.
We can also say that the rank of the matrix is defined as the maximum number of linearly independent column vectors in the matrix.
We can say that the 4 rows in this matrix represent the 4 equations of the matrix. The rank of this matrix is 3.
This says that only 3 equations are linearly independent with each other and the fourth equation is dependent on any one of the equations.
Thus for this matrix, the determinant of this matrix is zero.
The determinant of the matrix is the scalar value that can be computed from the elements of a square matrix and it gives some of the properties of the linear transformation.
The determinant of A is calculated as follows,
$A=\left[ \begin{matrix}
   1 & 2 & 3 & 0 \\
   2 & 4 & 3 & 2 \\
   3 & 2 & 1 & 3 \\
   6 & 8 & 7 & \alpha \\
\end{matrix} \right]$
Solving this matrix we get,
$A=1\times \left| \begin{matrix}
   4 & 3 & 2 \\
   2 & 1 & 3 \\
   8 & 7 & \alpha \\
\end{matrix} \right|-2\left| \begin{matrix}
   2 & 3 & 2 \\
   3 & 1 & 3 \\
   6 & 7 & \alpha \\
\end{matrix} \right|+3\left| \begin{matrix}
   2 & 4 & 2 \\
   3 & 2 & 3 \\
   6 & 8 & \alpha \\
\end{matrix} \right|-0\left| \begin{matrix}
   2 & 4 & 3 \\
   3 & 2 & 1 \\
   6 & 8 & 7 \\
\end{matrix} \right|$
The last term of the calculation is zero as the matrix is multiplied by zero. Any quantity multiplied by zero gives zero.
Therefore, the equation becomes,
$A=1\times \left| \begin{matrix}
   4 & 3 & 2 \\
   2 & 1 & 3 \\
   8 & 7 & \alpha \\
\end{matrix} \right|-2\left| \begin{matrix}
   2 & 3 & 2 \\
   3 & 1 & 3 \\
   6 & 7 & \alpha \\
\end{matrix} \right|+3\left| \begin{matrix}
   2 & 4 & 2 \\
   3 & 2 & 3 \\
   6 & 8 & \alpha \\
\end{matrix} \right|$
Solving the first term we get,
$\left| \begin{matrix}
   4 & 3 & 2 \\
   2 & 1 & 3 \\
   8 & 7 & \alpha \\
\end{matrix} \right|=4\left| \begin{matrix}
   1 & 3 \\
   7 & \alpha \\
\end{matrix} \right|-3\left| \begin{matrix}
   2 & 3 \\
   8 & \alpha \\
\end{matrix} \right|+2\left| \begin{matrix}
   2 & 1 \\
   8 & 7 \\
\end{matrix} \right|$
Calculating this further we get,
$\begin{align}
  & \left| \begin{matrix}
   4 & 3 & 2 \\
   2 & 1 & 3 \\
   8 & 7 & \alpha \\
\end{matrix} \right|=4\left( \alpha -21 \right)-3\left( 2\alpha -24 \right)+2\left( 14-8 \right) \\
 & =4\alpha -84-6\alpha +72+28-16 \\
 & =-2\alpha
\end{align}$
Solving for the second term,
$\left| \begin{matrix}
   2 & 3 & 2 \\
   3 & 1 & 3 \\
   6 & 7 & \alpha \\
\end{matrix} \right|=2\left| \begin{matrix}
   1 & 3 \\
   7 & \alpha \\
\end{matrix} \right|-3\left| \begin{matrix}
   3 & 3 \\
   6 & \alpha \\
\end{matrix} \right|+2\left| \begin{matrix}
   3 & 1 \\
   6 & 7 \\
\end{matrix} \right|$
Calculating this further we get,
$\begin{align}
  & \left| \begin{matrix}
   2 & 3 & 2 \\
   3 & 1 & 3 \\
   6 & 7 & \alpha \\
\end{matrix} \right|=2\left( \alpha -21 \right)-3\left( 3\alpha -18 \right)+2\left( 21-6 \right) \\
 & =2\alpha -42-9\alpha +54+42-12 \\
 & =-7\alpha +42
\end{align}$
Calculating the last term we get,
$\left| \begin{matrix}
   2 & 4 & 2 \\
   3 & 2 & 3 \\
   6 & 8 & \alpha \\
\end{matrix} \right|=2\left| \begin{matrix}
   2 & 3 \\
   8 & \alpha \\
\end{matrix} \right|-4\left| \begin{matrix}
   3 & 3 \\
   6 & \alpha \\
\end{matrix} \right|+2\left| \begin{matrix}
   3 & 2 \\
   6 & 8 \\
\end{matrix} \right|$
Solving this further we get,
$\begin{align}
  & \left| \begin{matrix}
   2 & 4 & 2 \\
   3 & 2 & 3 \\
   6 & 8 & \alpha \\
\end{matrix} \right|=2\left( 2\alpha -24 \right)-4\left( 3\alpha -18 \right)+2\left( 24-12 \right) \\
 & =4\alpha -48-12\alpha +72+48-24 \\
 & =-8\alpha +48
\end{align}$
Combining all the terms and substituting in the final equation we get,
$A=1\times \left( -2\alpha \right)-2\left( -7\alpha +42 \right)+3\left( -8\alpha +48 \right)$
Now, we need to equate this equation to zero as the determinant is zero.
Therefore, we get,
$1\times \left( -2\alpha \right)-2\left( -7\alpha +42 \right)+3\left( -8\alpha +48 \right)=0
\Rightarrow -2\alpha +14\alpha -84-24\alpha +144=0 \\
\Rightarrow -12\alpha +60=0 \\
\Rightarrow 12\alpha =60 \\
\Rightarrow \alpha =\dfrac{60}{12} \\
\Rightarrow \alpha =5 $
Therefore, the value of $\alpha $ is 5. Hence the option(b) is the anwer

Note: The calculation for this question seems to be quite complicated, as there is no alternative easier way. Also, we need to understand that while simplifying the matrix from a higher order to lower order there is an alternate negative sign starting from the second term. We need to equate the whole equation to zero always whenever the rank is less than the number of rows present in the matrix.