Formula for the number of diagonals in a regular polygon:

D = (n(n-3)/2)

Here D = number of diagonals which is 20

n = number of sides which is not known

20=(n^2-3n)/2

40=n^2-3n

n^2-3n-40= 0

By factorizing the equation you will get

n^2-8n+5n-40=0

n(n-8)+5(n-8)=0

(n-8)(n+5)= 0

So either

n-8=0 or n+5=0

n=8 or n=-5

n can not be negative therefore

n = 8 sides or regular Octagon

D = (n(n-3)/2)

Here D = number of diagonals which is 20

n = number of sides which is not known

20=(n^2-3n)/2

40=n^2-3n

n^2-3n-40= 0

By factorizing the equation you will get

n^2-8n+5n-40=0

n(n-8)+5(n-8)=0

(n-8)(n+5)= 0

So either

n-8=0 or n+5=0

n=8 or n=-5

n can not be negative therefore

n = 8 sides or regular Octagon