I have met many people from countries who were not colonized by Britain speaking good English. When I inquired how they said they learnt in secondary school.
How come that we were taught French in secondary school yet you can hardly see any Nigerian communicating in French just by learning in secondary school?

The English we speak is not learnt in school, it is actually learnt at home and everywhere else. It's only grammar and structure etc that school helps to straighten. You learn a language faster if everyone around you is speaking it for everyday use, not just as a subject in school. That's why most people still don't know how to speak French, because they treat it only as a subject (which starts and ends in school).