The case was brought by two men who challenged police use of rectal inspections after undergoing the procedure when being investigated for homosexuality, which is illegal in Kenya.
Being gay can carry a prison sentence of up to 14 years in the country, although prosecutions are rare.
“There was no other way evidence could have been obtained to ascertain that they are gay without carrying out anal analysis,” Judge Anyara Emukule said in a ruling at the High Court in the port city of Mombasa.
The men are expected to appeal the decision.
Eric Gitari, head of the National Gay and Lesbian Human Rights Commission in Kenya, called the tests “humiliating”.
The ban on homosexuality “has flooded Kenyan society with waters of prejudice, hatred and shame”, he wrote in Newsweek ahead of the ruling.
Homophobia is on the rise in Africa, and taking an anti-gay position while espousing evangelical Christian values is a major vote winner in many countries on the continent.
Gay rights activists have warned of rising intolerance in Kenya, including attacks on homosexuals and alleged cases of lesbians being raped to “cure” them.
Human Rights Watch wrote in a recent report that discrimination against homosexuals in Kenya “remains a major problem”, and that the authorities’ “response to mob attacks and other forms of anti-gay violence has been limited.