"My inappropriate remarks caused turmoil," he admitted.

Mori had been in charge of the Olympic Organising Committee since 2014, but had he remained in office, Olympic protocol would have required him to make a speech at July’s Opening Ceremony of a Games where 49 per cent of the competitors are expected to be women. This would have been awkward to say the least. The audience would also have included around 36 female members of the International Olympic Committee (IOC), the organisation which this week had branded his comments "absolutely inappropriate".

The Japan Times newspaper has reported that some 500 Games volunteers resigned and three runners have withdrawn from Tokyo’s Olympic Torch Relay, due to begin next month, because of Mori's comments.

Tokyo governor Yuriko Koike stayed away from an Olympic meeting this week and Masako Ōkawara, a member of the Japanese House of Representatives, instigated a protest where her colleagues wore white to "show that derogatory comments about women are unacceptable".

This year's Olympics are set to be the second to be held in Tokyo, but the city has been awarded the Games on three occasions. 

It had been selected to host the 1940 Olympics which never took place because of war. Then, every member of the Organising Committee was male. The only women to appear in the official report produced were clerical staff.

It was a similar story with the management of the 1964 Games, although these were hailed as a great success. Women such as American swimmer Donna de Varona, Czech gymnast Věra Čáslavská and Britain's Mary Rand in athletics were star performers.

Yet even in the "swinging sixties" the Olympic Movement remained a man’s world. It was not until 1981, almost 87 years after the foundation of the IOC, that the first two female members were chosen.

No women had taken part at the first Olympics of the modern era in Athens 125 years ago. Baron Pierre de Coubertin, the driving force in the early years of the Olympic Movement, insisted: "The only true Olympic hero is the individual male adult", a position he maintained for the whole of his life.

He wrote of a "wariness of feminism", but in spite of his opposition, women competed in croquet, golf, sailing and tennis at the 1900 Olympics.

The first female champion was Lottie Cooper from Britain in tennis singles and mixed doubles. American Margaret Abbott became the first female Olympic golf champion at the same Games.

At the 1908 Olympics in London, 53 year old Sybil "Queenie" Newall won archery gold, ahead of Lottie Dod, better known for her tennis exploits at Wimbledon.

Organisers presented their plans for the 1912 Stockholm Olympics during the 1910 IOC Session in Luxembourg. "Ladies competitions were proposed from certain quarters. In consequence of the wishes expressed at the Luxembourg meeting, the detailed programme was amplified as to include a 100 metres' swimming race for ladies and a ladies’ high diving competition", said the official report.

At the next IOC Session in Budapest, the programme was extended to include a women’s 4x100m freestyle relay and a diving competition.

Some 40 women eventually took part in aquatics. They were described as "lady competitors" and those from abroad were assigned "a caretaker who shall be responsible for the ladies’ dressing-rooms being in perfect order, and for the respective competitors being called in good time for the various events"

Australia’s Sarah "Fanny" Durack won the 100m freestyle and Greta Johansson of Sweden became the first female diving champion. She also competed in the swimming events but women were still not permitted in athletics.

In many countries they were also denied the vote. This was the case in Switzerland, where the Olympic headquarters were soon established, and in Coubertin’s homeland France. There, Alice Milliat, a rower, established a sporting association for women.

In 1921, she organised an international women’s event at the Casino Gardens in Monaco. She also founded an international women’s sport federation with Sophie Elliott Lynn, a pioneering athlete and aviator from Britain.

A more ambitious event still, described as "Une Olympiade Feminine", was held at the Stade Pershing in Paris in 1922. The use of the Olympic word was bound to catch the eye of the IOC.

Comte Justinien de Clary was soon advising his IOC colleagues that he had had to intervene to prevent the use of this unauthorised term. The activities of what the IOC described as "the feminist movement" were discussed extensively.

As a result the Olympic Charter now included the words: "Women are admitted to the Olympic Games. The programme will establish the events in which they may compete."

Female fencers competed in the foil for the first time in 1924, although not all the IOC members were convinced this should continue. At the 1926 IOC session in Lisbon, Sigfrid Edström, also president of World Athletics, advised members that his organisation had been asked to allow women to compete. He wanted the IOC’s agreement in principle.

The handwritten IOC minutes state "the committee will permit the admission of women in a restricted number of events at the Olympic Games."

Milliat had persistently lobbied Edström and his organisation, then known as the International Amateur Athletics Federation (IAAF). The governing body’s official history later asserted that "no other issue ignited and divided the membership to such an extent."

Agreement was reached that women’s athletics would come under IAAF jurisdiction and women would be allowed to take part in the 1928 Olympics, but the programme was very limited. The longest race was only over 800m. It was won by Germany’s Lina Radke. Her time, 2min 26.0sec, was a new world record.

A Dr Bergman, examining physician for the German team, said that "competition had not affected Radke’s system. Frau Radke cooked, sewed and kept house like any other house wife."

The silver medallist was Kinue Hitomi, the first Japanese woman to take part in the Olympics. Inga Gentzel of Sweden took bronze but the Gazette de Lausanne relayed that other runners "fell at the finish completely exhausted. The sight of these young women collapsing on the grass, exhausted, provoked various comments."

Amongst these was a description by John Tunis, later to become a novelist for New York evening papers, which described "eleven wretched women".

Harold Abrahams, Britain's 1924 Olympic 100m gold medallist, was also watching. He branded some reports sensationalist but still wrote: "Reaction has been hostile to repeating the women’s events provided during the ninth Olympiad. Women are apt to break down for reasons not instantly clear to the masculine understanding."

Abrahams suggested it was "a result of more psychological than physical causes."

Former IOC President Coubertin weighed in. "The experience of Amsterdam seems to have justified my opposition to allowing women into the Olympic Games."

When the IAAF met, there was a call for women’s athletics to be removed altogether. Although this was voted down, the 800m was taken off the programme. The longest distance that women were permitted to run was set at 200m, a restriction which remained in force until 1960.

In 1929, the year after Amsterdam, the IOC gathered in Lausanne. Lieutenant Colonel Andre Berdez administered the Session, assisted by the Mayor of Lausanne’s personal secretary, Marie-Louise Fauvannat. Her friend Lydia Zanchi agreed to help out, the start of an association which would last for almost 40 years.

"I ran up and down liaising between Colonel Berdez and my friend. I found it interesting," Zanchi later told the Olympic Review.

Zanchi was also an activist for female suffrage in Switzerland, still a distant prospect in 1929. Her thoughts on certain points of view can only be imagined.

Yvar Nyholm of Denmark sent a note to IOC President Henri Baillet-Latour which informed him of a meeting of Scandinavian sports officials.

"A resolution was passed urging a complete suppression of all women’s events from the Games."

Finland’s Ernst Krogius wrote in similar vein from Helsinki that "the Finland Olympic Committee had voted for the exclusion of women entirely from the Games."

Despite these moves, the IOC decided that in 1932, women could compete in ice skating, athletics, fencing, gymnastics and swimming.

Meanwhile, "a new feminine athletic marvel" was emerging. New York Times columnist Arthur Daley was writing about American Mildred "Babe" Didrikson. She won gold in 80m hurdles and javelin and a silver in the high jump. Later in the decade, Didrikson proved an outstanding golfer after being disbarred from athletics when her photograph appeared in a magazine advertisement.

Milliat’s women’s Games continued in the 1930s as the Olympic programme remained restricted.

The attitude of some was reflected by American sportswriter Paul Gallico.

"Females who run and jump in track meets are just wasting their time, and ours, because they can't run fast enough, jump high enough or throw things far enough to matter. If girls played basketball under men's rules, they would be taken away on stretchers after five minutes."

In May 1934, the IOC Executive Committee met in Brussels. The "admission of Women was asked for" by the federations for athletics, swimming, fencing, gymnastics and equestrianism.

When the full Session met in Athens that summer, there was further discussion. Nine IOC members also voted against women’s participation in athletics.

Women’s skiing was added although by a narrow margin, though a bid to include ice hockey was rejected.

When women did take part in the Games, they were subject to strict chaperoning. In 1936, swimmer Eleanor Holm was expelled from the American team for breaking curfew on the sea voyage.

The Riverside Daily Press, a Californian newspaper, reported that "Gay cocktail parties result in dismissal. Swimming queen defiant when told to keep training rules."

It is hard to envisage a male swimmer being reported in quite the same way.

When fully 30 years later, gender verification was introduced, this only applied to female competitors. Many regarded the "sex test" as invasive.

As late as 1953, an IOC agenda still included a proposal to exclude women’s events.

Although this was rejected "unanimously", IOC President Avery Brundage announced "women competitors should only be accepted in sports appropriated (sic) to them."

By now, women riders were permitted in equestrianism. They were allowed into dressage in 1952, jumping in 1956 and then finally eventing at Tokyo 1964. Those Games also saw the introduction of volleyball for both men and women.

The entry of women’s hockey was delayed until 1980 because of the delay in establishing a single International Federation which encompassed both men and women.

Women’s cycling did not appear until 1984 and many were frustrated that reaching agreement for the inclusion of a women’s marathon proved a long, drawn-out affair.

Weightlifting and wrestling, both on the original programme in 1896, had to await the new millennium before women were allowed to participate.

Not until Gianna Angelopoulos-Daskalaki took charge of Athens 2004 did a woman head an Organising Committee.

The IOC did not number any women amongst its members until 1981, but Monique Berlioux, an Olympian for France, held the powerful position of IOC director throughout the 1970s and early 1980s, a role which polarised opinion.

Swimmer Kirsty Coventry is the latest woman to head the IOC Athletes' Commission, but although there have been female vice-presidents, only American Anita DeFrantz has so far run for the IOC Presidency.

Italian Giuliana Minuzzo became the first woman to speak at an Olympic ceremony when she took the Olympic Oath at the 1956 Winter Olympics in Cortina d'Ampezzo.

That year women also took a leading role in carrying the Flame to the Equestrian Games in Stockholm before male dressage rider Hans Wikne ignited the main cauldron. Gymnast Karin Lindberg then carried a Flame to light a secondary bowl.

This was 12 years before Mexican runner Enriqueta Basilio carried the final Olympic Torch in Mexico City.

Last year, Greek shooter Anna Korakaki became the first woman to run with the Olympic Flame in the precincts of the ancient Olympic stadium. She handed the Flame to 2004 women’s marathon champion Mizuki Noguchi.

Another woman, 1996 Olympic swimmer Naoko Imoto, represented Tokyo in the Panathinaiko Stadium at the Flame handover.

The IOC is insistent that women will be prominent at the 2020 ceremonies. It has agreed that each country will be able to "to have their flag carried by one female and one male athlete at the Opening Ceremony."

It might come as a surprise to learn that a woman did so back at the Lake Placid Winter Olympics 1932. Britain’s Mollie Phillips was part of an all-female team at those Games.

That summer Mexico’s flag was carried by fencer Eugenia Escudero, one of only two women in their team.

For Tokyo 2020, the IOC has "requested" each National Olympic Committee to include "at least one female and one male athlete in their respective teams".

It might not come as a surprise if there are also joint lighters of the Olympic Cauldron this July.

The immediate concern for Tokyo 2020 is the choice of Mori’s successor. A Candidate Review Committee, which will have equal gender representation, has been set up to oversee the appointment of Mori's successor.

"All eyes are on the transparency of the selection process," Governor Koike said on Friday. "The result will largely impact the image Tokyo broadcasts to the rest of the world."