Baroness McIntosh of Pickering wants a law change so cyclists are treated just like car drivers when it comes to using a phone, so that offenders are prosecuted.
There is no specific offence for a cyclist using a phone, but bike-riders can be prosecuted for careless or dangerous cycling.
Fines of up to £2,500 can be dished out for such offences.
It comes after Highway Code changes, designed to protect pedestrians and cyclists, came into force in January.
Rule 149 of the code requires motorists to “exercise proper control of your vehicle at all times” and bans the use of mobile phones while driving.
Cyclists in London
Kirsty O'Connor
Highway Code changes have proved controversial
Jacob King
But Baroness McIntosh questioned why this rule does not apply to cyclists and other road-users.
She said she had recently been walking to the Houses of Parliament and as she was crossing the road, a cyclist travelled towards her using a mobile phone.
The cyclist had "one hand bicycling, one hand on the mobile phone, on the wrong side of the road".
Baroness McIntosh added: "I wasn't clear whether he was going to stop or not."
She alleged that some cyclists and e-scooter riders use their phones “inappropriately” as she asked why the Department of Transport has not addressed the issue.
The former minister said e-scooters were "even more of a concern" with some people left feeling "absolutely terrorised" by those using them "irresponsibly".
The 67-year-old called on the Government to "create criminal offences relating to dangerous, careless and inconsiderate cycling for those users of pedal bikes, electronically assisted bikes and e-scooters".
Transport minister Baroness Vere of Norbiton said existing laws already banned cyclists and e-scooter riders from using mobile phones.
She said: "It is not a specific offence to cycle and use a mobile phone or headphones, but cyclists and e-bike riders can be prosecuted by the police for careless and dangerous cycling with maximum fines of £1,000 and £2,500 respectively."
She added that the regulations in the Government's current e-scooter trial already state that is an offence to use a handheld mobile phone while operating one.
Rule-breakers could be fined and get six penalty points, she added.
It comes after a poll found almost two-thirds of motorists think that the Government’s new “hierarchy of road users” will cause more conflict on the road.
Road safety charity IAM Roadsmart surveyed 1,000 motorists and found 59 percent believed the new system will cause more conflict on the road, rather than make them safer.
Just six percent said things would improve, with 13 percent saying there would be no difference.
