Luckily the toilet had already been flushed both times. The first time I had it sitting on the sink next to the toilet and accidentally knocked it in and the second time I had it in my sweatshirt pocket and when I turned around to walk out it flew out of my pocket and landed right in the toilet. That's the one that finally broke it.
There's a decent gap between the counter and toilet. So if it did knock off it would fall in the trash.