Wearing size L, 182cm

I'm wearing a button-down shirt (size L) and I'm 182cm tall and weigh 70kg. I'm slim. If I'm wearing a jacket, size L would be fine, but if I want a more casual, loose fit, size XL would be fine.