As long as he can't log into your router and accesses your wi-fi on his own device shouldn't you be safe from him?