I'm surprised by how complex simple things could become. Alt+<key> in vim didn't work, so I copied a script from stackoverflow that makes vim poll the esc key every 50ms, if it is pressed together with another key in this period it is interpreted as Alt. My terminal sends esc+<key> instead of Alt+<key> so it works perfectly.