What seperates them?, the problem i think is, whatever definition traits you give to progressive metal, are also too synonymous with avant garde metal aswell.   Avant garde metal is not as explored as progressive metal, i could  name many prog metal bands, but only 4 avant garde bands.   The 4 avant garde bands i can name are Celtic Frost (possibly the band that started the genre), Borknager, Kekal and Sigh.   Now......apart from Celtic Frost, all the other bands i mentioned actually have pages on Progressive Archives.

This is what i think seperates Progressive metal from avant garde metal............Progressive metal is Traditional Heavy Metal or Power Metal based, Avant Garde is Thrash metal and/or Black metal based.   Voivid are possibly avant garde aswell, considering they started off being a thrash metal band.

Progressive metal refers to metal that has adopted characteristics from progressive rock i.e. time signatures, expansive concepts, sophisticated compositions etc

Avant simply refers to metal that falls outside of the parameters of any designated genres.

Of course many artists can be BOTH

RYM definition of avant

Avant-Garde Metal, also referred to as Experimental Metal, is a subgenre of https://rateyourmusic.com/genre/metal/" rel="nofollow - Metal music that first appeared in central Europe in the mid 1980s. The English translation of the French word Avant-Garde is literally vanguard, meaning the forefront of a movement or activity. As such, the genre incorporates a plethora of irregular and nonconventional sounds, song structures, and instrumentation, characterizing a vastly experimental sound that often integrates several genres into one.

It is important to note, however, that as the https://rateyourmusic.com/genre/metal/" rel="nofollow - Metal genre as a whole progresses, certain bands that are currently deemed "avant-garde" may not be considered so in the future, and may even be setting a precedent for a new sound altogether. The application of the term "avant-garde" is relative to time period and context, such that modern "avant-garde" bands may not be categorized as "avant-garde" in the future. For example, https://rateyourmusic.com/artist/celtic-frost" rel="nofollow - Celtic Frost 's earlier works were originally considered experimental and avant-garde for the time period, but two decades later we now consider the albums as simply Blackened https://rateyourmusic.com/genre/thrash-metal/" rel="nofollow - Thrash Metal . It should be clear that Avant-Garde Metal bands of today may not always be described as such in the future, and in certain cases may even be described as something more specific in the future if their sound is influential enough to generate a new genre or subgenre.

Sleepytime Gorilla Museum

Mr Bungle

Igorrr

Kayo Dot

Unexpect

Stolen Babies

Behold...The Arctopus

Five of my favorites.  Guess I'm an avant garde metal girl.  Never heard of Igorrr or Stolen Babies.