I'd like to suggest Japanese band Boris for either RIO/Avant-prog or Experimental/post-metal, or even prog related. Their albums from their early years lean extraordinarily towards the experimental side of rock-
Absolutego, Amplifier Worship, Flood and
Feedbacker are all composed of one massive, epic song that stays consistent in structure but shifts and piles melodies on top of each other as time goes on. They've also collaborated with noted experimental artists Keji Haino and Merzbow, and have a tendency to shift styles every other album:
Smile is a relatively straightforward noise/stoner doom album, but it was followed immediately by
Cloud Chamber, a collaboration with Michio Kurihara that can more aptly be described as a psychedelic rock album with classical overtones!
Anyway, here are a few Boris tracks from across their strange, prolific career, just to show you the vast multitude of styles they play in and show off their avant-garde nature:
Since Boris has over 30 full length releases including live albums and collaborations(and collaborative live albums) it's tough to condense their sound, but I think I've provided a good cross section. They may not be prog in the strictest definition of the word, but they embody the progressive spirit right down to its roots, and I've seen bands added here before for just that reason!
Whether you decide to include them or not, thanks a ton for considering them.