yes, i would remove a little at a time. a few a week or so. i would put some live rock rubble in there and maybe some chaeto macro.
on another note, i have read that putting rock rubble in pretty much the same as putting bio balls and should be avoided. anyone else read about this?