Don't pick up empty boxes from ground when in "Grab Mode"

If I use a box for sorting under a table, when I then grab from it I use "Grab Mode" as there is no other way to get items out of it. But when the box is empty I end up grabbing the box, then having to place it back down, sometimes hard due to limited reach and angle to place items.