If WOL is based only on the magic packet, only a specifically formed command will be able to wake up the box.
If conversely any packet sent to the box's IP address will trigger a resume condition, it's likely that on your local network there are sometimes broadcast packets (e.g. network discovery messages) that will be received and managed by all the connected devices, and this could cause the awakening from standby.