Is a black box over text in a PDF safe?
Often not: text under a drawn black box can still be copied. PDF by Zaweba finds unsafe redactions and turns them into proper ones that remove the text.

No, not usually. A black box drawn over text in a PDF hides the words on screen, but the text underneath is often still in the file and can be selected, copied or found by search. PDF by Zaweba has Check for unsafe redactions, which finds these boxes and offers to mark them so Apply redactions really removes the text. The app is free and works offline on Windows.
This mistake is common and has caught out plenty of people who meant well.
Why a box doesn't remove anything
A PDF page is built in layers. The text is one thing; a rectangle drawn on top is another. A black rectangle covers the words visually, the way a sticky label covers writing on paper, but the words are still underneath. Select across the box and copy, then paste into a text editor, and they often reappear.
The same goes for white boxes, which are even harder to spot, and for opaque comment boxes placed over text, since comments sit on top of the page and can be moved or hidden. And a document can have redaction marks that were placed but never applied, which look like redactions but have removed nothing.
How to check a document
Open it in PDF by Zaweba and run Check for unsafe redactions. It looks for black or white boxes drawn over text that can still be copied, opaque comment boxes over text, and redaction marks never applied. For each one it finds, it offers to mark it, so that Apply redactions removes the text properly.
You can also test by hand: try selecting text where the box is, or search for a word you expect to be hidden. If anything turns up, the redaction isn't safe.
Making it safe
Accept the marks the check offers, look over the rest of the document for anything else that should go, then Apply redactions. The text and picture pixels under the marks are removed from the file, not just covered. Save as a new file.
Then remove what else the file carries. Protect, Remove hidden information lists document details, XMP, attached files, comments, invisible text, content off the page, scripts and earlier versions, with counts. Earlier versions matter especially, because a file saved incrementally may still contain an older copy without the boxes. Sanitise document removes all of it in one step.
When to run the check
Run it on anything you're about to release that someone else redacted: disclosure bundles, responses to information requests, HR files, documents for court. Run it on your own work too, in case a box was drawn with a shape tool instead of a redaction tool.
It's also worth running on a document you've been sent, before you forward it. If the sender's redactions aren't safe, they'd want to know before it goes any further.
Doing it properly from the start
When you redact in PDF by Zaweba, use the redaction tools in the Protect menu rather than drawing shapes. Mark text, areas or pages, or use Find and mark for emails, card numbers, phone numbers and identity numbers, then Apply. Done that way, there is nothing left under the box to find.