Vtyjkyuk

I use the following definitions:

The set $Asubset X$ is said to be dense in $X$ if $bar A=X$.

The set $Asubset X$ is said to be nowhere dense in $X$ if $(bar A)^circ=emptyset$.

Yesterday I read a post saying that a set $A$ is nowhere dense in $X$ if and only if $A$ is not dense in any nonempty open subset of $X$.

If $A$ is nowhere dense, by the definition I follow, $(bar A)^circ=emptyset$. It implies that $bar A$ is not a nonempty open set. That is, $bar A≠G$ where $G≠emptyset$ is open. So $A$ is not dense in any nonempty open subset of $X$. I'm confused about the converse part :

Take $A=[0,1]$. Then $(bar A)^circ=(0,1)≠emptyset$. So $A$ isn't nowhere dense. Then $A$ (according to the post) must be dense in some nonempty open subset of $X$. What is that set? $bar A$ is not an open set. How can the statement be true?

Regarding your confusion for the converse part, $A$ is dense in the non-empty open set $G = (0,1)$. This is because by the definition of denseness, $A$ is dense in $G$ iff $A cap G$ is dense in $G$. That is, iff $G cap (overlineA cap G) = G$. Here, $A cap G = G$, and $overlineA cap G = A$, so $G cap (overlineA cap G) = G$.

Additionally, the proof you have given for the forward implication is slightly incorrect. You say that $(barA)^0 = emptyset implies barA$ is not a nonempty open set. But this is assuming too much; the correct implication is: $(barA)^0 = emptyset implies barA$ does not contain any nonempty open set.

To correct the proof, one could work as follows. Let $(barA)^0 = emptyset$. Then, $barA$ does not contain any nonempty open subset of $X$. Let $G$ be a nonempty open subset of $X$. If $A$ is dense in $G$, then $$
G cap (overlineA cap G) = G.
$$
But,
$$
A cap G subset A implies overlineA cap G subset barA.
$$
Therefore, $G cap (overlineA cap G) subset G cap barA subset G$. Hence,
$$
G cap (overlineA cap G) = G implies G cap barA = G implies G subset barA.
$$
But this contradicts that $barA$ does not contain any nonempty open subset of $X$. Hence, $A$ is not dense in $G$.

Further comments: Your proof breaks down at the point where you conclude from $barA$ not being a (nonempty) open set to the fact that $A$ is not dense in any nonempty open subset of $X$. In general this is not true.

To see why, we can take the same sets you considered for the reverse implication. Let $X = mathbbR$, $A = [0,1]$. Then $barA = A$, which is not a nonempty open set. But $A$ is still dense in the set $G = (0,1)$. So, just having that $barA$ is not a nonempty open set does not imply that $A$ is not dense in any nonempty subset of $X$.