I added an 18dBI omni-directional antenna to my network, using a 5400 Ovislink AP, to extend the range of my network...
to a nearby building. Unfortunately I found this did not add to the range or strength of my connection. The manufacturer referred to this as a common problem with high-gain antennas because of the near field effect which reduces the performance of this antenna to zero or near zero at short distances. Can you give me an explanation of this effect and problem?
A high-gain antenna is generally used to improve signal strength received over a longer distance. As you found, it rarely results in any improvement for receivers that are close to the transmitter. In fact, receivers that are very close are likely to miss a directional signal if the antenna is not aimed precisely at the receiver.
Why? Think of the directional signal as a cone-shaped wave front radiating from the antenna. A distant receiver is more likely to lie inside the cone, while a receiver very close to the antenna is likely to fall outside the cone. However, you may be able to improve your situation by reorienting the antenna to better focus on your intended receiver. To learn more about near field, see this Wikipedia entry.