I think you are looking for a technique called soft particles. From a Nvidia resource:
Particle sprites are widely used in most games. They are used for a variety of semitransparent effects such as fog, smoke, dust, magic spells and explosions. However, sprites commonly produce artifacts – unnaturally sharp edges – where they intersect the rest of the scene.
We present a simple way to fade out flat sprites against a 3D scene, softening the artificial edges. Two solutions are implemented: one uses the ability of DirectX10 to read the depth buffer as a texture; the other uses a more conventional second render target to store depth values.
@korner already experimented with soft particles right here: Soft Particles render