Finalizing the HBAO

This week I finished to fully implement the HBAO effect, following we have  the effect xml file:

<posteffect>
    <layer name="hbao" shader="/shader/postproc/HBAO/HBAO.xml" downsample="0">
        <parameter name="angle bias" type="float">0.52359877</parameter>
        <parameter name="sqr radius" type="float">0.01</parameter>
        <parameter name="inv radius" type="float">10</parameter>
        <parameter name="radius" type="float">0.1</parameter>
        <parameter name="num steps" type="float">25</parameter>
        <parameter name="num directions" type="float">32</parameter>
        <parameter name="contrast" type="float">2.3</parameter>
        <parameter name="attenuation" type="float">1.0</parameter>

        <input source="/data/posteffects/CosSinJitter.bmp" texname="tex csj"/>
        <output name="out" format="argb8" />
    </layer>

    <layer name="blurX" shader="/shader/postproc/HBAO/BilateralBlurX.xml">
        <parameter name="blur radius" type="float">7</parameter>
        <parameter name="blur falloff" type="float">0.010204081632</parameter> <!-- 1 / (2*radius*radius)-->
        <parameter name="sharpness" type="float">100000</parameter>

        <input layer="hbao.out" texname="tex diffuse"/>
        <output name="out" format="argb8" />
    </layer>

    <layer name="blurY" shader="/shader/postproc/HBAO/BilateralBlurY.xml">
        <parameter name="blur radius" type="float">7</parameter>
        <parameter name="blur falloff" type="float">0.010204081632</parameter> <!-- 1 / (2*radius*radius)-->
        <parameter name="sharpness" type="float">100000</parameter>

        <input layer="blurX.out" texname="tex diffuse"/>
        <output name="out" format="argb8" />
    </layer>

    <layer name="combine" shader="/shader/postproc/HBAO/combine.xml">
        <input layer="*screen" texname="tex diffuse"/>
        <input layer="blurY.out" texname="tex ao"/>
    </layer>
</posteffect>

So, it contains 4 layer:

• The HBAO - is the main layer, as the original effect uses a buffer(dx10) with precomputed random (cosine,sine,jitter) I baked it into a texture. In the parameters we have, the radius (in which the texture will be sampled), the number of directions and how much steps in each direction will be calculated. 
• Blur X and Y - is the bilateral blur, this is, a gaussian blur weigthed by the depth diferrence between the samples, the sharpnes parameter controls how much the difference affect the weight 
• Combine - this layers just combines the AO with the color, it can be done directly in the BlurY pass, but I keep it for now, just to make debug/test easy.

Now I will work on properly hook the depth buffer so that the effects can use it, I think there is some bug related to fbo's on intel gpus, because when I try to run the deferred rm on it the rm binds a D24S8 buffer and then the ShadowPSSM tries to bind a D32 buffer causing a invalid attachment error, but if I change the first to D32 it works fine. Maybe this bug was causing the invalid attachment when I tried to hook the depth buffer with a D24S8 format.

Other strange thing is that this same intel gpu supports dx10, but in the HBAO if I use "for" loops it says that I can't use "for", and also if I remove the "for"(unroll) it says that I exceded the max temporary registers! But the NVIDIA demo with almost the same shader runs fine.