As far as I can tell, the “forensic sense” refers to the act of God actually justifying someone, whereas the “declarative sense” refers to the way this justification is made known to everyone else. I could be wrong about this, but this is what I’ve gleaned so far from some searches…
This idea is based on the Protestant paradigm of what God actually does when He justifies you. In this paradigm, rather than actually making you righteous, God simply declares that you’re righteous. One common analogy is that you are declared innocent in God’s courtroom, regardless of your actual guilt, because Jesus paid the price for your guilt on the cross, and your faith in Him allows that payment to be applied to you so you can enter heaven.
This presents two problems. One is that it allows people to continue to sin all they like, knowing that their personal level of righteousness doesn’t influence their justification. The other is that there are a ton of places where the Bible says that we’ll be judged on the basis of our works. The terms you’ve presented are an attempt to get around these problems.
The idea appears to be this: When Scripture talks about “justification by faith apart from works of the law,” it means that our faith actually justifies us, because God rewards it with a change in our external legal status before Him. This is forensic justification; it refers to the external, forensic declaration of our righteousness.
To solve the problems I mentioned, one might claim that the only kind of faith that justifies is a kind that produces good works and keeps us away from sin. This way, surreptitious sinners who claim Christ don’t go to heaven, because they “never really had faith.” Thus, in order for us to show that our status really has been changed, we have to do good works. These works result in declarative justification: They declare that we’ve already been justified before God. They don’t actually justify us, but if they’re not present, we had better worry, because we then have no reason to think we’re actually justified.
So: Under this model, when James says Abraham was “justified by works,” what he means is that Abraham was already right with God, and showed that he was justified by his works. He was declaratively justified by those works. He was forensically justified by his faith, which he had in Genesis 15 (cf. Rom 4:3,9; Jas 2:23).
Hope that helps in some way.