I've tried to install Panda3D sdk, but I've received an error, saying the application isn't made by a known developer so I can't install it due to my security settings. I understand this precaution is made by Apple against malware. However I still want to install Panda3d, how can I be sure it isn't a malware or has malware in it?
Checking for malware is a complex thing. Something that might be a dangerous operation for a program to make (like deleting the entire contents of your hard drive), might be an unintended action (making the program malware), or it might be a feature of the program and exactly what you meant to do.
The way OS X generally decides if something is "unsafe" is by checking for a code signing certificate. Code signing certificates themselves don't guarantee that a program isn't malware. What they do is provide traceability back to the original developer that build the program, and verifies their identity through third parties. This is generally the same process used for certificate validation on websites.
That said, OS X still doesn't know if this is malware or not. It just knows that it can verify the program was built by someone who has verified their identity with a third party. Not have that verification only means the person who built it didn't have a certificate. It might still be someone you trust but the system can't tell.
Malware detection systems generally check code for copies of malicious code fragments that have been identified before. They will often also check to see if the code will do something it knows is dangerous (like deleting your entire hard drive, or accessing files it knows it shouldn't), but these things are still difficult to define.
The easiest way to avoid running malicious code is to use a virus scanner (I personally don't have one to suggest for OS X), but the best way is to do your own due diligence and only run code that comes from a trusted source. Be thorough about reading the address of any links you download software from and make sure they point the the website you think it should. Sometimes a software distributor will publish a hash (MD5 or SHA1/SHA2) of the file, which you can use the verify that the file you've downloaded is an exact match to that which was published.
In most cases, you will need to assess how much effort you need to take to verify that code is safe, which usually depends on how much is on the line. Are you running this code on your personal computer, or a hospital computer with patient records on it? Is this a computer you use for fun, or is the operating a nuclear power plant? These are things to consider when deciding how much effort you're willing to go to in order to stay safe from malware.