The short rule is, it's almost always OK to boo your own team.
If you feel the need to motivate your team to help them along and you have arrived at the point where you think a hearty boo or heckle is what's going to do the job, then you do it. You boo, Boo Bird. You boo to your hearts content and then maybe even do one of those tennis whistles or a belly-itcher chant. This is your right as a paying fan. You do it.
Indeed, some psychologists believe that being hostile to a batter may positively affect that batter's performance. This may be because booing tends to be a motivating factor, but also may have halo effects on the pitcher (see below). So when Matt Kemp comes up to bat and you want to motivate him by treating him as if he were some inept child that needs your warm wishes and hugs in order to excel, you are wrong. You boo him and tell him he's not worth your time. That's what he wants. He wants to be Anastasia and you need to be his Christian and show him like maybe thirty of the fifty shades of grey.
Now the caveats and execptions...
If you boo your own pitcher, expecting him to do better, you will likely not get the effect you want. Let's say you have it in your mind that a hearty boo will force Ian Kennedy to not lay up his next fastball and cast a one way ticket for Señor Pelota to home run land. Let's say you do this by chanting the aforementioned "Belly-Itcher" chant, perhaps of the variety: We want a pitcher, not a belly-itcher. If you do this, studies have shown that you are actually likely to hurt the performance of your pitcher.
[Jeers] did result in lower accuracy for both baseball pitchers. . . . Under these conditions,. . . jeers functioned as a punisher for throwing strikes. . . . These results may suggest the real value of fan behavior during a sporting event is in their antagonism of the opposing team (though the authors are not advocating encouraging this among fans). Interestingly, this also has implications for our understanding of the so-called homefield advantage.
Another case in which you should not boo your own team is if a relative happens to be playing on the team that you are booing. You are not allowed to boo your own family, you heartless sociopath.
Similarly, and perhaps a bit more common, you should not boo a player once you realize that the player's family is in attendance near you. This is called common courtesy.
Finally, the most common time to not boo is when children are around you. It's fair to eyeball it, but if you assume it's a child under the age of 12, then just don't do it. There's enough time for them to grow into the negativity that your calloused heart has developed after years of rejection and failure. Let them come to it in their own time. And absolutely don't boo a child trying to start the Wave. What kind of human monster are you? At least vampires sparkle.
So short answer: Boo (just about) whenever you want if it makes you feel better.
Slightly long answer: Boo your own batters if you want results.