A feeling of guilt is triggered when we realize that we have hurt someone we care about or when we otherwise fall short of the standards we feel we should live by. If we have done something wrong in the sight of God, and feelings of guilt come over us, it is good because it signals that we are aware of acceptable standards. It shows that we have a working conscience. It will help us not to commit the same wrongdoing.
Proverbs 14:9 says "Fools make fun of guilt, But among the upright, there is a willingness to reconcile.". A guilty conscience can and should move us to confession and other positive action. However, our basic reason for serving Jehovah should always be, not guilt, but love. The Bible assures us that when good people are encouraged with this in mind, they will do all they can. More important, they will be happy doing it.
So feelings of guilt is not all that bad as long as the individual is ready to do his best to stop that wrongdoing.