When I begin the process of a new composition, I try to find a way to organize my material and derive everything that I possibly can from the smallest compositional seed. The other process that happens somewhat simultaneously, is that I use my ear to pick out sonorities and elements that I find pleasing. This process is not a revelation by any means, but I find that balancing what I like to hear and what I can justify intellectually is not as simple as I often hope it will be.
Before I elaborate, I will clarify that I strongly believe that your ears and your heart should be the final judges. I don't see composition as an emotionally disengaged, intellectual process. Let me assure you that my favourite music is the kind that stirs my soul and makes me feel genuine human emotion (perhaps my insecurity of being perceived as a dispassionate theory nerd is showing just a tad).
Even though my ears guide me to what I want to hear, a curiosity I have about music is that there is a strong relationship between the degree of organization of compositional elements, and our emotional reaction to the composition. I don't think that organization on its own creates beauty, but definitely it is an element that exists in many beautiful things. And sometimes my ears will guide me to something that does not seem like it can be intellectually justified within the framework I have created. So what then?
Let's create a hypothetical scenario — I have just derived several scales and sonorities that I quite like. I understand there is an interval pattern that is consistent in deriving all these materials, and that interval pattern is what I have been using to create my motives, melodies, harmonic structure and even to generate the form of the piece. I reach a pivotal moment that is supposed to be very dramatic, but I realize that if I use the chord that I thought would sound good and also have intellectual significance does not turn out to be as stirring and beautiful as I hoped. For some reason that I can't understand or explain, a plain c major chord works incredibly well and sounds perfect for this moment. Why? It doesn't seem like it is incongruous with the rest of the material, yet I am still unable to make sense of why it sounds so much better. This can be alarming for me. I like to understand the significance of things in my music, and when I don't, I feel sometimes like the piece is running away from me. This is the battle I undergo to safeguard the "integrity" of the piece. Or at least I think that's what I'm doing.
Maybe what I can take from an experience like this is that it is futile to try controlling everything in your music. My compositional process is far from what could be considered serialism, but I still have a compulsion to seek things that are organically derived. I guess I have a foot in each camp. A little on the side of Mr. Spock, and a little on the side of Captain Kirk.
As a mentor recently told me, the audience doesn't care if your piece is organically derived from a (014) prime form set, or if there is a rhythmic canon. They came to hear the goods, and you're expected to deliver them. My initial hypothesis when I set out to write a piece is that these imperceptible elements cause the listener to have a greater emotional reaction. But if something sounds better to my ears, and I can't justify why, I'm still going to use it. I see writing as a process of discovering a piece that already exists, and revealing it in its optimal form. And the optimal form needs to satisfy my ears and my soul, not my analytical reasoning. I suppose if someone gives me a commission, or an assignment, the piece already exists in the future, but it is up to me to shape and guide its creation to the best of my abilities, keeping Mr. Spock on one shoulder, and Captain Kirk on the other.