Song interpretation of "Just the two of Us" - Bill Withers

Song: Just the Two of Us
Band: Bill Withers
Trivia: Will Smith did a cover of this song with some Rap. 
Source: Cover by the "Piano around the world" youtube channel (At the end of this post)

Interpretation 
I see the crystal raindrops fall
And the beauty of it all
Is when the sun comes shining through
To make those rainbows in my mind
When I think of you sometime
And I want to spend some time with you

Just the two of us
We can make it if we try
Just the two of us
Just the two of us
Building castles in the sky
Just the two of us
You and I
This song talks about how the character is looking back in time
about the times which he spent with his then friend. These memories
evoke a sense of peacefulness and serenity in his mind. He wants to
live in those memories again. He is aware that if he wants to be
with her again and it is sort of impossible \ dangerous but he still believes
that there is a chance. He wants to continue from where they left off,
and wants to kick of all the crazy plans they had made when they
were together and not be bothered about anyone or anything.

Bonnie and Clyde were American criminals who traveled the central United States with their gang during the Great Depression, robbing people and killing when cornered or confronted.
We look for love no time for tears
Wasted water's all that is
And it don't make no flowers grow

Good things might come to those who wait
Not for those who wait too late
We gotta go for all we know

And darling when the morning comes
And I see the morning sun
I want to be the one with you.
Crying over the past is not gonna help and self pity is not
going to blossom into something better. (So either he has found
someone new and better than his ex-lover\friend or that he wants
to get back with old friend.)
whatever it is , he wants the other person very much, to give their love
a chance so that it can blossom into something good, pure and clear
like the crystal rain drops - reflecting and emitting good things
from the inner core of their relationship while casting a delight
for others.

Alternative Meanings of this song:
The song be interpreted in a non-romantic way, ie where the character wants to make it big with his passion such as music\art etc 

Fav Cover & my source: